<img align="centre" alt="overlapping arrows" height="92" src="icons/mabel.svg" />
## mabel is a Data Engineering platform designed to run in serverless environments.
**mabel** just runs when you need it, scaling to zero, making it efficient and ideal for deployments to platforms like Kubernetes, GCP Cloud Run, AWS Fargate and Knative.
[](https://github.com/mabel-dev/mabel/blob/master/LICENSE)
[](https://github.com/mabel-dev/mabel)
[](https://github.com/mabel-dev/mabel/actions/workflows/regression_suite.yaml)
[](https://codecov.io/gh/mabl-dev/mabel)
[](https://github.com/mabel-dev/mabel/actions/workflows/static_analysis.yml)
[](https://pypi.org/project/mabel/)
[](https://sonarcloud.io/dashboard?id=joocer_mabel)
[](https://sonarcloud.io/dashboard?id=joocer_mabel)
[](https://snyk.io/advisor/python/mabel)
[](https://pepy.tech/project/mabel)
[](https://github.com/psf/black)
[](https://github.com/mabel-dev/mabel/commits)
[](https://github.com/mabel-dev/mabel/commits)
[](https://pypi.org/project/mabel/)
[](https://app.fossa.com/projects/git%2Bgithub.com%2Fmabel-dev%2Fmabel?ref=badge_shield)
- **Documentation** [GitHub Wiki](https://github.com/mabel-dev/mabel/wiki)
- **Bug Reports** [GitHub Issues](https://github.com/mabel-dev/mabel/issues/new/choose)
- **Feature Requests** [GitHub Issues](https://github.com/mabel-dev/mabel/issues/new/choose)
- **Source Code** [GitHub](https://github.com/mabel-dev/mabel)
- **Discussions** [GitHub Discussions](https://github.com/mabel-dev/mabel/discussions)
## Focus on What Matters
We've built **mabel** to enable Data Analysts to write complex data engineering tasks quickly and easily, so they could get on with doing what they do best.
~~~python
from mabel import Reader
data = Reader(dataset="test_data")
print(data.count())
~~~
## Key Features
- On-the-fly compression
- Low-memory requirements, even with terabytes of data
- Indexing and partitioning of data for fast reads
- Cursors for tracking reading position between processes
- Partial SQL DQL (Data Query Language) support
- Schema and [data_expectations](https://github.com/joocer/data_expectations) validation
## Installation
From PyPI (recommended)
~~~
pip install --upgrade mabel
~~~
From GitHub
~~~
pip install --upgrade git+https://github.com/mabel-dev/mabel
~~~
## Guides
[How to Read Data](https://github.com/mabel-dev/mabel/wiki/how_to_read_a_dataset)
## Dependencies
- **[orjson](https://github.com/ijl/orjson)** for JSON (de)serialization
- **[orso](https://github.com/mabel-dev/orso)** for data Schemas
- **[zstandard](https://github.com/indygreg/python-zstandard)** for real-time on disk compression
- **[LZ4](https://github.com/python-lz4/python-lz4)** for real-time in memory compression
There are a number of optional dependencies which are usually only required for specific features and functionality. These are listed in [tests/requirements.txt](https://github.com/mabel-dev/mabel/blob/main/tests/requirements.txt).
## Integrations
mabel comes with adapters for the following data services:
| | Service |
|-- |-- |
| <img align="centre" alt="GCP Storage" height="48" src="icons/gcs-logo.png" /> | Google Cloud Storage |
| <img align="centre" alt="MinIo" height="48" src="icons/minio-logo.png" /> | MinIO |
| <img align="centre" alt="AWS S3" height="48" src="icons/s3-logo.png" /> | AWS S3 |
| <img align="centre" alt="Azure" height="48" src="icons/azure.svg" /> | Azure Blob Storage |
| <img align="centre" alt="Local" height="48" src="icons/local-storage.png" /> | Local Storage |
Mabel is extensible with adapters for other data services as required.
## Deployment and Execution
mabel supports running on a range of platforms, including:
| | Platform |
|-- |-- |
| <img align="centre" alt="Docker" height="48" src="icons/docker-logo.png" /> | Docker
| <img align="centre" alt="Kubernetes" height="48" src="icons/kubernetes-logo.svg" /> | Kubernetes
| <img align="centre" alt="Windows" height="48" src="icons/windows-logo.png" /> | Windows (<img align="centre" alt="Notice" height="12" src="icons/note.svg" />1)
| <img align="centre" alt="Linux" height="48" src="icons/linux-logo.jpg" /> | Linux (<img align="centre" alt="Notice" height="12" src="icons/note.svg" />2)
| <img align="centre" alt="Linux" height="48" src="icons/mac-os.png" /> | Mac (<img align="centre" alt="Notice" height="12" src="icons/note.svg" />3)
<img align="centre" alt="Notice" height="12" src="icons/note.svg" />1 - Some non-core features are not available on Windows.
<img align="centre" alt="Notice" height="12" src="icons/note.svg" />2 - Tested on Debian (WSL) and Ubuntu.
<img align="centre" alt="Notice" height="12" src="icons/note.svg" />3 - Tested on Apple Silicon Macs.
## How Can I Contribute?
All contributions, bug reports, bug fixes, documentation improvements, enhancements, and ideas are welcome.
If you have a suggestion for an improvement or a bug, [raise a ticket](https://github.com/mabel-dev/mabel/issues/new/choose) or start a [discussion](https://github.com/mabel-dev/mabel/discussions).
Want to help build mabel? See the [contribution guidance](https://github.com/mabel-dev/mabel/blob/main/.github/CONTRIBUTING.md).
## License
[Apache 2.0](LICENSE)
[](https://app.fossa.com/projects/git%2Bgithub.com%2Fmabel-dev%2Fmabel?ref=badge_large)
Raw data
{
"_id": null,
"home_page": "https://github.com/mabel-dev/mabel/",
"name": "mabel",
"maintainer": "Joocer",
"docs_url": null,
"requires_python": null,
"maintainer_email": null,
"keywords": null,
"author": "joocer",
"author_email": "justin.joyce@joocer.com",
"download_url": "https://files.pythonhosted.org/packages/f8/a9/9e232a8f84bc73c10640e19adc70eae2d000e43cf3f9478fad4a9fecd591/mabel-0.6.20.tar.gz",
"platform": null,
"description": "<img align=\"centre\" alt=\"overlapping arrows\" height=\"92\" src=\"icons/mabel.svg\" />\n\n## mabel is a Data Engineering platform designed to run in serverless environments.\n\n**mabel** just runs when you need it, scaling to zero, making it efficient and ideal for deployments to platforms like Kubernetes, GCP Cloud Run, AWS Fargate and Knative.\n\n[](https://github.com/mabel-dev/mabel/blob/master/LICENSE)\n[](https://github.com/mabel-dev/mabel)\n[](https://github.com/mabel-dev/mabel/actions/workflows/regression_suite.yaml)\n[](https://codecov.io/gh/mabl-dev/mabel)\n[](https://github.com/mabel-dev/mabel/actions/workflows/static_analysis.yml)\n[](https://pypi.org/project/mabel/)\n[](https://sonarcloud.io/dashboard?id=joocer_mabel)\n[](https://sonarcloud.io/dashboard?id=joocer_mabel)\n[](https://snyk.io/advisor/python/mabel)\n[](https://pepy.tech/project/mabel)\n[](https://github.com/psf/black)\n[](https://github.com/mabel-dev/mabel/commits)\n[](https://github.com/mabel-dev/mabel/commits)\n[](https://pypi.org/project/mabel/)\n[](https://app.fossa.com/projects/git%2Bgithub.com%2Fmabel-dev%2Fmabel?ref=badge_shield)\n\n\n- **Documentation** [GitHub Wiki](https://github.com/mabel-dev/mabel/wiki) \n- **Bug Reports** [GitHub Issues](https://github.com/mabel-dev/mabel/issues/new/choose) \n- **Feature Requests** [GitHub Issues](https://github.com/mabel-dev/mabel/issues/new/choose) \n- **Source Code** [GitHub](https://github.com/mabel-dev/mabel) \n- **Discussions** [GitHub Discussions](https://github.com/mabel-dev/mabel/discussions)\n\n## Focus on What Matters\n\nWe've built **mabel** to enable Data Analysts to write complex data engineering tasks quickly and easily, so they could get on with doing what they do best.\n\n~~~python\nfrom mabel import Reader\n\ndata = Reader(dataset=\"test_data\")\nprint(data.count())\n~~~\n\n## Key Features\n\n- On-the-fly compression\n- Low-memory requirements, even with terabytes of data\n- Indexing and partitioning of data for fast reads \n- Cursors for tracking reading position between processes \n- Partial SQL DQL (Data Query Language) support \n- Schema and [data_expectations](https://github.com/joocer/data_expectations) validation\n\n## Installation\n\nFrom PyPI (recommended)\n~~~\npip install --upgrade mabel\n~~~\nFrom GitHub\n~~~\npip install --upgrade git+https://github.com/mabel-dev/mabel\n~~~\n\n\n## Guides\n\n[How to Read Data](https://github.com/mabel-dev/mabel/wiki/how_to_read_a_dataset)\n\n## Dependencies\n\n- **[orjson](https://github.com/ijl/orjson)** for JSON (de)serialization\n- **[orso](https://github.com/mabel-dev/orso)** for data Schemas\n- **[zstandard](https://github.com/indygreg/python-zstandard)** for real-time on disk compression\n- **[LZ4](https://github.com/python-lz4/python-lz4)** for real-time in memory compression\n\nThere are a number of optional dependencies which are usually only required for specific features and functionality. These are listed in [tests/requirements.txt](https://github.com/mabel-dev/mabel/blob/main/tests/requirements.txt).\n\n## Integrations\n\nmabel comes with adapters for the following data services:\n\n| | Service |\n|-- |-- |\n| <img align=\"centre\" alt=\"GCP Storage\" height=\"48\" src=\"icons/gcs-logo.png\" /> | Google Cloud Storage |\n| <img align=\"centre\" alt=\"MinIo\" height=\"48\" src=\"icons/minio-logo.png\" /> | MinIO |\n| <img align=\"centre\" alt=\"AWS S3\" height=\"48\" src=\"icons/s3-logo.png\" /> | AWS S3 | \n| <img align=\"centre\" alt=\"Azure\" height=\"48\" src=\"icons/azure.svg\" /> | Azure Blob Storage |\n| <img align=\"centre\" alt=\"Local\" height=\"48\" src=\"icons/local-storage.png\" /> | Local Storage |\n\nMabel is extensible with adapters for other data services as required.\n\n## Deployment and Execution\n\nmabel supports running on a range of platforms, including:\n\n| | Platform |\n|-- |-- |\n| <img align=\"centre\" alt=\"Docker\" height=\"48\" src=\"icons/docker-logo.png\" /> | Docker\n| <img align=\"centre\" alt=\"Kubernetes\" height=\"48\" src=\"icons/kubernetes-logo.svg\" /> | Kubernetes\n| <img align=\"centre\" alt=\"Windows\" height=\"48\" src=\"icons/windows-logo.png\" /> | Windows (<img align=\"centre\" alt=\"Notice\" height=\"12\" src=\"icons/note.svg\" />1)\n| <img align=\"centre\" alt=\"Linux\" height=\"48\" src=\"icons/linux-logo.jpg\" /> | Linux (<img align=\"centre\" alt=\"Notice\" height=\"12\" src=\"icons/note.svg\" />2)\n| <img align=\"centre\" alt=\"Linux\" height=\"48\" src=\"icons/mac-os.png\" /> | Mac (<img align=\"centre\" alt=\"Notice\" height=\"12\" src=\"icons/note.svg\" />3)\n\n<img align=\"centre\" alt=\"Notice\" height=\"12\" src=\"icons/note.svg\" />1 - Some non-core features are not available on Windows. \n<img align=\"centre\" alt=\"Notice\" height=\"12\" src=\"icons/note.svg\" />2 - Tested on Debian (WSL) and Ubuntu. \n<img align=\"centre\" alt=\"Notice\" height=\"12\" src=\"icons/note.svg\" />3 - Tested on Apple Silicon Macs.\n\n## How Can I Contribute?\n\nAll contributions, bug reports, bug fixes, documentation improvements, enhancements, and ideas are welcome.\n\nIf you have a suggestion for an improvement or a bug, [raise a ticket](https://github.com/mabel-dev/mabel/issues/new/choose) or start a [discussion](https://github.com/mabel-dev/mabel/discussions).\n\nWant to help build mabel? See the [contribution guidance](https://github.com/mabel-dev/mabel/blob/main/.github/CONTRIBUTING.md).\n\n## License\n\n[Apache 2.0](LICENSE)\n\n\n[](https://app.fossa.com/projects/git%2Bgithub.com%2Fmabel-dev%2Fmabel?ref=badge_large)\n",
"bugtrack_url": null,
"license": null,
"summary": "Python Data Libraries",
"version": "0.6.20",
"project_urls": {
"Homepage": "https://github.com/mabel-dev/mabel/"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "f617577645be985ec8931a49d039750692d5e3a60fb6e8c9117657f3e46ebc58",
"md5": "e946fa0aaaeeba8a163704ba6427a701",
"sha256": "dd4d48707c3bdc8a639492380b19cb87073654493da8c0f4bff65eaca969d7e6"
},
"downloads": -1,
"filename": "mabel-0.6.20-py3-none-any.whl",
"has_sig": false,
"md5_digest": "e946fa0aaaeeba8a163704ba6427a701",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": null,
"size": 93590,
"upload_time": "2024-04-07T15:30:41",
"upload_time_iso_8601": "2024-04-07T15:30:41.431057Z",
"url": "https://files.pythonhosted.org/packages/f6/17/577645be985ec8931a49d039750692d5e3a60fb6e8c9117657f3e46ebc58/mabel-0.6.20-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "f8a99e232a8f84bc73c10640e19adc70eae2d000e43cf3f9478fad4a9fecd591",
"md5": "0f017555194a7838862ad73e2d4d2bb2",
"sha256": "d0be1047f3fe2cc4fa428766d6a405b019de034c85ff5e71f4d5b5ed1f1bfb26"
},
"downloads": -1,
"filename": "mabel-0.6.20.tar.gz",
"has_sig": false,
"md5_digest": "0f017555194a7838862ad73e2d4d2bb2",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 92323,
"upload_time": "2024-04-07T15:30:44",
"upload_time_iso_8601": "2024-04-07T15:30:44.203465Z",
"url": "https://files.pythonhosted.org/packages/f8/a9/9e232a8f84bc73c10640e19adc70eae2d000e43cf3f9478fad4a9fecd591/mabel-0.6.20.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-04-07 15:30:44",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "mabel-dev",
"github_project": "mabel",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"requirements": [
{
"name": "bitarray",
"specs": []
},
{
"name": "lz4",
"specs": []
},
{
"name": "orjson",
"specs": []
},
{
"name": "orso",
"specs": [
[
">=",
"0.0.147"
]
]
},
{
"name": "siphashc",
"specs": []
},
{
"name": "zstandard",
"specs": []
}
],
"lcname": "mabel"
}
JSON
