# airbite
`airbite` brings the power of Airbyte to Python with an intuitive, user-friendly API. It builds upon the PyAirbyte project and adds additional features.
With only a few lines of code, `airbite` can:
- Read data from sources in either `full refresh` or `incremental refresh` mode (the source must support incremental refresh)
- Write to destinations with multiple write strategies: `append`, `merge`, or `replace`
## Supported Connectors
- Number of sources: 455
- Number of destinations: 23
More are being added regularly
The "friendly API" is heavily typed, so you don't need to refer to any connector documentation separately while coding. This also helps AI tools (like GitHub Copilot) to write correct code, saving you hours of reading.
## Installation
To install airbite, use pip:
`pip install airbite`
## Usage
There are a total of 5 steps you need to perform for moving data between sources and destiantions (after `import airbite`):
1. Create source config
2. Create source
3. Create destination config (optional)
4. Create destination (optional)
5. Write data into destination:
- `destination.write(source, ...)`
6. (Optional) You can also just read the data from the source for exploration:
- `source.read_to_pandas(...)`
- `source.read_to_iterator(...)`
In addition there are a few utility functions provided:
1. `list_connectors(kind: Literal["any", "source", "destination"] = 'any', filter_text: str = None)`
2. `get_env_variable(secret_name: str)`: use this to inject variables from your `.env` files
Below is a basic example of moving data from source to destination. For more examples head to the examples dir.
```py
import airbite
# step #1
s_config = airbite.SourceFileConfig(
dataset_name="data_a",
format="csv",
provider=SourceFile.LocalFilesystemLimited(storage="local"),
url="./data/a.csv",
)
# step #2
source = airbite.create_source(s_config)
# step #3
d_config = airbite.DestinationDuckDBConfig(db_path=".cache/my_cache")
# step #4
destination = airbite.create_destination(d_config)
# step #5
airbite.destination.write(source=source, streams= '*')
```
## Build and Publish
```bash
# this step MUST be executed once in local dev environment:
# install the pre-commit and post-commit scripts that will help generate the jsonschema, models and create "connector_configs/__init__.py"
./install-pre-n-post-commit-hook.sh
# to generate model locally:
python airbite/model_generator.py
# build locally:
# rm -rf airbite/airbite.egg-info airbite.egg-info dist build && python -m build -w
rm -rf airbite/airbite.egg-info airbite.egg-info dist build && uv build --wheel
# publish to test pypi
uv publish --publish-ur 'https://test.pypi.org/legacy/' dist/*.whl
# publish to pypi
uv publish --publish-ur 'https://upload.pypi.org/legacy/' dist/*.whl
# run tests
python -m unittest tests/*.py
# tagging and release (not used)
git tag -a vx.x.x -m ''
git push origin tag v0.1.0
# create a release
git fetch --tags origin
git tag
gh release create v0.1.8 --target feat-workflow --generate-notes
git fetch --tags origin
```
## License
This project is licensed under the MIT License. See the LICENSE file for details
Raw data
{
"_id": null,
"home_page": "https://github.com/ossmht/airbite_pub/",
"name": "airbite",
"maintainer": null,
"docs_url": null,
"requires_python": null,
"maintainer_email": null,
"keywords": "airbite, airbyte, data, etl, extract, transform, load, pyairbyte",
"author": "ossmht",
"author_email": "ossmht@gmail.com",
"download_url": null,
"platform": null,
"description": "# airbite\n\n`airbite` brings the power of Airbyte to Python with an intuitive, user-friendly API. It builds upon the PyAirbyte project and adds additional features.\n\nWith only a few lines of code, `airbite` can:\n\n- Read data from sources in either `full refresh` or `incremental refresh` mode (the source must support incremental refresh)\n- Write to destinations with multiple write strategies: `append`, `merge`, or `replace`\n\n## Supported Connectors\n\n- Number of sources: 455\n- Number of destinations: 23\nMore are being added regularly\n\nThe \"friendly API\" is heavily typed, so you don't need to refer to any connector documentation separately while coding. This also helps AI tools (like GitHub Copilot) to write correct code, saving you hours of reading.\n\n\n## Installation\n\nTo install airbite, use pip:\n`pip install airbite`\n\n\n## Usage\n\n\nThere are a total of 5 steps you need to perform for moving data between sources and destiantions (after `import airbite`):\n\n1. Create source config\n2. Create source\n3. Create destination config (optional)\n4. Create destination (optional)\n5. Write data into destination:\n - `destination.write(source, ...)`\n6. (Optional) You can also just read the data from the source for exploration:\n - `source.read_to_pandas(...)`\n - `source.read_to_iterator(...)`\n\nIn addition there are a few utility functions provided:\n\n1. `list_connectors(kind: Literal[\"any\", \"source\", \"destination\"] = 'any', filter_text: str = None)`\n2. `get_env_variable(secret_name: str)`: use this to inject variables from your `.env` files\n\n\nBelow is a basic example of moving data from source to destination. For more examples head to the examples dir.\n\n```py\n\nimport airbite\n\n# step #1\ns_config = airbite.SourceFileConfig( \n dataset_name=\"data_a\",\n format=\"csv\",\n provider=SourceFile.LocalFilesystemLimited(storage=\"local\"), \n url=\"./data/a.csv\",\n)\n# step #2\nsource = airbite.create_source(s_config)\n# step #3\nd_config = airbite.DestinationDuckDBConfig(db_path=\".cache/my_cache\")\n# step #4\ndestination = airbite.create_destination(d_config)\n# step #5\nairbite.destination.write(source=source, streams= '*')\n\n```\n\n## Build and Publish\n\n```bash\n\n# this step MUST be executed once in local dev environment:\n# install the pre-commit and post-commit scripts that will help generate the jsonschema, models and create \"connector_configs/__init__.py\"\n./install-pre-n-post-commit-hook.sh\n\n\n# to generate model locally:\npython airbite/model_generator.py\n\n# build locally:\n# rm -rf airbite/airbite.egg-info airbite.egg-info dist build && python -m build -w\nrm -rf airbite/airbite.egg-info airbite.egg-info dist build && uv build --wheel\n\n# publish to test pypi\nuv publish --publish-ur 'https://test.pypi.org/legacy/' dist/*.whl\n\n# publish to pypi\nuv publish --publish-ur 'https://upload.pypi.org/legacy/' dist/*.whl\n\n# run tests\npython -m unittest tests/*.py\n\n\n# tagging and release (not used)\ngit tag -a vx.x.x -m ''\ngit push origin tag v0.1.0\n\n\n# create a release\ngit fetch --tags origin\ngit tag\ngh release create v0.1.8 --target feat-workflow --generate-notes\ngit fetch --tags origin\n\n```\n\n## License\n\nThis project is licensed under the MIT License. See the LICENSE file for details\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "`airbite` brings the power of Airbyte to Python with an intuitive, user-friendly API. It builds upon the PyAirbyte project and adds additional features.",
"version": "0.1.15",
"project_urls": {
"Homepage": "https://github.com/ossmht/airbite_pub/"
},
"split_keywords": [
"airbite",
" airbyte",
" data",
" etl",
" extract",
" transform",
" load",
" pyairbyte"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "47204ffa64cc4ef5cb35cd2a87b33566cf30e1369aea69d3fa791fa01cd182c3",
"md5": "8f1b399f10d4e4d1f8baf18bfaf344ae",
"sha256": "f5891b5a097b53e9049ed9b696427bcfc6f4c2e27fac6770ac7eaf55f99cb631"
},
"downloads": -1,
"filename": "airbite-0.1.15-cp311-cp311-macosx_10_9_x86_64.whl",
"has_sig": false,
"md5_digest": "8f1b399f10d4e4d1f8baf18bfaf344ae",
"packagetype": "bdist_wheel",
"python_version": "cp311",
"requires_python": null,
"size": 295092,
"upload_time": "2024-12-17T04:08:18",
"upload_time_iso_8601": "2024-12-17T04:08:18.128162Z",
"url": "https://files.pythonhosted.org/packages/47/20/4ffa64cc4ef5cb35cd2a87b33566cf30e1369aea69d3fa791fa01cd182c3/airbite-0.1.15-cp311-cp311-macosx_10_9_x86_64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "e53c743198421eea46d41ff6d3fdf0ea039e1454f9d997766e50a584dc70083e",
"md5": "4fcb93d6ff822deb7ff7a9812c786a62",
"sha256": "5961c84465620a063e539a10553320415b876d0e02d106fa4449a121d4c7cfa1"
},
"downloads": -1,
"filename": "airbite-0.1.15-cp311-cp311-macosx_11_0_arm64.whl",
"has_sig": false,
"md5_digest": "4fcb93d6ff822deb7ff7a9812c786a62",
"packagetype": "bdist_wheel",
"python_version": "cp311",
"requires_python": null,
"size": 283178,
"upload_time": "2024-12-17T04:08:20",
"upload_time_iso_8601": "2024-12-17T04:08:20.133611Z",
"url": "https://files.pythonhosted.org/packages/e5/3c/743198421eea46d41ff6d3fdf0ea039e1454f9d997766e50a584dc70083e/airbite-0.1.15-cp311-cp311-macosx_11_0_arm64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "e39bc867e262fdeff02ec3d13b5f7ea3498c9af365ec0f4b62213d782d804b50",
"md5": "8b8c27b0c441a5ba8e155a81aded047f",
"sha256": "47632c8fa51b3e2b966cf85aa1f92ac816b91f259377d95d4c75b9693ba165c7"
},
"downloads": -1,
"filename": "airbite-0.1.15-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl",
"has_sig": false,
"md5_digest": "8b8c27b0c441a5ba8e155a81aded047f",
"packagetype": "bdist_wheel",
"python_version": "cp311",
"requires_python": null,
"size": 1447265,
"upload_time": "2024-12-17T04:08:22",
"upload_time_iso_8601": "2024-12-17T04:08:22.949825Z",
"url": "https://files.pythonhosted.org/packages/e3/9b/c867e262fdeff02ec3d13b5f7ea3498c9af365ec0f4b62213d782d804b50/airbite-0.1.15-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "b30d990107e38e1f2c30be123dfb94159ba1d785a5484bea4f2ee87965865c8e",
"md5": "9758f4c84a7c16504ae5ddfbe339f55b",
"sha256": "c0dacf3ab0cab92ec26b89e4a9a84188340d9658767a0661851a631ebc6be3a8"
},
"downloads": -1,
"filename": "airbite-0.1.15-cp311-cp311-win_amd64.whl",
"has_sig": false,
"md5_digest": "9758f4c84a7c16504ae5ddfbe339f55b",
"packagetype": "bdist_wheel",
"python_version": "cp311",
"requires_python": null,
"size": 289221,
"upload_time": "2024-12-17T04:08:26",
"upload_time_iso_8601": "2024-12-17T04:08:26.126973Z",
"url": "https://files.pythonhosted.org/packages/b3/0d/990107e38e1f2c30be123dfb94159ba1d785a5484bea4f2ee87965865c8e/airbite-0.1.15-cp311-cp311-win_amd64.whl",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-12-17 04:08:18",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "ossmht",
"github_project": "airbite_pub",
"github_not_found": true,
"lcname": "airbite"
}
JSON
