Name | flying-delta-readers-airbyte-cdk JSON |
Version |
0.1.0
JSON |
| download |
home_page | |
Summary | llama-index readers airbyte_cdk integration |
upload_time | 2024-01-26 16:57:20 |
maintainer | |
docs_url | None |
author | Your Name |
requires_python | >=3.8.1,<3.12 |
license | MIT |
keywords |
|
VCS |
|
bugtrack_url |
|
requirements |
No requirements were recorded.
|
Travis-CI |
No Travis.
|
coveralls test coverage |
No coveralls.
|
# Airbyte CDK Loader
The Airbyte CDK Loader is a shim for sources created using the [Airbyte Python CDK](https://docs.airbyte.com/connector-development/cdk-python/). It allows you to load data from any Airbyte source into LlamaIndex.
## Installation
- Install llama_hub: `pip install llama_hub`
- Install airbyte-cdk: `pip install airbyte-cdk`
- Install a source via git (or implement your own): `pip install git+https://github.com/airbytehq/airbyte.git@master#egg=source_github&subdirectory=airbyte-integrations/connectors/source-github`
## Usage
Implement and import your own source. You can find lots of resources for how to achieve this on the [Airbyte documentation page](https://docs.airbyte.com/connector-development/).
Here's an example usage of the AirbyteCdkReader.
```python
from llama_index import download_loader
from llama_hub.airbyte_cdk import AirbyteCDKReader
from source_github.source import (
SourceGithub,
) # this is just an example, you can use any source here - this one is loaded from the Airbyte Github repo via pip install git+https://github.com/airbytehq/airbyte.git@master#egg=source_github&subdirectory=airbyte-integrations/connectors/source-github`
github_config = {
# ...
}
reader = AirbyteCDKReader(source_class=SourceGithub, config=github_config)
documents = reader.load_data(stream_name="issues")
```
By default all fields are stored as metadata in the documents and the text is set to the JSON representation of all the fields. Construct the text of the document by passing a `record_handler` to the reader:
```python
def handle_record(record, id):
return Document(
doc_id=id, text=record.data["title"], extra_info=record.data
)
reader = AirbyteCDKReader(
source_class=SourceGithub,
config=github_config,
record_handler=handle_record,
)
```
## Lazy loads
The `reader.load_data` endpoint will collect all documents and return them as a list. If there are a large number of documents, this can cause issues. By using `reader.lazy_load_data` instead, an iterator is returned which can be consumed document by document without the need to keep all documents in memory.
## Incremental loads
If a stream supports it, this loader can be used to load data incrementally (only returning documents that weren't loaded last time or got updated in the meantime):
```python
reader = AirbyteCDKReader(source_class=SourceGithub, config=github_config)
documents = reader.load_data(stream_name="issues")
current_state = reader.last_state # can be pickled away or stored otherwise
updated_documents = reader.load_data(
stream_name="issues", state=current_state
) # only loads documents that were updated since last time
```
This loader is designed to be used as a way to load data into [LlamaIndex](https://github.com/run-llama/llama_index/tree/main/llama_index) and/or subsequently used as a Tool in a [LangChain](https://github.com/hwchase17/langchain) Agent. See [here](https://github.com/emptycrown/llama-hub/tree/main) for examples.
Raw data
{
"_id": null,
"home_page": "",
"name": "flying-delta-readers-airbyte-cdk",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.8.1,<3.12",
"maintainer_email": "",
"keywords": "",
"author": "Your Name",
"author_email": "you@example.com",
"download_url": "https://files.pythonhosted.org/packages/3b/7a/db9623d1eff6a91dac937dc2cde7e4ab20c1ba4fa90fb01b1eb2d248bc62/flying_delta_readers_airbyte_cdk-0.1.0.tar.gz",
"platform": null,
"description": "# Airbyte CDK Loader\n\nThe Airbyte CDK Loader is a shim for sources created using the [Airbyte Python CDK](https://docs.airbyte.com/connector-development/cdk-python/). It allows you to load data from any Airbyte source into LlamaIndex.\n\n## Installation\n\n- Install llama_hub: `pip install llama_hub`\n- Install airbyte-cdk: `pip install airbyte-cdk`\n- Install a source via git (or implement your own): `pip install git+https://github.com/airbytehq/airbyte.git@master#egg=source_github&subdirectory=airbyte-integrations/connectors/source-github`\n\n## Usage\n\nImplement and import your own source. You can find lots of resources for how to achieve this on the [Airbyte documentation page](https://docs.airbyte.com/connector-development/).\n\nHere's an example usage of the AirbyteCdkReader.\n\n```python\nfrom llama_index import download_loader\nfrom llama_hub.airbyte_cdk import AirbyteCDKReader\nfrom source_github.source import (\n SourceGithub,\n) # this is just an example, you can use any source here - this one is loaded from the Airbyte Github repo via pip install git+https://github.com/airbytehq/airbyte.git@master#egg=source_github&subdirectory=airbyte-integrations/connectors/source-github`\n\n\ngithub_config = {\n # ...\n}\nreader = AirbyteCDKReader(source_class=SourceGithub, config=github_config)\ndocuments = reader.load_data(stream_name=\"issues\")\n```\n\nBy default all fields are stored as metadata in the documents and the text is set to the JSON representation of all the fields. Construct the text of the document by passing a `record_handler` to the reader:\n\n```python\ndef handle_record(record, id):\n return Document(\n doc_id=id, text=record.data[\"title\"], extra_info=record.data\n )\n\n\nreader = AirbyteCDKReader(\n source_class=SourceGithub,\n config=github_config,\n record_handler=handle_record,\n)\n```\n\n## Lazy loads\n\nThe `reader.load_data` endpoint will collect all documents and return them as a list. If there are a large number of documents, this can cause issues. By using `reader.lazy_load_data` instead, an iterator is returned which can be consumed document by document without the need to keep all documents in memory.\n\n## Incremental loads\n\nIf a stream supports it, this loader can be used to load data incrementally (only returning documents that weren't loaded last time or got updated in the meantime):\n\n```python\nreader = AirbyteCDKReader(source_class=SourceGithub, config=github_config)\ndocuments = reader.load_data(stream_name=\"issues\")\ncurrent_state = reader.last_state # can be pickled away or stored otherwise\n\nupdated_documents = reader.load_data(\n stream_name=\"issues\", state=current_state\n) # only loads documents that were updated since last time\n```\n\nThis loader is designed to be used as a way to load data into [LlamaIndex](https://github.com/run-llama/llama_index/tree/main/llama_index) and/or subsequently used as a Tool in a [LangChain](https://github.com/hwchase17/langchain) Agent. See [here](https://github.com/emptycrown/llama-hub/tree/main) for examples.\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "llama-index readers airbyte_cdk integration",
"version": "0.1.0",
"project_urls": null,
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "0f4e9284d63f16f7069d1f8dca9a43eab26b865f57370953b9ffe4c3a1ffc4b0",
"md5": "2aff7d2447c62d38041e5618f92db450",
"sha256": "3803d15c2d8250a063336fa97bec675135868c0238f59781ca27583c8b7e6a1f"
},
"downloads": -1,
"filename": "flying_delta_readers_airbyte_cdk-0.1.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "2aff7d2447c62d38041e5618f92db450",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8.1,<3.12",
"size": 3683,
"upload_time": "2024-01-26T16:57:19",
"upload_time_iso_8601": "2024-01-26T16:57:19.160630Z",
"url": "https://files.pythonhosted.org/packages/0f/4e/9284d63f16f7069d1f8dca9a43eab26b865f57370953b9ffe4c3a1ffc4b0/flying_delta_readers_airbyte_cdk-0.1.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "3b7adb9623d1eff6a91dac937dc2cde7e4ab20c1ba4fa90fb01b1eb2d248bc62",
"md5": "28df449ed8911fe01f7fff7ffb5c06e9",
"sha256": "c7ce1d498c80d691ad5b7c79651d1c2a4bc92d1b8f3a7b2aa562276a6a7d21be"
},
"downloads": -1,
"filename": "flying_delta_readers_airbyte_cdk-0.1.0.tar.gz",
"has_sig": false,
"md5_digest": "28df449ed8911fe01f7fff7ffb5c06e9",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8.1,<3.12",
"size": 3286,
"upload_time": "2024-01-26T16:57:20",
"upload_time_iso_8601": "2024-01-26T16:57:20.332324Z",
"url": "https://files.pythonhosted.org/packages/3b/7a/db9623d1eff6a91dac937dc2cde7e4ab20c1ba4fa90fb01b1eb2d248bc62/flying_delta_readers_airbyte_cdk-0.1.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-01-26 16:57:20",
"github": false,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"lcname": "flying-delta-readers-airbyte-cdk"
}