# OpenDAL Loaders
## Base OpendalReader
This loader parses any file via [Apache OpenDAL](https://github.com/apache/incubator-opendal).
All files are temporarily downloaded locally and subsequently parsed with `SimpleDirectoryReader`. Hence, you may also specify a custom `file_extractor`, relying on any of the loaders in this library (or your own)!
### Usage
`OpendalReader` can read data from any supported storage services including `s3`, `azblob`, `gcs` and so on.
```python
from llama_index import download_loader
OpendalReader = download_loader("OpendalReader")
loader = OpendalReader(
scheme="s3",
bucket="bucket",
path="path/to/data/",
)
documents = loader.load_data()
```
We also provide `Opendal[S3|Gcs|Azblob]Reader` for convenience.
---
This loader is designed to be used as a way to load data into [LlamaIndex](https://github.com/run-llama/llama_index/tree/main/llama_index) and/or subsequently used as a Tool in a [LangChain](https://github.com/hwchase17/langchain) Agent. See [here](https://github.com/emptycrown/llama-hub/tree/main) for examples.
## Azblob Loader
This loader parses any file stored on Azblob.
All files are temporarily downloaded locally and subsequently parsed with `SimpleDirectoryReader`. Hence, you may also specify a custom `file_extractor`, relying on any of the loaders in this library (or your own)!
> Azblob loader is based on `OpendalReader`.
### Usage
```python
from llama_index import download_loader
OpendalAzblobReader = download_loader("OpendalAzblobReader")
loader = OpendalAzblobReader(
container="container",
path="path/to/data/",
endpoint="[endpoint]",
account_name="[account_name]",
account_key="[account_key]",
)
documents = loader.load_data()
```
---
This loader is designed to be used as a way to load data into [LlamaIndex](https://github.com/run-llama/llama_index/tree/main/llama_index) and/or subsequently used as a Tool in a [LangChain](https://github.com/hwchase17/langchain) Agent. See [here](https://github.com/emptycrown/llama-hub/tree/main) for examples.
## Gcs Loader
This loader parses any file stored on Gcs.
All files are temporarily downloaded locally and subsequently parsed with `SimpleDirectoryReader`. Hence, you may also specify a custom `file_extractor`, relying on any of the loaders in this library (or your own)!
> Gcs loader is based on `OpendalReader`.
### Usage
```python
from llama_index import download_loader
OpendalGcsReader = download_loader("OpendalGcsReader")
loader = OpendalGcsReader(
bucket="bucket",
path="path/to/data/",
endpoint="[endpoint]",
credentials="[credentials]",
)
documents = loader.load_data()
```
Note: if `credentials` is not provided, this loader to try to load from env.
---
This loader is designed to be used as a way to load data into [LlamaIndex](https://github.com/run-llama/llama_index/tree/main/llama_index) and/or subsequently used as a Tool in a [LangChain](https://github.com/hwchase17/langchain) Agent. See [here](https://github.com/emptycrown/llama-hub/tree/main) for examples.
## S3 Loader
This loader parses any file stored on S3. When initializing `S3Reader`, you may pass in your [AWS Access Key](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html). If none are found, the loader assumes they are stored in `~/.aws/credentials`.
All files are temporarily downloaded locally and subsequently parsed with `SimpleDirectoryReader`. Hence, you may also specify a custom `file_extractor`, relying on any of the loaders in this library (or your own)!
> S3 loader is based on `OpendalReader`.
### Usage
```python
from llama_index import download_loader
OpendalS3Reader = download_loader("OpendalS3Reader")
loader = OpendalS3Reader(
bucket="bucket",
path="path/to/data/",
access_key_id="[ACCESS_KEY_ID]",
secret_access_key="[ACCESS_KEY_SECRET]",
)
documents = loader.load_data()
```
Note: if `access_key_id` or `secret_access_key` is not provided, this loader to try to load from env.
Possible arguments includes:
- `endpoint`: Specify the endpoint of s3 service.
- `region`: Specify the region of s3 service.
---
This loader is designed to be used as a way to load data into [LlamaIndex](https://github.com/run-llama/llama_index/tree/main/llama_index) and/or subsequently used as a Tool in a [LangChain](https://github.com/hwchase17/langchain) Agent. See [here](https://github.com/emptycrown/llama-hub/tree/main) for examples.
Raw data
{
"_id": null,
"home_page": "",
"name": "llama-index-readers-opendal-reader",
"maintainer": "OpenDAL Contributors",
"docs_url": null,
"requires_python": ">=3.8.1,<4.0",
"maintainer_email": "",
"keywords": "azblob,gcs,s3,storage",
"author": "Your Name",
"author_email": "you@example.com",
"download_url": "https://files.pythonhosted.org/packages/01/94/af52fec620ba7fbd65d2cb89029c5de05b233c651c74832405515536969c/llama_index_readers_opendal_reader-0.1.3.tar.gz",
"platform": null,
"description": "# OpenDAL Loaders\n\n## Base OpendalReader\n\nThis loader parses any file via [Apache OpenDAL](https://github.com/apache/incubator-opendal).\n\nAll files are temporarily downloaded locally and subsequently parsed with `SimpleDirectoryReader`. Hence, you may also specify a custom `file_extractor`, relying on any of the loaders in this library (or your own)!\n\n### Usage\n\n`OpendalReader` can read data from any supported storage services including `s3`, `azblob`, `gcs` and so on.\n\n```python\nfrom llama_index import download_loader\n\nOpendalReader = download_loader(\"OpendalReader\")\n\nloader = OpendalReader(\n scheme=\"s3\",\n bucket=\"bucket\",\n path=\"path/to/data/\",\n)\ndocuments = loader.load_data()\n```\n\nWe also provide `Opendal[S3|Gcs|Azblob]Reader` for convenience.\n\n---\n\nThis loader is designed to be used as a way to load data into [LlamaIndex](https://github.com/run-llama/llama_index/tree/main/llama_index) and/or subsequently used as a Tool in a [LangChain](https://github.com/hwchase17/langchain) Agent. See [here](https://github.com/emptycrown/llama-hub/tree/main) for examples.\n\n## Azblob Loader\n\nThis loader parses any file stored on Azblob.\n\nAll files are temporarily downloaded locally and subsequently parsed with `SimpleDirectoryReader`. Hence, you may also specify a custom `file_extractor`, relying on any of the loaders in this library (or your own)!\n\n> Azblob loader is based on `OpendalReader`.\n\n### Usage\n\n```python\nfrom llama_index import download_loader\n\nOpendalAzblobReader = download_loader(\"OpendalAzblobReader\")\n\nloader = OpendalAzblobReader(\n container=\"container\",\n path=\"path/to/data/\",\n endpoint=\"[endpoint]\",\n account_name=\"[account_name]\",\n account_key=\"[account_key]\",\n)\ndocuments = loader.load_data()\n```\n\n---\n\nThis loader is designed to be used as a way to load data into [LlamaIndex](https://github.com/run-llama/llama_index/tree/main/llama_index) and/or subsequently used as a Tool in a [LangChain](https://github.com/hwchase17/langchain) Agent. See [here](https://github.com/emptycrown/llama-hub/tree/main) for examples.\n\n## Gcs Loader\n\nThis loader parses any file stored on Gcs.\n\nAll files are temporarily downloaded locally and subsequently parsed with `SimpleDirectoryReader`. Hence, you may also specify a custom `file_extractor`, relying on any of the loaders in this library (or your own)!\n\n> Gcs loader is based on `OpendalReader`.\n\n### Usage\n\n```python\nfrom llama_index import download_loader\n\nOpendalGcsReader = download_loader(\"OpendalGcsReader\")\n\nloader = OpendalGcsReader(\n bucket=\"bucket\",\n path=\"path/to/data/\",\n endpoint=\"[endpoint]\",\n credentials=\"[credentials]\",\n)\ndocuments = loader.load_data()\n```\n\nNote: if `credentials` is not provided, this loader to try to load from env.\n\n---\n\nThis loader is designed to be used as a way to load data into [LlamaIndex](https://github.com/run-llama/llama_index/tree/main/llama_index) and/or subsequently used as a Tool in a [LangChain](https://github.com/hwchase17/langchain) Agent. See [here](https://github.com/emptycrown/llama-hub/tree/main) for examples.\n\n## S3 Loader\n\nThis loader parses any file stored on S3. When initializing `S3Reader`, you may pass in your [AWS Access Key](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html). If none are found, the loader assumes they are stored in `~/.aws/credentials`.\n\nAll files are temporarily downloaded locally and subsequently parsed with `SimpleDirectoryReader`. Hence, you may also specify a custom `file_extractor`, relying on any of the loaders in this library (or your own)!\n\n> S3 loader is based on `OpendalReader`.\n\n### Usage\n\n```python\nfrom llama_index import download_loader\n\nOpendalS3Reader = download_loader(\"OpendalS3Reader\")\n\nloader = OpendalS3Reader(\n bucket=\"bucket\",\n path=\"path/to/data/\",\n access_key_id=\"[ACCESS_KEY_ID]\",\n secret_access_key=\"[ACCESS_KEY_SECRET]\",\n)\ndocuments = loader.load_data()\n```\n\nNote: if `access_key_id` or `secret_access_key` is not provided, this loader to try to load from env.\n\nPossible arguments includes:\n\n- `endpoint`: Specify the endpoint of s3 service.\n- `region`: Specify the region of s3 service.\n\n---\n\nThis loader is designed to be used as a way to load data into [LlamaIndex](https://github.com/run-llama/llama_index/tree/main/llama_index) and/or subsequently used as a Tool in a [LangChain](https://github.com/hwchase17/langchain) Agent. See [here](https://github.com/emptycrown/llama-hub/tree/main) for examples.\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "llama-index readers opendal_reader integration",
"version": "0.1.3",
"project_urls": null,
"split_keywords": [
"azblob",
"gcs",
"s3",
"storage"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "ababe8dbe2ea8954f2c3f129b5108fd3330b85a73256eb6a452aeae9b00e37a3",
"md5": "3e932b1a000da2b44453516ebf10d8d9",
"sha256": "430f3aafc00aef1e96fcd70e4d18d5dfa5ba2ec042925ad1cb35cc6bfcfc1513"
},
"downloads": -1,
"filename": "llama_index_readers_opendal_reader-0.1.3-py3-none-any.whl",
"has_sig": false,
"md5_digest": "3e932b1a000da2b44453516ebf10d8d9",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8.1,<4.0",
"size": 7487,
"upload_time": "2024-02-21T20:32:51",
"upload_time_iso_8601": "2024-02-21T20:32:51.116323Z",
"url": "https://files.pythonhosted.org/packages/ab/ab/e8dbe2ea8954f2c3f129b5108fd3330b85a73256eb6a452aeae9b00e37a3/llama_index_readers_opendal_reader-0.1.3-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "0194af52fec620ba7fbd65d2cb89029c5de05b233c651c74832405515536969c",
"md5": "33794cb4d7e0a176b5d3eb42791f4547",
"sha256": "a0f69d570383d570114d187c5f2644fdcd464b610fbb37f42f003617528da96e"
},
"downloads": -1,
"filename": "llama_index_readers_opendal_reader-0.1.3.tar.gz",
"has_sig": false,
"md5_digest": "33794cb4d7e0a176b5d3eb42791f4547",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8.1,<4.0",
"size": 4784,
"upload_time": "2024-02-21T20:32:52",
"upload_time_iso_8601": "2024-02-21T20:32:52.932449Z",
"url": "https://files.pythonhosted.org/packages/01/94/af52fec620ba7fbd65d2cb89029c5de05b233c651c74832405515536969c/llama_index_readers_opendal_reader-0.1.3.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-02-21 20:32:52",
"github": false,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"lcname": "llama-index-readers-opendal-reader"
}
JSON