# OpenDAL Loaders
```bash
pip install llama-index-readers-opendal
```
## Base OpendalReader
This loader parses any file via [Apache OpenDAL](https://github.com/apache/incubator-opendal).
All files are temporarily downloaded locally and subsequently parsed with `SimpleDirectoryReader`. Hence, you may also specify a custom `file_extractor`, relying on any of the loaders in this library (or your own)!
### Usage
`OpendalReader` can read data from any supported storage services including `s3`, `azblob`, `gcs` and so on.
```python
from llama_index.readers.opendal import OpendalReader
loader = OpendalReader(
scheme="s3",
bucket="bucket",
path="path/to/data/",
)
documents = loader.load_data()
```
We also provide `Opendal[S3|Gcs|Azblob]Reader` for convenience.
---
This loader is designed to be used as a way to load data into [LlamaIndex](https://github.com/run-llama/llama_index/).
## Azblob Loader
This loader parses any file stored on Azblob.
All files are temporarily downloaded locally and subsequently parsed with `SimpleDirectoryReader`. Hence, you may also specify a custom `file_extractor`, relying on any of the loaders in this library (or your own)!
> Azblob loader is based on `OpendalReader`.
### Usage
```python
from llama_index.readers.opendal import OpendalAzblobReader
loader = OpendalAzblobReader(
container="container",
path="path/to/data/",
endpoint="[endpoint]",
account_name="[account_name]",
account_key="[account_key]",
)
documents = loader.load_data()
```
---
This loader is designed to be used as a way to load data into [LlamaIndex](https://github.com/run-llama/llama_index/).
## Gcs Loader
This loader parses any file stored on Gcs.
All files are temporarily downloaded locally and subsequently parsed with `SimpleDirectoryReader`. Hence, you may also specify a custom `file_extractor`, relying on any of the loaders in this library (or your own)!
> Gcs loader is based on `OpendalReader`.
### Usage
```python
from llama_index.readers.opendal import OpendalGcsReader
loader = OpendalGcsReader(
bucket="bucket",
path="path/to/data/",
endpoint="[endpoint]",
credentials="[credentials]",
)
documents = loader.load_data()
```
Note: if `credentials` is not provided, this loader to try to load from env.
---
This loader is designed to be used as a way to load data into [LlamaIndex](https://github.com/run-llama/llama_index/).
## S3 Loader
This loader parses any file stored on S3. When initializing `S3Reader`, you may pass in your [AWS Access Key](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html). If none are found, the loader assumes they are stored in `~/.aws/credentials`.
All files are temporarily downloaded locally and subsequently parsed with `SimpleDirectoryReader`. Hence, you may also specify a custom `file_extractor`, relying on any of the loaders in this library (or your own)!
> S3 loader is based on `OpendalReader`.
### Usage
```python
loader = OpendalS3Reader(
bucket="bucket",
path="path/to/data/",
access_key_id="[ACCESS_KEY_ID]",
secret_access_key="[ACCESS_KEY_SECRET]",
)
documents = loader.load_data()
```
Note: if `access_key_id` or `secret_access_key` is not provided, this loader to try to load from env.
Possible arguments includes:
- `endpoint`: Specify the endpoint of s3 service.
- `region`: Specify the region of s3 service.
---
This loader is designed to be used as a way to load data into [LlamaIndex](https://github.com/run-llama/llama_index/).
Raw data
{
"_id": null,
"home_page": null,
"name": "llama-index-readers-opendal-reader",
"maintainer": "OpenDAL Contributors",
"docs_url": null,
"requires_python": "<4.0,>=3.8.1",
"maintainer_email": null,
"keywords": "azblob, gcs, s3, storage",
"author": "Your Name",
"author_email": "you@example.com",
"download_url": "https://files.pythonhosted.org/packages/3c/1f/4c6846c7b9c97cd7d879c449576bdf42e55726d1ed52011cc1954615d888/llama_index_readers_opendal_reader-0.2.1.tar.gz",
"platform": null,
"description": "# OpenDAL Loaders\n\n```bash\npip install llama-index-readers-opendal\n```\n\n## Base OpendalReader\n\nThis loader parses any file via [Apache OpenDAL](https://github.com/apache/incubator-opendal).\n\nAll files are temporarily downloaded locally and subsequently parsed with `SimpleDirectoryReader`. Hence, you may also specify a custom `file_extractor`, relying on any of the loaders in this library (or your own)!\n\n### Usage\n\n`OpendalReader` can read data from any supported storage services including `s3`, `azblob`, `gcs` and so on.\n\n```python\nfrom llama_index.readers.opendal import OpendalReader\n\nloader = OpendalReader(\n scheme=\"s3\",\n bucket=\"bucket\",\n path=\"path/to/data/\",\n)\ndocuments = loader.load_data()\n```\n\nWe also provide `Opendal[S3|Gcs|Azblob]Reader` for convenience.\n\n---\n\nThis loader is designed to be used as a way to load data into [LlamaIndex](https://github.com/run-llama/llama_index/).\n\n## Azblob Loader\n\nThis loader parses any file stored on Azblob.\n\nAll files are temporarily downloaded locally and subsequently parsed with `SimpleDirectoryReader`. Hence, you may also specify a custom `file_extractor`, relying on any of the loaders in this library (or your own)!\n\n> Azblob loader is based on `OpendalReader`.\n\n### Usage\n\n```python\nfrom llama_index.readers.opendal import OpendalAzblobReader\n\nloader = OpendalAzblobReader(\n container=\"container\",\n path=\"path/to/data/\",\n endpoint=\"[endpoint]\",\n account_name=\"[account_name]\",\n account_key=\"[account_key]\",\n)\ndocuments = loader.load_data()\n```\n\n---\n\nThis loader is designed to be used as a way to load data into [LlamaIndex](https://github.com/run-llama/llama_index/).\n\n## Gcs Loader\n\nThis loader parses any file stored on Gcs.\n\nAll files are temporarily downloaded locally and subsequently parsed with `SimpleDirectoryReader`. Hence, you may also specify a custom `file_extractor`, relying on any of the loaders in this library (or your own)!\n\n> Gcs loader is based on `OpendalReader`.\n\n### Usage\n\n```python\nfrom llama_index.readers.opendal import OpendalGcsReader\n\nloader = OpendalGcsReader(\n bucket=\"bucket\",\n path=\"path/to/data/\",\n endpoint=\"[endpoint]\",\n credentials=\"[credentials]\",\n)\ndocuments = loader.load_data()\n```\n\nNote: if `credentials` is not provided, this loader to try to load from env.\n\n---\n\nThis loader is designed to be used as a way to load data into [LlamaIndex](https://github.com/run-llama/llama_index/).\n\n## S3 Loader\n\nThis loader parses any file stored on S3. When initializing `S3Reader`, you may pass in your [AWS Access Key](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html). If none are found, the loader assumes they are stored in `~/.aws/credentials`.\n\nAll files are temporarily downloaded locally and subsequently parsed with `SimpleDirectoryReader`. Hence, you may also specify a custom `file_extractor`, relying on any of the loaders in this library (or your own)!\n\n> S3 loader is based on `OpendalReader`.\n\n### Usage\n\n```python\nloader = OpendalS3Reader(\n bucket=\"bucket\",\n path=\"path/to/data/\",\n access_key_id=\"[ACCESS_KEY_ID]\",\n secret_access_key=\"[ACCESS_KEY_SECRET]\",\n)\ndocuments = loader.load_data()\n```\n\nNote: if `access_key_id` or `secret_access_key` is not provided, this loader to try to load from env.\n\nPossible arguments includes:\n\n- `endpoint`: Specify the endpoint of s3 service.\n- `region`: Specify the region of s3 service.\n\n---\n\nThis loader is designed to be used as a way to load data into [LlamaIndex](https://github.com/run-llama/llama_index/).\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "llama-index readers opendal_reader integration",
"version": "0.2.1",
"project_urls": null,
"split_keywords": [
"azblob",
" gcs",
" s3",
" storage"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "75c721c683f79cef4358c1d2371d12b15caf1d6d989fd1bae7a7b963c19aa9c2",
"md5": "4e86ca48afa3e8df3e75bfe3aaf5fdb3",
"sha256": "a5f449c3c3203abacbd2ec2aab86638ac558b967e396114e3733ce858f6d0df1"
},
"downloads": -1,
"filename": "llama_index_readers_opendal_reader-0.2.1-py3-none-any.whl",
"has_sig": false,
"md5_digest": "4e86ca48afa3e8df3e75bfe3aaf5fdb3",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": "<4.0,>=3.8.1",
"size": 7371,
"upload_time": "2024-08-22T20:32:30",
"upload_time_iso_8601": "2024-08-22T20:32:30.938944Z",
"url": "https://files.pythonhosted.org/packages/75/c7/21c683f79cef4358c1d2371d12b15caf1d6d989fd1bae7a7b963c19aa9c2/llama_index_readers_opendal_reader-0.2.1-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "3c1f4c6846c7b9c97cd7d879c449576bdf42e55726d1ed52011cc1954615d888",
"md5": "594df749e201f64ab3074a4d4728cf96",
"sha256": "54d6dd66481560f2252c75548c192a86ec382674a6ac2b00ae2f3f2de78d105b"
},
"downloads": -1,
"filename": "llama_index_readers_opendal_reader-0.2.1.tar.gz",
"has_sig": false,
"md5_digest": "594df749e201f64ab3074a4d4728cf96",
"packagetype": "sdist",
"python_version": "source",
"requires_python": "<4.0,>=3.8.1",
"size": 4184,
"upload_time": "2024-08-22T20:32:31",
"upload_time_iso_8601": "2024-08-22T20:32:31.824038Z",
"url": "https://files.pythonhosted.org/packages/3c/1f/4c6846c7b9c97cd7d879c449576bdf42e55726d1ed52011cc1954615d888/llama_index_readers_opendal_reader-0.2.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-08-22 20:32:31",
"github": false,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"lcname": "llama-index-readers-opendal-reader"
}
JSON