[](LICENSE)
[](https://travis-ci.com/polyaxon/polystores)
[](https://badge.fury.io/py/polystores)
[](https://www.codacy.com/app/polyaxon/polystores?utm_source=github.com&utm_medium=referral&utm_content=polyaxon/polystores&utm_campaign=Badge_Grade)
[](https://join.slack.com/t/polyaxon/shared_invite/enQtMzQ0ODc2MDg1ODc0LWY2ZTdkMTNmZjBlZmRmNjQxYmYwMTBiMDZiMWJhODI2ZTk0MDU4Mjg5YzA5M2NhYzc5ZjhiMjczMDllYmQ2MDg)
# polystores
Polystores is an abstraction and a collection of clients to interact with cloud storages.
## Install
```bash
$ pip install -U polystores
```
N.B. this module does not include by default the cloud storage's client requirements
to keep the library lightweight, the user needs to install the appropriate module to use with `polystores`.
### Install S3
```bash
pip install -U polystores[s3]
```
### Install GCS
```bash
pip install -U polystores[gcs]
```
### Install Azure Storage
```bash
pip install -U polystores[azure]
```
## Stores
This module includes clients and stores abstraction that can be used to interact with AWS S3, Azure Storage, and Google Cloud Storage.
## S3
### Normal instantiation
```python
from polystores.stores.s3_store import S3Store
s3_store = S3Store(endpoint_url=...,
access_key_id=...,
secret_access_key=...,
session_token=...,
region=...)
```
### Using env vars
```bash
export AWS_ENDPOINT_URL=...
export AWS_ACCESS_KEY_ID=...
export AWS_SECRET_ACCESS_KEY=...
export AWS_SECURITY_TOKEN=...
exprot AWS_REGION=...
```
And then you can instantiate the store
```python
from polystores.stores.s3_store import S3Store
s3_store = S3Store()
```
### Using a client
```python
from polystores.stores.s3_store import S3Store
s3_store = S3Store(client=client)
```
### Important methods
```python
s3_store.list(bucket_name, prefix='', delimiter='', page_size=None, max_items=None, keys=True, prefixes=True)
s3_store.list_prefixes(bucket_name, prefix='', delimiter='', page_size=None, max_items=None)
s3_store.list_keys(bucket_name, prefix='', delimiter='', page_size=None, max_items=None)
s3_store.check_key(key, bucket_name=None)
s3_store.get_key(key, bucket_name=None)
s3_store.read_key(key, bucket_name=None)
s3_store.upload_bytes(bytes_data, key, bucket_name=None, overwrite=False, encrypt=False, acl=None)
s3_store.upload_string(string_data, key, bucket_name=None, overwrite=False, encrypt=False, acl=None, encoding='utf-8')
s3_store.upload_file(filename, key, bucket_name=None, overwrite=False, encrypt=False, acl=None, use_basename=True)
s3_store.download_file(key, local_path, bucket_name=None, use_basename=True)
s3_store.upload_dir(dirname, key, bucket_name=None, overwrite=False, encrypt=False, acl=None, use_basename=True)
s3_store.download_dir(key, local_path, bucket_name=None, use_basename=True)
```
## GCS
### Normal instantiation
```python
from polystores.stores.gcs_store import GCSStore
gcs_store = GCSStore(project_id=...,
credentials=...,
key_path=...,
key_path=...,
scopes=...)
```
### Using a client
```python
from polystores.stores.gcs_store import GCSStore
gcs_store = GCSStore(client=client)
```
### Important methods
```python
gcs_store.list(key, bucket_name=None, path=None, delimiter='/', blobs=True, prefixes=True)
gcs_store.upload_file(filename, blob, bucket_name=None, use_basename=True)
gcs_store.download_file(blob, local_path, bucket_name=None, use_basename=True)
gcs_store.upload_dir(dirname, blob, bucket_name=None, use_basename=True)
gcs_store.download_dir(blob, local_path, bucket_name=None, use_basename=True)
```
## Azure Storage
### Normal instantiation
```python
from polystores.stores.azure_store import AzureStore
az_store = AzureStore(account_name=...,
account_key=...,
connection_string=...)
```
### Using env vars
```bash
export AZURE_ACCOUNT_NAME=...
export AZURE_ACCOUNT_KEY=...
export AZURE_CONNECTION_STRING=...
```
And then you can instantiate the store
```python
from polystores.stores.azure_store import AzureStore
az_store = AzureStore()
```
### Using a client
```python
from polystores.stores.azure_store import AzureStore
az_store = AzureStore(client=client)
```
### Important methods
```python
az_store.list(key, container_name=None, path=None, delimiter='/', blobs=True, prefixes=True)
az_store.upload_file(filename, blob, container_name=None, use_basename=True)
az_store.download_file(blob, local_path, container_name=None, use_basename=True)
az_store.upload_dir(dirname, blob, container_name=None, use_basename=True)
az_store.download_dir(blob, local_path, container_name=None, use_basename=True)
```
## Running tests
```
pytest
```
## License
[](https://app.fossa.io/projects/git%2Bgithub.com%2Fpolyaxon%2Fpolystores?ref=badge_large)
Raw data
{
"_id": null,
"home_page": "https://github.com/polyaxon/polystores",
"name": "polystores",
"maintainer": "Mourad Mourafiq",
"docs_url": null,
"requires_python": "",
"maintainer_email": "mourad@polyaxon.com",
"keywords": "polyaxon,aws,s3,microsoft,azure,google cloud storage,gcs,machine-learning,data-science,artificial-intelligence,ai,reinforcement-learning,kubernetes,docker",
"author": "Mourad Mourafiq",
"author_email": "mourad@polyaxon.com",
"download_url": "https://files.pythonhosted.org/packages/f1/1d/cee828e30e3bba6f8baf1787963aa3dc90ec8ff8ee088606fcc75bac25c3/polystores-0.2.5.tar.gz",
"platform": "any",
"description": "[](LICENSE)\n[](https://travis-ci.com/polyaxon/polystores)\n[](https://badge.fury.io/py/polystores)\n[](https://www.codacy.com/app/polyaxon/polystores?utm_source=github.com&utm_medium=referral&utm_content=polyaxon/polystores&utm_campaign=Badge_Grade)\n[](https://join.slack.com/t/polyaxon/shared_invite/enQtMzQ0ODc2MDg1ODc0LWY2ZTdkMTNmZjBlZmRmNjQxYmYwMTBiMDZiMWJhODI2ZTk0MDU4Mjg5YzA5M2NhYzc5ZjhiMjczMDllYmQ2MDg)\n\n\n# polystores\n\nPolystores is an abstraction and a collection of clients to interact with cloud storages.\n\n\n## Install\n\n```bash\n$ pip install -U polystores\n```\n\nN.B. this module does not include by default the cloud storage's client requirements \nto keep the library lightweight, the user needs to install the appropriate module to use with `polystores`.\n\n### Install S3\n\n```bash\npip install -U polystores[s3]\n```\n\n### Install GCS\n\n\n```bash\npip install -U polystores[gcs]\n```\n\n### Install Azure Storage\n\n\n```bash\npip install -U polystores[azure]\n```\n\n## Stores\n\nThis module includes clients and stores abstraction that can be used to interact with AWS S3, Azure Storage, and Google Cloud Storage.\n\n\n## S3\n\n### Normal instantiation\n\n```python\nfrom polystores.stores.s3_store import S3Store\n\ns3_store = S3Store(endpoint_url=..., \n access_key_id=...,\n secret_access_key=...,\n session_token=...,\n region=...)\n```\n\n\n### Using env vars\n\n```bash\nexport AWS_ENDPOINT_URL=...\nexport AWS_ACCESS_KEY_ID=...\nexport AWS_SECRET_ACCESS_KEY=...\nexport AWS_SECURITY_TOKEN=...\nexprot AWS_REGION=...\n```\n\nAnd then you can instantiate the store\n```python\nfrom polystores.stores.s3_store import S3Store\n\ns3_store = S3Store()\n```\n\n### Using a client\n\n```python\nfrom polystores.stores.s3_store import S3Store\n\ns3_store = S3Store(client=client)\n```\n\n### Important methods\n\n```python\ns3_store.list(bucket_name, prefix='', delimiter='', page_size=None, max_items=None, keys=True, prefixes=True)\ns3_store.list_prefixes(bucket_name, prefix='', delimiter='', page_size=None, max_items=None)\ns3_store.list_keys(bucket_name, prefix='', delimiter='', page_size=None, max_items=None)\ns3_store.check_key(key, bucket_name=None)\ns3_store.get_key(key, bucket_name=None)\ns3_store.read_key(key, bucket_name=None)\ns3_store.upload_bytes(bytes_data, key, bucket_name=None, overwrite=False, encrypt=False, acl=None)\ns3_store.upload_string(string_data, key, bucket_name=None, overwrite=False, encrypt=False, acl=None, encoding='utf-8')\ns3_store.upload_file(filename, key, bucket_name=None, overwrite=False, encrypt=False, acl=None, use_basename=True)\ns3_store.download_file(key, local_path, bucket_name=None, use_basename=True)\ns3_store.upload_dir(dirname, key, bucket_name=None, overwrite=False, encrypt=False, acl=None, use_basename=True)\ns3_store.download_dir(key, local_path, bucket_name=None, use_basename=True)\n```\n\n\n## GCS\n\n### Normal instantiation\n\n```python\nfrom polystores.stores.gcs_store import GCSStore\n\ngcs_store = GCSStore(project_id=..., \n credentials=...,\n key_path=...,\n key_path=...,\n scopes=...)\n```\n\n### Using a client\n\n```python\nfrom polystores.stores.gcs_store import GCSStore\n\ngcs_store = GCSStore(client=client)\n```\n\n### Important methods\n\n```python\ngcs_store.list(key, bucket_name=None, path=None, delimiter='/', blobs=True, prefixes=True)\ngcs_store.upload_file(filename, blob, bucket_name=None, use_basename=True)\ngcs_store.download_file(blob, local_path, bucket_name=None, use_basename=True)\ngcs_store.upload_dir(dirname, blob, bucket_name=None, use_basename=True)\ngcs_store.download_dir(blob, local_path, bucket_name=None, use_basename=True)\n```\n\n## Azure Storage\n\n### Normal instantiation\n\n```python\nfrom polystores.stores.azure_store import AzureStore\n\naz_store = AzureStore(account_name=..., \n account_key=...,\n connection_string=...)\n```\n\n### Using env vars\n\n```bash\nexport AZURE_ACCOUNT_NAME=...\nexport AZURE_ACCOUNT_KEY=...\nexport AZURE_CONNECTION_STRING=...\n```\n\nAnd then you can instantiate the store\n```python\nfrom polystores.stores.azure_store import AzureStore\n\naz_store = AzureStore()\n```\n\n### Using a client\n\n```python\nfrom polystores.stores.azure_store import AzureStore\n\naz_store = AzureStore(client=client)\n```\n\n### Important methods\n\n```python\naz_store.list(key, container_name=None, path=None, delimiter='/', blobs=True, prefixes=True)\naz_store.upload_file(filename, blob, container_name=None, use_basename=True)\naz_store.download_file(blob, local_path, container_name=None, use_basename=True)\naz_store.upload_dir(dirname, blob, container_name=None, use_basename=True)\naz_store.download_dir(blob, local_path, container_name=None, use_basename=True)\n```\n\n## Running tests\n\n```\npytest\n```\n\n\n## License\n\n[](https://app.fossa.io/projects/git%2Bgithub.com%2Fpolyaxon%2Fpolystores?ref=badge_large)\n\n\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "Polystores is an abstraction and a collection of clients to interact with cloud storages.",
"version": "0.2.5",
"project_urls": {
"Homepage": "https://github.com/polyaxon/polystores"
},
"split_keywords": [
"polyaxon",
"aws",
"s3",
"microsoft",
"azure",
"google cloud storage",
"gcs",
"machine-learning",
"data-science",
"artificial-intelligence",
"ai",
"reinforcement-learning",
"kubernetes",
"docker"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "3e976ca1485410bd0912bdae48c9ccaf75f92dc1d63aa11e9fe8c8726a7a510e",
"md5": "da969acddaf12dd13e9079a43eb2d82e",
"sha256": "afcfd4487ac22e80f462b78b1f4080d6a6a677df91d01d86762f9dffa7a1e226"
},
"downloads": -1,
"filename": "polystores-0.2.5-py2.py3-none-any.whl",
"has_sig": false,
"md5_digest": "da969acddaf12dd13e9079a43eb2d82e",
"packagetype": "bdist_wheel",
"python_version": "py2.py3",
"requires_python": null,
"size": 31783,
"upload_time": "2020-01-12T18:09:29",
"upload_time_iso_8601": "2020-01-12T18:09:29.485850Z",
"url": "https://files.pythonhosted.org/packages/3e/97/6ca1485410bd0912bdae48c9ccaf75f92dc1d63aa11e9fe8c8726a7a510e/polystores-0.2.5-py2.py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "f11dcee828e30e3bba6f8baf1787963aa3dc90ec8ff8ee088606fcc75bac25c3",
"md5": "d98b4fee965e3cee8df9a2797f3df61e",
"sha256": "0e09a9c4ac5a7547429c89326664fa79b055107c81e32f7833467e2d433f653d"
},
"downloads": -1,
"filename": "polystores-0.2.5.tar.gz",
"has_sig": false,
"md5_digest": "d98b4fee965e3cee8df9a2797f3df61e",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 25982,
"upload_time": "2020-01-12T18:09:31",
"upload_time_iso_8601": "2020-01-12T18:09:31.657819Z",
"url": "https://files.pythonhosted.org/packages/f1/1d/cee828e30e3bba6f8baf1787963aa3dc90ec8ff8ee088606fcc75bac25c3/polystores-0.2.5.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2020-01-12 18:09:31",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "polyaxon",
"github_project": "polystores",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "polystores"
}
JSON
