# CS Python File Wrapper
## Quick Start
### Install
#### Pip
```
pip install
```
To resolve packages using pip, add the following to `~/.pip/pip.conf`:
```python
[global]
index-url = https://<USERNAME>:<PASSWORD>@bergsalex.jfrog.io/artifactory/api/pypi/bergsalex-pypi/simple
```
If credentials are required they should be embedded in the URL. To resolve packages using pip, run:
```
pip install "jax-cs-storage"
```
#### Poetry
In your project's `pyproject.toml` file, add:
```
[[tool.poetry.source]]
name = "jfrogbergsalex"
url = "https://bergsalex.jfrog.io/artifactory/api/pypi/bergsalex-pypi/simple"
```
If the repository is password protected, add:
```
poetry config http-basic.jfrogbergsalex <YOUR-EMAIL>
```
You will be prompted for your password:
```
Password:
```
Then you can run:
```
poetry add jax-cs-storage
```
#### Docker
##### Pip
Add the following to the top of your `requirements.txt` file:
```
--extra-index-url "https://bergsalex.jfrog.io/artifactory/api/pypi/bergsalex-pypi/simple"
```
##### Poetry
The `[[tool.poetry.source]]` configuration above should handle resolving to the right repository.
##### Access restricted repositories
TBD
### Setup
The file wrapper uses pydantic to get configuration from a .env file by default. The steps to
customize this behavior are discussed in more detail in later sections.
For now, you'll need to at least have the following in you `.env`
```dotenv
JAX_CS_STORAGE_GCS_BUCKET = 'YOUR_BUCKET_NAME'
```
### Usage
#### Create a file from in memory content
```python
from jax.cs.storage import StorageObjectIngest
# You could also configure the package with `jax.cs.storage.init` in a namespace of you choice
# from .files import StorageObject, StorageObjectIngest
content = 'This is some in memory content'
filename = 'SomeGeneratedName.txt'
wrapped = StorageObjectIngest(content, from_memory=True).ingest(filename=filename)
```
#### Allow a user to create a file in Google Cloud Storage
Google cloud storage allows you to generate a signed url to allow user uploads to a specific
location in object storage, without providing the user with explicit credentials to write to
that object storage bucket.
```python
from jax.cs.storage import StorageObject, ObjectSvcGCS
desired_gs_url = 'gs://some_gs_url'
wrapped = StorageObject(desired_gs_url, file_svc=ObjectSvcGCS)
user_upload_url = wrapped.user_upload_url(content_type='image/png')
```
#### Look up a file by name
```python
# `ObjectSvcGCS` is where `jax.cs.storage stores files in the default configuration,
# you should use whichever service you use with you StorageObjectIngest
from jax.cs.storage.object.services.gcs import ObjectSvcGCS
filename = 'SomeGeneratedName.txt'
all_files = ObjectSvcGCS.all()
found = filename in all_files
```
#### Cache by unique file name
```python
from jax.cs.storage import StorageObject, StorageObjectIngest
# `ObjectSvcGCS` is where `jax.cs.storage stores files in the default configuration,
# you should use whichever service you use with you StorageObjectIngest
from jax.cs.storage.object.services.gcs import ObjectSvcGCS
def check_if_exists():
filename = 'SomeGeneratedName.txt'
all_files = ObjectSvcGCS.all()
found = filename in all_files
if found:
wrapped = StorageObject(filename)
else:
content = 'some_generated_content'
wrapped = StorageObjectIngest(content, from_memory=True).ingest(filename=filename)
return wrapped.user_url
```
#### Get a wrapped version of an existing file
```python
from jax.cs.storage import StorageObject
def get_file_for_user(known_file_location: str):
# Get the wrapped file
wrapped = StorageObject(known_file_location)
# Return information about the file to a user
return {'name': wrapped.user_name, 'location': wrapped.user_url}
```
## Configuration
### Settings
The following pydantic Settings class is used to configure the library. By
default, these values will be populated automatically from a `.env` config file.
The `.env` can have both your application configuration as well as the library
configuration side by side:
```dotenv
LOG_LEVEL = 'DEBUG'
JAX_CS_STORAGE_GCS_BUCKET = 'fake-bucket'
JAX_CS_STORAGE_GCS_PREFIX_DIR = None
JAX_CS_STORAGE_GCS_CHECK_BLOB_EXISTS = True
JAX_CS_STORAGE_R2_BASE_URL 'https://r2.jax.org
# You should comingle application and library config variables
APPLICATION_SPECIFIC_CONFIG = 'something'
APPLICATION_SPECIFIC_CONFIG_2 = 'something_else'
JAX_CS_STORAGE_IO_FILE_ROOT = '/path/to/local/storage/root'
```
If you need to manually set the configuration of the library, you can do so with
the special `init` entrypoint method.
```python
from jax.cs.storage import init
from jax.cs.storage.config import StorageConfig
StorageObject, StorageObjectIngest = init(
StorageConfig.parse_obj({
'GCS_BUCKET': '1',
'R2_BASE_URL': 'https://r2.jax.org/test',
'GCS_PREFIX_DIR': '3',
'GCS_CHECK_BLOB_EXISTS': False,
'IO_FILE_ROOT': '/path/to/local/storage/root'
})
)
```
You can use standard Pydantic methods to create the settings instance.
E.g. from a dictionary:
```python
from jax.cs.storage import init
from jax.cs.storage.config import StorageConfig
my_dict_config = {
'GCS_BUCKET': '1',
'R2_BASE_URL': 'https://r2.jax.org/test',
'GCS_PREFIX_DIR': '3',
'GCS_CHECK_BLOB_EXISTS': False}
StorageObject, StorageObjectIngest = init(StorageConfig.parse_obj(my_dict_config))
```
The underlying settings class looks like:
```python
from typing import Optional
from pydantic import AnyHttpUrl, BaseSettings
class StorageConfig(BaseSettings):
"""The pydantic configuration class definition for the jax.cs.storage package."""
LOG_LEVEL: str = 'DEBUG'
GCS_BUCKET: Optional[str] = None
GCS_PREFIX_DIR: Optional[str] = None
GCS_URL_EXPIRE_HOURS: int = 24
GCS_CHECK_BLOB_EXISTS: bool = True
R2_BASE_URL: AnyHttpUrl = 'https://r2.jax.org'
IO_FILE_ROOT: Optional[str] = '/'
class Config:
"""The config class for pydantic object.
Used here to configure the default means of determining settings for the package.
"""
env_prefix = 'JAX_CS_STORAGE_'
case_sensitive = True
env_file = ".env"
```
### Wrapper Config
#### Default Service
The default service is the concrete ObjectSvc implementation that the StorageObject will fall back to if
no other service can validate the file. To set the default wrapper service set the `default_svc`
argument on the `init` call:
```python
from jax.cs.storage import init, ObjectSvcGCS
StorageObject, StorageObjectIngest = init(default_svc=ObjectSvcGCS)
```
#### Available Services
The available services are an ordered list of concrete ObjectSvc implementations in which the order of
the list is the order of precedence of the services. To set the list of available services, use the
`services` argument on the `init` call:
```python
from jax.cs.storage import init, ObjectSvcGCS, ObjectSvcR2, ObjectSvcIO
StorageObject, StorageObjectIngest = init(
services=[ObjectSvcGCS, ObjectSvcIO, ObjectSvcR2])
```
#### Ingestion Service
To set how files are ingested using the StorageObjectIngest, pass an alternate concrete implementation
of the ObjectIngestSvc abstract class as the `ingestion_svc` argument on the `init` call:
```python
from jax.cs.storage import init, ObjectIngestSvcGCS
StorageObject, StorageObjectIngest = init(ingestion_svc=ObjectIngestSvcGCS)
```
#### Configure app instance to use a specific Ingestion service
This example shows how you could dynamically configure the library to use different file ingestion
services in different scenarios.
```python
from jax.cs.storage import init
from jax.cs.storage.object.services.io import ObjectIngestSvcIO
from jax.cs.storage.object.services.gcs import ObjectIngestSvcGCS
# This could be a boolean value taken from your app config
use_gcs = False
ingestion_svc = ObjectIngestSvcGCS if use_gcs else ObjectIngestSvcIO
StorageObject, StorageObjectIngest = init(
ingestion_svc=ingestion_svc
)
```
## Contributing
### Static Checkers
#### Python syntax and style: `flake8`
Flake8 is pre-configured with the .flake8 file from this repository. Just run the following.
```bash
python -m flake8 src/jax
```
#### Docstring existence and format: `pydocstyle`
Service implementations inherit their docstrings from their abstract class, we ignore D102 for the
service.py files.
Pydocstyle is configured in the `pyproject.toml` file at the root of this repository.
##### Check non-service files
````bash
pydocstyle src/jax/cs/storage/
````
##### Check services
```bash
pydocstyle src/jax/cs/storage/object/services --match='service.py' --add-ignore=D102
```
#### Known security vulnerabilities: `bandit`
```bash
bandit -r src/jax/cs/storage
```
e.g.
```
$ bandit -r src/jax/cs/storage
[main] INFO profile include tests: None
[main] INFO profile exclude tests: None
[main] INFO cli include tests: None
[main] INFO cli exclude tests: None
[main] INFO running on Python 3.7.12
Run started:2022-01-21 14:02:01.153764
Test results:
No issues identified.
Code scanned:
Total lines of code: 1143
Total lines skipped (#nosec): 0
Run metrics:
Total issues (by severity):
Undefined: 0.0
Low: 0.0
Medium: 0.0
High: 0.0
Total issues (by confidence):
Undefined: 0.0
Low: 0.0
Medium: 0.0
High: 0.0
Files skipped (0):
```
#### Dependency license analysis: `liccheck`
Liccheck is configured in the `[tools.liccheck]` section of the `pyproject.toml` file.
```
liccheck
```
e.g.
```bash
$ liccheck
gathering licenses...
8 packages and dependencies.
check authorized packages...
8 packages.
```
Raw data
{
"_id": null,
"home_page": "https://bitbucket.org/jacksonlaboratory/jax-cs-storage/",
"name": "jax-cs-storage",
"maintainer": null,
"docs_url": null,
"requires_python": "<4.0,>=3.8",
"maintainer_email": null,
"keywords": null,
"author": "Alexander Berger",
"author_email": "Alexander.Berger@jax.org",
"download_url": "https://files.pythonhosted.org/packages/67/80/957b8de7f4ab0bf11cf96b16853e17029425dc74f8f8bd2ffd1a5bc067d9/jax_cs_storage-0.9.2.tar.gz",
"platform": null,
"description": "# CS Python File Wrapper\n\n## Quick Start\n### Install\n\n#### Pip\n```\npip install \n```\n\nTo resolve packages using pip, add the following to `~/.pip/pip.conf`:\n```python\n[global]\nindex-url = https://<USERNAME>:<PASSWORD>@bergsalex.jfrog.io/artifactory/api/pypi/bergsalex-pypi/simple\n```\nIf credentials are required they should be embedded in the URL. To resolve packages using pip, run:\n```\npip install \"jax-cs-storage\"\n```\n\n#### Poetry\nIn your project's `pyproject.toml` file, add:\n```\n[[tool.poetry.source]]\nname = \"jfrogbergsalex\"\nurl = \"https://bergsalex.jfrog.io/artifactory/api/pypi/bergsalex-pypi/simple\"\n```\n\nIf the repository is password protected, add:\n```\npoetry config http-basic.jfrogbergsalex <YOUR-EMAIL>\n```\nYou will be prompted for your password:\n```\nPassword: \n```\n\nThen you can run:\n```\npoetry add jax-cs-storage\n```\n\n#### Docker\n##### Pip\nAdd the following to the top of your `requirements.txt` file:\n```\n--extra-index-url \"https://bergsalex.jfrog.io/artifactory/api/pypi/bergsalex-pypi/simple\"\n```\n\n##### Poetry\nThe `[[tool.poetry.source]]` configuration above should handle resolving to the right repository.\n\n##### Access restricted repositories\nTBD\n\n### Setup\nThe file wrapper uses pydantic to get configuration from a .env file by default. The steps to \ncustomize this behavior are discussed in more detail in later sections.\n\nFor now, you'll need to at least have the following in you `.env`\n```dotenv\nJAX_CS_STORAGE_GCS_BUCKET = 'YOUR_BUCKET_NAME'\n```\n### Usage\n\n#### Create a file from in memory content\n\n```python\nfrom jax.cs.storage import StorageObjectIngest\n\n# You could also configure the package with `jax.cs.storage.init` in a namespace of you choice\n# from .files import StorageObject, StorageObjectIngest\n\ncontent = 'This is some in memory content'\nfilename = 'SomeGeneratedName.txt'\nwrapped = StorageObjectIngest(content, from_memory=True).ingest(filename=filename)\n```\n\n#### Allow a user to create a file in Google Cloud Storage\nGoogle cloud storage allows you to generate a signed url to allow user uploads to a specific \nlocation in object storage, without providing the user with explicit credentials to write to\nthat object storage bucket.\n\n```python\nfrom jax.cs.storage import StorageObject, ObjectSvcGCS\n\ndesired_gs_url = 'gs://some_gs_url'\nwrapped = StorageObject(desired_gs_url, file_svc=ObjectSvcGCS)\nuser_upload_url = wrapped.user_upload_url(content_type='image/png')\n```\n\n#### Look up a file by name\n\n```python\n# `ObjectSvcGCS` is where `jax.cs.storage stores files in the default configuration,\n# you should use whichever service you use with you StorageObjectIngest\nfrom jax.cs.storage.object.services.gcs import ObjectSvcGCS\n\nfilename = 'SomeGeneratedName.txt'\nall_files = ObjectSvcGCS.all()\nfound = filename in all_files\n```\n\n#### Cache by unique file name\n\n```python\nfrom jax.cs.storage import StorageObject, StorageObjectIngest\n# `ObjectSvcGCS` is where `jax.cs.storage stores files in the default configuration,\n# you should use whichever service you use with you StorageObjectIngest\nfrom jax.cs.storage.object.services.gcs import ObjectSvcGCS\n\n\ndef check_if_exists():\n filename = 'SomeGeneratedName.txt'\n all_files = ObjectSvcGCS.all()\n found = filename in all_files\n\n if found:\n wrapped = StorageObject(filename)\n else:\n content = 'some_generated_content'\n wrapped = StorageObjectIngest(content, from_memory=True).ingest(filename=filename)\n\n return wrapped.user_url\n```\n\n#### Get a wrapped version of an existing file\n\n```python\nfrom jax.cs.storage import StorageObject\n\n\ndef get_file_for_user(known_file_location: str):\n # Get the wrapped file\n wrapped = StorageObject(known_file_location)\n # Return information about the file to a user\n return {'name': wrapped.user_name, 'location': wrapped.user_url}\n```\n\n\n## Configuration\n\n### Settings\nThe following pydantic Settings class is used to configure the library. By \ndefault, these values will be populated automatically from a `.env` config file. \nThe `.env` can have both your application configuration as well as the library \nconfiguration side by side:\n```dotenv\nLOG_LEVEL = 'DEBUG'\nJAX_CS_STORAGE_GCS_BUCKET = 'fake-bucket'\nJAX_CS_STORAGE_GCS_PREFIX_DIR = None\nJAX_CS_STORAGE_GCS_CHECK_BLOB_EXISTS = True\nJAX_CS_STORAGE_R2_BASE_URL 'https://r2.jax.org\n# You should comingle application and library config variables\nAPPLICATION_SPECIFIC_CONFIG = 'something'\nAPPLICATION_SPECIFIC_CONFIG_2 = 'something_else'\nJAX_CS_STORAGE_IO_FILE_ROOT = '/path/to/local/storage/root'\n```\n\nIf you need to manually set the configuration of the library, you can do so with\nthe special `init` entrypoint method.\n\n```python\nfrom jax.cs.storage import init\nfrom jax.cs.storage.config import StorageConfig\n\nStorageObject, StorageObjectIngest = init(\n StorageConfig.parse_obj({\n 'GCS_BUCKET': '1',\n 'R2_BASE_URL': 'https://r2.jax.org/test',\n 'GCS_PREFIX_DIR': '3',\n 'GCS_CHECK_BLOB_EXISTS': False,\n 'IO_FILE_ROOT': '/path/to/local/storage/root'\n })\n)\n```\n\nYou can use standard Pydantic methods to create the settings instance. \nE.g. from a dictionary:\n\n```python\nfrom jax.cs.storage import init\nfrom jax.cs.storage.config import StorageConfig\n\nmy_dict_config = {\n 'GCS_BUCKET': '1',\n 'R2_BASE_URL': 'https://r2.jax.org/test',\n 'GCS_PREFIX_DIR': '3',\n 'GCS_CHECK_BLOB_EXISTS': False}\nStorageObject, StorageObjectIngest = init(StorageConfig.parse_obj(my_dict_config))\n```\n\nThe underlying settings class looks like:\n```python\nfrom typing import Optional\nfrom pydantic import AnyHttpUrl, BaseSettings\n\nclass StorageConfig(BaseSettings):\n \"\"\"The pydantic configuration class definition for the jax.cs.storage package.\"\"\"\n\n LOG_LEVEL: str = 'DEBUG'\n\n GCS_BUCKET: Optional[str] = None\n GCS_PREFIX_DIR: Optional[str] = None\n GCS_URL_EXPIRE_HOURS: int = 24\n GCS_CHECK_BLOB_EXISTS: bool = True\n R2_BASE_URL: AnyHttpUrl = 'https://r2.jax.org'\n IO_FILE_ROOT: Optional[str] = '/'\n\n class Config:\n \"\"\"The config class for pydantic object.\n\n Used here to configure the default means of determining settings for the package.\n \"\"\"\n\n env_prefix = 'JAX_CS_STORAGE_'\n case_sensitive = True\n env_file = \".env\"\n```\n\n### Wrapper Config\n#### Default Service\nThe default service is the concrete ObjectSvc implementation that the StorageObject will fall back to if\nno other service can validate the file. To set the default wrapper service set the `default_svc` \nargument on the `init` call:\n\n```python\nfrom jax.cs.storage import init, ObjectSvcGCS\n\nStorageObject, StorageObjectIngest = init(default_svc=ObjectSvcGCS)\n```\n\n#### Available Services\nThe available services are an ordered list of concrete ObjectSvc implementations in which the order of\nthe list is the order of precedence of the services. To set the list of available services, use the \n`services` argument on the `init` call:\n\n```python\nfrom jax.cs.storage import init, ObjectSvcGCS, ObjectSvcR2, ObjectSvcIO\n\nStorageObject, StorageObjectIngest = init(\n services=[ObjectSvcGCS, ObjectSvcIO, ObjectSvcR2])\n```\n\n#### Ingestion Service\nTo set how files are ingested using the StorageObjectIngest, pass an alternate concrete implementation\nof the ObjectIngestSvc abstract class as the `ingestion_svc` argument on the `init` call:\n\n```python\nfrom jax.cs.storage import init, ObjectIngestSvcGCS\n\nStorageObject, StorageObjectIngest = init(ingestion_svc=ObjectIngestSvcGCS)\n```\n\n#### Configure app instance to use a specific Ingestion service\nThis example shows how you could dynamically configure the library to use different file ingestion\nservices in different scenarios.\n\n```python\nfrom jax.cs.storage import init\nfrom jax.cs.storage.object.services.io import ObjectIngestSvcIO\nfrom jax.cs.storage.object.services.gcs import ObjectIngestSvcGCS\n\n# This could be a boolean value taken from your app config\nuse_gcs = False\ningestion_svc = ObjectIngestSvcGCS if use_gcs else ObjectIngestSvcIO\nStorageObject, StorageObjectIngest = init(\n ingestion_svc=ingestion_svc\n)\n```\n\n## Contributing\n\n### Static Checkers\n\n#### Python syntax and style: `flake8`\nFlake8 is pre-configured with the .flake8 file from this repository. Just run the following.\n```bash\npython -m flake8 src/jax\n```\n\n#### Docstring existence and format: `pydocstyle`\nService implementations inherit their docstrings from their abstract class, we ignore D102 for the\nservice.py files.\n\nPydocstyle is configured in the `pyproject.toml` file at the root of this repository.\n\n##### Check non-service files\n````bash\npydocstyle src/jax/cs/storage/ \n````\n\n##### Check services\n```bash\npydocstyle src/jax/cs/storage/object/services --match='service.py' --add-ignore=D102\n```\n\n#### Known security vulnerabilities: `bandit`\n```bash\nbandit -r src/jax/cs/storage\n ```\n\ne.g.\n```\n$ bandit -r src/jax/cs/storage\n[main] INFO profile include tests: None\n[main] INFO profile exclude tests: None\n[main] INFO cli include tests: None\n[main] INFO cli exclude tests: None\n[main] INFO running on Python 3.7.12\nRun started:2022-01-21 14:02:01.153764\n\nTest results:\n No issues identified.\n\nCode scanned:\n Total lines of code: 1143\n Total lines skipped (#nosec): 0\n\nRun metrics:\n Total issues (by severity):\n Undefined: 0.0\n Low: 0.0\n Medium: 0.0\n High: 0.0\n Total issues (by confidence):\n Undefined: 0.0\n Low: 0.0\n Medium: 0.0\n High: 0.0\nFiles skipped (0):\n```\n\n#### Dependency license analysis: `liccheck`\nLiccheck is configured in the `[tools.liccheck]` section of the `pyproject.toml` file.\n\n```\nliccheck\n```\n\ne.g.\n```bash\n$ liccheck\ngathering licenses...\n8 packages and dependencies.\ncheck authorized packages...\n8 packages.\n```",
"bugtrack_url": null,
"license": "Apache-2.0",
"summary": "A python wrapper to make file backends seamless",
"version": "0.9.2",
"project_urls": {
"Homepage": "https://bitbucket.org/jacksonlaboratory/jax-cs-storage/",
"Repository": "https://bitbucket.org/jacksonlaboratory/jax-cs-storage/"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "0bdb4f7a90e47295b343f8d93053c5fbf8900b0c72124d2596c4d42c9b0fd31f",
"md5": "a88bdaca5800533d7c0b1e949e88c252",
"sha256": "a1725dd46191affad2565d3b8d892b7a617bb9d6f312bd047ab053634679fa7b"
},
"downloads": -1,
"filename": "jax_cs_storage-0.9.2-py3-none-any.whl",
"has_sig": false,
"md5_digest": "a88bdaca5800533d7c0b1e949e88c252",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": "<4.0,>=3.8",
"size": 41604,
"upload_time": "2024-08-28T13:16:20",
"upload_time_iso_8601": "2024-08-28T13:16:20.791394Z",
"url": "https://files.pythonhosted.org/packages/0b/db/4f7a90e47295b343f8d93053c5fbf8900b0c72124d2596c4d42c9b0fd31f/jax_cs_storage-0.9.2-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "6780957b8de7f4ab0bf11cf96b16853e17029425dc74f8f8bd2ffd1a5bc067d9",
"md5": "31916bed7e0f72136e7cdceddd779364",
"sha256": "df8e30ef1a6243dfaecf866229746d747fe61ab5ce2377bedc704b7bc35fa4f6"
},
"downloads": -1,
"filename": "jax_cs_storage-0.9.2.tar.gz",
"has_sig": false,
"md5_digest": "31916bed7e0f72136e7cdceddd779364",
"packagetype": "sdist",
"python_version": "source",
"requires_python": "<4.0,>=3.8",
"size": 29942,
"upload_time": "2024-08-28T13:16:22",
"upload_time_iso_8601": "2024-08-28T13:16:22.322350Z",
"url": "https://files.pythonhosted.org/packages/67/80/957b8de7f4ab0bf11cf96b16853e17029425dc74f8f8bd2ffd1a5bc067d9/jax_cs_storage-0.9.2.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-08-28 13:16:22",
"github": false,
"gitlab": false,
"bitbucket": true,
"codeberg": false,
"bitbucket_user": "jacksonlaboratory",
"bitbucket_project": "jax-cs-storage",
"lcname": "jax-cs-storage"
}
JSON