# Streamlit FilesConnection
Connect to cloud (or local) file storage from your Streamlit app. Powered by `st.experimental_connection()` and [fsspec](https://filesystem-spec.readthedocs.io/en/latest/). Works with Streamlit >= 1.22.
Any fsspec compatible protocol should work, it just needs to be installed. Read more about Streamlit Connections in the
[official docs](https://docs.streamlit.io/library/api-reference/connections).
## Quickstart
See the example directory for a fuller example using S3 and/or GCS.
```sh
pip install streamlit
pip install git+https://github.com/streamlit/files-connection
```
**Note:** Install from pypi coming soon
```python
import streamlit as st
from st_files_connection import FilesConnection
"# Minimal FilesConnection example"
with st.echo():
conn = st.experimental_connection('my_connection', type=FilesConnection)
# Write a file to local directory if it doesn't exist
test_file = "test.txt"
try:
_ = conn.read(test_file, input_format='text')
except FileNotFoundError:
with conn.open(test_file, "wt") as f:
f.write("Hello, world!")
with st.echo():
# Read back the contents of the file
st.write(conn.read(test_file, input_format='text'))
```
## Using for cloud file storage
You can pass the protocol name into `st.experimental_connection()` as the first argument, for any known fsspec protocol:
```python
# Create an S3 connection
conn = st.experimental_connection('s3', type=FilesConnection)
# Create a GCS connection
conn = st.experimental_connection('gcs', type=FilesConnection)
# Create a Weights & Biases connection
conn = st.experimental_connection('wandb', type=FilesConnection)
```
For cloud file storage tools (or anything that needs config / credentials) you can specify it in two ways:
- Using the native configuration / credential approach of the underlying library (e.g. config file or environment variables)
- Using [Streamlit secrets](https://docs.streamlit.io/library/advanced-features/secrets-management).
For Streamlit secrets, create a section called `[connections.<name>]` in your `.streamlit/secrets.toml` file, and add parameters
there. You can pass in anything you would pass to an fsspec file system constructor. Additionally:
- For GCS, the contents of secrets are assumed to be the keys to a token file (e.g. it is passed as a `{"token":{<secrets>}}` dict)
## Main methods
### read()
`conn.read("path/to/file", input_format="text|csv|parquet|json|jsonl" or None, ttl=None) -> pd.DataFrame`
Specify a path to file and input format. Optionally specify a TTL for caching.
Valid values for `input_format=`:
- `text` returns a string
- `json` returns a dict or list (depending on the JSON object) - only one object per file is supported
- `csv`, `parquet`, `jsonl` return a pandas DataFrame
- `None` will attempt to infer the input format from file extension of `path`
- Anything else (or unrecognized inferred type) raises a `ValueError`
```python
conn = st.experimental_connection("s3", type=FilesConnection)
df = conn.read(f"my-s3-bucket/path/to/file.parquet", input_format='parquet')
st.dataframe(df)
```
**Note:** We want to add a `format=` argument to specify output format with more options, contributions welcome!
### open()
`conn.open("path/to/file", mode="rb", *args, **kwargs) -> Iterator[TextIOWrapper | AbstractBufferedFile]`
Works just like fsspec [AbstractFileSystem.open()](https://filesystem-spec.readthedocs.io/en/latest/api.html#fsspec.spec.AbstractFileSystem.open).
### fs
Use `conn.fs` to access the [underlying FileSystem object API](https://filesystem-spec.readthedocs.io/en/latest/api.html#fsspec.spec.AbstractFileSystem).
## Contributing
Contributions to this repo are welcome. We are still figuring it out and expect it may get some usage over time. We want to keep the API pretty simple
and not increase the maintenance surface area too much. If you are interested in helping to maintain it, reach out to us. Thanks for your patience if
it takes a few days to respond.
The best way to submit ideas you might want to work on is to open an issue and tag `@sfc-gh-jcarroll` and/or any other listed contributors.
Please don't spend a bunch of time working on a PR without checking with us first, since it risks the work being wasted and leaving you frustrated.
Also note, the Streamlit `experimental_connection()` interface is open for 3rd party packages and we look forward to promoting high quality ones in
the ecosystem. If you have an idea that differs from our direction here, we would love for you to fork / clone, build it, and share it with us and
the wider community. Thank you!
Raw data
{
"_id": null,
"home_page": "https://github.com/streamlit/files-connection",
"name": "st-files-connection",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": "",
"keywords": "",
"author": "Snowflake Inc",
"author_email": "hello@streamlit.io",
"download_url": "https://files.pythonhosted.org/packages/76/22/0aaab0013bf311cb37a0262e71964d298783432375201c2ee05515547f45/st-files-connection-0.1.0.tar.gz",
"platform": null,
"description": "# Streamlit FilesConnection\n\nConnect to cloud (or local) file storage from your Streamlit app. Powered by `st.experimental_connection()` and [fsspec](https://filesystem-spec.readthedocs.io/en/latest/). Works with Streamlit >= 1.22.\n\nAny fsspec compatible protocol should work, it just needs to be installed. Read more about Streamlit Connections in the\n[official docs](https://docs.streamlit.io/library/api-reference/connections).\n\n## Quickstart\n\nSee the example directory for a fuller example using S3 and/or GCS.\n\n```sh\npip install streamlit\npip install git+https://github.com/streamlit/files-connection\n```\n\n**Note:** Install from pypi coming soon\n\n```python\nimport streamlit as st\nfrom st_files_connection import FilesConnection\n\n\"# Minimal FilesConnection example\"\nwith st.echo():\n conn = st.experimental_connection('my_connection', type=FilesConnection)\n\n# Write a file to local directory if it doesn't exist\ntest_file = \"test.txt\"\ntry:\n _ = conn.read(test_file, input_format='text')\nexcept FileNotFoundError:\n with conn.open(test_file, \"wt\") as f:\n f.write(\"Hello, world!\")\n\nwith st.echo():\n # Read back the contents of the file\n st.write(conn.read(test_file, input_format='text'))\n```\n\n## Using for cloud file storage\n\nYou can pass the protocol name into `st.experimental_connection()` as the first argument, for any known fsspec protocol:\n\n```python\n# Create an S3 connection\nconn = st.experimental_connection('s3', type=FilesConnection)\n\n# Create a GCS connection\nconn = st.experimental_connection('gcs', type=FilesConnection)\n\n# Create a Weights & Biases connection\nconn = st.experimental_connection('wandb', type=FilesConnection)\n```\n\nFor cloud file storage tools (or anything that needs config / credentials) you can specify it in two ways:\n\n- Using the native configuration / credential approach of the underlying library (e.g. config file or environment variables)\n- Using [Streamlit secrets](https://docs.streamlit.io/library/advanced-features/secrets-management).\n\nFor Streamlit secrets, create a section called `[connections.<name>]` in your `.streamlit/secrets.toml` file, and add parameters\nthere. You can pass in anything you would pass to an fsspec file system constructor. Additionally:\n\n- For GCS, the contents of secrets are assumed to be the keys to a token file (e.g. it is passed as a `{\"token\":{<secrets>}}` dict)\n\n## Main methods\n\n### read()\n\n`conn.read(\"path/to/file\", input_format=\"text|csv|parquet|json|jsonl\" or None, ttl=None) -> pd.DataFrame`\n\nSpecify a path to file and input format. Optionally specify a TTL for caching.\n\nValid values for `input_format=`:\n\n- `text` returns a string\n- `json` returns a dict or list (depending on the JSON object) - only one object per file is supported\n- `csv`, `parquet`, `jsonl` return a pandas DataFrame\n- `None` will attempt to infer the input format from file extension of `path`\n- Anything else (or unrecognized inferred type) raises a `ValueError`\n\n```python\nconn = st.experimental_connection(\"s3\", type=FilesConnection)\ndf = conn.read(f\"my-s3-bucket/path/to/file.parquet\", input_format='parquet')\nst.dataframe(df)\n```\n\n**Note:** We want to add a `format=` argument to specify output format with more options, contributions welcome!\n\n### open()\n\n`conn.open(\"path/to/file\", mode=\"rb\", *args, **kwargs) -> Iterator[TextIOWrapper | AbstractBufferedFile]`\n\nWorks just like fsspec [AbstractFileSystem.open()](https://filesystem-spec.readthedocs.io/en/latest/api.html#fsspec.spec.AbstractFileSystem.open).\n\n### fs\n\nUse `conn.fs` to access the [underlying FileSystem object API](https://filesystem-spec.readthedocs.io/en/latest/api.html#fsspec.spec.AbstractFileSystem).\n\n## Contributing\n\nContributions to this repo are welcome. We are still figuring it out and expect it may get some usage over time. We want to keep the API pretty simple\nand not increase the maintenance surface area too much. If you are interested in helping to maintain it, reach out to us. Thanks for your patience if\nit takes a few days to respond.\n\nThe best way to submit ideas you might want to work on is to open an issue and tag `@sfc-gh-jcarroll` and/or any other listed contributors.\nPlease don't spend a bunch of time working on a PR without checking with us first, since it risks the work being wasted and leaving you frustrated.\n\nAlso note, the Streamlit `experimental_connection()` interface is open for 3rd party packages and we look forward to promoting high quality ones in\nthe ecosystem. If you have an idea that differs from our direction here, we would love for you to fork / clone, build it, and share it with us and\nthe wider community. Thank you!\n",
"bugtrack_url": null,
"license": "Apache License 2.0",
"summary": "Streamlit Connection for cloud and remote file storage.",
"version": "0.1.0",
"project_urls": {
"Homepage": "https://github.com/streamlit/files-connection",
"Source Code": "https://github.com/streamlit/files-connection"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "1e66df147106c2000b5db525a4a3ae7ca99cc28a837b6528f95d4091d528cb06",
"md5": "a7fad11b0254bc1bbe5c236c6904bab2",
"sha256": "869ae5cbc0f5eb65778b133c4348d1debdddb9acef483c0e37c4bc1f9012d29c"
},
"downloads": -1,
"filename": "st_files_connection-0.1.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "a7fad11b0254bc1bbe5c236c6904bab2",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8",
"size": 10831,
"upload_time": "2023-09-26T17:21:41",
"upload_time_iso_8601": "2023-09-26T17:21:41.836711Z",
"url": "https://files.pythonhosted.org/packages/1e/66/df147106c2000b5db525a4a3ae7ca99cc28a837b6528f95d4091d528cb06/st_files_connection-0.1.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "76220aaab0013bf311cb37a0262e71964d298783432375201c2ee05515547f45",
"md5": "8e59e4aeddd2faccf2194a30ca708cf7",
"sha256": "6ebc672e466ccd981d77a962a31fd7ab2615545f0b3acda691382f1eb7724f0c"
},
"downloads": -1,
"filename": "st-files-connection-0.1.0.tar.gz",
"has_sig": false,
"md5_digest": "8e59e4aeddd2faccf2194a30ca708cf7",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8",
"size": 9758,
"upload_time": "2023-09-26T17:21:43",
"upload_time_iso_8601": "2023-09-26T17:21:43.552734Z",
"url": "https://files.pythonhosted.org/packages/76/22/0aaab0013bf311cb37a0262e71964d298783432375201c2ee05515547f45/st-files-connection-0.1.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-09-26 17:21:43",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "streamlit",
"github_project": "files-connection",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "st-files-connection"
}
JSON
