| Name | lakestream JSON |
| Version |
0.0.3
JSON |
| download |
| home_page | |
| Summary | Python interface for Lakestream |
| upload_time | 2023-05-07 22:09:53 |
| maintainer | |
| docs_url | None |
| author | |
| requires_python | >=3.7 |
| license | MIT |
| keywords |
serverless
data
api
|
| VCS |
|
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
Lakestream
==========
Lakestream is a tool for interacting with object stores such as S3. It is built from the ground up in Rust, with APIs available for both Python and the web via JS/WASM.
The idea behind Lakestream is to create a high-performance and future-proof data tool that can scale with new (AI-driven) networking and usage patterns. This includes the ability to work in both client and service mode, and a modular design to allow compute functions on the network.
In the short term, the focus is on implementing basic features such as List, Copy, and Delete. The current version (0.0.2) enables listing and searching items on an S3 bucket or Local Filesystem.
Prerequisites
-------------
- Python or Rust
- Optional: S3 account with valid access key and secret key
Installation
------------
Lakestream can be used via Python (API) or directly via Rust (CLI).
A (local-first) browser-based version is on the short-term roadmap.
Python (API)
~~~~~~~~~~~~~~~~~~~~~~
Only Linux and MacOS wheels are pre-compiled. A Windows version should follow soon.
.. code-block:: console
pip install lakestream
Rust (CLI)
~~~~~~~~~~~~~~~~~~~~
Clone the repository and compile the project using Cargo:
.. code-block:: console
git clone https://github.com/serverlessnext/lakestream.git
cd lakestream
cargo build --release
Next, copy the binary from `./target/release/lakestream` to your local path.
Usage
-----
Quickstart
~~~~~~~~~~~~~~
.. code-block:: console
# for s3://buckets: AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY must be set
export AWS_ACCESS_KEY_ID=your_access_key
export AWS_SECRET_ACCESS_KEY=your_secret_key
export AWS_REGION=us-east-1 # optional
.. code-block:: console
# Find all files in the "reports" directory, with names containing "2023" and
# modified within the last 30 days, in a given S3 bucket.
lakestream ls s3://bucket-name/reports/ --name "*2023*" --mtime "-30D
# Find all files in the current directory, larger than 100 MB and modified
# within the last 2 days.
lakestream ls . --size "+100M" --mtime "-2D"
Find all files larger than 1 megabyte (MB) in a given S3 Bucket
lakestream ls s3://bucket-name/ --size "+1M" --recursive
# Find all files modified more than 1 hour ago, recursively
lakestream ls . --mtime "+1h" --recursive
More **CLI** examples `here <https://lakestream.dev/cli_list.html>`__.
Python can also be used as a CLI. Arguments are mapped 1:1 to the Rust library.
.. code-block:: console
# Python
python -m lakestream ls ./
# Rust
lakestream ls ./
Python API
~~~~~~~~~~
.. code-block:: python
import lakestream
client = lakestream.Client()
# Define a filter dictionary
filter_dict = {
"name": "example.txt",
"size": "5",
"mtime": "1D",
}
# List the contents of a storage location with the filter
result = client.list("s3://your-bucket", recursive=True, filter_dict=filter_dict)
print(result)
Python API Documentation `here <https://lakestream.dev/python_api.html>`__.
Contributing
------------
Contributions to the Lakestream project are welcome. Please open an issue or submit a pull request on the GitHub repository.
License
-------
Lakestream is released under the MIT license. See LICENSE for more details.
Links
-----
Documentation: https://lakestream.dev
Raw data
{
"_id": null,
"home_page": "",
"name": "lakestream",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.7",
"maintainer_email": "Anthony Potappel <aprxi@lakestream.dev>",
"keywords": "serverless,data,api",
"author": "",
"author_email": "",
"download_url": "https://files.pythonhosted.org/packages/8d/54/a03a5f4a20b4b69b3b21d71136b2da1399e9022243225cb744348c7a56d0/lakestream-0.0.3.tar.gz",
"platform": null,
"description": "\nLakestream\n==========\n\nLakestream is a tool for interacting with object stores such as S3. It is built from the ground up in Rust, with APIs available for both Python and the web via JS/WASM.\n\nThe idea behind Lakestream is to create a high-performance and future-proof data tool that can scale with new (AI-driven) networking and usage patterns. This includes the ability to work in both client and service mode, and a modular design to allow compute functions on the network.\n\nIn the short term, the focus is on implementing basic features such as List, Copy, and Delete. The current version (0.0.2) enables listing and searching items on an S3 bucket or Local Filesystem.\n\nPrerequisites\n-------------\n\n- Python or Rust\n- Optional: S3 account with valid access key and secret key\n\nInstallation\n------------\n\nLakestream can be used via Python (API) or directly via Rust (CLI).\nA (local-first) browser-based version is on the short-term roadmap.\n\nPython (API)\n~~~~~~~~~~~~~~~~~~~~~~\n\nOnly Linux and MacOS wheels are pre-compiled. A Windows version should follow soon.\n\n.. code-block:: console\n\n pip install lakestream\n\nRust (CLI)\n~~~~~~~~~~~~~~~~~~~~\n\nClone the repository and compile the project using Cargo:\n\n.. code-block:: console\n\n git clone https://github.com/serverlessnext/lakestream.git\n cd lakestream\n cargo build --release\n\nNext, copy the binary from `./target/release/lakestream` to your local path.\n\nUsage\n-----\n\nQuickstart\n~~~~~~~~~~~~~~\n\n.. code-block:: console\n\n # for s3://buckets: AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY must be set\n export AWS_ACCESS_KEY_ID=your_access_key\n export AWS_SECRET_ACCESS_KEY=your_secret_key\n export AWS_REGION=us-east-1 # optional\n\n.. code-block:: console\n\n # Find all files in the \"reports\" directory, with names containing \"2023\" and\n # modified within the last 30 days, in a given S3 bucket.\n lakestream ls s3://bucket-name/reports/ --name \"*2023*\" --mtime \"-30D\n\n # Find all files in the current directory, larger than 100 MB and modified\n # within the last 2 days.\n lakestream ls . --size \"+100M\" --mtime \"-2D\"\n\n Find all files larger than 1 megabyte (MB) in a given S3 Bucket\n lakestream ls s3://bucket-name/ --size \"+1M\" --recursive\n\n # Find all files modified more than 1 hour ago, recursively\n lakestream ls . --mtime \"+1h\" --recursive\n\nMore **CLI** examples `here <https://lakestream.dev/cli_list.html>`__.\n\nPython can also be used as a CLI. Arguments are mapped 1:1 to the Rust library.\n\n.. code-block:: console\n\n # Python\n python -m lakestream ls ./\n\n # Rust\n lakestream ls ./\n\nPython API\n~~~~~~~~~~\n\n.. code-block:: python\n\n import lakestream\n\n client = lakestream.Client()\n\n # Define a filter dictionary\n filter_dict = {\n \"name\": \"example.txt\",\n \"size\": \"5\",\n \"mtime\": \"1D\",\n }\n\n # List the contents of a storage location with the filter\n result = client.list(\"s3://your-bucket\", recursive=True, filter_dict=filter_dict)\n\n print(result)\n\n\nPython API Documentation `here <https://lakestream.dev/python_api.html>`__.\n\n\nContributing\n------------\n\nContributions to the Lakestream project are welcome. Please open an issue or submit a pull request on the GitHub repository.\n\nLicense\n-------\n\nLakestream is released under the MIT license. See LICENSE for more details.\n\nLinks\n-----\n\nDocumentation: https://lakestream.dev\n\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "Python interface for Lakestream",
"version": "0.0.3",
"project_urls": null,
"split_keywords": [
"serverless",
"data",
"api"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "689803a2c04dc7db5321cacb3f8f949a88d97ee057248eac0fea6695c0fa2382",
"md5": "b246edd4e31e5ad0d74435f384ad277e",
"sha256": "7be340c3f23cbfbd55a4285dea6c9c2ee227761b2a8e4fbfe165107c0ec34b2b"
},
"downloads": -1,
"filename": "lakestream-0.0.3-cp310-cp310-macosx_10_7_x86_64.whl",
"has_sig": false,
"md5_digest": "b246edd4e31e5ad0d74435f384ad277e",
"packagetype": "bdist_wheel",
"python_version": "cp310",
"requires_python": ">=3.7",
"size": 1183496,
"upload_time": "2023-05-07T22:09:31",
"upload_time_iso_8601": "2023-05-07T22:09:31.243472Z",
"url": "https://files.pythonhosted.org/packages/68/98/03a2c04dc7db5321cacb3f8f949a88d97ee057248eac0fea6695c0fa2382/lakestream-0.0.3-cp310-cp310-macosx_10_7_x86_64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "5a4f3d1d7d6340b3140d9ae6a4cd53046947a620ed403ca5b7b6fa7f139736c0",
"md5": "4f55fb5095edfc73feec8f004a56a620",
"sha256": "9d9a50adbd4cb37173c7127d82ca6b72e3d3461825d7a9a872757a1f9e7f3602"
},
"downloads": -1,
"filename": "lakestream-0.0.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl",
"has_sig": false,
"md5_digest": "4f55fb5095edfc73feec8f004a56a620",
"packagetype": "bdist_wheel",
"python_version": "cp310",
"requires_python": ">=3.7",
"size": 2357171,
"upload_time": "2023-05-07T22:09:34",
"upload_time_iso_8601": "2023-05-07T22:09:34.338465Z",
"url": "https://files.pythonhosted.org/packages/5a/4f/3d1d7d6340b3140d9ae6a4cd53046947a620ed403ca5b7b6fa7f139736c0/lakestream-0.0.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "77fa1ee8895bc4e9128c648a97a482802486e0decfbb6d59c27b4f22c3d102b9",
"md5": "88b2f0f519bf75ab64596445bc1a8beb",
"sha256": "b44c82fe8bfb7da98ce1e8d6188064e8838d2488078620c5c78e0d70d1a93282"
},
"downloads": -1,
"filename": "lakestream-0.0.3-cp311-cp311-macosx_10_7_x86_64.whl",
"has_sig": false,
"md5_digest": "88b2f0f519bf75ab64596445bc1a8beb",
"packagetype": "bdist_wheel",
"python_version": "cp311",
"requires_python": ">=3.7",
"size": 1183459,
"upload_time": "2023-05-07T22:09:36",
"upload_time_iso_8601": "2023-05-07T22:09:36.362832Z",
"url": "https://files.pythonhosted.org/packages/77/fa/1ee8895bc4e9128c648a97a482802486e0decfbb6d59c27b4f22c3d102b9/lakestream-0.0.3-cp311-cp311-macosx_10_7_x86_64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "f22ff415f572e03f3ae2ed37d131e7aab174be8e22a0565f3b4adf672ed68f75",
"md5": "c82ad26fd73cb0dc5e329c9c38327e15",
"sha256": "7bbff3297f7c888f78199dcecd2c137a2acd0fd576dd70521447fd822488c939"
},
"downloads": -1,
"filename": "lakestream-0.0.3-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl",
"has_sig": false,
"md5_digest": "c82ad26fd73cb0dc5e329c9c38327e15",
"packagetype": "bdist_wheel",
"python_version": "cp311",
"requires_python": ">=3.7",
"size": 2357171,
"upload_time": "2023-05-07T22:09:39",
"upload_time_iso_8601": "2023-05-07T22:09:39.104985Z",
"url": "https://files.pythonhosted.org/packages/f2/2f/f415f572e03f3ae2ed37d131e7aab174be8e22a0565f3b4adf672ed68f75/lakestream-0.0.3-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "d7d41b295dce874be64bdfe08593744d00fc74aec2ca428a8662e82cc551074b",
"md5": "30045c3bef071def7c3b12bd455a05b3",
"sha256": "fb75b600343dc16afdf3c1f75e72db61b8b60d45f2b596cb329a133588054e97"
},
"downloads": -1,
"filename": "lakestream-0.0.3-cp37-cp37m-macosx_10_7_x86_64.whl",
"has_sig": false,
"md5_digest": "30045c3bef071def7c3b12bd455a05b3",
"packagetype": "bdist_wheel",
"python_version": "cp37",
"requires_python": ">=3.7",
"size": 1183703,
"upload_time": "2023-05-07T22:09:40",
"upload_time_iso_8601": "2023-05-07T22:09:40.854491Z",
"url": "https://files.pythonhosted.org/packages/d7/d4/1b295dce874be64bdfe08593744d00fc74aec2ca428a8662e82cc551074b/lakestream-0.0.3-cp37-cp37m-macosx_10_7_x86_64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "c5a8519886833556af0d11219444ec9994a1a2e544672d82720734f19a43e8c3",
"md5": "07eccd5844ce50561ddc8124321bd830",
"sha256": "92fdb6646bb86f6c91e2a5f01268a65187754a465dbbca7ab43ff6068cc80fe7"
},
"downloads": -1,
"filename": "lakestream-0.0.3-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl",
"has_sig": false,
"md5_digest": "07eccd5844ce50561ddc8124321bd830",
"packagetype": "bdist_wheel",
"python_version": "cp37",
"requires_python": ">=3.7",
"size": 2357254,
"upload_time": "2023-05-07T22:09:43",
"upload_time_iso_8601": "2023-05-07T22:09:43.613526Z",
"url": "https://files.pythonhosted.org/packages/c5/a8/519886833556af0d11219444ec9994a1a2e544672d82720734f19a43e8c3/lakestream-0.0.3-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "4eafc1533d0a82fb368a987c37e42984cdac18e0e62a07c67247007d20f4c681",
"md5": "8a70106f5eeb4c67c8407289295122a8",
"sha256": "8a006ae9b087b4d0fd1d2d858405ab00982dc44b0a99866e749838b152ecc827"
},
"downloads": -1,
"filename": "lakestream-0.0.3-cp38-cp38-macosx_10_7_x86_64.whl",
"has_sig": false,
"md5_digest": "8a70106f5eeb4c67c8407289295122a8",
"packagetype": "bdist_wheel",
"python_version": "cp38",
"requires_python": ">=3.7",
"size": 1183696,
"upload_time": "2023-05-07T22:09:45",
"upload_time_iso_8601": "2023-05-07T22:09:45.712618Z",
"url": "https://files.pythonhosted.org/packages/4e/af/c1533d0a82fb368a987c37e42984cdac18e0e62a07c67247007d20f4c681/lakestream-0.0.3-cp38-cp38-macosx_10_7_x86_64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "c2466760b885f8dffbb3df00752745f4e0be2d46ad03b1aff7449da5b21911b1",
"md5": "40f2d4222b3c7ce90dcf2293a684c6ff",
"sha256": "dbcbbeac5fa29a959c2f2bc145fab4ac929c9666653c82d8b64979ddb8b324c0"
},
"downloads": -1,
"filename": "lakestream-0.0.3-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl",
"has_sig": false,
"md5_digest": "40f2d4222b3c7ce90dcf2293a684c6ff",
"packagetype": "bdist_wheel",
"python_version": "cp38",
"requires_python": ">=3.7",
"size": 2357347,
"upload_time": "2023-05-07T22:09:47",
"upload_time_iso_8601": "2023-05-07T22:09:47.510069Z",
"url": "https://files.pythonhosted.org/packages/c2/46/6760b885f8dffbb3df00752745f4e0be2d46ad03b1aff7449da5b21911b1/lakestream-0.0.3-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "705a86d6ef570cf28894abde61f87531ac30d5444310f74ebfc5646d25eb912b",
"md5": "b5be62bfe3e7eebe42155717aeac43ff",
"sha256": "e065a34d29c64fdaed313cdcf741d070453a236d83660c36396d47c9a91846a6"
},
"downloads": -1,
"filename": "lakestream-0.0.3-cp39-cp39-macosx_10_7_x86_64.whl",
"has_sig": false,
"md5_digest": "b5be62bfe3e7eebe42155717aeac43ff",
"packagetype": "bdist_wheel",
"python_version": "cp39",
"requires_python": ">=3.7",
"size": 1183528,
"upload_time": "2023-05-07T22:09:49",
"upload_time_iso_8601": "2023-05-07T22:09:49.155396Z",
"url": "https://files.pythonhosted.org/packages/70/5a/86d6ef570cf28894abde61f87531ac30d5444310f74ebfc5646d25eb912b/lakestream-0.0.3-cp39-cp39-macosx_10_7_x86_64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "58b973838e50ca73e28c6e3b7f66b5f1a1c0d0f1c20b344f18f6cfb2f8ef2160",
"md5": "7e8701d2446287a5ad8f85d5589fa688",
"sha256": "6d5f788efcb88af079a040dc3121ba11771622541d5d7109e737a40a7f02ef41"
},
"downloads": -1,
"filename": "lakestream-0.0.3-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl",
"has_sig": false,
"md5_digest": "7e8701d2446287a5ad8f85d5589fa688",
"packagetype": "bdist_wheel",
"python_version": "cp39",
"requires_python": ">=3.7",
"size": 2357170,
"upload_time": "2023-05-07T22:09:51",
"upload_time_iso_8601": "2023-05-07T22:09:51.446820Z",
"url": "https://files.pythonhosted.org/packages/58/b9/73838e50ca73e28c6e3b7f66b5f1a1c0d0f1c20b344f18f6cfb2f8ef2160/lakestream-0.0.3-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "8d54a03a5f4a20b4b69b3b21d71136b2da1399e9022243225cb744348c7a56d0",
"md5": "0e4c1ef0013e3c1688c418d5294ca427",
"sha256": "a4bcb23445ff223bacc347d4a264e31ff1d09f46e0abc9d4f3f719ebc8341305"
},
"downloads": -1,
"filename": "lakestream-0.0.3.tar.gz",
"has_sig": false,
"md5_digest": "0e4c1ef0013e3c1688c418d5294ca427",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.7",
"size": 54848,
"upload_time": "2023-05-07T22:09:53",
"upload_time_iso_8601": "2023-05-07T22:09:53.109407Z",
"url": "https://files.pythonhosted.org/packages/8d/54/a03a5f4a20b4b69b3b21d71136b2da1399e9022243225cb744348c7a56d0/lakestream-0.0.3.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-05-07 22:09:53",
"github": false,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"lcname": "lakestream"
}
