# Pachyderm's Python SDK
Official Python client/SDK for Pachyderm.
The successor to https://github.com/pachyderm/python-pachyderm.
This library provides the autogenerated gRPC/protobuf code for Pachyderm,
generated using [a fork of the betterproto package](https://github.com/pachyderm/python-betterproto),
along with higher-level functionality.
## Installation
```bash
pip install pachyderm_sdk
```
## A Small Taste
Here's an example that creates a repo and adds a file:
```python
from pachyderm_sdk import Client
from pachyderm_sdk.api import pfs
# Connects to a pachyderm cluster using your local config
# at ~/.pachyderm/config.json
client = Client.from_config()
# Creates a pachyderm repo called `test`
repo = pfs.Repo(name="test")
client.pfs.create_repo(repo=repo)
# Create a new commit in `test@master` and upload a file.
branch = pfs.Branch.from_uri("test@master")
with client.pfs.commit(branch=branch) as commit:
file = commit.put_file_from_bytes(path="/data/file.dat", data=b"DATA")
# Retrieve the uploaded file.
with client.pfs.pfs_file(file) as f:
print(f.readall())
```
How to load a CAST file into a pandas dataframe
```python
from pachyderm_sdk import Client
from pachyderm_sdk.api import pfs
import pandas as pd
client = Client.from_config()
file = pfs.File.from_uri("test@master:/path/to/data.csv")
with client.pfs.pfs_file(file) as f:
df = pd.read_csv(f)
```
## Changes from Python-Pachyderm
This package is a successor to the python-pachyderm package.
Listed below are some of the notable changes:
1. Organization of the API
* Methods and Message objects are now organized according to the
service they are associated with, i.e. auth, pfs (pachyderm file-system),
pps (pachyderm pipelining-system).
* Message objects can be found within their respective submodule of the
`pachyder_sdk.api` module, i.e. `pachyderm_sdk.api.pfs`.
* Methods can be found within their respective attribute of the `Client`
class, i.e. `client.pps.create_pipeline`.
* Some methods have been renamed to remove redundancy due to this organization, i.e.
`python_pachyderm.Client.get_enterprise_state` -> `pachyderm_sdk.Client.enterprise.get_state`
2. The autogenerated code is generated using a fork of the betterproto compiler.
* Messages are now python dataclasses.
* Methods require keyword arguments.
* Pachyderm resources are specified using types.
- python-pachyderm (old): `client.create_repo("test")`
- pachyderm_sdk (new): `client.pfs.create_repo(repo=pfs.Repo(name="test"))`
## Contributing
Please see [the contributing guide](./CONTRIBUTING.md) for more info (including testing instructions)
## Developer Guide
Generate python APIs from protobuf:
```
./generate-protos.sh
```
Generate HTML documentation (writes to docs/pachyderm_sdk):
```bash
make docs
```
Running Tests:
```
pytest -vvv tests
```
Raw data
{
"_id": null,
"home_page": "https://github.com/pachyderm/pachyderm/tree/master/python-sdk",
"name": "pachyderm-sdk",
"maintainer": null,
"docs_url": null,
"requires_python": "<4.0,>=3.9",
"maintainer_email": null,
"keywords": "pachyderm",
"author": "Pachyderm Integrations",
"author_email": "integrations@pachyderm.io",
"download_url": "https://files.pythonhosted.org/packages/ae/d9/91be873d242f5db6732cb77e218dcc5018f23369aefc25ba5c76e9ee264f/pachyderm_sdk-2.11.6.tar.gz",
"platform": null,
"description": "# Pachyderm's Python SDK\n\nOfficial Python client/SDK for Pachyderm.\nThe successor to https://github.com/pachyderm/python-pachyderm.\n\nThis library provides the autogenerated gRPC/protobuf code for Pachyderm,\n generated using [a fork of the betterproto package](https://github.com/pachyderm/python-betterproto),\n along with higher-level functionality.\n\n## Installation\n```bash\npip install pachyderm_sdk\n```\n\n## A Small Taste\nHere's an example that creates a repo and adds a file:\n```python\nfrom pachyderm_sdk import Client\nfrom pachyderm_sdk.api import pfs\n\n# Connects to a pachyderm cluster using your local config\n# at ~/.pachyderm/config.json\nclient = Client.from_config()\n\n# Creates a pachyderm repo called `test`\nrepo = pfs.Repo(name=\"test\")\nclient.pfs.create_repo(repo=repo)\n\n# Create a new commit in `test@master` and upload a file.\nbranch = pfs.Branch.from_uri(\"test@master\")\nwith client.pfs.commit(branch=branch) as commit:\n file = commit.put_file_from_bytes(path=\"/data/file.dat\", data=b\"DATA\")\n\n# Retrieve the uploaded file.\nwith client.pfs.pfs_file(file) as f:\n print(f.readall())\n```\n\nHow to load a CAST file into a pandas dataframe\n```python\nfrom pachyderm_sdk import Client\nfrom pachyderm_sdk.api import pfs\nimport pandas as pd\n\nclient = Client.from_config()\nfile = pfs.File.from_uri(\"test@master:/path/to/data.csv\")\nwith client.pfs.pfs_file(file) as f:\n df = pd.read_csv(f)\n```\n\n## Changes from Python-Pachyderm\nThis package is a successor to the python-pachyderm package.\nListed below are some of the notable changes:\n1. Organization of the API\n * Methods and Message objects are now organized according to the \n service they are associated with, i.e. auth, pfs (pachyderm file-system),\n pps (pachyderm pipelining-system).\n * Message objects can be found within their respective submodule of the\n `pachyder_sdk.api` module, i.e. `pachyderm_sdk.api.pfs`.\n * Methods can be found within their respective attribute of the `Client`\n class, i.e. `client.pps.create_pipeline`.\n * Some methods have been renamed to remove redundancy due to this organization, i.e.\n `python_pachyderm.Client.get_enterprise_state` -> `pachyderm_sdk.Client.enterprise.get_state`\n2. The autogenerated code is generated using a fork of the betterproto compiler.\n * Messages are now python dataclasses.\n * Methods require keyword arguments.\n * Pachyderm resources are specified using types.\n - python-pachyderm (old): `client.create_repo(\"test\")`\n - pachyderm_sdk (new): `client.pfs.create_repo(repo=pfs.Repo(name=\"test\"))`\n\n## Contributing\nPlease see [the contributing guide](./CONTRIBUTING.md) for more info (including testing instructions)\n\n\n## Developer Guide\n\nGenerate python APIs from protobuf:\n```\n./generate-protos.sh\n```\n\nGenerate HTML documentation (writes to docs/pachyderm_sdk):\n```bash\nmake docs\n```\n\nRunning Tests:\n\n```\npytest -vvv tests\n```",
"bugtrack_url": null,
"license": "Apache 2.0",
"summary": "Python Pachyderm Client",
"version": "2.11.6",
"project_urls": {
"Documentation": "https://docs.pachyderm.com/latest/sdk/python/",
"Homepage": "https://github.com/pachyderm/pachyderm/tree/master/python-sdk",
"Repository": "https://github.com/pachyderm/pachyderm/tree/master/python-sdk"
},
"split_keywords": [
"pachyderm"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "e5032021846399ad35de25eb2c3952c91b89ca7a4174be3e95371bf8bd9f11cb",
"md5": "ecdc9ff0c315af2931f8530e090f5a4a",
"sha256": "961b8c9fd88dd341765f19c8265165d067ace129980a762f08f97e8f288d947f"
},
"downloads": -1,
"filename": "pachyderm_sdk-2.11.6-py3-none-any.whl",
"has_sig": false,
"md5_digest": "ecdc9ff0c315af2931f8530e090f5a4a",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": "<4.0,>=3.9",
"size": 86905,
"upload_time": "2024-12-13T17:42:09",
"upload_time_iso_8601": "2024-12-13T17:42:09.367541Z",
"url": "https://files.pythonhosted.org/packages/e5/03/2021846399ad35de25eb2c3952c91b89ca7a4174be3e95371bf8bd9f11cb/pachyderm_sdk-2.11.6-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "aed991be873d242f5db6732cb77e218dcc5018f23369aefc25ba5c76e9ee264f",
"md5": "cdcbfa391d204f72c362b325861abdf0",
"sha256": "f87a4a969c115796457f5d41129733da3c01761415218518fc984f4714862de4"
},
"downloads": -1,
"filename": "pachyderm_sdk-2.11.6.tar.gz",
"has_sig": false,
"md5_digest": "cdcbfa391d204f72c362b325861abdf0",
"packagetype": "sdist",
"python_version": "source",
"requires_python": "<4.0,>=3.9",
"size": 71336,
"upload_time": "2024-12-13T17:42:10",
"upload_time_iso_8601": "2024-12-13T17:42:10.611417Z",
"url": "https://files.pythonhosted.org/packages/ae/d9/91be873d242f5db6732cb77e218dcc5018f23369aefc25ba5c76e9ee264f/pachyderm_sdk-2.11.6.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-12-13 17:42:10",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "pachyderm",
"github_project": "pachyderm",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"circle": true,
"lcname": "pachyderm-sdk"
}
JSON
