## PFIO
PFIO is an IO abstraction library developed by PFN, optimized for deep
learning training with batteries included. It supports
- Filesystem API abstraction with unified error semantics,
- Explicit user-land caching system,
- IO performance tracing and metrics stats, and
- Fileset container utilities to save metadata.
## Dependency
- HDFS client and libhdfs for HDFS access
- CPython >= 3.9
## Installation and Document build
Installation
```shell
$ git clone https://github.com/pfnet/pfio.git
$ cd pfio
$ pip install .
```
Documentation
```sh
$ cd pfio/docs
$ make html
$ open build/html/index.html
```
## How to use
Please refer to the [official document](https://pfio.readthedocs.io) for more information about the usage.
## Release
Check [the official document](https://packaging.python.org/tutorials/packaging-projects/) for latest release procedure.
Run tests locally:
```sh
$ pip install tox
$ tox
```
Bump version numbers in `pfio/version.py` .
Push and open a pull request to invoke CI. Once CI passed and the pull request merged,
tag a release:
```sh
$ git tag -s X.Y.Z
$ git push --tags
```
Build:
```sh
$ rm -rf dist
$ pip3 install --user build
$ python3 -m build
```
Release to PyPI:
```sh
$ python3 -m pip install --user --upgrade twine
$ python3 -m twine upload --repository testpypi dist/*
```
Raw data
{
"_id": null,
"home_page": "http://github.com/pfnet/pfio",
"name": "pfio",
"maintainer": null,
"docs_url": null,
"requires_python": null,
"maintainer_email": null,
"keywords": "filesystem hdfs chainer development",
"author": "Tianqi Xu, Kota Uenishi",
"author_email": "tianqi@preferred.jp, kota@preferred.jp",
"download_url": "https://files.pythonhosted.org/packages/51/20/d9a873a2b552baff2b2983a7ed67c506bfcfad45f178a2b593fdeb5bd568/pfio-2.9.0.tar.gz",
"platform": null,
"description": "## PFIO\n\n\n\nPFIO is an IO abstraction library developed by PFN, optimized for deep\nlearning training with batteries included. It supports\n\n- Filesystem API abstraction with unified error semantics,\n- Explicit user-land caching system,\n- IO performance tracing and metrics stats, and\n- Fileset container utilities to save metadata.\n\n\n## Dependency\n\n- HDFS client and libhdfs for HDFS access\n- CPython >= 3.9\n\n## Installation and Document build\n\nInstallation\n\n```shell\n$ git clone https://github.com/pfnet/pfio.git\n$ cd pfio\n$ pip install .\n```\n\nDocumentation\n```sh\n$ cd pfio/docs\n$ make html\n$ open build/html/index.html\n```\n\n## How to use\n\nPlease refer to the [official document](https://pfio.readthedocs.io) for more information about the usage.\n\n## Release\n\nCheck [the official document](https://packaging.python.org/tutorials/packaging-projects/) for latest release procedure.\n\nRun tests locally:\n\n```sh\n$ pip install tox\n$ tox\n```\n\nBump version numbers in `pfio/version.py` .\n\nPush and open a pull request to invoke CI. Once CI passed and the pull request merged,\ntag a release:\n\n```sh\n$ git tag -s X.Y.Z\n$ git push --tags\n```\n\nBuild:\n\n```sh\n$ rm -rf dist\n$ pip3 install --user build\n$ python3 -m build\n```\n\nRelease to PyPI:\n\n```sh\n$ python3 -m pip install --user --upgrade twine\n$ python3 -m twine upload --repository testpypi dist/*\n```\n",
"bugtrack_url": null,
"license": null,
"summary": "PFN IO library",
"version": "2.9.0",
"project_urls": {
"Homepage": "http://github.com/pfnet/pfio"
},
"split_keywords": [
"filesystem",
"hdfs",
"chainer",
"development"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "fd0f18a64949c3e8bb2e309c1d2f4670e2d23aa9fbd1a9e08eaf224725a4429f",
"md5": "76b5a1d48826406e7ee5eb118e2373a0",
"sha256": "41e6319ae51df5a58eef217896fc941f728328274a9233b5b645cfc62114b02d"
},
"downloads": -1,
"filename": "pfio-2.9.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "76b5a1d48826406e7ee5eb118e2373a0",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": null,
"size": 48533,
"upload_time": "2024-10-29T08:15:47",
"upload_time_iso_8601": "2024-10-29T08:15:47.946508Z",
"url": "https://files.pythonhosted.org/packages/fd/0f/18a64949c3e8bb2e309c1d2f4670e2d23aa9fbd1a9e08eaf224725a4429f/pfio-2.9.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "5120d9a873a2b552baff2b2983a7ed67c506bfcfad45f178a2b593fdeb5bd568",
"md5": "98147f49d6d01fe95a24b8fdfd3728b7",
"sha256": "0a23d3bbc7baf85cad6cdb22e1582ad59a786baef96bbaac699edc115080ab70"
},
"downloads": -1,
"filename": "pfio-2.9.0.tar.gz",
"has_sig": false,
"md5_digest": "98147f49d6d01fe95a24b8fdfd3728b7",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 41178,
"upload_time": "2024-10-29T08:15:49",
"upload_time_iso_8601": "2024-10-29T08:15:49.439852Z",
"url": "https://files.pythonhosted.org/packages/51/20/d9a873a2b552baff2b2983a7ed67c506bfcfad45f178a2b593fdeb5bd568/pfio-2.9.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-10-29 08:15:49",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "pfnet",
"github_project": "pfio",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"tox": true,
"lcname": "pfio"
}
JSON
