## PFIO
PFIO is an IO abstraction library developed by PFN, optimized for deep
learning training with batteries included. It supports
- Filesystem API abstraction with unified error semantics,
- Explicit user-land caching system,
- IO performance tracing and metrics stats, and
- Fileset container utilities to save metadata.
## Dependency
- HDFS client and libhdfs for HDFS access
- CPython >= 3.8
## Installation and Document build
Installation
```shell
$ git clone https://github.com/pfnet/pfio.git
$ cd pfio
$ pip install .
```
Documentation
```sh
$ cd pfio/docs
$ make html
$ open build/html/index.html
```
## How to use
Please refer to the [official document](https://pfio.readthedocs.io) for more information about the usage.
## Release
Check [the official document](https://packaging.python.org/tutorials/packaging-projects/) for latest release procedure.
Run tests locally:
```sh
$ pip install tox
$ tox
```
Bump version numbers in `pfio/version.py` .
Push and open a pull request to invoke CI. Once CI passed and the pull request merged,
tag a release:
```sh
$ git tag -s X.Y.Z
$ git push --tags
```
Build:
```sh
$ rm -rf dist
$ pip3 install --user build
$ python3 -m build
```
Release to PyPI:
```sh
$ python3 -m pip install --user --upgrade twine
$ python3 -m twine upload --repository testpypi dist/*
```
Raw data
{
"_id": null,
"home_page": "http://github.com/pfnet/pfio",
"name": "pfio",
"maintainer": null,
"docs_url": null,
"requires_python": null,
"maintainer_email": null,
"keywords": "filesystem hdfs chainer development",
"author": "Tianqi Xu, Kota Uenishi",
"author_email": "tianqi@preferred.jp, kota@preferred.jp",
"download_url": "https://files.pythonhosted.org/packages/20/f1/e99bda28b4dab8901ce07d17c33d58f05aeeed138e5853d6813b24455474/pfio-2.8.0.tar.gz",
"platform": null,
"description": "## PFIO\n\n\n\nPFIO is an IO abstraction library developed by PFN, optimized for deep\nlearning training with batteries included. It supports\n\n- Filesystem API abstraction with unified error semantics,\n- Explicit user-land caching system,\n- IO performance tracing and metrics stats, and\n- Fileset container utilities to save metadata.\n\n\n## Dependency\n\n- HDFS client and libhdfs for HDFS access\n- CPython >= 3.8\n\n## Installation and Document build\n\nInstallation\n\n```shell\n$ git clone https://github.com/pfnet/pfio.git\n$ cd pfio\n$ pip install .\n```\n\nDocumentation\n```sh\n$ cd pfio/docs\n$ make html\n$ open build/html/index.html\n```\n\n## How to use\n\nPlease refer to the [official document](https://pfio.readthedocs.io) for more information about the usage.\n\n## Release\n\nCheck [the official document](https://packaging.python.org/tutorials/packaging-projects/) for latest release procedure.\n\nRun tests locally:\n\n```sh\n$ pip install tox\n$ tox\n```\n\nBump version numbers in `pfio/version.py` .\n\nPush and open a pull request to invoke CI. Once CI passed and the pull request merged,\ntag a release:\n\n```sh\n$ git tag -s X.Y.Z\n$ git push --tags\n```\n\nBuild:\n\n```sh\n$ rm -rf dist\n$ pip3 install --user build\n$ python3 -m build\n```\n\nRelease to PyPI:\n\n```sh\n$ python3 -m pip install --user --upgrade twine\n$ python3 -m twine upload --repository testpypi dist/*\n```\n",
"bugtrack_url": null,
"license": null,
"summary": "PFN IO library",
"version": "2.8.0",
"project_urls": {
"Homepage": "http://github.com/pfnet/pfio"
},
"split_keywords": [
"filesystem",
"hdfs",
"chainer",
"development"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "927a6ecd508d28d8804a67fea4724d887924884f26a660cb79de5a6f671a2090",
"md5": "6494ae16e7b0a09f5123e94c98a5826f",
"sha256": "3b70596f683cb316da385299d137de29fe972cfde448f495c60203de77c4dc5f"
},
"downloads": -1,
"filename": "pfio-2.8.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "6494ae16e7b0a09f5123e94c98a5826f",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": null,
"size": 46381,
"upload_time": "2024-04-17T00:08:43",
"upload_time_iso_8601": "2024-04-17T00:08:43.744918Z",
"url": "https://files.pythonhosted.org/packages/92/7a/6ecd508d28d8804a67fea4724d887924884f26a660cb79de5a6f671a2090/pfio-2.8.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "20f1e99bda28b4dab8901ce07d17c33d58f05aeeed138e5853d6813b24455474",
"md5": "bc416efb77c66fba382e496c645aa058",
"sha256": "7f609549fc688a82c416f1c9edd5b0fcbe1e6ab469b737d858f026fa5e4586cf"
},
"downloads": -1,
"filename": "pfio-2.8.0.tar.gz",
"has_sig": false,
"md5_digest": "bc416efb77c66fba382e496c645aa058",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 39033,
"upload_time": "2024-04-17T00:08:33",
"upload_time_iso_8601": "2024-04-17T00:08:33.591811Z",
"url": "https://files.pythonhosted.org/packages/20/f1/e99bda28b4dab8901ce07d17c33d58f05aeeed138e5853d6813b24455474/pfio-2.8.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-04-17 00:08:33",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "pfnet",
"github_project": "pfio",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"tox": true,
"lcname": "pfio"
}
JSON
