# iden
<p align="center">
<a href="https://github.com/durandtibo/iden/actions">
<img alt="CI" src="https://github.com/durandtibo/iden/workflows/CI/badge.svg">
</a>
<a href="https://github.com/durandtibo/iden/actions">
<img alt="Nightly Tests" src="https://github.com/durandtibo/iden/workflows/Nightly%20Tests/badge.svg">
</a>
<a href="https://github.com/durandtibo/iden/actions">
<img alt="Nightly Package Tests" src="https://github.com/durandtibo/iden/workflows/Nightly%20Package%20Tests/badge.svg">
</a>
<br/>
<a href="https://durandtibo.github.io/iden/">
<img alt="Documentation" src="https://github.com/durandtibo/iden/workflows/Documentation%20(stable)/badge.svg">
</a>
<a href="https://durandtibo.github.io/iden/">
<img alt="Documentation" src="https://github.com/durandtibo/iden/workflows/Documentation%20(unstable)/badge.svg">
</a>
<br/>
<a href="https://codecov.io/gh/durandtibo/iden">
<img alt="Codecov" src="https://codecov.io/gh/durandtibo/iden/branch/main/graph/badge.svg">
</a>
<a href="https://codeclimate.com/github/durandtibo/iden/maintainability">
<img src="https://api.codeclimate.com/v1/badges/e1d6a491463b53dde654/maintainability" />
</a>
<a href="https://codeclimate.com/github/durandtibo/iden/test_coverage">
<img src="https://api.codeclimate.com/v1/badges/e1d6a491463b53dde654/test_coverage" />
</a>
<br/>
<a href="https://github.com/psf/black">
<img alt="Code style: black" src="https://img.shields.io/badge/code%20style-black-000000.svg">
</a>
<a href="https://google.github.io/styleguide/pyguide.html#s3.8-comments-and-docstrings">
<img alt="Doc style: google" src="https://img.shields.io/badge/%20style-google-3666d6.svg">
</a>
<a href="https://github.com/astral-sh/ruff">
<img src="https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json" alt="Ruff" style="max-width:100%;">
</a>
<a href="https://github.com/guilatrova/tryceratops">
<img alt="Doc style: google" src="https://img.shields.io/badge/try%2Fexcept%20style-tryceratops%20%F0%9F%A6%96%E2%9C%A8-black">
</a>
<br/>
<a href="https://pypi.org/project/iden/">
<img alt="PYPI version" src="https://img.shields.io/pypi/v/iden">
</a>
<a href="https://pypi.org/project/iden/">
<img alt="Python" src="https://img.shields.io/pypi/pyversions/iden.svg">
</a>
<a href="https://opensource.org/licenses/BSD-3-Clause">
<img alt="BSD-3-Clause" src="https://img.shields.io/pypi/l/iden">
</a>
<br/>
<a href="https://pepy.tech/project/iden">
<img alt="Downloads" src="https://static.pepy.tech/badge/iden">
</a>
<a href="https://pepy.tech/project/iden">
<img alt="Monthly downloads" src="https://static.pepy.tech/badge/iden/month">
</a>
<br/>
</p>
## Overview
`iden` is a simple Python library to manage a dataset of shards when training a machine learning
model.
`iden` uses a lazy loading approach to load the shard's data, so it is easy to manage shards without
loading their data.
`iden` supports different format to store shards on disk.
## Installation
We highly recommend installing
a [virtual environment](https://packaging.python.org/guides/installing-using-pip-and-virtual-environments/).
`iden` can be installed from pip using the following command:
```shell
pip install iden
```
To make the package as slim as possible, only the minimal packages required to use `iden` are
installed.
To include all the dependencies, you can use the following command:
```shell
pip install iden[all]
```
Please check the [get started page](https://durandtibo.github.io/iden/get_started) to see how to
install only some specific dependencies or other alternatives to install the library.
The following is the corresponding `iden` versions and tested dependencies.
| `iden` | `coola` | `objectory` | `numpy`<sup>*</sup> | `pyyaml`<sup>*</sup> | `safetensors`<sup>*</sup> | `torch`<sup>*</sup> | `python` |
|---------|--------------|--------------|---------------------|----------------------|---------------------------|---------------------|---------------|
| `main` | `>=0.3,<1.0` | `>=0.1,<1.0` | `>=1.22,<2.0` | `>=6.0,<7.0` | `>=0.4,<1.0` | `>=2.0,<3.0` | `>=3.9,<3.12` |
| `0.0.2` | `>=0.4,<1.0` | `>=0.1,<1.0` | `>=1.22,<2.0` | `>=6.0,<7.0` | `>=0.4,<1.0` | `>=2.0,<2.1` | `>=3.9,<3.12` |
| `0.0.1` | `>=0.4,<1.0` | `>=0.1,<1.0` | `>=1.22,<2.0` | `>=6.0,<7.0` | `>=0.4,<1.0` | `>=2.0,<2.1` | `>=3.9,<3.12` |
<sup>*</sup> indicates an optional dependency
## Contributing
Please check the instructions in [CONTRIBUTING.md](.github/CONTRIBUTING.md).
## Suggestions and Communication
Everyone is welcome to contribute to the community.
If you have any questions or suggestions, you can
submit [Github Issues](https://github.com/durandtibo/iden/issues).
We will reply to you as soon as possible. Thank you very much.
## API stability
:warning: While `iden` is in development stage, no API is guaranteed to be stable from one
release to the next.
In fact, it is very likely that the API will change multiple times before a stable 1.0.0 release.
In practice, this means that upgrading `iden` to a new version will possibly break any code that
was using the old version of `iden`.
## License
`iden` is licensed under BSD 3-Clause "New" or "Revised" license available in [LICENSE](LICENSE)
file.
Raw data
{
"_id": null,
"home_page": "https://github.com/durandtibo/iden",
"name": "iden",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.9,<3.13",
"maintainer_email": "",
"keywords": "dataset,shard",
"author": "Thibaut Durand",
"author_email": "durand.tibo+gh@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/ff/ab/36a08a6d2617d67f4b566692cbbcfa6586c3767f60904672ce8aa8e28e6a/iden-0.0.3.tar.gz",
"platform": null,
"description": "# iden\n\n<p align=\"center\">\n <a href=\"https://github.com/durandtibo/iden/actions\">\n <img alt=\"CI\" src=\"https://github.com/durandtibo/iden/workflows/CI/badge.svg\">\n </a>\n <a href=\"https://github.com/durandtibo/iden/actions\">\n <img alt=\"Nightly Tests\" src=\"https://github.com/durandtibo/iden/workflows/Nightly%20Tests/badge.svg\">\n </a>\n <a href=\"https://github.com/durandtibo/iden/actions\">\n <img alt=\"Nightly Package Tests\" src=\"https://github.com/durandtibo/iden/workflows/Nightly%20Package%20Tests/badge.svg\">\n </a>\n <br/>\n <a href=\"https://durandtibo.github.io/iden/\">\n <img alt=\"Documentation\" src=\"https://github.com/durandtibo/iden/workflows/Documentation%20(stable)/badge.svg\">\n </a>\n <a href=\"https://durandtibo.github.io/iden/\">\n <img alt=\"Documentation\" src=\"https://github.com/durandtibo/iden/workflows/Documentation%20(unstable)/badge.svg\">\n </a>\n <br/>\n <a href=\"https://codecov.io/gh/durandtibo/iden\">\n <img alt=\"Codecov\" src=\"https://codecov.io/gh/durandtibo/iden/branch/main/graph/badge.svg\">\n </a>\n <a href=\"https://codeclimate.com/github/durandtibo/iden/maintainability\">\n <img src=\"https://api.codeclimate.com/v1/badges/e1d6a491463b53dde654/maintainability\" />\n </a>\n <a href=\"https://codeclimate.com/github/durandtibo/iden/test_coverage\">\n <img src=\"https://api.codeclimate.com/v1/badges/e1d6a491463b53dde654/test_coverage\" />\n </a>\n <br/>\n <a href=\"https://github.com/psf/black\">\n <img alt=\"Code style: black\" src=\"https://img.shields.io/badge/code%20style-black-000000.svg\">\n </a>\n <a href=\"https://google.github.io/styleguide/pyguide.html#s3.8-comments-and-docstrings\">\n <img alt=\"Doc style: google\" src=\"https://img.shields.io/badge/%20style-google-3666d6.svg\">\n </a>\n <a href=\"https://github.com/astral-sh/ruff\">\n <img src=\"https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json\" alt=\"Ruff\" style=\"max-width:100%;\">\n </a>\n <a href=\"https://github.com/guilatrova/tryceratops\">\n <img alt=\"Doc style: google\" src=\"https://img.shields.io/badge/try%2Fexcept%20style-tryceratops%20%F0%9F%A6%96%E2%9C%A8-black\">\n </a>\n <br/>\n <a href=\"https://pypi.org/project/iden/\">\n <img alt=\"PYPI version\" src=\"https://img.shields.io/pypi/v/iden\">\n </a>\n <a href=\"https://pypi.org/project/iden/\">\n <img alt=\"Python\" src=\"https://img.shields.io/pypi/pyversions/iden.svg\">\n </a>\n <a href=\"https://opensource.org/licenses/BSD-3-Clause\">\n <img alt=\"BSD-3-Clause\" src=\"https://img.shields.io/pypi/l/iden\">\n </a>\n <br/>\n <a href=\"https://pepy.tech/project/iden\">\n <img alt=\"Downloads\" src=\"https://static.pepy.tech/badge/iden\">\n </a>\n <a href=\"https://pepy.tech/project/iden\">\n <img alt=\"Monthly downloads\" src=\"https://static.pepy.tech/badge/iden/month\">\n </a>\n <br/>\n</p>\n\n## Overview\n\n`iden` is a simple Python library to manage a dataset of shards when training a machine learning\nmodel.\n`iden` uses a lazy loading approach to load the shard's data, so it is easy to manage shards without\nloading their data.\n`iden` supports different format to store shards on disk.\n\n## Installation\n\nWe highly recommend installing\na [virtual environment](https://packaging.python.org/guides/installing-using-pip-and-virtual-environments/).\n`iden` can be installed from pip using the following command:\n\n```shell\npip install iden\n```\n\nTo make the package as slim as possible, only the minimal packages required to use `iden` are\ninstalled.\nTo include all the dependencies, you can use the following command:\n\n```shell\npip install iden[all]\n```\n\nPlease check the [get started page](https://durandtibo.github.io/iden/get_started) to see how to\ninstall only some specific dependencies or other alternatives to install the library.\nThe following is the corresponding `iden` versions and tested dependencies.\n\n| `iden` | `coola` | `objectory` | `numpy`<sup>*</sup> | `pyyaml`<sup>*</sup> | `safetensors`<sup>*</sup> | `torch`<sup>*</sup> | `python` |\n|---------|--------------|--------------|---------------------|----------------------|---------------------------|---------------------|---------------|\n| `main` | `>=0.3,<1.0` | `>=0.1,<1.0` | `>=1.22,<2.0` | `>=6.0,<7.0` | `>=0.4,<1.0` | `>=2.0,<3.0` | `>=3.9,<3.12` |\n| `0.0.2` | `>=0.4,<1.0` | `>=0.1,<1.0` | `>=1.22,<2.0` | `>=6.0,<7.0` | `>=0.4,<1.0` | `>=2.0,<2.1` | `>=3.9,<3.12` |\n| `0.0.1` | `>=0.4,<1.0` | `>=0.1,<1.0` | `>=1.22,<2.0` | `>=6.0,<7.0` | `>=0.4,<1.0` | `>=2.0,<2.1` | `>=3.9,<3.12` |\n\n<sup>*</sup> indicates an optional dependency\n\n## Contributing\n\nPlease check the instructions in [CONTRIBUTING.md](.github/CONTRIBUTING.md).\n\n## Suggestions and Communication\n\nEveryone is welcome to contribute to the community.\nIf you have any questions or suggestions, you can\nsubmit [Github Issues](https://github.com/durandtibo/iden/issues).\nWe will reply to you as soon as possible. Thank you very much.\n\n## API stability\n\n:warning: While `iden` is in development stage, no API is guaranteed to be stable from one\nrelease to the next.\nIn fact, it is very likely that the API will change multiple times before a stable 1.0.0 release.\nIn practice, this means that upgrading `iden` to a new version will possibly break any code that\nwas using the old version of `iden`.\n\n## License\n\n`iden` is licensed under BSD 3-Clause \"New\" or \"Revised\" license available in [LICENSE](LICENSE)\nfile.\n",
"bugtrack_url": null,
"license": "BSD-3-Clause",
"summary": "simple library to manage a dataset of shards to train machine learning models",
"version": "0.0.3",
"project_urls": {
"Homepage": "https://github.com/durandtibo/iden",
"Repository": "https://github.com/durandtibo/iden"
},
"split_keywords": [
"dataset",
"shard"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "5ec0df0939da5ae37a3965357bdf26db76522760ebf4cc59cf78126ec16727f3",
"md5": "8d7ae7a15be7e19338024cbed5acd14a",
"sha256": "a674274f2621bedd6f2550e7bf9a476d3294258f5e8bf0730a4b4508a20ce059"
},
"downloads": -1,
"filename": "iden-0.0.3-py3-none-any.whl",
"has_sig": false,
"md5_digest": "8d7ae7a15be7e19338024cbed5acd14a",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.9,<3.13",
"size": 72655,
"upload_time": "2024-03-17T23:26:52",
"upload_time_iso_8601": "2024-03-17T23:26:52.382759Z",
"url": "https://files.pythonhosted.org/packages/5e/c0/df0939da5ae37a3965357bdf26db76522760ebf4cc59cf78126ec16727f3/iden-0.0.3-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "ffab36a08a6d2617d67f4b566692cbbcfa6586c3767f60904672ce8aa8e28e6a",
"md5": "649bb1d84360f5f7f5d5b12ca407d68c",
"sha256": "eaeb5bf37ec975771386fae17e224e50386fce279f031a54b888610381b7adc3"
},
"downloads": -1,
"filename": "iden-0.0.3.tar.gz",
"has_sig": false,
"md5_digest": "649bb1d84360f5f7f5d5b12ca407d68c",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.9,<3.13",
"size": 36809,
"upload_time": "2024-03-17T23:26:53",
"upload_time_iso_8601": "2024-03-17T23:26:53.850963Z",
"url": "https://files.pythonhosted.org/packages/ff/ab/36a08a6d2617d67f4b566692cbbcfa6586c3767f60904672ce8aa8e28e6a/iden-0.0.3.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-03-17 23:26:53",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "durandtibo",
"github_project": "iden",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "iden"
}
JSON
