# h5mapper
``h5mapper`` is a pythonic ORM-like tool for reading and writing HDF5 data.
It is built on top of `h5py` and lets you define types of **.h5 files as python classes** which you can then easily
**create from raw sources** (e.g. files, urls...), **serve** (use as ``Dataset`` for a ``Dataloader``),
or dynamically populate (logs, checkpoints of an experiment).
## Content
- [Installation](#Installation)
- [Quickstart](#Quickstart)
- [TypedFile](#TypedFile)
- [Feature](#Feature)
- [Examples](#Examples)
- [Development](#Development)
- [License](#License)
## Installation
### ``pip``
``h5mapper`` is on pypi, to install it, one only needs to
```bash
pip install h5mapper
```
### developer install
for playing around with the internals of the package, a good solution is to first
```bash
git clone https://github.com/ktonal/h5mapper.git
```
and then
```bash
pip install -e h5mapper/
```
which installs the repo in editable mode.
## Quickstart
### TypedFile
``h5m`` assumes that you want to store collections of contiguous arrays in single datasets and that you want several such concatenated datasets in a file.
Thus, ``TypedFile`` allows you to create and read files that maintain a 2-d reference system, where contiguous arrays are stored within features and indexed by their source's id.
Such a file might then look like
```bash
<Experiment "experiment.h5">
----------------------------------------------------> sources' ids axis
| "planes/01.jpeg" | "train"
| |
| data/ |
| images/ (32, 32) | None
| labels/ (1, ) | None
| logs/ |
| loss/ None | (10000,)
| ...
V
features axis
```
where the entries correspond to the shapes of arrays or their absence (`None`).
> Note that this is a different approach than storing each file or image in a separate dataset.
> In this case, there would be an `h5py.Dataset` located at `data/images/planes/01.jpeg` although in our
> example, the only dataset is at `data/images/` and one of its regions is indexed by the id `"planes/01.jpeg"`
For interacting with files that follow this particular structure, simply define a class
```python
import h5mapper as h5m
class Experiment(h5m.TypedFile):
data = h5m.Group(
# your custom h5m.Feature classes:
images=Image(),
labels=DirLabels()
)
logs = h5m.Group(
loss=h5m.Array()
)
```
#### ``create``, ``add``
now, create an instance, load data from files through parallel jobs and add data on the fly :
```python
# create instance from raw sources
exp = Experiment.create("experiment.h5",
# those are then used as ids :
sources=["planes/01.jpeg", "planes/02.jpeg"],
n_workers=8)
...
# add id <-> data on the fly :
exp.logs.add("train", dict(loss=losses_array))
```
#### ``get``, ``refs`` and ``__getitem__``
There are 3 main options to read data from a ``TypedFile`` or one of its ``Proxy``
1/ By their id
```python
>> exp.logs.get("train")
Out: {"loss": np.array([...])}
# which, in this case, is equivalent to
>> exp.logs["train"]
Out: {"loss": np.array([...])}
# because `exp.logs` is a Group and Groups only support id-based indexing
```
2/ By the index of their ids through their ``refs`` attribute :
```python
>> exp.data.images[exp.data.images.refs[0]].shape
Out: (32, 32)
```
Which works because `exp.data.images` is a `Dataset` and only `Datasets` have `refs`
3/ with any ``item`` supported by the ``h5py.Dataset``
```python
>> exp.data.labels[:32]
Out: np.array([0, 0, ....])
```
Which only works for `Dataset`s - not for `Group`s.
> Note that, in this last case, you are indexing into the **concatenation of all sub-arrays along their first axis**.
> The same interface is also implemented for ``set(source, data)`` and ``__setitem__``
### Feature
``h5m`` exposes a class that helps you configure the behaviour of your ``TypedFile`` classes and the properties of the .h5 they create.
the ``Feature`` class helps you define :
- how sources' ids are loaded into arrays (``feature.load(source)``)
- which types of files are supported
- how the data is stored by ``h5py`` (compression, chunks)
- which extraction parameters need to be stored with the data (e.g. sample rate of audio files)
- custom-methods relevant to this kind of data
Once you defined a `Feature` class, attach it to the class dict of a ``TypedFile``, that's it!
For example :
```python
import h5mapper as h5m
class MyFeature(h5m.Feature):
# only sources matching this pattern will be passed to load(...)
__re__ = r".special$"
# args for the h5py.Dataset
__ds_kwargs__ = dict(compression='lzf', chunks=(1, 350))
def __init__(self, my_extraction_param=0):
self.my_extraction_param = my_extraction_param
@property
def attrs(self):
# those are then written in the h5py.Group.attrs
return {"p": self.my_extraction_param}
def load(self, source):
"""your method to get an np.ndarray or a dict thereof
from a path, an url, whatever sources you have..."""
return data
def plot(self, data):
"""custom plotting method for this kind of data"""
# ...
# attach it
class Data(h5m.TypedFile):
feat = MyFeature(47)
# load sources...
f = Data.create(....)
# read your data through __getitem__
batch = f.feat[4:8]
# access your method
f.feat.plot(batch)
# modify the file through __setitem__
f.feat[4:8] = batch ** 2
```
for more examples, checkout `h5mapper/h5mapper/features.py`.
#### ``serve``
Primarly designed with `pytorch` users in mind, `h5m` plays very nicely with the `Dataset` class :
```python
class MyDS(h5m.TypedFile, torch.utils.data.Dataset):
x = MyInputFeature(42)
def __getitem__(self, item):
return self.x[item], self.labels[item]
def __len__(self):
return len(self.x)
ds = MyDS.create("train.h5", sources, keep_open=True)
dl = torch.utils.data.DataLoader(ds, batch_size=16, num_workers=8, pin_memory=True)
```
`TypedFile` even have a method that takes the Dataloader args and a batch object filled with `BatchItems` and returns
a Dataloader that will yield such batch objects.
Example :
```python
f = TypedFile("train.h5", keep_open=True)
loader = f.serve(
# batch object :
dict(
x=h5m.Input(key='data/image', getter=h5m.GetId()),
labels=h5m.Target(key='data/labels', getter=h5m.GetId())
),
# Dataloader kwargs :
num_workers=8, pin_memory=True, batch_size=32, shuffle=True
)
```
### Examples
in ``h5mapper/examples`` you'll find for now
- a train script with data, checkpoints and logs in `dataset_and_logs.py`
- a script for benchmarking batch-loading times of different options
### Development
`h5mapper` is just getting started and you're welcome to contribute!
You'll find some tests you can run from the root of the repo with a simple
```bash
pytest
```
If you'd like to get involved, just drop us an email : ktonalberlin@gmail.com
### License
`h5mapper` is distributed under the terms of the MIT License.
Raw data
{
"_id": null,
"home_page": null,
"name": "h5mapper",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.10",
"maintainer_email": null,
"keywords": "ORM, deep-learning, h5py, hdf5, machine-learning",
"author": null,
"author_email": "Antoine Daurat <antoinedaurat@gmail.com>",
"download_url": "https://files.pythonhosted.org/packages/21/49/47f5d645e8f5cf22d4fe55500512bea60b680c4eebd1a7f75d0bd3838465/h5mapper-0.3.3.tar.gz",
"platform": null,
"description": "# h5mapper\n\n``h5mapper`` is a pythonic ORM-like tool for reading and writing HDF5 data.\n\nIt is built on top of `h5py` and lets you define types of **.h5 files as python classes** which you can then easily \n**create from raw sources** (e.g. files, urls...), **serve** (use as ``Dataset`` for a ``Dataloader``), \nor dynamically populate (logs, checkpoints of an experiment).\n\n## Content\n- [Installation](#Installation)\n- [Quickstart](#Quickstart)\n - [TypedFile](#TypedFile)\n - [Feature](#Feature)\n- [Examples](#Examples)\n- [Development](#Development)\n- [License](#License)\n \n## Installation\n\n### ``pip``\n\n``h5mapper`` is on pypi, to install it, one only needs to \n\n```bash\npip install h5mapper\n```\n\n### developer install\n\nfor playing around with the internals of the package, a good solution is to first\n\n```bash\ngit clone https://github.com/ktonal/h5mapper.git\n```\nand then \n\n```bash\npip install -e h5mapper/\n```\nwhich installs the repo in editable mode.\n\n## Quickstart\n\n### TypedFile\n\n``h5m`` assumes that you want to store collections of contiguous arrays in single datasets and that you want several such concatenated datasets in a file.\n\nThus, ``TypedFile`` allows you to create and read files that maintain a 2-d reference system, where contiguous arrays are stored within features and indexed by their source's id.\n\nSuch a file might then look like \n```bash\n<Experiment \"experiment.h5\">\n----------------------------------------------------> sources' ids axis\n| \"planes/01.jpeg\" | \"train\"\n| |\n| data/ |\n| images/ (32, 32) | None\n| labels/ (1, ) | None\n| logs/ |\n| loss/ None | (10000,)\n| ...\nV\nfeatures axis\n``` \nwhere the entries correspond to the shapes of arrays or their absence (`None`).\n\n> Note that this is a different approach than storing each file or image in a separate dataset. \n> In this case, there would be an `h5py.Dataset` located at `data/images/planes/01.jpeg` although in our\n> example, the only dataset is at `data/images/` and one of its regions is indexed by the id `\"planes/01.jpeg\"` \n\nFor interacting with files that follow this particular structure, simply define a class\n\n```python\nimport h5mapper as h5m\n\nclass Experiment(h5m.TypedFile):\n\n data = h5m.Group(\n # your custom h5m.Feature classes:\n images=Image(),\n labels=DirLabels()\n )\n logs = h5m.Group(\n loss=h5m.Array()\n )\n```\n#### ``create``, ``add``\n\nnow, create an instance, load data from files through parallel jobs and add data on the fly :\n\n```python\n# create instance from raw sources\nexp = Experiment.create(\"experiment.h5\",\n # those are then used as ids :\n sources=[\"planes/01.jpeg\", \"planes/02.jpeg\"],\n n_workers=8)\n...\n# add id <-> data on the fly :\nexp.logs.add(\"train\", dict(loss=losses_array))\n``` \n\n#### ``get``, ``refs`` and ``__getitem__`` \n\nThere are 3 main options to read data from a ``TypedFile`` or one of its ``Proxy``\n\n1/ By their id\n\n```python\n>> exp.logs.get(\"train\")\nOut: {\"loss\": np.array([...])}\n# which, in this case, is equivalent to \n>> exp.logs[\"train\"]\nOut: {\"loss\": np.array([...])}\n# because `exp.logs` is a Group and Groups only support id-based indexing\n```\n\n2/ By the index of their ids through their ``refs`` attribute :\n\n```python\n>> exp.data.images[exp.data.images.refs[0]].shape\nOut: (32, 32)\n```\nWhich works because `exp.data.images` is a `Dataset` and only `Datasets` have `refs`\n\n3/ with any ``item`` supported by the ``h5py.Dataset``\n```python\n>> exp.data.labels[:32]\nOut: np.array([0, 0, ....])\n```\nWhich only works for `Dataset`s - not for `Group`s.\n\n> Note that, in this last case, you are indexing into the **concatenation of all sub-arrays along their first axis**.\n\n> The same interface is also implemented for ``set(source, data)`` and ``__setitem__``\n\n### Feature\n\n``h5m`` exposes a class that helps you configure the behaviour of your ``TypedFile`` classes and the properties of the .h5 they create.\n\nthe ``Feature`` class helps you define :\n- how sources' ids are loaded into arrays (``feature.load(source)``)\n- which types of files are supported\n- how the data is stored by ``h5py`` (compression, chunks)\n- which extraction parameters need to be stored with the data (e.g. sample rate of audio files)\n- custom-methods relevant to this kind of data\n\nOnce you defined a `Feature` class, attach it to the class dict of a ``TypedFile``, that's it!\n\nFor example :\n\n```python\nimport h5mapper as h5m\n\n\nclass MyFeature(h5m.Feature):\n\n # only sources matching this pattern will be passed to load(...)\n __re__ = r\".special$\"\n\n # args for the h5py.Dataset\n __ds_kwargs__ = dict(compression='lzf', chunks=(1, 350))\n \n def __init__(self, my_extraction_param=0):\n self.my_extraction_param = my_extraction_param\n\n @property\n def attrs(self):\n # those are then written in the h5py.Group.attrs\n return {\"p\": self.my_extraction_param}\n\n def load(self, source):\n \"\"\"your method to get an np.ndarray or a dict thereof\n from a path, an url, whatever sources you have...\"\"\" \n return data\n\n def plot(self, data):\n \"\"\"custom plotting method for this kind of data\"\"\"\n # ...\n\n# attach it\nclass Data(h5m.TypedFile):\n feat = MyFeature(47)\n\n# load sources...\nf = Data.create(....)\n\n# read your data through __getitem__ \nbatch = f.feat[4:8]\n\n# access your method \nf.feat.plot(batch)\n\n# modify the file through __setitem__\nf.feat[4:8] = batch ** 2 \n```\n\nfor more examples, checkout `h5mapper/h5mapper/features.py`.\n\n#### ``serve``\n\nPrimarly designed with `pytorch` users in mind, `h5m` plays very nicely with the `Dataset` class :\n\n```python\nclass MyDS(h5m.TypedFile, torch.utils.data.Dataset):\n \n x = MyInputFeature(42)\n \n def __getitem__(self, item):\n return self.x[item], self.labels[item]\n \n def __len__(self):\n return len(self.x)\n\nds = MyDS.create(\"train.h5\", sources, keep_open=True)\n\ndl = torch.utils.data.DataLoader(ds, batch_size=16, num_workers=8, pin_memory=True)\n```\n\n`TypedFile` even have a method that takes the Dataloader args and a batch object filled with `BatchItems` and returns \na Dataloader that will yield such batch objects.\n\nExample :\n\n```python\nf = TypedFile(\"train.h5\", keep_open=True)\nloader = f.serve(\n # batch object :\n dict(\n x=h5m.Input(key='data/image', getter=h5m.GetId()),\n labels=h5m.Target(key='data/labels', getter=h5m.GetId())\n ),\n # Dataloader kwargs :\n num_workers=8, pin_memory=True, batch_size=32, shuffle=True\n)\n``` \n\n### Examples\n\nin ``h5mapper/examples`` you'll find for now\n- a train script with data, checkpoints and logs in `dataset_and_logs.py`\n- a script for benchmarking batch-loading times of different options\n\n### Development\n\n`h5mapper` is just getting started and you're welcome to contribute!\n\nYou'll find some tests you can run from the root of the repo with a simple\n```bash\npytest\n```\n\nIf you'd like to get involved, just drop us an email : ktonalberlin@gmail.com\n\n\n### License\n\n`h5mapper` is distributed under the terms of the MIT License. ",
"bugtrack_url": null,
"license": null,
"summary": "pythonic ORM tool for reading and writing HDF5 data",
"version": "0.3.3",
"project_urls": {
"Download": "https://github.com/ktonal/h5mapper",
"Homepage": "https://github.com/ktonal/h5mapper"
},
"split_keywords": [
"orm",
" deep-learning",
" h5py",
" hdf5",
" machine-learning"
],
"urls": [
{
"comment_text": null,
"digests": {
"blake2b_256": "119b00caf19b588a0d16af1b8f707abfb8290c7bde457df1da7b8f7a7315cf5d",
"md5": "ef4b453627c4e249f6e4f1cb9337a7e1",
"sha256": "96bf3cca9d989e0986b4d7772976aacfafaaa39cc03af55b21b650913a28a8dd"
},
"downloads": -1,
"filename": "h5mapper-0.3.3-py3-none-any.whl",
"has_sig": false,
"md5_digest": "ef4b453627c4e249f6e4f1cb9337a7e1",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.10",
"size": 19784,
"upload_time": "2025-01-10T05:22:45",
"upload_time_iso_8601": "2025-01-10T05:22:45.859071Z",
"url": "https://files.pythonhosted.org/packages/11/9b/00caf19b588a0d16af1b8f707abfb8290c7bde457df1da7b8f7a7315cf5d/h5mapper-0.3.3-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "214947f5d645e8f5cf22d4fe55500512bea60b680c4eebd1a7f75d0bd3838465",
"md5": "544246aebb80c7f2afa0418393cc9f75",
"sha256": "5975d3a7a709e4e51c1763977456e5eba8ba3fa9c7f712a77194fd630bbb9dce"
},
"downloads": -1,
"filename": "h5mapper-0.3.3.tar.gz",
"has_sig": false,
"md5_digest": "544246aebb80c7f2afa0418393cc9f75",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.10",
"size": 16242,
"upload_time": "2025-01-10T05:22:43",
"upload_time_iso_8601": "2025-01-10T05:22:43.430183Z",
"url": "https://files.pythonhosted.org/packages/21/49/47f5d645e8f5cf22d4fe55500512bea60b680c4eebd1a7f75d0bd3838465/h5mapper-0.3.3.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-01-10 05:22:43",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "ktonal",
"github_project": "h5mapper",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"lcname": "h5mapper"
}
JSON
