<h1 align="center">
<img src="docs/assets/logo.png" width="64" title="zappend">
zappend
</h1>
<div align="center">
[](https://github.com/bcdev/zappend/actions/workflows/tests.yml)
[](https://codecov.io/gh/bcdev/zappend)
[](https://pypi.org/project/zappend/)
[](https://anaconda.org/conda-forge/zappend)
[](https://github.com/psf/black)
[](https://mybinder.org/v2/gh/bcdev/zappend/HEAD?labpath=examples%2Fzappend-demo.ipynb)
[](https://github.com/bcdev/zappend)
</div>
<!--- Align following sections with docs/index.md -->
---
`zappend` is a tool written in Python that is used for robustly creating and
updating Zarr datacubes from smaller dataset slices. It is built on top of the
awesome Python packages [xarray](https://docs.xarray.dev/) and [zarr](https://zarr.readthedocs.io/).
## Motivation
The objective of `zappend` is enabling geodata scientists and developers to
robustly create large data cubes. The tool performs transaction-based dataset
appends to existing data cubes in the
[Zarr format](https://zarr.readthedocs.io/en/stable/spec/v2.html). If an error
occurs during an append step — typically due to I/O problems or out-of-memory
conditions — `zappend` will automatically roll back the operation, ensuring that
the existing data cube maintains its structural integrity. The design drivers
behind zappend are first ease of use and secondly, high configurability
regarding filesystems, data source types, data cube outline and encoding.
The tool comprises a command-line interface, a Python API for programmatic
control, and a comprehensible documentation to guide users effectively.
You can easily install `zappend` as a plain Python package using either
`pip install zappend` or `conda install -conda-forge zappend`.
## Features
The `zappend` tool provides the following features:
* **Locking**: While the target dataset is being modified, a file lock is
created, effectively preventing concurrent dataset modifications.
* **Transaction-based dataset appends**: On failure during an append step,
the transaction is rolled back, so that the target dataset remains valid and
preserves its integrity.
* **Filesystem transparency**: The target dataset may be generated and updated
in any writable filesystems supported by the
[fsspec](https://filesystem-spec.readthedocs.io/) package.
The same holds for the slice datasets to be appended.
* **Dataset polling**: The tool can be configured to wait for slice datasets to
become available.
* **Dynamic attributes**: Use syntax `{{ expression }}` to update the target
dataset with dynamically computed attribute values.
* **CLI and Python API**: The tool can be used in a shell using the
[`zappend`](cli.md) command or from Python. When used from Python using the
[`zappend()`](api.md) function, slice datasets can be passed as local file
paths, URIs, as datasets of type
[xarray.Dataset](https://docs.xarray.dev/en/stable/generated/xarray.Dataset.html), or as custom
[slice sources](https://bcdev.github.io/zappend/guide/#slice-sources).
More about zappend can be found in its
[documentation](https://bcdev.github.io/zappend/).
Raw data
{
"_id": null,
"home_page": null,
"name": "zappend",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.10",
"maintainer_email": null,
"keywords": "analysis ready data, data science, datacube, xarray, zarr",
"author": null,
"author_email": "Norman Fomferra <norman.fomferra@brockmann-consult.de>",
"download_url": "https://files.pythonhosted.org/packages/f2/a9/68eb6234381213005078c47109fb65dd02a37310228d7689c1ccc9f27429/zappend-0.8.0.tar.gz",
"platform": null,
"description": "<h1 align=\"center\">\n <img src=\"docs/assets/logo.png\" width=\"64\" title=\"zappend\">\n zappend\n</h1>\n\n<div align=\"center\">\n\n[](https://github.com/bcdev/zappend/actions/workflows/tests.yml)\n[](https://codecov.io/gh/bcdev/zappend)\n[](https://pypi.org/project/zappend/)\n[](https://anaconda.org/conda-forge/zappend)\n[](https://github.com/psf/black)\n[](https://mybinder.org/v2/gh/bcdev/zappend/HEAD?labpath=examples%2Fzappend-demo.ipynb)\n[](https://github.com/bcdev/zappend)\n\n</div>\n\n<!--- Align following sections with docs/index.md -->\n\n---\n\n`zappend` is a tool written in Python that is used for robustly creating and \nupdating Zarr datacubes from smaller dataset slices. It is built on top of the \nawesome Python packages [xarray](https://docs.xarray.dev/) and [zarr](https://zarr.readthedocs.io/).\n\n## Motivation\n\nThe objective of `zappend` is enabling geodata scientists and developers to \nrobustly create large data cubes. The tool performs transaction-based dataset \nappends to existing data cubes in the \n[Zarr format](https://zarr.readthedocs.io/en/stable/spec/v2.html). If an error \noccurs during an append step \u2014 typically due to I/O problems or out-of-memory \nconditions \u2014 `zappend` will automatically roll back the operation, ensuring that \nthe existing data cube maintains its structural integrity. The design drivers \nbehind zappend are first ease of use and secondly, high configurability \nregarding filesystems, data source types, data cube outline and encoding. \n\nThe tool comprises a command-line interface, a Python API for programmatic \ncontrol, and a comprehensible documentation to guide users effectively. \nYou can easily install `zappend` as a plain Python package using either \n`pip install zappend` or `conda install -conda-forge zappend`.\n\n## Features\n\nThe `zappend` tool provides the following features:\n\n* **Locking**: While the target dataset is being modified, a file lock is \n created, effectively preventing concurrent dataset modifications.\n* **Transaction-based dataset appends**: On failure during an append step, \n the transaction is rolled back, so that the target dataset remains valid and \n preserves its integrity.\n* **Filesystem transparency**: The target dataset may be generated and updated \n in any writable filesystems supported by the \n [fsspec](https://filesystem-spec.readthedocs.io/) package. \n The same holds for the slice datasets to be appended.\n* **Dataset polling**: The tool can be configured to wait for slice datasets to \n become available. \n* **Dynamic attributes**: Use syntax `{{ expression }}` to update the target \n dataset with dynamically computed attribute values. \n* **CLI and Python API**: The tool can be used in a shell using the \n [`zappend`](cli.md) command or from Python. When used from Python using the \n [`zappend()`](api.md) function, slice datasets can be passed as local file \n paths, URIs, as datasets of type \n [xarray.Dataset](https://docs.xarray.dev/en/stable/generated/xarray.Dataset.html), or as custom\n [slice sources](https://bcdev.github.io/zappend/guide/#slice-sources).\n\n \nMore about zappend can be found in its \n[documentation](https://bcdev.github.io/zappend/).\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "Tool for robustly creating and updating Zarr datacubes from smaller slices",
"version": "0.8.0",
"project_urls": {
"Changelog": "https://github.com/bcdev/zappend/blob/main/CHANGES.md",
"Documentation": "https://bcdev.github.io/zappend/",
"Issues": "https://github.com/bcdev/zappend/issues",
"Repository": "https://github.com/bcdev/zappend"
},
"split_keywords": [
"analysis ready data",
" data science",
" datacube",
" xarray",
" zarr"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "de0d236d3c6ecef57eee7edbafb21c9ae3e2df04dd99bd006a9be223566ab5ea",
"md5": "26f05c9201c1ef39968b169e6d0bc67d",
"sha256": "029181570968bae00c160842aab483defa7941813e0e2e3d9a08fdccc903d77f"
},
"downloads": -1,
"filename": "zappend-0.8.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "26f05c9201c1ef39968b169e6d0bc67d",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.10",
"size": 52704,
"upload_time": "2024-10-04T10:26:03",
"upload_time_iso_8601": "2024-10-04T10:26:03.271655Z",
"url": "https://files.pythonhosted.org/packages/de/0d/236d3c6ecef57eee7edbafb21c9ae3e2df04dd99bd006a9be223566ab5ea/zappend-0.8.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "f2a968eb6234381213005078c47109fb65dd02a37310228d7689c1ccc9f27429",
"md5": "52199747fa47d5c11dfa50f9b5e5c2ba",
"sha256": "ec4f80fca8b36e1a442f007f1fd8d7fe792e1d741c267e595a5de95649a2fdd4"
},
"downloads": -1,
"filename": "zappend-0.8.0.tar.gz",
"has_sig": false,
"md5_digest": "52199747fa47d5c11dfa50f9b5e5c2ba",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.10",
"size": 52571,
"upload_time": "2024-10-04T10:26:04",
"upload_time_iso_8601": "2024-10-04T10:26:04.546691Z",
"url": "https://files.pythonhosted.org/packages/f2/a9/68eb6234381213005078c47109fb65dd02a37310228d7689c1ccc9f27429/zappend-0.8.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-10-04 10:26:04",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "bcdev",
"github_project": "zappend",
"travis_ci": false,
"coveralls": true,
"github_actions": true,
"lcname": "zappend"
}
JSON
