quiltcore


Namequiltcore JSON
Version 0.6.3 PyPI version JSON
download
home_page
Summarylow-level plubming to read/write Quilt packages
upload_time2023-12-13 19:38:00
maintainer
docs_urlNone
authorErnest Prabhakar
requires_python>=3.10,<4.0
license
keywords
VCS
bugtrack_url
requirements No requirements were recorded.
Travis-CI No Travis.
coveralls test coverage No coveralls.
            # QuiltCore

QuiltCore is a library for building and running [Quilt](https://quiltdata.com) data packages.
It is designed to leverage standard open source technology and YAML configuration files
so that it can easily be ported to other languages and platforms.

This initial implementation is in Python.

## Key Technologies

- Apache [Arrow](https://arrow.apache.org/) for reading, writing, and representing manifests
  - [PyArrow](https://arrow.apache.org/docs/python/) for Python bindings to Arrow
- fsspec [filesystems](https://filesystem-spec.readthedocs.io/en/latest/)
  for reading and writing files from various sources
- [PyYAML](https://pyyaml.org/) for reading and writing YAML configuration files

## Example

```bash
poetry install
```

```python
#!/usr/bin/env python
import os
from quiltcore import Domain, UDI
from tempfile import TemporaryDirectory
from upath import UPath

TEST_BKT = "s3://quilt-example"
TEST_PKG = "akarve/amazon-reviews"
TEST_TAG = "1570503102"
TEST_HASH = "ffe323137d0a84a9d1d6f200cecd616f434e121b3f53a8891a5c8d70f82244c2"
TEST_KEY = "camera-reviews"
WRITE_BKT = os.environ.get("WRITE_BUCKET")
SOURCE_URI = f"quilt+{TEST_BKT}#package={TEST_PKG}:{TEST_TAG}"
DEST_URI = f"quilt+{TEST_BKT}#package={TEST_PKG}:{TEST_TAG}"
```

### Get Manifest

<!--pytest-codeblocks:cont-->
```python
remote = UDI.FromUri(SOURCE_URI)
print(f"remote: {remote}")
with TemporaryDirectory() as tmpdir:
    local = UPath(tmpdir)
    domain = Domain.FromLocalPath(local)
    print(f"domain: {domain}")
    folder = domain.pull(remote)
    print(f"folder: {folder}")
    if WRITE_BKT:
        tag = domain.push(folder, remote=UDI.FromUri(DEST_URI))
        print(f"tag: {tag}")
```

            

Raw data

            {
    "_id": null,
    "home_page": "",
    "name": "quiltcore",
    "maintainer": "",
    "docs_url": null,
    "requires_python": ">=3.10,<4.0",
    "maintainer_email": "",
    "keywords": "",
    "author": "Ernest Prabhakar",
    "author_email": "ernest@quiltdata.io",
    "download_url": "https://files.pythonhosted.org/packages/5d/4d/7fb0f607bd929861cba1514362a0d8d1aaea57aef40507c8d494e4e344bc/quiltcore-0.6.3.tar.gz",
    "platform": null,
    "description": "# QuiltCore\n\nQuiltCore is a library for building and running [Quilt](https://quiltdata.com) data packages.\nIt is designed to leverage standard open source technology and YAML configuration files\nso that it can easily be ported to other languages and platforms.\n\nThis initial implementation is in Python.\n\n## Key Technologies\n\n- Apache [Arrow](https://arrow.apache.org/) for reading, writing, and representing manifests\n  - [PyArrow](https://arrow.apache.org/docs/python/) for Python bindings to Arrow\n- fsspec [filesystems](https://filesystem-spec.readthedocs.io/en/latest/)\n  for reading and writing files from various sources\n- [PyYAML](https://pyyaml.org/) for reading and writing YAML configuration files\n\n## Example\n\n```bash\npoetry install\n```\n\n```python\n#!/usr/bin/env python\nimport os\nfrom quiltcore import Domain, UDI\nfrom tempfile import TemporaryDirectory\nfrom upath import UPath\n\nTEST_BKT = \"s3://quilt-example\"\nTEST_PKG = \"akarve/amazon-reviews\"\nTEST_TAG = \"1570503102\"\nTEST_HASH = \"ffe323137d0a84a9d1d6f200cecd616f434e121b3f53a8891a5c8d70f82244c2\"\nTEST_KEY = \"camera-reviews\"\nWRITE_BKT = os.environ.get(\"WRITE_BUCKET\")\nSOURCE_URI = f\"quilt+{TEST_BKT}#package={TEST_PKG}:{TEST_TAG}\"\nDEST_URI = f\"quilt+{TEST_BKT}#package={TEST_PKG}:{TEST_TAG}\"\n```\n\n### Get Manifest\n\n<!--pytest-codeblocks:cont-->\n```python\nremote = UDI.FromUri(SOURCE_URI)\nprint(f\"remote: {remote}\")\nwith TemporaryDirectory() as tmpdir:\n    local = UPath(tmpdir)\n    domain = Domain.FromLocalPath(local)\n    print(f\"domain: {domain}\")\n    folder = domain.pull(remote)\n    print(f\"folder: {folder}\")\n    if WRITE_BKT:\n        tag = domain.push(folder, remote=UDI.FromUri(DEST_URI))\n        print(f\"tag: {tag}\")\n```\n",
    "bugtrack_url": null,
    "license": "",
    "summary": "low-level plubming to read/write Quilt packages",
    "version": "0.6.3",
    "project_urls": null,
    "split_keywords": [],
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "blake2b_256": "9fa52a1c246d67e4a879dc57b218a13ed400f573fcd6b7a6024cbd61356599cf",
                "md5": "3453dd585c7717329f8881ff786548f8",
                "sha256": "5a777a5cd94afd293d48682317232712e592bdc2fb081837b4adb342a0e7c172"
            },
            "downloads": -1,
            "filename": "quiltcore-0.6.3-py3-none-any.whl",
            "has_sig": false,
            "md5_digest": "3453dd585c7717329f8881ff786548f8",
            "packagetype": "bdist_wheel",
            "python_version": "py3",
            "requires_python": ">=3.10,<4.0",
            "size": 34770,
            "upload_time": "2023-12-13T19:37:59",
            "upload_time_iso_8601": "2023-12-13T19:37:59.573187Z",
            "url": "https://files.pythonhosted.org/packages/9f/a5/2a1c246d67e4a879dc57b218a13ed400f573fcd6b7a6024cbd61356599cf/quiltcore-0.6.3-py3-none-any.whl",
            "yanked": false,
            "yanked_reason": null
        },
        {
            "comment_text": "",
            "digests": {
                "blake2b_256": "5d4d7fb0f607bd929861cba1514362a0d8d1aaea57aef40507c8d494e4e344bc",
                "md5": "ff5de7f207e04f5baf25844cb45b03b8",
                "sha256": "e38a373bc31eb7a76b53bb63c20106317b82154989efa27bead80cd9f4a36742"
            },
            "downloads": -1,
            "filename": "quiltcore-0.6.3.tar.gz",
            "has_sig": false,
            "md5_digest": "ff5de7f207e04f5baf25844cb45b03b8",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": ">=3.10,<4.0",
            "size": 26418,
            "upload_time": "2023-12-13T19:38:00",
            "upload_time_iso_8601": "2023-12-13T19:38:00.927137Z",
            "url": "https://files.pythonhosted.org/packages/5d/4d/7fb0f607bd929861cba1514362a0d8d1aaea57aef40507c8d494e4e344bc/quiltcore-0.6.3.tar.gz",
            "yanked": false,
            "yanked_reason": null
        }
    ],
    "upload_time": "2023-12-13 19:38:00",
    "github": false,
    "gitlab": false,
    "bitbucket": false,
    "codeberg": false,
    "lcname": "quiltcore"
}
        
Elapsed time: 0.14612s