squirrel-core


Namesquirrel-core JSON
Version 0.14.2 PyPI version JSON
download
home_page
SummarySquirrel is a Python library that enables ML teams to share, load, and transform data in a collaborative, flexible, and efficient way.
upload_time2022-06-23 10:07:54
maintainer
docs_urlNone
authorMerantix Momentum
requires_python>=3.8.0
licenseApache 2.0
keywords
VCS
bugtrack_url
requirements No requirements were recorded.
Travis-CI No Travis.
coveralls test coverage No coveralls.
            <div align="center">
  
# <img src="https://raw.githubusercontent.com/merantix-momentum/squirrel-core/main/docs/_static/logo.png" width="150px"> Squirrel Core
  
**Share, load, and transform data in a collaborative, flexible, and efficient way**

[![Python](https://img.shields.io/pypi/pyversions/squirrel-core.svg?style=plastic)](https://badge.fury.io/py/squirrel-core)
[![PyPI](https://img.shields.io/pypi/v/squirrel-core?label=pypi%20package)](https://pypi.org/project/squirrel-core/)  
[![Conda](https://img.shields.io/conda/vn/conda-forge/squirrel-core)](https://anaconda.org/conda-forge/squirrel-core)
[![Documentation Status](https://readthedocs.org/projects/squirrel-core/badge/?version=latest)](https://squirrel-core.readthedocs.io/en/latest)
[![Downloads](https://static.pepy.tech/personalized-badge/squirrel-core?period=total&units=international_system&left_color=grey&right_color=blue&left_text=Downloads)](https://pepy.tech/project/squirrel-core)
[![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://raw.githubusercontent.com/merantix-momentum/squirrel-core/main/LICENSE)
[![DOI](https://zenodo.org/badge/458099869.svg)](https://zenodo.org/badge/latestdoi/458099869)
[![Generic badge](https://img.shields.io/badge/Website-Merantix%20Momentum-blue)](https://merantix-momentum.com)
[![Slack](https://img.shields.io/badge/slack-chat-green.svg?logo=slack)](https://join.slack.com/t/squirrel-core/shared_invite/zt-14k6sk6sw-zQPHfqAI8Xq5WYd~UqgNFw)

</div>

---

# What is Squirrel?

Squirrel is a Python library that enables ML teams to share, load, and transform data in a collaborative, flexible, and efficient way.

1. **SPEED:** Avoid data stall, i.e. the expensive GPU will not be idle while waiting for the data. 

2. **COSTS:** First, avoid GPU stalling, and second allow to shard & cluster your data and store & load it in bundles, decreasing the cost for your data bucket cloud storage.

3. **FLEXIBILITY:** Work with a flexible standard data scheme which is adaptable to any setting, including multimodal data.

4. **COLLABORATION:** Make it easier to share data & code between teams and projects in a self-service model.

Stream data from anywhere to your machine learning model as easy as:
```python
it = (Catalog.from_plugins()["imagenet"].get_driver()
      .get_iter("train")
      .map(lambda r: (augment(r["image"]), r["label"]))
      .batched(100))
```

Check out our full [getting started](https://github.com/merantix-momentum/squirrel-datasets-core/blob/main/examples/01.Getting_Started.ipynb) tutorial notebook. If you have any questions or would like to contribute, join our [Slack community](https://join.slack.com/t/squirrel-core/shared_invite/zt-14k6sk6sw-zQPHfqAI8Xq5WYd~UqgNFw).

# Installation
You can install `squirrel-core` by 
```shell
pip install "squirrel-core[all]"
```

# Documentation

Read our documentation at [ReadTheDocs](https://squirrel-core.readthedocs.io/en/latest)

# Example Notebooks
Check out the [Squirrel-datasets repository](https://github.com/merantix-momentum/squirrel-datasets-core/tree/main/examples) for open source and community-contributed tutorial and example notebooks of using Squirrel.

# Contributing
Squirrel is open source and community contributions are welcome!

Check out the [contribution guide](https://squirrel-core.readthedocs.io/en/latest/usage/contribute.html) to learn how to get involved.

# The humans behind Squirrel
We are [Merantix Momentum](https://merantix-momentum.com/), a team of ~30 machine learning engineers, developing machine learning solutions for industry and research. Each project comes with its own challenges, data types and learnings, but one issue we always faced was scalable data loading, transforming and sharing. We were looking for a solution that would allow us to load the data in a fast and cost-efficient way, while keeping the flexibility to work with any possible dataset and integrate with any API. That's why we build Squirrel – and we hope you'll find it as useful as we do! By the way, [we are hiring](https://merantix-momentum.com/about#jobs)!


# Citation

If you use Squirrel in your research, please cite it using:
```bibtex
@article{2022squirrelcore,
  title={Squirrel: A Python library that enables ML teams to share, load, and transform data in a collaborative, flexible, and efficient way.},
  author={Squirrel Developer Team},
  journal={GitHub. Note: https://github.com/merantix-momentum/squirrel-core},
  doi={10.5281/zenodo.6418280},
  year={2022}
}
```

            

Raw data

            {
    "_id": null,
    "home_page": "",
    "name": "squirrel-core",
    "maintainer": "",
    "docs_url": null,
    "requires_python": ">=3.8.0",
    "maintainer_email": "",
    "keywords": "",
    "author": "Merantix Momentum",
    "author_email": "",
    "download_url": "https://files.pythonhosted.org/packages/53/ce/3fbc64078b53ffa1ce09b9be6ba3ec4fb612d072db26c0e4eb5d879ec1ec/squirrel-core-0.14.2.tar.gz",
    "platform": null,
    "description": "<div align=\"center\">\n  \n# <img src=\"https://raw.githubusercontent.com/merantix-momentum/squirrel-core/main/docs/_static/logo.png\" width=\"150px\"> Squirrel Core\n  \n**Share, load, and transform data in a collaborative, flexible, and efficient way**\n\n[![Python](https://img.shields.io/pypi/pyversions/squirrel-core.svg?style=plastic)](https://badge.fury.io/py/squirrel-core)\n[![PyPI](https://img.shields.io/pypi/v/squirrel-core?label=pypi%20package)](https://pypi.org/project/squirrel-core/)  \n[![Conda](https://img.shields.io/conda/vn/conda-forge/squirrel-core)](https://anaconda.org/conda-forge/squirrel-core)\n[![Documentation Status](https://readthedocs.org/projects/squirrel-core/badge/?version=latest)](https://squirrel-core.readthedocs.io/en/latest)\n[![Downloads](https://static.pepy.tech/personalized-badge/squirrel-core?period=total&units=international_system&left_color=grey&right_color=blue&left_text=Downloads)](https://pepy.tech/project/squirrel-core)\n[![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://raw.githubusercontent.com/merantix-momentum/squirrel-core/main/LICENSE)\n[![DOI](https://zenodo.org/badge/458099869.svg)](https://zenodo.org/badge/latestdoi/458099869)\n[![Generic badge](https://img.shields.io/badge/Website-Merantix%20Momentum-blue)](https://merantix-momentum.com)\n[![Slack](https://img.shields.io/badge/slack-chat-green.svg?logo=slack)](https://join.slack.com/t/squirrel-core/shared_invite/zt-14k6sk6sw-zQPHfqAI8Xq5WYd~UqgNFw)\n\n</div>\n\n---\n\n# What is Squirrel?\n\nSquirrel is a Python library that enables ML teams to share, load, and transform data in a collaborative, flexible, and efficient way.\n\n1. **SPEED:** Avoid data stall, i.e. the expensive GPU will not be idle while waiting for the data. \n\n2. **COSTS:** First, avoid GPU stalling, and second allow to shard & cluster your data and store & load it in bundles, decreasing the cost for your data bucket cloud storage.\n\n3. **FLEXIBILITY:** Work with a flexible standard data scheme which is adaptable to any setting, including multimodal data.\n\n4. **COLLABORATION:** Make it easier to share data & code between teams and projects in a self-service model.\n\nStream data from anywhere to your machine learning model as easy as:\n```python\nit = (Catalog.from_plugins()[\"imagenet\"].get_driver()\n      .get_iter(\"train\")\n      .map(lambda r: (augment(r[\"image\"]), r[\"label\"]))\n      .batched(100))\n```\n\nCheck out our full [getting started](https://github.com/merantix-momentum/squirrel-datasets-core/blob/main/examples/01.Getting_Started.ipynb) tutorial notebook. If you have any questions or would like to contribute, join our [Slack community](https://join.slack.com/t/squirrel-core/shared_invite/zt-14k6sk6sw-zQPHfqAI8Xq5WYd~UqgNFw).\n\n# Installation\nYou can install `squirrel-core` by \n```shell\npip install \"squirrel-core[all]\"\n```\n\n# Documentation\n\nRead our documentation at [ReadTheDocs](https://squirrel-core.readthedocs.io/en/latest)\n\n# Example Notebooks\nCheck out the [Squirrel-datasets repository](https://github.com/merantix-momentum/squirrel-datasets-core/tree/main/examples) for open source and community-contributed tutorial and example notebooks of using Squirrel.\n\n# Contributing\nSquirrel is open source and community contributions are welcome!\n\nCheck out the [contribution guide](https://squirrel-core.readthedocs.io/en/latest/usage/contribute.html) to learn how to get involved.\n\n# The humans behind Squirrel\nWe are [Merantix Momentum](https://merantix-momentum.com/), a team of ~30 machine learning engineers, developing machine learning solutions for industry and research. Each project comes with its own challenges, data types and learnings, but one issue we always faced was scalable data loading, transforming and sharing. We were looking for a solution that would allow us to load the data in a fast and cost-efficient way, while keeping the flexibility to work with any possible dataset and integrate with any API. That's why we build Squirrel \u2013 and we hope you'll find it as useful as we do! By the way, [we are hiring](https://merantix-momentum.com/about#jobs)!\n\n\n# Citation\n\nIf you use Squirrel in your research, please cite it using:\n```bibtex\n@article{2022squirrelcore,\n  title={Squirrel: A Python library that enables ML teams to share, load, and transform data in a collaborative, flexible, and efficient way.},\n  author={Squirrel Developer Team},\n  journal={GitHub. Note: https://github.com/merantix-momentum/squirrel-core},\n  doi={10.5281/zenodo.6418280},\n  year={2022}\n}\n```\n",
    "bugtrack_url": null,
    "license": "Apache 2.0",
    "summary": "Squirrel is a Python library that enables ML teams to share, load, and transform data in a collaborative, flexible, and efficient way.",
    "version": "0.14.2",
    "split_keywords": [],
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "md5": "b5cd039112cb6eaa1d76bbf0b8181dde",
                "sha256": "54e8260983c1eb2e60378c69c03d8ff12f90703c0e85d9e7a2f6ece427743501"
            },
            "downloads": -1,
            "filename": "squirrel_core-0.14.2-py3-none-any.whl",
            "has_sig": false,
            "md5_digest": "b5cd039112cb6eaa1d76bbf0b8181dde",
            "packagetype": "bdist_wheel",
            "python_version": "py3",
            "requires_python": ">=3.8.0",
            "size": 66566,
            "upload_time": "2022-06-23T10:07:52",
            "upload_time_iso_8601": "2022-06-23T10:07:52.980886Z",
            "url": "https://files.pythonhosted.org/packages/9d/a1/de214e55cc4ef5e276e2afeaa33f7d6e4cad130f67de184418493ef3e258/squirrel_core-0.14.2-py3-none-any.whl",
            "yanked": false,
            "yanked_reason": null
        },
        {
            "comment_text": "",
            "digests": {
                "md5": "cc46e7ac1182b812ff6a4ac9ea3e7035",
                "sha256": "49dd4b233685677e6e82f9865740bb881028a6465b396eaa005749ce3c7133e5"
            },
            "downloads": -1,
            "filename": "squirrel-core-0.14.2.tar.gz",
            "has_sig": false,
            "md5_digest": "cc46e7ac1182b812ff6a4ac9ea3e7035",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": ">=3.8.0",
            "size": 50843,
            "upload_time": "2022-06-23T10:07:54",
            "upload_time_iso_8601": "2022-06-23T10:07:54.760896Z",
            "url": "https://files.pythonhosted.org/packages/53/ce/3fbc64078b53ffa1ce09b9be6ba3ec4fb612d072db26c0e4eb5d879ec1ec/squirrel-core-0.14.2.tar.gz",
            "yanked": false,
            "yanked_reason": null
        }
    ],
    "upload_time": "2022-06-23 10:07:54",
    "github": false,
    "gitlab": false,
    "bitbucket": false,
    "lcname": "squirrel-core"
}
        
Elapsed time: 0.49650s