kedro


Namekedro JSON
Version 0.17.4 PyPI version JSON
download
home_pagehttps://github.com/quantumblacklabs/kedro
SummaryKedro helps you build production-ready data and analytics pipelines
upload_time2021-06-16 09:10:34
maintainer
docs_urlNone
authorQuantumBlack Labs
requires_python>=3.6, <3.9
licenseApache Software License (Apache 2.0)
keywords pipelines machine learning data pipelines data science data engineering
VCS
bugtrack_url
requirements No requirements were recorded.
Travis-CI No Travis.
coveralls test coverage No coveralls.
            ![Kedro Logo Banner](https://raw.githubusercontent.com/quantumblacklabs/kedro/develop/static/img/kedro_banner.png)

[![Python version](https://img.shields.io/badge/python-3.6%20%7C%203.7%20%7C%203.8-blue.svg)](https://pypi.org/project/kedro/)
[![PyPI version](https://badge.fury.io/py/kedro.svg)](https://pypi.org/project/kedro/)
[![Conda version](https://img.shields.io/conda/vn/conda-forge/kedro.svg)](https://anaconda.org/conda-forge/kedro)
[![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/quantumblacklabs/kedro/blob/master/LICENSE.md)
[![Discord Server](https://img.shields.io/discord/778216384475693066.svg?color=7289da&label=Kedro%20Discord&logo=discord&style=flat-square)](https://discord.gg/akJDeVaxnB)
![CircleCI - Master Branch](https://img.shields.io/circleci/build/github/quantumblacklabs/kedro/master?label=master)
![Develop Branch Build](https://img.shields.io/circleci/build/github/quantumblacklabs/kedro/develop?label=develop)
[![Documentation](https://readthedocs.org/projects/kedro/badge/?version=stable)](https://kedro.readthedocs.io/)
[![DOI](https://zenodo.org/badge/182067506.svg)](https://zenodo.org/badge/latestdoi/182067506)


## What is Kedro?

Kedro is an open-source Python framework for creating reproducible, maintainable and modular data science code. It borrows concepts from software engineering and applies them to machine-learning code; applied concepts include modularity, separation of concerns and versioning.


## How do I install Kedro?

To install Kedro from the Python Package Index (PyPI) simply run:

```
pip install kedro
```

It is also possible to install Kedro using `conda`:

```
conda install -c conda-forge kedro
```

Our [Get Started guide](https://kedro.readthedocs.io/en/stable/02_get_started/01_prerequisites.html) contains full installation instructions, and includes how to set up Python virtual environments.


## What are the main features of Kedro?

![Kedro-Viz Pipeline Visualisation](https://raw.githubusercontent.com/quantumblacklabs/kedro/develop/static/img/pipeline_visualisation.png)
*A pipeline visualisation generated using [Kedro-Viz](https://github.com/quantumblacklabs/kedro-viz)*


| Feature | What is this? |
|----------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| Project Template | A standard, modifiable and easy-to-use project template based on [Cookiecutter Data Science](https://github.com/drivendata/cookiecutter-data-science/). |
| Data Catalog | A series of lightweight data connectors used to save and load data across many different file formats and file systems, including local and network file systems, cloud object stores, and HDFS. The Data Catalog also includes data and model versioning for file-based systems. |
| Pipeline Abstraction | Automatic resolution of dependencies between pure Python functions and data pipeline visualisation using [Kedro-Viz](https://github.com/quantumblacklabs/kedro-viz). |
| Coding Standards | Test-driven development using [`pytest`](https://github.com/pytest-dev/pytest), produce well-documented code using [Sphinx](http://www.sphinx-doc.org/en/master/), create linted code with support for [`flake8`](https://github.com/PyCQA/flake8), [`isort`](https://github.com/PyCQA/isort) and [`black`](https://github.com/psf/black) and make use of the standard Python logging library. |
| Flexible Deployment | Deployment strategies that include single or distributed-machine deployment as well as additional support for deploying on Argo, Prefect, Kubeflow, AWS Batch and Databricks. |


## How do I use Kedro?

The [Kedro documentation](https://kedro.readthedocs.io/en/stable/) includes three examples to help get you started:
- A typical "Hello World" example, for an [entry-level description of the main Kedro concepts](https://kedro.readthedocs.io/en/stable/02_get_started/03_hello_kedro.html)
- An [introduction to the project template](https://kedro.readthedocs.io/en/stable/02_get_started/05_example_project.html) using the Iris dataset
- A more detailed [spaceflights tutorial](https://kedro.readthedocs.io/en/stable/03_tutorial/02_tutorial_template.html) to give you hands-on experience


## Why does Kedro exist?

Kedro is built upon our collective best-practice (and mistakes) trying to deliver real-world ML applications that have vast amounts of raw unvetted data. We developed Kedro to achieve the following:
 - To address the main shortcomings of Jupyter notebooks, one-off scripts, and glue-code because there is a focus on
  creating **maintainable data science code**
 - To enhance **team collaboration** when different team members have varied exposure to software engineering concepts
 - To increase efficiency, because applied concepts like modularity and separation of concerns inspire the creation of
  **reusable analytics code**


## The humans behind Kedro

Kedro is maintained by a [product team from QuantumBlack](https://kedro.readthedocs.io/en/stable/12_faq/01_faq.html) and a number of [contributors from across the world](https://github.com/quantumblacklabs/kedro/releases).


## Can I contribute?

Yes! Want to help build Kedro? Check out our [guide to contributing to Kedro](https://github.com/quantumblacklabs/kedro/blob/master/CONTRIBUTING.md).


## Where can I learn more?

There is a growing community around Kedro. Have a look at the [Kedro FAQs](https://kedro.readthedocs.io/en/stable/12_faq/01_faq.html#how-can-i-find-out-more-about-kedro) to find projects using Kedro and links to articles, podcasts and talks.


## Who likes Kedro?

There are Kedro users across the world, who work at start-ups, major enterprises and academic institutions like [Absa](https://www.absa.co.za/),
[Acensi](https://acensi.eu/page/home),
[AI Singapore](https://makerspace.aisingapore.org/2020/08/leveraging-kedro-in-100e/),
[AXA UK](https://www.axa.co.uk/),
[Belfius](https://www.linkedin.com/posts/vangansen_mlops-machinelearning-kedro-activity-6772379995953238016-JUmo),
[Caterpillar](https://www.caterpillar.com/),
[CRIM](https://www.crim.ca/en/),
[Dendra Systems](https://www.dendra.io/),
[Element AI](https://www.elementai.com/),
[GMO](https://recruit.gmo.jp/engineer/jisedai/engineer/jisedai/engineer/jisedai/engineer/jisedai/engineer/jisedai/blog/kedro_and_mlflow_tracking/),
[Imperial College London](https://github.com/dssg/barefoot-winnie-public),
[Jungle Scout](https://junglescouteng.medium.com/jungle-scout-case-study-kedro-airflow-and-mlflow-use-on-production-code-150d7231d42e),
[Helvetas](https://www.linkedin.com/posts/lionel-trebuchon_mlflow-kedro-ml-ugcPost-6747074322164154368-umKw),
[Leapfrog](https://www.lftechnology.com/blog/ai-pipeline-kedro/),
[McKinsey & Company](https://www.mckinsey.com/alumni/news-and-insights/global-news/firm-news/kedro-from-proprietary-to-open-source),
[Mercado Libre Argentina](https://www.mercadolibre.com.ar),
[Modec](https://www.modec.com/),
[Mosaic Data Science](https://www.youtube.com/watch?v=fCWGevB366g),
[NaranjaX](https://www.youtube.com/watch?v=_0kMmRfltEQ),
[Open Data Science LatAm](https://www.odesla.org/),
[Prediqt](https://prediqt.co/),
[QuantumBlack](https://medium.com/quantumblack/introducing-kedro-the-open-source-library-for-production-ready-machine-learning-code-d1c6d26ce2cf),
[Retrieva](https://tech.retrieva.jp/entry/2020/07/28/181414), [Roche](https://www.roche.com/),
[Sber](https://www.linkedin.com/posts/seleznev-artem_welcome-to-kedros-documentation-kedro-activity-6767523561109385216-woTt),
[Telkomsel](https://www.linkedin.com/feed/update/urn:li:activity:6749338226403766272/updateEntityUrn=urn%3Ali%3Afs_feedUpdate%3A%28V2%2Curn%3Ali%3Aactivity%3A6749338226403766272%29),
[Universidad Rey Juan Carlos](https://github.com/vchaparro/MasterThesis-wind-power-forecasting/blob/master/thesis.pdf),
[UrbanLogiq](https://urbanlogiq.com/),
[Wildlife Studios](https://wildlifestudios.com),
[WovenLight](https://www.wovenlight.com/) and
[XP](https://youtu.be/wgnGOVNkXqU?t=2210).

Kedro has also won [Best Technical Tool or Framework for AI](https://awards.ai/the-awards/previous-awards/the-4th-ai-award-winners/) in the 2019 Awards AI competition and a merit award for the 2020 [UK Technical Communication Awards](https://uktcawards.com/announcing-the-award-winners-for-2020/). It is listed on the 2020 [ThoughtWorks Technology Radar](https://www.thoughtworks.com/radar/languages-and-frameworks/kedro) and the 2020 [Data & AI Landscape](https://mattturck.com/data2020/).

## How can I cite Kedro?

If you're an academic, Kedro can also help you, for example, as a tool to solve the problem of reproducible research. Find our citation reference on [Zenodo](https://zenodo.org/record/4336685).



            

Raw data

            {
    "_id": null,
    "home_page": "https://github.com/quantumblacklabs/kedro",
    "name": "kedro",
    "maintainer": "",
    "docs_url": null,
    "requires_python": ">=3.6, <3.9",
    "maintainer_email": "",
    "keywords": "pipelines,machine learning,data pipelines,data science,data engineering",
    "author": "QuantumBlack Labs",
    "author_email": "",
    "download_url": "https://files.pythonhosted.org/packages/44/5d/f7ae2ad79c1b200028bd0f84521c77a1202a37423d3c2050edc476f05abf/kedro-0.17.4.tar.gz",
    "platform": "",
    "description": "![Kedro Logo Banner](https://raw.githubusercontent.com/quantumblacklabs/kedro/develop/static/img/kedro_banner.png)\n\n[![Python version](https://img.shields.io/badge/python-3.6%20%7C%203.7%20%7C%203.8-blue.svg)](https://pypi.org/project/kedro/)\n[![PyPI version](https://badge.fury.io/py/kedro.svg)](https://pypi.org/project/kedro/)\n[![Conda version](https://img.shields.io/conda/vn/conda-forge/kedro.svg)](https://anaconda.org/conda-forge/kedro)\n[![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/quantumblacklabs/kedro/blob/master/LICENSE.md)\n[![Discord Server](https://img.shields.io/discord/778216384475693066.svg?color=7289da&label=Kedro%20Discord&logo=discord&style=flat-square)](https://discord.gg/akJDeVaxnB)\n![CircleCI - Master Branch](https://img.shields.io/circleci/build/github/quantumblacklabs/kedro/master?label=master)\n![Develop Branch Build](https://img.shields.io/circleci/build/github/quantumblacklabs/kedro/develop?label=develop)\n[![Documentation](https://readthedocs.org/projects/kedro/badge/?version=stable)](https://kedro.readthedocs.io/)\n[![DOI](https://zenodo.org/badge/182067506.svg)](https://zenodo.org/badge/latestdoi/182067506)\n\n\n## What is Kedro?\n\nKedro is an open-source Python framework for creating reproducible, maintainable and modular data science code. It borrows concepts from software engineering and applies them to machine-learning code; applied concepts include modularity, separation of concerns and versioning.\n\n\n## How do I install Kedro?\n\nTo install Kedro from the Python Package Index (PyPI) simply run:\n\n```\npip install kedro\n```\n\nIt is also possible to install Kedro using `conda`:\n\n```\nconda install -c conda-forge kedro\n```\n\nOur [Get Started guide](https://kedro.readthedocs.io/en/stable/02_get_started/01_prerequisites.html) contains full installation instructions, and includes how to set up Python virtual environments.\n\n\n## What are the main features of Kedro?\n\n![Kedro-Viz Pipeline Visualisation](https://raw.githubusercontent.com/quantumblacklabs/kedro/develop/static/img/pipeline_visualisation.png)\n*A pipeline visualisation generated using [Kedro-Viz](https://github.com/quantumblacklabs/kedro-viz)*\n\n\n| Feature | What is this? |\n|----------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|\n| Project Template | A standard, modifiable and easy-to-use project template based on [Cookiecutter Data Science](https://github.com/drivendata/cookiecutter-data-science/). |\n| Data Catalog | A series of lightweight data connectors used to save and load data across many different file formats and file systems, including local and network file systems, cloud object stores, and HDFS. The Data Catalog also includes data and model versioning for file-based systems. |\n| Pipeline Abstraction | Automatic resolution of dependencies between pure Python functions and data pipeline visualisation using [Kedro-Viz](https://github.com/quantumblacklabs/kedro-viz). |\n| Coding Standards | Test-driven development using [`pytest`](https://github.com/pytest-dev/pytest), produce well-documented code using [Sphinx](http://www.sphinx-doc.org/en/master/), create linted code with support for [`flake8`](https://github.com/PyCQA/flake8), [`isort`](https://github.com/PyCQA/isort) and [`black`](https://github.com/psf/black) and make use of the standard Python logging library. |\n| Flexible Deployment | Deployment strategies that include single or distributed-machine deployment as well as additional support for deploying on Argo, Prefect, Kubeflow, AWS Batch and Databricks. |\n\n\n## How do I use Kedro?\n\nThe [Kedro documentation](https://kedro.readthedocs.io/en/stable/) includes three examples to help get you started:\n- A typical \"Hello World\" example, for an [entry-level description of the main Kedro concepts](https://kedro.readthedocs.io/en/stable/02_get_started/03_hello_kedro.html)\n- An [introduction to the project template](https://kedro.readthedocs.io/en/stable/02_get_started/05_example_project.html) using the Iris dataset\n- A more detailed [spaceflights tutorial](https://kedro.readthedocs.io/en/stable/03_tutorial/02_tutorial_template.html) to give you hands-on experience\n\n\n## Why does Kedro exist?\n\nKedro is built upon our collective best-practice (and mistakes) trying to deliver real-world ML applications that have vast amounts of raw unvetted data. We developed Kedro to achieve the following:\n - To address the main shortcomings of Jupyter notebooks, one-off scripts, and glue-code because there is a focus on\n  creating **maintainable data science code**\n - To enhance **team collaboration** when different team members have varied exposure to software engineering concepts\n - To increase efficiency, because applied concepts like modularity and separation of concerns inspire the creation of\n  **reusable analytics code**\n\n\n## The humans behind Kedro\n\nKedro is maintained by a [product team from QuantumBlack](https://kedro.readthedocs.io/en/stable/12_faq/01_faq.html) and a number of [contributors from across the world](https://github.com/quantumblacklabs/kedro/releases).\n\n\n## Can I contribute?\n\nYes! Want to help build Kedro? Check out our [guide to contributing to Kedro](https://github.com/quantumblacklabs/kedro/blob/master/CONTRIBUTING.md).\n\n\n## Where can I learn more?\n\nThere is a growing community around Kedro. Have a look at the [Kedro FAQs](https://kedro.readthedocs.io/en/stable/12_faq/01_faq.html#how-can-i-find-out-more-about-kedro) to find projects using Kedro and links to articles, podcasts and talks.\n\n\n## Who likes Kedro?\n\nThere are Kedro users across the world, who work at start-ups, major enterprises and academic institutions like [Absa](https://www.absa.co.za/),\n[Acensi](https://acensi.eu/page/home),\n[AI Singapore](https://makerspace.aisingapore.org/2020/08/leveraging-kedro-in-100e/),\n[AXA UK](https://www.axa.co.uk/),\n[Belfius](https://www.linkedin.com/posts/vangansen_mlops-machinelearning-kedro-activity-6772379995953238016-JUmo),\n[Caterpillar](https://www.caterpillar.com/),\n[CRIM](https://www.crim.ca/en/),\n[Dendra Systems](https://www.dendra.io/),\n[Element AI](https://www.elementai.com/),\n[GMO](https://recruit.gmo.jp/engineer/jisedai/engineer/jisedai/engineer/jisedai/engineer/jisedai/engineer/jisedai/blog/kedro_and_mlflow_tracking/),\n[Imperial College London](https://github.com/dssg/barefoot-winnie-public),\n[Jungle Scout](https://junglescouteng.medium.com/jungle-scout-case-study-kedro-airflow-and-mlflow-use-on-production-code-150d7231d42e),\n[Helvetas](https://www.linkedin.com/posts/lionel-trebuchon_mlflow-kedro-ml-ugcPost-6747074322164154368-umKw),\n[Leapfrog](https://www.lftechnology.com/blog/ai-pipeline-kedro/),\n[McKinsey & Company](https://www.mckinsey.com/alumni/news-and-insights/global-news/firm-news/kedro-from-proprietary-to-open-source),\n[Mercado Libre Argentina](https://www.mercadolibre.com.ar),\n[Modec](https://www.modec.com/),\n[Mosaic Data Science](https://www.youtube.com/watch?v=fCWGevB366g),\n[NaranjaX](https://www.youtube.com/watch?v=_0kMmRfltEQ),\n[Open Data Science LatAm](https://www.odesla.org/),\n[Prediqt](https://prediqt.co/),\n[QuantumBlack](https://medium.com/quantumblack/introducing-kedro-the-open-source-library-for-production-ready-machine-learning-code-d1c6d26ce2cf),\n[Retrieva](https://tech.retrieva.jp/entry/2020/07/28/181414), [Roche](https://www.roche.com/),\n[Sber](https://www.linkedin.com/posts/seleznev-artem_welcome-to-kedros-documentation-kedro-activity-6767523561109385216-woTt),\n[Telkomsel](https://www.linkedin.com/feed/update/urn:li:activity:6749338226403766272/updateEntityUrn=urn%3Ali%3Afs_feedUpdate%3A%28V2%2Curn%3Ali%3Aactivity%3A6749338226403766272%29),\n[Universidad Rey Juan Carlos](https://github.com/vchaparro/MasterThesis-wind-power-forecasting/blob/master/thesis.pdf),\n[UrbanLogiq](https://urbanlogiq.com/),\n[Wildlife Studios](https://wildlifestudios.com),\n[WovenLight](https://www.wovenlight.com/) and\n[XP](https://youtu.be/wgnGOVNkXqU?t=2210).\n\nKedro has also won [Best Technical Tool or Framework for AI](https://awards.ai/the-awards/previous-awards/the-4th-ai-award-winners/) in the 2019 Awards AI competition and a merit award for the 2020 [UK Technical Communication Awards](https://uktcawards.com/announcing-the-award-winners-for-2020/). It is listed on the 2020 [ThoughtWorks Technology Radar](https://www.thoughtworks.com/radar/languages-and-frameworks/kedro) and the 2020 [Data & AI Landscape](https://mattturck.com/data2020/).\n\n## How can I cite Kedro?\n\nIf you're an academic, Kedro can also help you, for example, as a tool to solve the problem of reproducible research. Find our citation reference on [Zenodo](https://zenodo.org/record/4336685).\n\n\n",
    "bugtrack_url": null,
    "license": "Apache Software License (Apache 2.0)",
    "summary": "Kedro helps you build production-ready data and analytics pipelines",
    "version": "0.17.4",
    "split_keywords": [
        "pipelines",
        "machine learning",
        "data pipelines",
        "data science",
        "data engineering"
    ],
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "md5": "ae6482fd02136c7268f68560326f012c",
                "sha256": "a949e4f87f1bebbb26d3cb859aaae9288a8db108d93a90fcb9dfa621c278c6c0"
            },
            "downloads": -1,
            "filename": "kedro-0.17.4-py3-none-any.whl",
            "has_sig": false,
            "md5_digest": "ae6482fd02136c7268f68560326f012c",
            "packagetype": "bdist_wheel",
            "python_version": "py3",
            "requires_python": ">=3.6, <3.9",
            "size": 17414892,
            "upload_time": "2021-06-16T09:10:32",
            "upload_time_iso_8601": "2021-06-16T09:10:32.303709Z",
            "url": "https://files.pythonhosted.org/packages/0f/8c/aae41d77295e36dd1220a276fb69d3aa16f14160586528dc08d3944f897c/kedro-0.17.4-py3-none-any.whl",
            "yanked": false,
            "yanked_reason": null
        },
        {
            "comment_text": "",
            "digests": {
                "md5": "216e1f073855616bfd47d5524d9858df",
                "sha256": "7bd30a622599ade5771d0b9b58e67bf67455b2309e5bf016e53d6dd856f9bc17"
            },
            "downloads": -1,
            "filename": "kedro-0.17.4.tar.gz",
            "has_sig": false,
            "md5_digest": "216e1f073855616bfd47d5524d9858df",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": ">=3.6, <3.9",
            "size": 134371,
            "upload_time": "2021-06-16T09:10:34",
            "upload_time_iso_8601": "2021-06-16T09:10:34.351893Z",
            "url": "https://files.pythonhosted.org/packages/44/5d/f7ae2ad79c1b200028bd0f84521c77a1202a37423d3c2050edc476f05abf/kedro-0.17.4.tar.gz",
            "yanked": false,
            "yanked_reason": null
        }
    ],
    "upload_time": "2021-06-16 09:10:34",
    "github": true,
    "gitlab": false,
    "bitbucket": false,
    "github_user": null,
    "github_project": "quantumblacklabs",
    "error": "Could not fetch GitHub repository",
    "lcname": "kedro"
}
        
Elapsed time: 0.26356s