mage-ai


Namemage-ai JSON
Version 0.4.15 PyPI version JSON
download
home_pagehttps://github.com/mage-ai/mage-ai
SummaryMage is a tool for building and deploying data pipelines.
upload_time2022-10-03 08:51:37
maintainer
docs_urlNone
authorMage
requires_python>=3.6
license
keywords
VCS
bugtrack_url
requirements No requirements were recorded.
Travis-CI No Travis.
coveralls test coverage No coveralls.
            <img alt="Fire mage" height="160" src="media/mage-fire-charging-up.svg" />

[![PyPi](https://img.shields.io/pypi/v/mage-ai?color=orange)](https://pypi.org/project/mage-ai/)
[![mage-ai](https://img.shields.io/circleci/build/gh/mage-ai/mage-ai?color=%23159946&label=CircleCI&logo=circleci)](https://app.circleci.com/pipelines/github/mage-ai/mage-ai?branch=master&filter=all)
[![License](https://img.shields.io/github/license/mage-ai/mage-ai?color=red)](https://opensource.org/licenses/Apache-2.0)
[![Join Slack](https://img.shields.io/badge/Slack-Join%20Slack-blueviolet?logo=slack)](https://join.slack.com/t/mageai/shared_invite/zt-1adn34w4m-t~TcnPTlo3~5~d_0raOp6A)
<img
  referrerpolicy="no-referrer-when-downgrade"
  src="https://static.scarf.sh/a.png?x-pxid=b3c96d79-b8f0-414b-a687-8bfc164b4b7a"
/>

# ๐Ÿง™ Mage

Mage is an open-source tool for [building](docs/features/README.md#notebook-for-building-data-pipelines)
and [running](docs/features/orchestration/README.md)
data pipelines that <b>transform</b> your data.

<br />

<b>Here is a sample data pipeline defined across 3 files:</b>

```python
# data_loaders/load_data_from_file.py
@data_loader
def load_csv_from_file():
    return pd.read_csv('default_repo/titanic.csv')
```

```python
# transformers/select_columns.py
@transformer
def select_columns_from_df(df, *args):
    return df[['Age', 'Fare', 'Survived']]
```

```python
# data_exporters/export_to_file.py
@data_exporter
def export_titanic_data_to_disk(df) -> None:
    df.to_csv('default_repo/titanic_transformed.csv')
```

<br />

<b>What the data pipeline looks like in the UI:</b>

<img
  alt="data pipeline overview"
  src="media/data-pipeline-overview.jpg"
/>

New? We recommend reading about [blocks](docs/blocks/README.md) and
learning from a [hands-on tutorial](docs/tutorials/quick_start/etl_restaurant/README.md).

<br />

[![Join us on Slack](https://img.shields.io/badge/%20-Join%20us%20on%20Slack-black?style=for-the-badge&logo=slack&labelColor=6B50D7)](https://www.mage.ai/chat)

## Table of contents

1. [Quick start](#%EF%B8%8F-quick-start)
1. [Demo](#-demo)
1. [Tutorials](#-tutorials)
1. [Features](#-features)
1. [Core design principles](#%EF%B8%8F-core-design-principles)
1. [Core abstractions](#-core-abstractions)
1. [Documentation](docs/README.md)

# ๐Ÿƒโ€โ™€๏ธ Quick start

Install Mage using Docker or `pip`:

### Using Docker

Create a new project and launch tool (change `demo_project` to any other name if you want):

```bash
docker run -it -p 6789:6789 -v $(pwd):/home/src \
  mageai/mageai mage start demo_project
```

<sub>Follow the [guide](docs/spark/setup/README.md) if you want to use PySpark kernel in your notebook.</sub>

### Using `pip` or `conda`

##### 1. Install Mage
```bash
pip install mage-ai
```

or

```bash
conda install -c conda-forge mage-ai
```

<sub>For additional packages (e.g. `spark`, `postgres`, etc), please see [Installing extra packages](docs/README.md#installing-extra-packages).</sub>

<sub>If you run into errors, please see [Install errors](docs/tutorials/quick_start/setup.md#errors).</sub>

##### 2. Create new project and launch tool (change `demo_project` to any other name if you want):

```bash
mage start demo_project
```

### Open tool in browser

Open [http://localhost:6789](http://localhost:6789) in your browser and build a pipeline.

<br />

# ๐ŸŽฎ Demo

## Live demo

Try a hosted version of the tool here: [http://demo.mage.ai](http://demo.mage.ai/).

> WARNING
>
> The live demo is public, please donโ€™t save anything sensitive.

## Demo video (2 min)

[![Mage quick start demo](media/mage-youtube-preview.jpg)](https://www.youtube.com/watch?v=hrsErfPDits "Mage quick start demo")

<sub><i>Click the image to play video</i></sub>

<br />

# ๐Ÿ‘ฉโ€๐Ÿซ Tutorials

- [Train model on Titanic dataset](docs/tutorials/quick_start/train_titanic_model/README.md)
- [Load data from API, transform it, and export it to PostgreSQL](docs/tutorials/quick_start/etl_restaurant/README.md)
- [Integrate Mage into an existing Airflow project](docs/tutorials/airflow/integrate_into_existing_project/README.md)

<br />

# ๐Ÿ”ฎ Features

- [Data pipeline management](docs/features/orchestration/README.md)
- [Notebook for building data pipelines](docs/features/README.md#notebook-for-building-data-pipelines)

<sub>Read more [<b>here</b>](docs/features/README.md).</sub>

<br />

# ๐Ÿ”๏ธ Core design principles

Every user experience and technical design decision adheres to these principles.

1. [Easy developer experience](docs/core/design_principles.md#-easy-developer-experience)
1. [Engineering best practices built-in](docs/core/design_principles.md#engineering-best-practices-built-in)
1. [Data is a first-class citizen](docs/core/design_principles.md#data-is-a-first-class-citizen)
1. [Scaling made simple](docs/core/design_principles.md#scaling-is-made-simple)

<sub>Read more [<b>here</b>](docs/core/design_principles.md).</sub>

<br />

# ๐Ÿ›ธ Core abstractions

These are the fundamental concepts that Mage uses to operate.

- [Project](docs/core/abstractions.md#project)
- [Pipeline](docs/core/abstractions.md#pipeline)
- [Block](docs/core/abstractions.md#block)
- [Data product](docs/core/abstractions.md#data-product)
- [Trigger](docs/core/abstractions.md#trigger)
- [Run](docs/core/abstractions.md#run)

<sub>Read more [<b>here</b>](docs/core/abstractions.md).</sub>

<br />

# ๐Ÿ“š Documentation

Read more [<b>here</b>](docs/README.md).

<br />

# ๐Ÿ™‹โ€โ™€๏ธ Contributing

Check out the [๐ŸŽ contributing guide](/docs/contributing/README.md) to get started
by setting up your development environment and
exploring the code base.

<br />

# ๐Ÿค” Frequently Asked Questions (FAQs)

Check out our [FAQ page](https://www.notion.so/mageai/Mage-FAQs-33d93ee65f934ed39568f8a4bc823b39) to find answers to some of our most asked questions.

<br />

# ๐Ÿง™ Community
We love the community of Magers (`/หˆmฤjษ™r/`);
a group of mages who help each other realize their full potential!

To live chat with the Mage team and community,
please join the free Mage [<img alt="Slack" height="20" src="https://thepostsportsbar.com/wp-content/uploads/2017/02/Slack-Logo.png" style="position: relative; top: 4px;" /> Slack](https://www.mage.ai/chat)
channel.

[![Join us on Slack](https://img.shields.io/badge/%20-Join%20us%20on%20Slack-black?style=for-the-badge&logo=slack&labelColor=6B50D7)](https://www.mage.ai/chat)

For real-time news and fun memes, check out the Mage
[<img alt="Twitter" height="20" src="https://upload.wikimedia.org/wikipedia/commons/thumb/4/4f/Twitter-logo.svg/2491px-Twitter-logo.svg.png" style="position: relative; top: 4px;" /> Twitter](https://twitter.com/mage_ai).

To report bugs or add your awesome code for others to enjoy,
visit [GitHub](https://github.com/mage-ai/mage-ai).

<br />

# ๐Ÿชช License
See the [LICENSE](LICENSE) file for licensing information.

[<img alt="Water mage casting spell" height="300" src="media/mage-water-charging-up.svg" />](https://www.mage.ai/)

            

Raw data

            {
    "_id": null,
    "home_page": "https://github.com/mage-ai/mage-ai",
    "name": "mage-ai",
    "maintainer": "",
    "docs_url": null,
    "requires_python": ">=3.6",
    "maintainer_email": "",
    "keywords": "",
    "author": "Mage",
    "author_email": "eng@mage.ai",
    "download_url": "https://files.pythonhosted.org/packages/4d/34/e22927a0429a9ef6155afeeafa801a3c10f12ccd0ad8184f26685e739454/mage-ai-0.4.15.tar.gz",
    "platform": null,
    "description": "<img alt=\"Fire mage\" height=\"160\" src=\"media/mage-fire-charging-up.svg\" />\n\n[![PyPi](https://img.shields.io/pypi/v/mage-ai?color=orange)](https://pypi.org/project/mage-ai/)\n[![mage-ai](https://img.shields.io/circleci/build/gh/mage-ai/mage-ai?color=%23159946&label=CircleCI&logo=circleci)](https://app.circleci.com/pipelines/github/mage-ai/mage-ai?branch=master&filter=all)\n[![License](https://img.shields.io/github/license/mage-ai/mage-ai?color=red)](https://opensource.org/licenses/Apache-2.0)\n[![Join Slack](https://img.shields.io/badge/Slack-Join%20Slack-blueviolet?logo=slack)](https://join.slack.com/t/mageai/shared_invite/zt-1adn34w4m-t~TcnPTlo3~5~d_0raOp6A)\n<img\n  referrerpolicy=\"no-referrer-when-downgrade\"\n  src=\"https://static.scarf.sh/a.png?x-pxid=b3c96d79-b8f0-414b-a687-8bfc164b4b7a\"\n/>\n\n# \ud83e\uddd9 Mage\n\nMage is an open-source tool for [building](docs/features/README.md#notebook-for-building-data-pipelines)\nand [running](docs/features/orchestration/README.md)\ndata pipelines that <b>transform</b> your data.\n\n<br />\n\n<b>Here is a sample data pipeline defined across 3 files:</b>\n\n```python\n# data_loaders/load_data_from_file.py\n@data_loader\ndef load_csv_from_file():\n    return pd.read_csv('default_repo/titanic.csv')\n```\n\n```python\n# transformers/select_columns.py\n@transformer\ndef select_columns_from_df(df, *args):\n    return df[['Age', 'Fare', 'Survived']]\n```\n\n```python\n# data_exporters/export_to_file.py\n@data_exporter\ndef export_titanic_data_to_disk(df) -> None:\n    df.to_csv('default_repo/titanic_transformed.csv')\n```\n\n<br />\n\n<b>What the data pipeline looks like in the UI:</b>\n\n<img\n  alt=\"data pipeline overview\"\n  src=\"media/data-pipeline-overview.jpg\"\n/>\n\nNew? We recommend reading about [blocks](docs/blocks/README.md) and\nlearning from a [hands-on tutorial](docs/tutorials/quick_start/etl_restaurant/README.md).\n\n<br />\n\n[![Join us on Slack](https://img.shields.io/badge/%20-Join%20us%20on%20Slack-black?style=for-the-badge&logo=slack&labelColor=6B50D7)](https://www.mage.ai/chat)\n\n## Table of contents\n\n1. [Quick start](#%EF%B8%8F-quick-start)\n1. [Demo](#-demo)\n1. [Tutorials](#-tutorials)\n1. [Features](#-features)\n1. [Core design principles](#%EF%B8%8F-core-design-principles)\n1. [Core abstractions](#-core-abstractions)\n1. [Documentation](docs/README.md)\n\n# \ud83c\udfc3\u200d\u2640\ufe0f Quick start\n\nInstall Mage using Docker or `pip`:\n\n### Using Docker\n\nCreate a new project and launch tool (change `demo_project` to any other name if you want):\n\n```bash\ndocker run -it -p 6789:6789 -v $(pwd):/home/src \\\n  mageai/mageai mage start demo_project\n```\n\n<sub>Follow the [guide](docs/spark/setup/README.md) if you want to use PySpark kernel in your notebook.</sub>\n\n### Using `pip` or `conda`\n\n##### 1. Install Mage\n```bash\npip install mage-ai\n```\n\nor\n\n```bash\nconda install -c conda-forge mage-ai\n```\n\n<sub>For additional packages (e.g. `spark`, `postgres`, etc), please see [Installing extra packages](docs/README.md#installing-extra-packages).</sub>\n\n<sub>If you run into errors, please see [Install errors](docs/tutorials/quick_start/setup.md#errors).</sub>\n\n##### 2. Create new project and launch tool (change `demo_project` to any other name if you want):\n\n```bash\nmage start demo_project\n```\n\n### Open tool in browser\n\nOpen [http://localhost:6789](http://localhost:6789) in your browser and build a pipeline.\n\n<br />\n\n# \ud83c\udfae Demo\n\n## Live demo\n\nTry a hosted version of the tool here: [http://demo.mage.ai](http://demo.mage.ai/).\n\n> WARNING\n>\n> The live demo is public, please don\u2019t save anything sensitive.\n\n## Demo video (2 min)\n\n[![Mage quick start demo](media/mage-youtube-preview.jpg)](https://www.youtube.com/watch?v=hrsErfPDits \"Mage quick start demo\")\n\n<sub><i>Click the image to play video</i></sub>\n\n<br />\n\n# \ud83d\udc69\u200d\ud83c\udfeb Tutorials\n\n- [Train model on Titanic dataset](docs/tutorials/quick_start/train_titanic_model/README.md)\n- [Load data from API, transform it, and export it to PostgreSQL](docs/tutorials/quick_start/etl_restaurant/README.md)\n- [Integrate Mage into an existing Airflow project](docs/tutorials/airflow/integrate_into_existing_project/README.md)\n\n<br />\n\n# \ud83d\udd2e Features\n\n- [Data pipeline management](docs/features/orchestration/README.md)\n- [Notebook for building data pipelines](docs/features/README.md#notebook-for-building-data-pipelines)\n\n<sub>Read more [<b>here</b>](docs/features/README.md).</sub>\n\n<br />\n\n# \ud83c\udfd4\ufe0f Core design principles\n\nEvery user experience and technical design decision adheres to these principles.\n\n1. [Easy developer experience](docs/core/design_principles.md#-easy-developer-experience)\n1. [Engineering best practices built-in](docs/core/design_principles.md#engineering-best-practices-built-in)\n1. [Data is a first-class citizen](docs/core/design_principles.md#data-is-a-first-class-citizen)\n1. [Scaling made simple](docs/core/design_principles.md#scaling-is-made-simple)\n\n<sub>Read more [<b>here</b>](docs/core/design_principles.md).</sub>\n\n<br />\n\n# \ud83d\udef8 Core abstractions\n\nThese are the fundamental concepts that Mage uses to operate.\n\n- [Project](docs/core/abstractions.md#project)\n- [Pipeline](docs/core/abstractions.md#pipeline)\n- [Block](docs/core/abstractions.md#block)\n- [Data product](docs/core/abstractions.md#data-product)\n- [Trigger](docs/core/abstractions.md#trigger)\n- [Run](docs/core/abstractions.md#run)\n\n<sub>Read more [<b>here</b>](docs/core/abstractions.md).</sub>\n\n<br />\n\n# \ud83d\udcda Documentation\n\nRead more [<b>here</b>](docs/README.md).\n\n<br />\n\n# \ud83d\ude4b\u200d\u2640\ufe0f Contributing\n\nCheck out the [\ud83c\udf81 contributing guide](/docs/contributing/README.md) to get started\nby setting up your development environment and\nexploring the code base.\n\n<br />\n\n# \ud83e\udd14 Frequently Asked Questions (FAQs)\n\nCheck out our [FAQ page](https://www.notion.so/mageai/Mage-FAQs-33d93ee65f934ed39568f8a4bc823b39) to find answers to some of our most asked questions.\n\n<br />\n\n# \ud83e\uddd9 Community\nWe love the community of Magers (`/\u02c8m\u0101j\u0259r/`);\na group of mages who help each other realize their full potential!\n\nTo live chat with the Mage team and community,\nplease join the free Mage [<img alt=\"Slack\" height=\"20\" src=\"https://thepostsportsbar.com/wp-content/uploads/2017/02/Slack-Logo.png\" style=\"position: relative; top: 4px;\" /> Slack](https://www.mage.ai/chat)\nchannel.\n\n[![Join us on Slack](https://img.shields.io/badge/%20-Join%20us%20on%20Slack-black?style=for-the-badge&logo=slack&labelColor=6B50D7)](https://www.mage.ai/chat)\n\nFor real-time news and fun memes, check out the Mage\n[<img alt=\"Twitter\" height=\"20\" src=\"https://upload.wikimedia.org/wikipedia/commons/thumb/4/4f/Twitter-logo.svg/2491px-Twitter-logo.svg.png\" style=\"position: relative; top: 4px;\" /> Twitter](https://twitter.com/mage_ai).\n\nTo report bugs or add your awesome code for others to enjoy,\nvisit [GitHub](https://github.com/mage-ai/mage-ai).\n\n<br />\n\n# \ud83e\udeaa License\nSee the [LICENSE](LICENSE) file for licensing information.\n\n[<img alt=\"Water mage casting spell\" height=\"300\" src=\"media/mage-water-charging-up.svg\" />](https://www.mage.ai/)\n",
    "bugtrack_url": null,
    "license": "",
    "summary": "Mage is a tool for building and deploying data pipelines.",
    "version": "0.4.15",
    "split_keywords": [],
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "md5": "c88c5aa39f5334ca6f8a7d42f13f2425",
                "sha256": "7e967e10e208dd88c32b149b1b635d543c78ff342368c9b93c55cdf4686a28f2"
            },
            "downloads": -1,
            "filename": "mage_ai-0.4.15-py3-none-any.whl",
            "has_sig": false,
            "md5_digest": "c88c5aa39f5334ca6f8a7d42f13f2425",
            "packagetype": "bdist_wheel",
            "python_version": "py3",
            "requires_python": ">=3.6",
            "size": 6467423,
            "upload_time": "2022-10-03T08:51:35",
            "upload_time_iso_8601": "2022-10-03T08:51:35.735585Z",
            "url": "https://files.pythonhosted.org/packages/4d/d9/5f8f2de87e3777196f530a32a071c924c44af85dac28c960d23e20f856ad/mage_ai-0.4.15-py3-none-any.whl",
            "yanked": false,
            "yanked_reason": null
        },
        {
            "comment_text": "",
            "digests": {
                "md5": "0abddf437ec995d0f1b978353126dc05",
                "sha256": "36a947bf98a497689a3b45bc499a310c941be93fc65d2161fd68cbe836598b37"
            },
            "downloads": -1,
            "filename": "mage-ai-0.4.15.tar.gz",
            "has_sig": false,
            "md5_digest": "0abddf437ec995d0f1b978353126dc05",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": ">=3.6",
            "size": 6225798,
            "upload_time": "2022-10-03T08:51:37",
            "upload_time_iso_8601": "2022-10-03T08:51:37.883459Z",
            "url": "https://files.pythonhosted.org/packages/4d/34/e22927a0429a9ef6155afeeafa801a3c10f12ccd0ad8184f26685e739454/mage-ai-0.4.15.tar.gz",
            "yanked": false,
            "yanked_reason": null
        }
    ],
    "upload_time": "2022-10-03 08:51:37",
    "github": true,
    "gitlab": false,
    "bitbucket": false,
    "github_user": "mage-ai",
    "github_project": "mage-ai",
    "travis_ci": false,
    "coveralls": false,
    "github_actions": true,
    "requirements": [],
    "lcname": "mage-ai"
}
        
Elapsed time: 0.49239s