<h1 align="center">
<a
target="_blank"
href="https://mage.ai"
>
<img
align="center"
alt="Mage"
src="https://github.com/mage-ai/assets/blob/main/mascots/mascots-shorter.jpeg?raw=true"
style="width:100%;"
/>
</a>
</h1>
<p align="center">
๐ง A modern replacement for Airflow.
</p>
<p align="center">
<a
href="https://docs.mage.ai"
target="_blank"
><b>Documentation</b></a> ๐ช๏ธ
<a
href="https://youtu.be/GswOdShLGmg"
target="_blank"
><b>Get a 5 min overview</b></a> ๐
<a
href="https://demo.mage.ai"
target="_blank"
><b>Play with live tool</b></a> ๐ฅ
<a
href="https://www.mage.ai/chat"
target="_blank"
>
<b>Get instant help</b>
</a>
</p>
<div align="center">
<a
href="https://pypi.org/project/mage-ai/"
target="_blank"
>
<img alt="PyPi" src="https://img.shields.io/pypi/v/mage-ai?color=orange" />
</a>
<a
href="https://anaconda.org/conda-forge/mage-ai"
target="_blank"
>
<img src="https://anaconda.org/conda-forge/mage-ai/badges/version.svg" />
</a>
<a
href="https://opensource.org/licenses/Apache-2.0"
target="_blank"
>
<img alt="License" src="https://img.shields.io/github/license/mage-ai/mage-ai?color=red" />
</a>
<a
href="https://www.mage.ai/chat"
target="_blank"
>
<img alt="Slack" src="https://img.shields.io/badge/Slack-Join%20Slack-blueviolet?logo=slack" />
</a>
<a
href="https://github.com/mage-ai/mage-ai"
target="_blank"
>
<img alt="GitHub Stars" src="https://img.shields.io/github/stars/mage-ai/mage-ai?logo=github">
</a>
<a
href="https://hub.docker.com/r/mageai/mageai"
target="_blank"
>
<img alt="Docker pulls" src="https://img.shields.io/docker/pulls/mageai/mageai.svg">
</a>
<a
href="https://pepy.tech/project/mage-ai"
target="_blank"
>
<img alt="pip installs" src="https://static.pepy.tech/personalized-badge/mage-ai?period=total&units=international_system&left_color=grey&right_color=blue&left_text=pip%20installs">
</a>
</div>
<img
referrerpolicy="no-referrer-when-downgrade"
src="https://static.scarf.sh/a.png?x-pxid=b3c96d79-b8f0-414b-a687-8bfc164b4b7a"
/>
<div align="center">
### Give your data team `magical` powers
</div>
<p align="center">
<b>Integrate</b> and synchronize data from 3rd party sources
</p>
<p align="center">
Build real-time and batch pipelines to <b>transform</b> data using Python, SQL, and R
</p>
<p align="center">
Run, monitor, and <b>orchestrate</b> thousands of pipelines without losing sleep
</p>
<br />
<p align="center">1๏ธโฃ ๐๏ธ</p>
<h1 align="center">Build</h1>
<p align="center">
Have you met anyone who said they loved developing in Airflow?
<br />
Thatโs why we designed an easy developer experience that youโll enjoy.
</p>
| | |
| --- | --- |
| <b>Easy developer experience</b><br />Start developing locally with a single command or launch a dev environment in your cloud using Terraform.<br /><br/><b>Language of choice</b><br />Write code in Python, SQL, or R in the same data pipeline for ultimate flexibility.<br /><br /><b>Engineering best practices built-in</b><br />Each step in your pipeline is a standalone file containing modular code thatโs reusable and testable with data validations. No more DAGs with spaghetti code. | <img src="https://github.com/mage-ai/assets/blob/main/overview/mage-build.gif?raw=true" /> |
<p align="center">
โ
</p>
<p align="center">2๏ธโฃ ๐ฎ</p>
<h1 align="center">Preview</h1>
<p align="center">
Stop wasting time waiting around for your DAGs to finish testing.
<br />
Get instant feedback from your code each time you run it.
</p>
| | |
| --- | --- |
| <b>Interactive code</b><br />Immediately see results from your codeโs output with an interactive notebook UI.<br /><br/><b>Data is a first-class citizen</b><br />Each block of code in your pipeline produces data that can be versioned, partitioned, and cataloged for future use.<br /><br /><b>Collaborate on cloud</b><br />Develop collaboratively on cloud resources, version control with Git, and test pipelines without waiting for an available shared staging environment. | <img src="https://github.com/mage-ai/assets/blob/main/overview/mage-preview.gif?raw=True" /> |
<p align="center">
โ
</p>
<p align="center">3๏ธโฃ ๐</p>
<h1 align="center">Launch</h1>
<p align="center">
Donโt have a large team dedicated to Airflow?
<br />
Mage makes it easy for a single developer or small team to scale up and manage thousands of pipelines.
</p>
| | |
| --- | --- |
| <b>Fast deploy</b><br />Deploy Mage to AWS, GCP, or Azure with only 2 commands using maintained Terraform templates.<br /><br/><b>Scaling made simple</b><br />Transform very large datasets directly in your data warehouse or through a native integration with Spark.<br /><br /><b>Observability</b><br />Operationalize your pipelines with built-in monitoring, alerting, and observability through an intuitive UI. | <img src="https://github.com/mage-ai/assets/blob/main/overview/observability.gif?raw=True" /> |
<br />
# ๐ง Intro
Mage is an open-source data pipeline tool for transforming and integrating data.
1. [Install](#%EF%B8%8F-install)
1. [Demo](#-demo)
1. [Tutorials](#-tutorials)
1. [Documentation](https://docs.mage.ai)
1. [Features](#-features)
1. [Core design principles](https://docs.mage.ai/design/core-design-principles)
1. [Core abstractions](https://docs.mage.ai/design/core-abstractions)
1. [Contributing](https://docs.mage.ai/community/contributing)
<br />
# ๐โโ๏ธ Install
The recommended way to install the latest version of Mage is through Docker with the following command:
```bash
docker pull mageai/mageai:latest
```
You can also install Mage using pip or conda, though this may cause dependency issues without the proper environment.
```bash
pip install mage-ai
```
```bash
conda install -c conda-forge mage-ai
```
Looking for help? The _fastest_ way to get started is by checking out our documentation [here](https://docs.mage.ai/getting-started/setup).
Looking for quick examples? Open a [demo](https://demo.mage.ai/) project right in your browser or check out our [guides](https://docs.mage.ai/guides/overview).
# ๐ฎ Demo
### Live demo
Build and run a data pipeline with our <b>[demo app](https://demo.mage.ai/)</b>.
> WARNING
>
> The live demo is public to everyone, please donโt save anything sensitive (e.g. passwords, secrets, etc).
### Demo video (5 min)
[](https://youtu.be/GswOdShLGmg)
<sub><i>Click the image to play video</i></sub>
<br />
# ๐ฉโ๐ซ Tutorials
- [Load data from API, transform it, and export it to PostgreSQL](https://docs.mage.ai/guides/load-api-data)
- [Integrate Mage into an existing Airflow project](https://docs.mage.ai/integrations/airflow)
- [Train model on Titanic dataset](https://docs.mage.ai/guides/train-model)
- [Set up dbt models and orchestrate dbt runs](https://docs.mage.ai/integrations/dbt-models)
<img alt="Fire mage" height="160" src="https://github.com/mage-ai/assets/blob/main/mage-fire-charging-up.svg?raw=True" />
<br />
# ๐ฎ [Features](https://docs.mage.ai/about/features)
| | | |
| --- | --- | --- |
| ๐ถ | <b>[Orchestration](https://docs.mage.ai/design/data-pipeline-management)</b> | Schedule and manage data pipelines with observability. |
| ๐ | <b>[Notebook](https://docs.mage.ai/about/features#notebook-for-building-data-pipelines)</b> | Interactive Python, SQL, & R editor for coding data pipelines. |
| ๐๏ธ | <b>[Data integrations](https://docs.mage.ai/data-integrations/overview)</b> | Synchronize data from 3rd party sources to your internal destinations. |
| ๐ฐ | <b>[Streaming pipelines](https://docs.mage.ai/guides/streaming-pipeline)</b> | Ingest and transform real-time data. |
| โ | <b>[dbt](https://docs.mage.ai/dbt/overview)</b> | Build, run, and manage your dbt models with Mage. |
<b>A sample data pipeline defined across 3 files โ</b>
1. Load data โ
```python
@data_loader
def load_csv_from_file():
return pd.read_csv('default_repo/titanic.csv')
```
1. Transform data โ
```python
@transformer
def select_columns_from_df(df, *args):
return df[['Age', 'Fare', 'Survived']]
```
1. Export data โ
```python
@data_exporter
def export_titanic_data_to_disk(df) -> None:
df.to_csv('default_repo/titanic_transformed.csv')
```
<b>What the data pipeline looks like in the UI โ</b>
<img
alt="data pipeline overview"
src="https://github.com/mage-ai/assets/blob/main/data-pipeline-overview.png?raw=True"
/>
New? We recommend reading about <b>[blocks](https://docs.mage.ai/design/blocks)</b> and
learning from a <b>[hands-on tutorial](https://docs.mage.ai/guides/load-api-data)</b>.
[](https://www.mage.ai/chat)
<br />
# ๐๏ธ [Core design principles](https://docs.mage.ai/design/core-design-principles)
Every user experience and technical design decision adheres to these principles.
| | | |
| --- | --- | --- |
| ๐ป | [Easy developer experience](https://docs.mage.ai/design/core-design-principles#easy-developer-experience) | Open-source engine that comes with a custom notebook UI for building data pipelines. |
| ๐ข | [Engineering best practices built-in](https://docs.mage.ai/design/core-design-principles#engineering-best-practices-built-in) | Build and deploy data pipelines using modular code. No more writing throwaway code or trying to turn notebooks into scripts. |
| ๐ณ | [Data is a first-class citizen](https://docs.mage.ai/design/core-design-principles#data-is-a-first-class-citizen) | Designed from the ground up specifically for running data-intensive workflows. |
| ๐ช | [Scaling is made simple](https://docs.mage.ai/design/core-design-principles#scaling-is-made-simple) | Analyze and process large data quickly for rapid iteration. |
<br />
# ๐ธ [Core abstractions](https://docs.mage.ai/design/core-abstractions)
These are the fundamental concepts that Mage uses to operate.
| | |
| --- | --- |
| [Project](https://docs.mage.ai/design/core-abstractions#project) | Like a repository on GitHub; this is where you write all your code. |
| [Pipeline](https://docs.mage.ai/design/core-abstractions#pipeline) | Contains references to all the blocks of code you want to run, charts for visualizing data, and organizes the dependency between each block of code. |
| [Block](https://docs.mage.ai/design/core-abstractions#block) | A file with code that can be executed independently or within a pipeline. |
| [Data product](https://docs.mage.ai/design/core-abstractions#data-product) | Every block produces data after it's been executed. These are called data products in Mage. |
| [Trigger](https://docs.mage.ai/design/core-abstractions#trigger) | A set of instructions that determine when or how a pipeline should run. |
| [Run](https://docs.mage.ai/design/core-abstractions#run) | Stores information about when it was started, its status, when it was completed, any runtime variables used in the execution of the pipeline or block, etc. |
<br />
# ๐โโ๏ธ Contributing and developing
Add features and instantly improve the experience for everyone.
Check out the <b>[contributing guide](https://docs.mage.ai/community/contributing)</b>
to set up your development environment and start building.
<br />
# ๐จโ๐ฉโ๐งโ๐ฆ Community
Individually, weโre a mage.
> ๐ง Mage
>
> Magic is indistinguishable from advanced technology.
> A mage is someone who uses magic (aka advanced technology).
Together, weโre Magers!
> ๐งโโ๏ธ๐ง Magers (`/หmฤjษr/`)
>
> A group of mages who help each other realize their full potential!
Letโs hang out and chat together โ
[](https://www.mage.ai/chat)
For real-time news, fun memes, data engineering topics, and more, join us on โ
| | |
| --- | --- |
| <img alt="Twitter" height="20" src="https://user-images.githubusercontent.com/78053898/198755056-a15c4439-c07f-41ea-ba35-bc4bfdd09f1a.png" /> | [Twitter](https://twitter.com/mage_ai) |
| <img alt="LinkedIn" height="20" src="https://user-images.githubusercontent.com/78053898/198755052-2777d6ae-c161-4a4b-9ece-4fd7bd458e26.png" /> | [LinkedIn](https://www.linkedin.com/company/magetech/mycompany) |
| <img alt="GitHub" height="20" src="https://user-images.githubusercontent.com/78053898/198755053-5c3971b1-9c49-4888-8a8e-1599f0fc6646.png" /> | [GitHub](https://github.com/mage-ai/mage-ai) |
| <img alt="Slack" height="20" src="https://user-images.githubusercontent.com/78053898/198755054-03d47bfc-18b6-45a5-9593-7b496eb927f3.png" /> | [Slack](https://www.mage.ai/chat) |
<br />
# ๐ค Frequently Asked Questions (FAQs)
Check out our [FAQ page](https://docs.mage.ai/about/frequently-asked-questions) to find answers to some of our most asked questions.
<br />
# ๐ชช License
See the [LICENSE](LICENSE) file for licensing information.
[<img alt="Water mage casting spell" height="300" src="https://github.com/mage-ai/assets/blob/main/mage-water-charging-up.svg?raw=True" />](https://www.mage.ai/)
<br />
Raw data
{
"_id": null,
"home_page": "https://github.com/mage-ai/mage-ai",
"name": "mage-ai",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.7",
"maintainer_email": null,
"keywords": null,
"author": "Mage",
"author_email": "eng@mage.ai",
"download_url": "https://files.pythonhosted.org/packages/56/80/ceb3284ef24f89444723e3a1a1cbbe70836a4f86d9bd87a8aa34691bf542/mage-ai-0.9.69.tar.gz",
"platform": null,
"description": "<h1 align=\"center\">\n <a\n target=\"_blank\"\n href=\"https://mage.ai\"\n >\n <img\n align=\"center\"\n alt=\"Mage\"\n src=\"https://github.com/mage-ai/assets/blob/main/mascots/mascots-shorter.jpeg?raw=true\"\n style=\"width:100%;\"\n />\n </a>\n</h1>\n<p align=\"center\">\n \ud83e\uddd9 A modern replacement for Airflow.\n</p>\n\n<p align=\"center\">\n <a\n href=\"https://docs.mage.ai\"\n target=\"_blank\"\n ><b>Documentation</b></a> \ud83c\udf2a\ufe0f \n <a\n href=\"https://youtu.be/GswOdShLGmg\"\n target=\"_blank\"\n ><b>Get a 5 min overview</b></a> \ud83c\udf0a \n <a\n href=\"https://demo.mage.ai\"\n target=\"_blank\"\n ><b>Play with live tool</b></a> \ud83d\udd25 \n <a\n href=\"https://www.mage.ai/chat\"\n target=\"_blank\"\n >\n <b>Get instant help</b>\n </a>\n</p>\n<div align=\"center\">\n <a\n href=\"https://pypi.org/project/mage-ai/\"\n target=\"_blank\"\n >\n <img alt=\"PyPi\" src=\"https://img.shields.io/pypi/v/mage-ai?color=orange\" />\n </a>\n <a\n href=\"https://anaconda.org/conda-forge/mage-ai\"\n target=\"_blank\"\n >\n <img src=\"https://anaconda.org/conda-forge/mage-ai/badges/version.svg\" />\n </a>\n <a\n href=\"https://opensource.org/licenses/Apache-2.0\"\n target=\"_blank\"\n >\n <img alt=\"License\" src=\"https://img.shields.io/github/license/mage-ai/mage-ai?color=red\" />\n </a>\n <a\n href=\"https://www.mage.ai/chat\"\n target=\"_blank\"\n >\n <img alt=\"Slack\" src=\"https://img.shields.io/badge/Slack-Join%20Slack-blueviolet?logo=slack\" />\n </a>\n <a\n href=\"https://github.com/mage-ai/mage-ai\"\n target=\"_blank\"\n >\n <img alt=\"GitHub Stars\" src=\"https://img.shields.io/github/stars/mage-ai/mage-ai?logo=github\">\n </a>\n <a\n href=\"https://hub.docker.com/r/mageai/mageai\"\n target=\"_blank\"\n >\n <img alt=\"Docker pulls\" src=\"https://img.shields.io/docker/pulls/mageai/mageai.svg\">\n </a>\n <a\n href=\"https://pepy.tech/project/mage-ai\"\n target=\"_blank\"\n >\n <img alt=\"pip installs\" src=\"https://static.pepy.tech/personalized-badge/mage-ai?period=total&units=international_system&left_color=grey&right_color=blue&left_text=pip%20installs\">\n </a>\n</div>\n<img\n referrerpolicy=\"no-referrer-when-downgrade\"\n src=\"https://static.scarf.sh/a.png?x-pxid=b3c96d79-b8f0-414b-a687-8bfc164b4b7a\"\n/>\n\n<div align=\"center\">\n\n### Give your data team `magical` powers\n\n</div>\n\n<p align=\"center\">\n <b>Integrate</b> and synchronize data from 3rd party sources\n</p>\n\n<p align=\"center\">\n Build real-time and batch pipelines to <b>transform</b> data using Python, SQL, and R\n</p>\n\n<p align=\"center\">\n Run, monitor, and <b>orchestrate</b> thousands of pipelines without losing sleep\n</p>\n\n<br />\n\n<p align=\"center\">1\ufe0f\u20e3 \ud83c\udfd7\ufe0f</p>\n<h1 align=\"center\">Build</h1>\n<p align=\"center\">\n Have you met anyone who said they loved developing in Airflow?\n <br />\n That\u2019s why we designed an easy developer experience that you\u2019ll enjoy.\n</p>\n\n| | |\n| --- | --- |\n| <b>Easy developer experience</b><br />Start developing locally with a single command or launch a dev environment in your cloud using Terraform.<br /><br/><b>Language of choice</b><br />Write code in Python, SQL, or R in the same data pipeline for ultimate flexibility.<br /><br /><b>Engineering best practices built-in</b><br />Each step in your pipeline is a standalone file containing modular code that\u2019s reusable and testable with data validations. No more DAGs with spaghetti code. | <img src=\"https://github.com/mage-ai/assets/blob/main/overview/mage-build.gif?raw=true\" /> |\n\n<p align=\"center\">\n \u2193\n</p>\n\n<p align=\"center\">2\ufe0f\u20e3 \ud83d\udd2e</p>\n<h1 align=\"center\">Preview</h1>\n<p align=\"center\">\n Stop wasting time waiting around for your DAGs to finish testing.\n <br />\n Get instant feedback from your code each time you run it.\n</p>\n\n| | |\n| --- | --- |\n| <b>Interactive code</b><br />Immediately see results from your code\u2019s output with an interactive notebook UI.<br /><br/><b>Data is a first-class citizen</b><br />Each block of code in your pipeline produces data that can be versioned, partitioned, and cataloged for future use.<br /><br /><b>Collaborate on cloud</b><br />Develop collaboratively on cloud resources, version control with Git, and test pipelines without waiting for an available shared staging environment. | <img src=\"https://github.com/mage-ai/assets/blob/main/overview/mage-preview.gif?raw=True\" /> |\n\n<p align=\"center\">\n \u2193\n</p>\n\n<p align=\"center\">3\ufe0f\u20e3 \ud83d\ude80</p>\n<h1 align=\"center\">Launch</h1>\n<p align=\"center\">\n Don\u2019t have a large team dedicated to Airflow?\n <br />\n Mage makes it easy for a single developer or small team to scale up and manage thousands of pipelines.\n</p>\n\n| | |\n| --- | --- |\n| <b>Fast deploy</b><br />Deploy Mage to AWS, GCP, or Azure with only 2 commands using maintained Terraform templates.<br /><br/><b>Scaling made simple</b><br />Transform very large datasets directly in your data warehouse or through a native integration with Spark.<br /><br /><b>Observability</b><br />Operationalize your pipelines with built-in monitoring, alerting, and observability through an intuitive UI. | <img src=\"https://github.com/mage-ai/assets/blob/main/overview/observability.gif?raw=True\" /> |\n\n<br />\n\n# \ud83e\uddd9 Intro\n\nMage is an open-source data pipeline tool for transforming and integrating data.\n\n1. [Install](#%EF%B8%8F-install)\n1. [Demo](#-demo)\n1. [Tutorials](#-tutorials)\n1. [Documentation](https://docs.mage.ai)\n1. [Features](#-features)\n1. [Core design principles](https://docs.mage.ai/design/core-design-principles)\n1. [Core abstractions](https://docs.mage.ai/design/core-abstractions)\n1. [Contributing](https://docs.mage.ai/community/contributing)\n\n<br />\n\n# \ud83c\udfc3\u200d\u2640\ufe0f Install\n\nThe recommended way to install the latest version of Mage is through Docker with the following command:\n\n```bash\ndocker pull mageai/mageai:latest\n```\n\nYou can also install Mage using pip or conda, though this may cause dependency issues without the proper environment.\n\n```bash\npip install mage-ai\n```\n```bash\nconda install -c conda-forge mage-ai\n```\n\nLooking for help? The _fastest_ way to get started is by checking out our documentation [here](https://docs.mage.ai/getting-started/setup).\n\nLooking for quick examples? Open a [demo](https://demo.mage.ai/) project right in your browser or check out our [guides](https://docs.mage.ai/guides/overview).\n\n# \ud83c\udfae Demo\n\n### Live demo\n\nBuild and run a data pipeline with our <b>[demo app](https://demo.mage.ai/)</b>.\n\n> WARNING\n>\n> The live demo is public to everyone, please don\u2019t save anything sensitive (e.g. passwords, secrets, etc).\n### Demo video (5 min)\n\n[](https://youtu.be/GswOdShLGmg)\n\n<sub><i>Click the image to play video</i></sub>\n\n<br />\n\n# \ud83d\udc69\u200d\ud83c\udfeb Tutorials\n\n- [Load data from API, transform it, and export it to PostgreSQL](https://docs.mage.ai/guides/load-api-data)\n- [Integrate Mage into an existing Airflow project](https://docs.mage.ai/integrations/airflow)\n- [Train model on Titanic dataset](https://docs.mage.ai/guides/train-model)\n- [Set up dbt models and orchestrate dbt runs](https://docs.mage.ai/integrations/dbt-models)\n\n<img alt=\"Fire mage\" height=\"160\" src=\"https://github.com/mage-ai/assets/blob/main/mage-fire-charging-up.svg?raw=True\" />\n\n<br />\n\n# \ud83d\udd2e [Features](https://docs.mage.ai/about/features)\n\n| | | |\n| --- | --- | --- |\n| \ud83c\udfb6 | <b>[Orchestration](https://docs.mage.ai/design/data-pipeline-management)</b> | Schedule and manage data pipelines with observability. |\n| \ud83d\udcd3 | <b>[Notebook](https://docs.mage.ai/about/features#notebook-for-building-data-pipelines)</b> | Interactive Python, SQL, & R editor for coding data pipelines. |\n| \ud83c\udfd7\ufe0f | <b>[Data integrations](https://docs.mage.ai/data-integrations/overview)</b> | Synchronize data from 3rd party sources to your internal destinations. |\n| \ud83d\udeb0 | <b>[Streaming pipelines](https://docs.mage.ai/guides/streaming-pipeline)</b> | Ingest and transform real-time data. |\n| \u274e | <b>[dbt](https://docs.mage.ai/dbt/overview)</b> | Build, run, and manage your dbt models with Mage. |\n\n<b>A sample data pipeline defined across 3 files \u279d</b>\n\n1. Load data \u279d\n ```python\n @data_loader\n def load_csv_from_file():\n return pd.read_csv('default_repo/titanic.csv')\n ```\n1. Transform data \u279d\n ```python\n @transformer\n def select_columns_from_df(df, *args):\n return df[['Age', 'Fare', 'Survived']]\n ```\n1. Export data \u279d\n ```python\n @data_exporter\n def export_titanic_data_to_disk(df) -> None:\n df.to_csv('default_repo/titanic_transformed.csv')\n ```\n\n<b>What the data pipeline looks like in the UI \u279d</b>\n\n<img\n alt=\"data pipeline overview\"\n src=\"https://github.com/mage-ai/assets/blob/main/data-pipeline-overview.png?raw=True\"\n/>\n\nNew? We recommend reading about <b>[blocks](https://docs.mage.ai/design/blocks)</b> and\nlearning from a <b>[hands-on tutorial](https://docs.mage.ai/guides/load-api-data)</b>.\n\n[](https://www.mage.ai/chat)\n\n<br />\n\n# \ud83c\udfd4\ufe0f [Core design principles](https://docs.mage.ai/design/core-design-principles)\n\nEvery user experience and technical design decision adheres to these principles.\n\n| | | |\n| --- | --- | --- |\n| \ud83d\udcbb | [Easy developer experience](https://docs.mage.ai/design/core-design-principles#easy-developer-experience) | Open-source engine that comes with a custom notebook UI for building data pipelines. |\n| \ud83d\udea2 | [Engineering best practices built-in](https://docs.mage.ai/design/core-design-principles#engineering-best-practices-built-in) | Build and deploy data pipelines using modular code. No more writing throwaway code or trying to turn notebooks into scripts. |\n| \ud83d\udcb3 | [Data is a first-class citizen](https://docs.mage.ai/design/core-design-principles#data-is-a-first-class-citizen) | Designed from the ground up specifically for running data-intensive workflows. |\n| \ud83e\ude90 | [Scaling is made simple](https://docs.mage.ai/design/core-design-principles#scaling-is-made-simple) | Analyze and process large data quickly for rapid iteration. |\n\n<br />\n\n# \ud83d\udef8 [Core abstractions](https://docs.mage.ai/design/core-abstractions)\n\nThese are the fundamental concepts that Mage uses to operate.\n\n| | |\n| --- | --- |\n| [Project](https://docs.mage.ai/design/core-abstractions#project) | Like a repository on GitHub; this is where you write all your code. |\n| [Pipeline](https://docs.mage.ai/design/core-abstractions#pipeline) | Contains references to all the blocks of code you want to run, charts for visualizing data, and organizes the dependency between each block of code. |\n| [Block](https://docs.mage.ai/design/core-abstractions#block) | A file with code that can be executed independently or within a pipeline. |\n| [Data product](https://docs.mage.ai/design/core-abstractions#data-product) | Every block produces data after it's been executed. These are called data products in Mage. |\n| [Trigger](https://docs.mage.ai/design/core-abstractions#trigger) | A set of instructions that determine when or how a pipeline should run. |\n| [Run](https://docs.mage.ai/design/core-abstractions#run) | Stores information about when it was started, its status, when it was completed, any runtime variables used in the execution of the pipeline or block, etc. |\n\n<br />\n\n# \ud83d\ude4b\u200d\u2640\ufe0f Contributing and developing\n\nAdd features and instantly improve the experience for everyone.\n\nCheck out the <b>[contributing guide](https://docs.mage.ai/community/contributing)</b>\nto set up your development environment and start building.\n\n<br />\n\n# \ud83d\udc68\u200d\ud83d\udc69\u200d\ud83d\udc67\u200d\ud83d\udc66 Community\nIndividually, we\u2019re a mage.\n\n> \ud83e\uddd9 Mage\n>\n> Magic is indistinguishable from advanced technology.\n> A mage is someone who uses magic (aka advanced technology).\nTogether, we\u2019re Magers!\n\n> \ud83e\uddd9\u200d\u2642\ufe0f\ud83e\uddd9 Magers (`/\u02c8m\u0101j\u0259r/`)\n>\n> A group of mages who help each other realize their full potential!\nLet\u2019s hang out and chat together \u279d\n\n[](https://www.mage.ai/chat)\n\nFor real-time news, fun memes, data engineering topics, and more, join us on \u279d\n\n| | |\n| --- | --- |\n| <img alt=\"Twitter\" height=\"20\" src=\"https://user-images.githubusercontent.com/78053898/198755056-a15c4439-c07f-41ea-ba35-bc4bfdd09f1a.png\" /> | [Twitter](https://twitter.com/mage_ai) |\n| <img alt=\"LinkedIn\" height=\"20\" src=\"https://user-images.githubusercontent.com/78053898/198755052-2777d6ae-c161-4a4b-9ece-4fd7bd458e26.png\" /> | [LinkedIn](https://www.linkedin.com/company/magetech/mycompany) |\n| <img alt=\"GitHub\" height=\"20\" src=\"https://user-images.githubusercontent.com/78053898/198755053-5c3971b1-9c49-4888-8a8e-1599f0fc6646.png\" /> | [GitHub](https://github.com/mage-ai/mage-ai) |\n| <img alt=\"Slack\" height=\"20\" src=\"https://user-images.githubusercontent.com/78053898/198755054-03d47bfc-18b6-45a5-9593-7b496eb927f3.png\" /> | [Slack](https://www.mage.ai/chat) |\n\n<br />\n\n# \ud83e\udd14 Frequently Asked Questions (FAQs)\n\nCheck out our [FAQ page](https://docs.mage.ai/about/frequently-asked-questions) to find answers to some of our most asked questions.\n\n<br />\n\n# \ud83e\udeaa License\nSee the [LICENSE](LICENSE) file for licensing information.\n\n[<img alt=\"Water mage casting spell\" height=\"300\" src=\"https://github.com/mage-ai/assets/blob/main/mage-water-charging-up.svg?raw=True\" />](https://www.mage.ai/)\n\n<br />\n",
"bugtrack_url": null,
"license": null,
"summary": "Mage is a tool for building and deploying data pipelines.",
"version": "0.9.69",
"project_urls": {
"Homepage": "https://github.com/mage-ai/mage-ai"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "a213318ad418ba29c93b1a6bca8050b2bea7d07b5bc1a51098bf232e2cedbbcd",
"md5": "570e6790f49c8afe7f201c537b441277",
"sha256": "4797c58aac1a1e573cd82cbfe76cbc09a34a52f9c5b857cc26203fc3327ca4e9"
},
"downloads": -1,
"filename": "mage_ai-0.9.69-py3-none-any.whl",
"has_sig": false,
"md5_digest": "570e6790f49c8afe7f201c537b441277",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.7",
"size": 20630975,
"upload_time": "2024-04-11T09:28:12",
"upload_time_iso_8601": "2024-04-11T09:28:12.773464Z",
"url": "https://files.pythonhosted.org/packages/a2/13/318ad418ba29c93b1a6bca8050b2bea7d07b5bc1a51098bf232e2cedbbcd/mage_ai-0.9.69-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "5680ceb3284ef24f89444723e3a1a1cbbe70836a4f86d9bd87a8aa34691bf542",
"md5": "b41b7f34497312ffd4335887c0acca18",
"sha256": "2ef0c82bf3ebd2cc4edbff677451b2af8eac12d697e3cbd0980ce555947c96e5"
},
"downloads": -1,
"filename": "mage-ai-0.9.69.tar.gz",
"has_sig": false,
"md5_digest": "b41b7f34497312ffd4335887c0acca18",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.7",
"size": 19239659,
"upload_time": "2024-04-11T09:28:19",
"upload_time_iso_8601": "2024-04-11T09:28:19.332545Z",
"url": "https://files.pythonhosted.org/packages/56/80/ceb3284ef24f89444723e3a1a1cbbe70836a4f86d9bd87a8aa34691bf542/mage-ai-0.9.69.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-04-11 09:28:19",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "mage-ai",
"github_project": "mage-ai",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"circle": true,
"requirements": [
{
"name": "Faker",
"specs": [
[
"==",
"4.14.0"
]
]
},
{
"name": "GitPython",
"specs": [
[
"==",
"3.1.41"
]
]
},
{
"name": "Jinja2",
"specs": [
[
"==",
"3.1.3"
]
]
},
{
"name": "PyGithub",
"specs": [
[
"==",
"1.59.0"
]
]
},
{
"name": "PyJWT",
"specs": [
[
"==",
"2.6.0"
]
]
},
{
"name": "aiofiles",
"specs": [
[
"==",
"22.1.0"
]
]
},
{
"name": "alembic",
"specs": [
[
">=",
"1.7.5"
]
]
},
{
"name": "bcrypt",
"specs": [
[
"==",
"4.0.1"
]
]
},
{
"name": "croniter",
"specs": [
[
"==",
"1.3.7"
]
]
},
{
"name": "cryptography",
"specs": [
[
"==",
"41.0.6"
]
]
},
{
"name": "dask",
"specs": [
[
">=",
"2022.2.0"
]
]
},
{
"name": "datadog",
"specs": [
[
"==",
"0.44.0"
]
]
},
{
"name": "freezegun",
"specs": [
[
"==",
"1.2.2"
]
]
},
{
"name": "great-expectations",
"specs": [
[
"==",
"0.18.12"
]
]
},
{
"name": "httpx",
"specs": [
[
"==",
"0.23.1"
]
]
},
{
"name": "inflection",
"specs": [
[
"==",
"0.5.1"
]
]
},
{
"name": "ipykernel",
"specs": [
[
"==",
"6.15.0"
]
]
},
{
"name": "ipython",
"specs": [
[
"==",
"8.10.0"
]
]
},
{
"name": "itsdangerous",
"specs": [
[
"~=",
"1.1.0"
]
]
},
{
"name": "joblib",
"specs": [
[
">=",
"1.1.0"
]
]
},
{
"name": "jupyter-server-proxy",
"specs": [
[
"==",
"3.2.1"
]
]
},
{
"name": "jupyter-server",
"specs": [
[
"==",
"1.23.5"
]
]
},
{
"name": "jupyter_client",
"specs": [
[
"==",
"7.4.4"
]
]
},
{
"name": "ldap3",
"specs": [
[
"==",
"2.9.1"
]
]
},
{
"name": "newrelic",
"specs": [
[
"==",
"8.8.0"
]
]
},
{
"name": "numpy",
"specs": [
[
">=",
"1.22.0"
]
]
},
{
"name": "pandas",
"specs": [
[
">=",
"1.3.0"
]
]
},
{
"name": "polars",
"specs": [
[
">=",
"0.18.0"
],
[
"<",
"0.19.2"
]
]
},
{
"name": "protobuf",
"specs": [
[
"~=",
"4.21.12"
]
]
},
{
"name": "pyarrow",
"specs": [
[
"==",
"14.0.1"
]
]
},
{
"name": "python-dateutil",
"specs": [
[
"==",
"2.8.2"
]
]
},
{
"name": "pytz",
"specs": [
[
"==",
"2022.2.1"
]
]
},
{
"name": "pyyaml",
"specs": [
[
"~=",
"6.0"
]
]
},
{
"name": "redis",
"specs": [
[
"~=",
"5.0.1"
]
]
},
{
"name": "requests",
"specs": [
[
"~=",
"2.31.0"
]
]
},
{
"name": "ruamel.yaml",
"specs": [
[
"==",
"0.17.17"
]
]
},
{
"name": "scikit-learn",
"specs": [
[
">=",
"1.0"
]
]
},
{
"name": "sentry-sdk",
"specs": [
[
"==",
"1.19.1"
]
]
},
{
"name": "simplejson",
"specs": []
},
{
"name": "six",
"specs": [
[
">=",
"1.15.0"
]
]
},
{
"name": "sqlalchemy",
"specs": [
[
"<",
"2.0.0"
],
[
">=",
"1.4.20"
]
]
},
{
"name": "sqlglot",
"specs": []
},
{
"name": "terminado",
"specs": [
[
"==",
"0.17.1"
]
]
},
{
"name": "thefuzz",
"specs": [
[
"==",
"0.19.0"
]
]
},
{
"name": "tornado",
"specs": [
[
"==",
"6.3.3"
]
]
},
{
"name": "typer",
"specs": [
[
"==",
"0.9.0"
]
]
},
{
"name": "typing_extensions",
"specs": [
[
"==",
"4.10.0"
]
]
},
{
"name": "watchdog",
"specs": [
[
"==",
"4.0.0"
]
]
},
{
"name": "astor",
"specs": [
[
">=",
"0.8.1"
]
]
},
{
"name": "aws-secretsmanager-caching",
"specs": [
[
"==",
"1.1.1.5"
]
]
},
{
"name": "boto3",
"specs": [
[
"==",
"1.26.60"
]
]
},
{
"name": "botocore",
"specs": [
[
"==",
"1.29.60"
]
]
},
{
"name": "clickhouse-connect",
"specs": [
[
"~=",
"0.6.23"
]
]
},
{
"name": "db-dtypes",
"specs": [
[
"==",
"1.0.5"
]
]
},
{
"name": "duckdb",
"specs": [
[
"==",
"0.9.2"
]
]
},
{
"name": "google-api-core",
"specs": [
[
"~=",
"2.15.0"
]
]
},
{
"name": "google-api-python-client",
"specs": [
[
"~=",
"2.70.0"
]
]
},
{
"name": "google-cloud-bigquery",
"specs": [
[
"~=",
"3.14.1"
]
]
},
{
"name": "google-cloud-iam",
"specs": [
[
"~=",
"2.13.0"
]
]
},
{
"name": "google-cloud-pubsub",
"specs": [
[
"~=",
"2.19.0"
]
]
},
{
"name": "google-cloud-run",
"specs": [
[
"~=",
"0.10.1"
]
]
},
{
"name": "google-cloud-storage",
"specs": [
[
"~=",
"2.5.0"
]
]
},
{
"name": "gspread",
"specs": [
[
"==",
"5.7.2"
]
]
},
{
"name": "kubernetes",
"specs": [
[
">=",
"28.1.0"
]
]
},
{
"name": "langchain",
"specs": [
[
"==",
"0.1.6"
]
]
},
{
"name": "langchain_community",
"specs": [
[
"<",
"0.0.20"
]
]
},
{
"name": "mysql-connector-python",
"specs": [
[
"~=",
"8.2.0"
]
]
},
{
"name": "openai",
"specs": [
[
"<",
"1.0.0"
],
[
">=",
"0.27.8"
]
]
},
{
"name": "opentelemetry-api",
"specs": [
[
"==",
"1.22.0"
]
]
},
{
"name": "opentelemetry-exporter-prometheus",
"specs": [
[
"==",
"0.43b0"
]
]
},
{
"name": "opentelemetry-instrumentation-tornado",
"specs": [
[
"==",
"0.43b0"
]
]
},
{
"name": "opentelemetry-exporter-otlp",
"specs": [
[
"==",
"1.22.0"
]
]
},
{
"name": "opentelemetry-instrumentation-sqlalchemy",
"specs": [
[
"==",
"0.43b0"
]
]
},
{
"name": "oracledb",
"specs": [
[
"==",
"1.3.1"
]
]
},
{
"name": "pinotdb",
"specs": [
[
"==",
"5.1.0"
]
]
},
{
"name": "prometheus_client",
"specs": [
[
">=",
"0.18.0"
]
]
},
{
"name": "psycopg2",
"specs": [
[
"==",
"2.9.3"
]
]
},
{
"name": "psycopg2-binary",
"specs": [
[
"==",
"2.9.3"
]
]
},
{
"name": "pydruid",
"specs": [
[
"==",
"0.6.5"
]
]
},
{
"name": "pyodbc",
"specs": [
[
"==",
"4.0.35"
]
]
},
{
"name": "redshift-connector",
"specs": [
[
"==",
"2.0.915"
]
]
},
{
"name": "lxml",
"specs": [
[
"==",
"4.9.4"
]
]
},
{
"name": "snowflake-connector-python",
"specs": [
[
"==",
"3.5.0"
]
]
},
{
"name": "sshtunnel",
"specs": [
[
"==",
"0.4.0"
]
]
},
{
"name": "tables",
"specs": [
[
"==",
"3.7.0"
]
]
},
{
"name": "trino",
"specs": [
[
"~=",
"0.326"
]
]
},
{
"name": "azure-eventhub",
"specs": [
[
"==",
"5.11.2"
]
]
},
{
"name": "azure-identity",
"specs": [
[
"==",
"1.12.0"
]
]
},
{
"name": "azure-keyvault-secrets",
"specs": [
[
"==",
"4.6.0"
]
]
},
{
"name": "azure-keyvault-certificates",
"specs": [
[
"==",
"4.6.0"
]
]
},
{
"name": "azure-mgmt-containerinstance",
"specs": [
[
"==",
"10.1.0"
]
]
},
{
"name": "azure-storage-blob",
"specs": [
[
"==",
"12.14.1"
]
]
},
{
"name": "confluent-avro",
"specs": [
[
"~=",
"1.8.0"
]
]
},
{
"name": "elasticsearch",
"specs": [
[
"==",
"8.9.0"
]
]
},
{
"name": "influxdb_client",
"specs": [
[
"==",
"1.36.1"
]
]
},
{
"name": "kafka-python",
"specs": [
[
"==",
"2.0.2"
]
]
},
{
"name": "nats-py",
"specs": [
[
"==",
"2.6.0"
]
]
},
{
"name": "nkeys",
"specs": [
[
"~=",
"0.1.0"
]
]
},
{
"name": "opensearch-py",
"specs": [
[
"==",
"2.0.0"
]
]
},
{
"name": "pika",
"specs": [
[
"==",
"1.3.1"
]
]
},
{
"name": "pymongo",
"specs": [
[
"==",
"4.3.3"
]
]
},
{
"name": "requests_aws4auth",
"specs": [
[
"==",
"1.1.2"
]
]
},
{
"name": "stomp.py",
"specs": [
[
"==",
"8.1.0"
]
]
},
{
"name": "dbt-bigquery",
"specs": [
[
"==",
"1.7.2"
]
]
},
{
"name": "dbt-clickhouse",
"specs": [
[
"==",
"1.7.1"
]
]
},
{
"name": "dbt-core",
"specs": [
[
"==",
"1.7.4"
]
]
},
{
"name": "dbt-duckdb",
"specs": [
[
"==",
"1.7.0"
]
]
},
{
"name": "dbt-postgres",
"specs": [
[
"==",
"1.7.4"
]
]
},
{
"name": "dbt-redshift",
"specs": [
[
"==",
"1.7.1"
]
]
},
{
"name": "dbt-snowflake",
"specs": [
[
"==",
"1.7.1"
]
]
},
{
"name": "dbt-spark",
"specs": [
[
"==",
"1.7.1"
]
]
},
{
"name": "dbt-sqlserver",
"specs": [
[
"==",
"1.3.1"
]
]
},
{
"name": "dbt-trino",
"specs": [
[
"==",
"1.7.1"
]
]
},
{
"name": "attrs",
"specs": [
[
"==",
"17.4.0"
]
]
},
{
"name": "backoff",
"specs": []
},
{
"name": "clickhouse_sqlalchemy",
"specs": []
},
{
"name": "deltalake",
"specs": [
[
"==",
"0.7.0"
]
]
},
{
"name": "facebook_business",
"specs": [
[
"==",
"17.0.2"
]
]
},
{
"name": "gnupg",
"specs": [
[
"==",
"2.3.1"
]
]
},
{
"name": "google-analytics-data",
"specs": [
[
"==",
"0.14.2"
]
]
},
{
"name": "google-cloud-bigquery-storage",
"specs": [
[
"==",
"2.16.2"
]
]
},
{
"name": "httpx",
"specs": []
},
{
"name": "pendulum",
"specs": [
[
"==",
"2.1.0"
]
]
},
{
"name": "pymongo_schema",
"specs": [
[
"==",
"0.4.1"
]
]
},
{
"name": "python-magic",
"specs": [
[
"==",
"0.4.27"
]
]
},
{
"name": "redshift-connector",
"specs": [
[
"~=",
"2.0.915"
]
]
},
{
"name": "requests_mock",
"specs": [
[
"==",
"1.10.0"
]
]
},
{
"name": "simple_salesforce",
"specs": [
[
"~=",
"1.12.5"
]
]
},
{
"name": "singer_sdk",
"specs": [
[
"~=",
"0.34.1"
]
]
},
{
"name": "stripe",
"specs": [
[
"==",
"5.5.0"
]
]
},
{
"name": "terminaltables",
"specs": [
[
"==",
"3.1.10"
]
]
},
{
"name": "twitter-ads",
"specs": [
[
"==",
"11.0.0"
]
]
},
{
"name": "tzlocal",
"specs": [
[
"==",
"4.2"
]
]
},
{
"name": "xlsx2csv",
"specs": [
[
"==",
"0.8.1"
]
]
},
{
"name": "xmltodict",
"specs": [
[
"==",
"0.13.0"
]
]
},
{
"name": "zenpy",
"specs": [
[
"==",
"2.0.25"
]
]
}
],
"lcname": "mage-ai"
}
