| Name | astronomer-nova JSON |
| Version |
0.1.3
JSON |
| download |
| home_page | |
| Summary | Automate migration from Airflow 1 to Airflow 2 |
| upload_time | 2022-12-16 18:06:51 |
| maintainer | |
| docs_url | None |
| author | |
| requires_python | <4,>=3.9 |
| license | |
| keywords |
airflow
migration
|
| VCS |
|
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
# Astronomer Nova
### (Formerly Airflow V2 Upgrader)
This project aims to automate common tasks when upgrading DAGs from Airflow 1 to Airflow 2,
making use of [RedBaron](https://redbaron.readthedocs.io/en/latest/) to parse and manipulate the Python syntax tree.
# Features
- Automatically upgrade DAG source code from Airflow 1 to Airflow 2
- Scan Dags and report changes (Coming Soon)
# Installation
You can install the Astronomer Nova CLI through Pip
```shell
pip install astronomer-nova
```
The simplest way to get started with Astronomer Nova is by using the astro cli.
Simply add `astronomer-nova` to your list of dependencies in `requirements.txt`.
Then you can run it with the following commands:
```shell
astro dev start
astro dev bash
nova
```
## Installation from sources
To install astronomer-nova in [development mode](https://pip.pypa.io/en/latest/cli/pip_install/#install-editable):
```shell
pip install --editable .[lint,test]
```
To build from sources:
```
python -m build
```
OR
```
python -m pip wheel .
```
## Running tests:
```shell
python -m pytest
```
# How to Use
```shell
nova dags/ dags/upgraded/
```
For more information, you can access the help menu with:
```shell
nova --help
```
# How it works
Astronomer Nova performs the upgrade by:
1. Building an import map for classes in `airflow.hooks`, `airflow.operators`, `airflow.sensors`, and `airflow.providers`, which maps the classes to the current Airflow version import path.
2. Modifying Python files in a DAG directory to update
- Imports
- DAG access controls
- XCOM push
- XCOM pull
3. Writing updated DAG to either the same DAG directory or a parameter output directory with `_upgraded` added to the filename.
The script is configured to run against `./dags` and to write output to `./dags/upgraded` by default.
## Caveats
In order to load the Airflow import map, Airflow provider packages must be installed. There are often cross-dependencies within Airflow providers (eg `airflow.operators.s3_to_hive_operator` required the Hive provider), so make sure the `requirements.txt` file includes all necessary .
The script currently tries to correct imports that do not need to be changed going from Airflow 1 to 2, and logging does not include which files are currently being acted against.
The code uses Astronomer-centric opinions, like removing DAG-level RBAC. In the future, I'd rather have these decisions configurable at runtime, as well as generating per-DAG reports on the DAGs that can be used for further remediation, as well as understanding of DAG requirements, like the use of Connections and Variables.
# Contribute
- Issue Tracker: https://github.com/astronomer/astronomer-nova/issues
- Source Code: https://github.com/astronomer/astronomer-nova
# Contact
cabella@astronomer.io
Raw data
{
"_id": null,
"home_page": "",
"name": "astronomer-nova",
"maintainer": "",
"docs_url": null,
"requires_python": "<4,>=3.9",
"maintainer_email": "",
"keywords": "Airflow,Migration",
"author": "",
"author_email": "",
"download_url": "https://files.pythonhosted.org/packages/5f/f3/347fd147db739f9da1769d4f5d0e4c10e40cee1dad598a6960c5dccd2161/astronomer-nova-0.1.3.tar.gz",
"platform": null,
"description": "# Astronomer Nova\n### (Formerly Airflow V2 Upgrader)\n\nThis project aims to automate common tasks when upgrading DAGs from Airflow 1 to Airflow 2,\nmaking use of [RedBaron](https://redbaron.readthedocs.io/en/latest/) to parse and manipulate the Python syntax tree.\n\n# Features\n\n- Automatically upgrade DAG source code from Airflow 1 to Airflow 2\n- Scan Dags and report changes (Coming Soon)\n\n# Installation\nYou can install the Astronomer Nova CLI through Pip\n\n```shell\npip install astronomer-nova\n```\n\nThe simplest way to get started with Astronomer Nova is by using the astro cli.\nSimply add `astronomer-nova` to your list of dependencies in `requirements.txt`.\nThen you can run it with the following commands:\n\n```shell\nastro dev start\nastro dev bash\nnova\n```\n\n## Installation from sources\n\nTo install astronomer-nova in [development mode](https://pip.pypa.io/en/latest/cli/pip_install/#install-editable):\n\n```shell\npip install --editable .[lint,test]\n```\n\nTo build from sources:\n\n```\npython -m build\n```\n\nOR\n\n```\npython -m pip wheel .\n```\n\n## Running tests:\n\n```shell\npython -m pytest\n```\n\n# How to Use\n\n```shell\nnova dags/ dags/upgraded/\n```\n\nFor more information, you can access the help menu with:\n\n```shell\nnova --help\n```\n\n# How it works\n\nAstronomer Nova performs the upgrade by:\n\n1. Building an import map for classes in `airflow.hooks`, `airflow.operators`, `airflow.sensors`, and `airflow.providers`, which maps the classes to the current Airflow version import path.\n2. Modifying Python files in a DAG directory to update\n - Imports\n - DAG access controls\n - XCOM push\n - XCOM pull\n3. Writing updated DAG to either the same DAG directory or a parameter output directory with `_upgraded` added to the filename.\n\nThe script is configured to run against `./dags` and to write output to `./dags/upgraded` by default.\n\n## Caveats\n\nIn order to load the Airflow import map, Airflow provider packages must be installed. There are often cross-dependencies within Airflow providers (eg `airflow.operators.s3_to_hive_operator` required the Hive provider), so make sure the `requirements.txt` file includes all necessary .\n\nThe script currently tries to correct imports that do not need to be changed going from Airflow 1 to 2, and logging does not include which files are currently being acted against.\n\nThe code uses Astronomer-centric opinions, like removing DAG-level RBAC. In the future, I'd rather have these decisions configurable at runtime, as well as generating per-DAG reports on the DAGs that can be used for further remediation, as well as understanding of DAG requirements, like the use of Connections and Variables.\n\n\n# Contribute\n\n- Issue Tracker: https://github.com/astronomer/astronomer-nova/issues\n- Source Code: https://github.com/astronomer/astronomer-nova\n\n# Contact\n\ncabella@astronomer.io\n",
"bugtrack_url": null,
"license": "",
"summary": "Automate migration from Airflow 1 to Airflow 2",
"version": "0.1.3",
"split_keywords": [
"airflow",
"migration"
],
"urls": [
{
"comment_text": "",
"digests": {
"md5": "201aa0967d19922bb9da36e4ea4f9a22",
"sha256": "1cb8f2cdd7db4df3962f25c868c61a649ea3cdeaaf7351ef193a81a3f804a135"
},
"downloads": -1,
"filename": "astronomer_nova-0.1.3-py3-none-any.whl",
"has_sig": false,
"md5_digest": "201aa0967d19922bb9da36e4ea4f9a22",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": "<4,>=3.9",
"size": 7812,
"upload_time": "2022-12-16T18:06:50",
"upload_time_iso_8601": "2022-12-16T18:06:50.200121Z",
"url": "https://files.pythonhosted.org/packages/bf/3a/8f85b63abf63c917f10780bf03a02ae738d7fa1171dabadfbfd143d28f8e/astronomer_nova-0.1.3-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"md5": "ba6548c39e7391f488c08801db516e79",
"sha256": "9cccc325e4908c120e6c31be5665dcfbf428d1cf23aff42eb1c8312251bcf731"
},
"downloads": -1,
"filename": "astronomer-nova-0.1.3.tar.gz",
"has_sig": false,
"md5_digest": "ba6548c39e7391f488c08801db516e79",
"packagetype": "sdist",
"python_version": "source",
"requires_python": "<4,>=3.9",
"size": 6640,
"upload_time": "2022-12-16T18:06:51",
"upload_time_iso_8601": "2022-12-16T18:06:51.390056Z",
"url": "https://files.pythonhosted.org/packages/5f/f3/347fd147db739f9da1769d4f5d0e4c10e40cee1dad598a6960c5dccd2161/astronomer-nova-0.1.3.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2022-12-16 18:06:51",
"github": false,
"gitlab": false,
"bitbucket": false,
"lcname": "astronomer-nova"
}
