dagsonar


Namedagsonar JSON
Version 0.0.3 PyPI version JSON
download
home_pageNone
SummaryDeep visibility into your Airflow task changes
upload_time2025-02-20 10:20:53
maintainerNone
docs_urlNone
authorNone
requires_python>=3.8
licenseNone
keywords airflow changes dag monitoring task tracking
VCS
bugtrack_url
requirements No requirements were recorded.
Travis-CI No Travis.
coveralls test coverage No coveralls.
            # DagSonar

Deep visibility into your Airflow task changes through AST parsing and tracking

[![Python](https://img.shields.io/badge/python-3.7+-blue.svg)](https://www.python.org/downloads/)
[![Apache Airflow](https://img.shields.io/badge/apache--airflow-2.0+-yellow.svg)](https://airflow.apache.org/)
[![License](https://img.shields.io/badge/license-Apache%202.0-green.svg)](LICENSE)

## What is DagSonar?

DagSonar is a monitoring tool that provides deep visibility into your Airflow DAG tasks by tracking changes through AST (Abstract Syntax Tree) parsing. It detects modifications in task definitions, external variables, shell scripts, and function calls, ensuring you never miss critical changes to your DAGs.

## Key Features

- **AST-Based Detection**: Tracks changes by parsing the Abstract Syntax Tree of your DAG files
- **Task Reference Tracking**: Monitors task definitions, external variables, and function calls
- **Shell Script Integration**: Tracks associated shell scripts referenced in BashOperator tasks
- **Change History**: Maintains a JSON-based history of all task modifications
- **Task Hash Generation**: Generates unique hashes for each task state to detect changes
- **Support for Multiple DAGs**: Track tasks across multiple DAG configurations

## Installation

```bash
pip install dagsonar
```

## Basic Usage

```python
from pathlib import Path
from dagsonar import TaskTracker, DagConfig

# Initialize the tracker
tracker = TaskTracker(history_file=Path("task_history.json"))

# Configure DAGs to track
dag_configs = {
    "example_dag": DagConfig(
        path=Path("/path/to/dag.py"),
        tasks=["task1", "task2"]  # Optional: specify tasks to track
    )
}

# Track tasks and get references
references = tracker.track_tasks(dag_configs)

# Check for changes
changes, references = tracker.check_for_changes(references)

# Save the new state
tracker.save_history(references)
```

## Features in Detail

### Task Reference Tracking

DagSonar tracks several aspects of your tasks:
- Task content and structure through AST
- External variable references
- Called functions
- Shell scripts referenced in bash tasks
- Task-specific hashes for change detection

### Supported Task Types

Currently supports tracking of:
- Function-based task definitions
- BashOperator task instances
- Referenced shell scripts
- External variable dependencies

## Configuration

### DagConfig
```python
from dagsonar import DagConfig
from pathlib import Path

config = DagConfig(
    path=Path("/path/to/dag.py"),  # Path to DAG file
    tasks=["task1", "task2"]       # Optional: List of specific tasks to track
)
```

### Task History

Task history is stored in JSON format with the following structure:
```json
[
  {
    "dag_id": "example_dag",
    "reference": {
      "dag_id": "example_dag",
      "task_history": [
        {
          "task_id": "task1",
          "content": "<ast_content>",
          "hash": "<computed_hash>",
          "external_variables": [],
          "called_functions": [],
          "shell_scripts": []
        }
      ]
    }
  }
]
```

## Contributing

We welcome contributions! Please check out our [Contributing Guide](CONTRIBUTING.md) to get started.

### Development Setup

1. Clone the repository:
```bash
git clone https://github.com/pesnik/dagsonar.git
cd dagsonar
```

2. Create a virtual environment:
```bash
python -m venv venv
source venv/bin/activate  # Linux/Mac
# or
.\venv\Scripts\activate  # Windows
```

3. Install development dependencies:
```bash
pip install -e ".[dev]"
```

### Running Tests
```bash
pytest tests/
```

## License

This project is licensed under the Apache License 2.0 - see the [LICENSE](LICENSE) file for details.

## Acknowledgments

- Apache Airflow community
- All contributors and users providing valuable feedback

---
Built for the Airflow community

            

Raw data

            {
    "_id": null,
    "home_page": null,
    "name": "dagsonar",
    "maintainer": null,
    "docs_url": null,
    "requires_python": ">=3.8",
    "maintainer_email": null,
    "keywords": "airflow, changes, dag, monitoring, task, tracking",
    "author": null,
    "author_email": "\"Md. Rakibul Hasan\" <hasanrakibul.masum@gmail.com>",
    "download_url": "https://files.pythonhosted.org/packages/06/f4/3de279eabdc9a256e9d440e28414ae5a9c91661af75e0cc936ad45ecf3e1/dagsonar-0.0.3.tar.gz",
    "platform": null,
    "description": "# DagSonar\n\nDeep visibility into your Airflow task changes through AST parsing and tracking\n\n[![Python](https://img.shields.io/badge/python-3.7+-blue.svg)](https://www.python.org/downloads/)\n[![Apache Airflow](https://img.shields.io/badge/apache--airflow-2.0+-yellow.svg)](https://airflow.apache.org/)\n[![License](https://img.shields.io/badge/license-Apache%202.0-green.svg)](LICENSE)\n\n## What is DagSonar?\n\nDagSonar is a monitoring tool that provides deep visibility into your Airflow DAG tasks by tracking changes through AST (Abstract Syntax Tree) parsing. It detects modifications in task definitions, external variables, shell scripts, and function calls, ensuring you never miss critical changes to your DAGs.\n\n## Key Features\n\n- **AST-Based Detection**: Tracks changes by parsing the Abstract Syntax Tree of your DAG files\n- **Task Reference Tracking**: Monitors task definitions, external variables, and function calls\n- **Shell Script Integration**: Tracks associated shell scripts referenced in BashOperator tasks\n- **Change History**: Maintains a JSON-based history of all task modifications\n- **Task Hash Generation**: Generates unique hashes for each task state to detect changes\n- **Support for Multiple DAGs**: Track tasks across multiple DAG configurations\n\n## Installation\n\n```bash\npip install dagsonar\n```\n\n## Basic Usage\n\n```python\nfrom pathlib import Path\nfrom dagsonar import TaskTracker, DagConfig\n\n# Initialize the tracker\ntracker = TaskTracker(history_file=Path(\"task_history.json\"))\n\n# Configure DAGs to track\ndag_configs = {\n    \"example_dag\": DagConfig(\n        path=Path(\"/path/to/dag.py\"),\n        tasks=[\"task1\", \"task2\"]  # Optional: specify tasks to track\n    )\n}\n\n# Track tasks and get references\nreferences = tracker.track_tasks(dag_configs)\n\n# Check for changes\nchanges, references = tracker.check_for_changes(references)\n\n# Save the new state\ntracker.save_history(references)\n```\n\n## Features in Detail\n\n### Task Reference Tracking\n\nDagSonar tracks several aspects of your tasks:\n- Task content and structure through AST\n- External variable references\n- Called functions\n- Shell scripts referenced in bash tasks\n- Task-specific hashes for change detection\n\n### Supported Task Types\n\nCurrently supports tracking of:\n- Function-based task definitions\n- BashOperator task instances\n- Referenced shell scripts\n- External variable dependencies\n\n## Configuration\n\n### DagConfig\n```python\nfrom dagsonar import DagConfig\nfrom pathlib import Path\n\nconfig = DagConfig(\n    path=Path(\"/path/to/dag.py\"),  # Path to DAG file\n    tasks=[\"task1\", \"task2\"]       # Optional: List of specific tasks to track\n)\n```\n\n### Task History\n\nTask history is stored in JSON format with the following structure:\n```json\n[\n  {\n    \"dag_id\": \"example_dag\",\n    \"reference\": {\n      \"dag_id\": \"example_dag\",\n      \"task_history\": [\n        {\n          \"task_id\": \"task1\",\n          \"content\": \"<ast_content>\",\n          \"hash\": \"<computed_hash>\",\n          \"external_variables\": [],\n          \"called_functions\": [],\n          \"shell_scripts\": []\n        }\n      ]\n    }\n  }\n]\n```\n\n## Contributing\n\nWe welcome contributions! Please check out our [Contributing Guide](CONTRIBUTING.md) to get started.\n\n### Development Setup\n\n1. Clone the repository:\n```bash\ngit clone https://github.com/pesnik/dagsonar.git\ncd dagsonar\n```\n\n2. Create a virtual environment:\n```bash\npython -m venv venv\nsource venv/bin/activate  # Linux/Mac\n# or\n.\\venv\\Scripts\\activate  # Windows\n```\n\n3. Install development dependencies:\n```bash\npip install -e \".[dev]\"\n```\n\n### Running Tests\n```bash\npytest tests/\n```\n\n## License\n\nThis project is licensed under the Apache License 2.0 - see the [LICENSE](LICENSE) file for details.\n\n## Acknowledgments\n\n- Apache Airflow community\n- All contributors and users providing valuable feedback\n\n---\nBuilt for the Airflow community\n",
    "bugtrack_url": null,
    "license": null,
    "summary": "Deep visibility into your Airflow task changes",
    "version": "0.0.3",
    "project_urls": {
        "Documentation": "https://github.com/pesnik/dagsonar#readme",
        "Issues": "https://github.com/pesnik/dagsonar/issues",
        "Source": "https://github.com/pesnik/dagsonar"
    },
    "split_keywords": [
        "airflow",
        " changes",
        " dag",
        " monitoring",
        " task",
        " tracking"
    ],
    "urls": [
        {
            "comment_text": null,
            "digests": {
                "blake2b_256": "fdcdc36910151ee0eb1000e273dd0a266488bc3485708c5e583e1ef7da7bfd70",
                "md5": "d00984b48d646394a06fa575891309b4",
                "sha256": "06c495b19da0d48f8ba9d6dad01acce84eb8996c970a0bae5a3cbd66dc36fb7a"
            },
            "downloads": -1,
            "filename": "dagsonar-0.0.3-py3-none-any.whl",
            "has_sig": false,
            "md5_digest": "d00984b48d646394a06fa575891309b4",
            "packagetype": "bdist_wheel",
            "python_version": "py3",
            "requires_python": ">=3.8",
            "size": 9379,
            "upload_time": "2025-02-20T10:20:51",
            "upload_time_iso_8601": "2025-02-20T10:20:51.788796Z",
            "url": "https://files.pythonhosted.org/packages/fd/cd/c36910151ee0eb1000e273dd0a266488bc3485708c5e583e1ef7da7bfd70/dagsonar-0.0.3-py3-none-any.whl",
            "yanked": false,
            "yanked_reason": null
        },
        {
            "comment_text": null,
            "digests": {
                "blake2b_256": "06f43de279eabdc9a256e9d440e28414ae5a9c91661af75e0cc936ad45ecf3e1",
                "md5": "29a7d1ea55abbd2e8242697bcf77efe0",
                "sha256": "83b4b67e7db01556d6974a2991afba0e455a06bd290933362bc00eb43957244f"
            },
            "downloads": -1,
            "filename": "dagsonar-0.0.3.tar.gz",
            "has_sig": false,
            "md5_digest": "29a7d1ea55abbd2e8242697bcf77efe0",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": ">=3.8",
            "size": 12315,
            "upload_time": "2025-02-20T10:20:53",
            "upload_time_iso_8601": "2025-02-20T10:20:53.294765Z",
            "url": "https://files.pythonhosted.org/packages/06/f4/3de279eabdc9a256e9d440e28414ae5a9c91661af75e0cc936ad45ecf3e1/dagsonar-0.0.3.tar.gz",
            "yanked": false,
            "yanked_reason": null
        }
    ],
    "upload_time": "2025-02-20 10:20:53",
    "github": true,
    "gitlab": false,
    "bitbucket": false,
    "codeberg": false,
    "github_user": "pesnik",
    "github_project": "dagsonar#readme",
    "travis_ci": false,
    "coveralls": false,
    "github_actions": false,
    "lcname": "dagsonar"
}
        
Elapsed time: 0.73755s