| Name | noetl JSON |
| Version |
0.1.24
JSON |
| download |
| home_page | None |
| Summary | A framework to build and run data pipelines and workflows. |
| upload_time | 2025-07-14 06:05:35 |
| maintainer | None |
| docs_url | None |
| author | None |
| requires_python | >=3.12 |
| license | None |
| keywords |
etl
data
pipeline
workflow
automation
|
| VCS |
 |
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
# Not Only ETL
__NoETL__ is an automation framework for data processing and MLOps orchestration.
[](https://badge.fury.io/py/noetl)
[](https://pypi.org/project/noetl/)
[](https://github.com/noetl/noetl/blob/main/LICENSE)
## Quick Start
### Installation
- Install NoETL from PyPI:
```bash
pip install noetl
```
For development or specific versions:
- Install in a virtual environment
```bash
python -m venv .venv
source .venv/bin/activate
pip install noetl
```
- For Windows users (in PowerShell)
```bash
python -m venv .venv
Set-ExecutionPolicy -Scope Process -ExecutionPolicy Bypass
.venv\Scripts\Activate.ps1
pip install noetl
```
- Install a specific version
```bash
pip install noetl==0.1.24
```
### Prerequisites
- Python 3.11+
- For full functionality:
- Postgres database (mandatory, for event log persistent storage and NoETL system metadata)
- Docker (optional, for containerized development and deployment)
## Basic Usage
After installing NoETL:
### 1. Run the NoETL Server
Start the NoETL server to access the web UI and REST API:
```bash
noetl server
```
This starts the server on http://localhost:8082 by default.
### 2. Using the Command Line
NoETL has a command-line interface for executing playbooks:
- Register a playbook in the catalog
```bash
noetl playbook --register ./<path to playbook folder>/playbook.yaml
```
- Execute a playbook from the catalog
```bash
noetl playbook --execute --path "workflows/example/playbook"
```
- Execute a playbook directly
```bash
noetl agent -f ./<path to playbook folder>/playbook.yaml
```
### 3. Docker Deployment
For containerized deployment:
```bash
docker pull noetl/noetl:latest
docker run -p 8082:8082 noetl/noetl:latest
```
## Workflow DSL Structure
NoETL uses a declarative YAML-based Domain Specific Language (DSL) for defining workflows. The key components of a NoETL playbook include:
- **Metadata**: Version, path, and description of the playbook
- **Workload**: Input data and parameters for the workflow
- **Workflow**: A list of steps that make up the workflow, where each step is defined with `step: step_name`, including:
- **Steps**: Individual operations in the workflow
- **Tasks**: Actions performed at each step (HTTP requests, database operations, Python code)
- **Transitions**: Rules for moving between steps
- **Conditions**: Logic for branching the workflow
- **Workbook**: Reusable task definitions that can be called from workflow steps, including:
- **Task Types**: Python, HTTP, DuckDB, PostgreSQL, Secret.
- **Parameters**: Input parameters for the tasks
- **Code**: Implementation of the tasks
For examples of NoETL playbooks and detailed explanations, see the [Examples Guide](https://github.com/noetl/noetl/blob/master/docs/examples.md).
To run a playbook:
```bash
noetl agent -f path/to/playbook.yaml
```
## Documentation
For more detailed information, please refer to the following documentation:
> **Note:**
> When installed from PyPI, the `docs` folder is included in your local package.
> You can find all documentation files in the `docs/` directory of your installed package.
### Getting Started
- [Installation Guide](https://github.com/noetl/noetl/blob/master/docs/installation.md) - Installation instructions
- [CLI Usage Guide](https://github.com/noetl/noetl/blob/master/docs/cli_usage.md) - Commandline interface usage
- [API Usage Guide](https://github.com/noetl/noetl/blob/master/docs/api_usage.md) - REST API usage
- [Docker Usage Guide](https://github.com/noetl/noetl/blob/master/docs/docker_usage.md) - Docker deployment
### Core Concepts
- [Playbook Structure](https://github.com/noetl/noetl/blob/master/docs/playbook_structure.md) - Structure of NoETL playbooks
- [Workflow Tasks](https://github.com/noetl/noetl/blob/master/docs/action_type.md) - Action types and parameters
- [Environment Configuration](https://github.com/noetl/noetl/blob/master/docs/environment_variables.md) - Setting up environment variables
### Advanced Examples
NoETL includes several example playbooks that demonstrate more advanced capabilities:
- **Weather API Integration** - Fetches and processes weather data from external APIs
- **Database Operations** - Demonstrates Postgres and DuckDB integration
- **Google Cloud Storage** - Shows secure cloud storage operations with Google Cloud
- **Secrets Management** - Illustrates secure handling of credentials and sensitive data
- **Multi-Playbook Workflows** - Demonstrates complex workflow orchestration
For detailed examples, see the [Examples Guide](https://github.com/noetl/noetl/blob/master/docs/examples.md).
## Development
For information about contributing to NoETL or building from source:
- [Development Guide](https://github.com/noetl/noetl/blob/master/docs/development.md) - Setting up a development environment
- [PyPI Publishing Guide](https://github.com/noetl/noetl/blob/master/docs/pypi_manual.md) - Building and publishing to PyPI
## Community & Support
- **GitHub Issues**: [Report bugs or request features](https://github.com/noetl/noetl/issues)
- **Documentation**: [Full documentation](https://noetl.io/docs)
- **Website**: [https://noetl.io](https://noetl.io)
## License
NoETL is released under the MIT License. See the [LICENSE](LICENSE) file for details.
Raw data
{
"_id": null,
"home_page": null,
"name": "noetl",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.12",
"maintainer_email": null,
"keywords": "etl, data, pipeline, workflow, automation",
"author": null,
"author_email": "Kadyapam <182583029+kadyapam@users.noreply.github.com>",
"download_url": "https://files.pythonhosted.org/packages/74/88/d7d42a3be982179c119369f310e5e50265c36bc6e9c66dab43e4461fdae3/noetl-0.1.24.tar.gz",
"platform": null,
"description": "# Not Only ETL\n\n__NoETL__ is an automation framework for data processing and MLOps orchestration.\n\n[](https://badge.fury.io/py/noetl)\n[](https://pypi.org/project/noetl/)\n[](https://github.com/noetl/noetl/blob/main/LICENSE)\n\n## Quick Start\n\n### Installation\n\n- Install NoETL from PyPI:\n ```bash\n pip install noetl\n ```\n\nFor development or specific versions:\n- Install in a virtual environment\n ```bash\n python -m venv .venv\n source .venv/bin/activate\n pip install noetl\n ```\n- For Windows users (in PowerShell)\n ```bash\n python -m venv .venv\n Set-ExecutionPolicy -Scope Process -ExecutionPolicy Bypass\n .venv\\Scripts\\Activate.ps1\n pip install noetl\n ```\n- Install a specific version\n ```bash\n pip install noetl==0.1.24\n ```\n\n### Prerequisites\n\n- Python 3.11+\n- For full functionality:\n - Postgres database (mandatory, for event log persistent storage and NoETL system metadata)\n - Docker (optional, for containerized development and deployment)\n\n## Basic Usage\n\nAfter installing NoETL:\n\n### 1. Run the NoETL Server\n\nStart the NoETL server to access the web UI and REST API:\n\n```bash\nnoetl server\n```\n\nThis starts the server on http://localhost:8082 by default.\n\n### 2. Using the Command Line\n\nNoETL has a command-line interface for executing playbooks:\n\n- Register a playbook in the catalog\n```bash\nnoetl playbook --register ./<path to playbook folder>/playbook.yaml\n```\n- Execute a playbook from the catalog\n```bash\nnoetl playbook --execute --path \"workflows/example/playbook\"\n```\n- Execute a playbook directly\n```bash\nnoetl agent -f ./<path to playbook folder>/playbook.yaml\n```\n\n### 3. Docker Deployment\n\nFor containerized deployment:\n\n```bash\ndocker pull noetl/noetl:latest\ndocker run -p 8082:8082 noetl/noetl:latest\n```\n\n## Workflow DSL Structure\n\nNoETL uses a declarative YAML-based Domain Specific Language (DSL) for defining workflows. The key components of a NoETL playbook include:\n\n- **Metadata**: Version, path, and description of the playbook\n- **Workload**: Input data and parameters for the workflow\n- **Workflow**: A list of steps that make up the workflow, where each step is defined with `step: step_name`, including:\n - **Steps**: Individual operations in the workflow\n - **Tasks**: Actions performed at each step (HTTP requests, database operations, Python code)\n - **Transitions**: Rules for moving between steps\n - **Conditions**: Logic for branching the workflow\n- **Workbook**: Reusable task definitions that can be called from workflow steps, including:\n - **Task Types**: Python, HTTP, DuckDB, PostgreSQL, Secret.\n - **Parameters**: Input parameters for the tasks\n - **Code**: Implementation of the tasks\n\nFor examples of NoETL playbooks and detailed explanations, see the [Examples Guide](https://github.com/noetl/noetl/blob/master/docs/examples.md).\n\nTo run a playbook:\n\n```bash\nnoetl agent -f path/to/playbook.yaml\n```\n\n## Documentation\n\nFor more detailed information, please refer to the following documentation:\n\n> **Note:** \n> When installed from PyPI, the `docs` folder is included in your local package. \n> You can find all documentation files in the `docs/` directory of your installed package.\n\n### Getting Started\n- [Installation Guide](https://github.com/noetl/noetl/blob/master/docs/installation.md) - Installation instructions\n- [CLI Usage Guide](https://github.com/noetl/noetl/blob/master/docs/cli_usage.md) - Commandline interface usage\n- [API Usage Guide](https://github.com/noetl/noetl/blob/master/docs/api_usage.md) - REST API usage\n- [Docker Usage Guide](https://github.com/noetl/noetl/blob/master/docs/docker_usage.md) - Docker deployment\n\n### Core Concepts\n- [Playbook Structure](https://github.com/noetl/noetl/blob/master/docs/playbook_structure.md) - Structure of NoETL playbooks\n- [Workflow Tasks](https://github.com/noetl/noetl/blob/master/docs/action_type.md) - Action types and parameters\n- [Environment Configuration](https://github.com/noetl/noetl/blob/master/docs/environment_variables.md) - Setting up environment variables\n\n\n### Advanced Examples\n\nNoETL includes several example playbooks that demonstrate more advanced capabilities:\n\n- **Weather API Integration** - Fetches and processes weather data from external APIs\n- **Database Operations** - Demonstrates Postgres and DuckDB integration\n- **Google Cloud Storage** - Shows secure cloud storage operations with Google Cloud\n- **Secrets Management** - Illustrates secure handling of credentials and sensitive data\n- **Multi-Playbook Workflows** - Demonstrates complex workflow orchestration\n\nFor detailed examples, see the [Examples Guide](https://github.com/noetl/noetl/blob/master/docs/examples.md).\n\n## Development\n\nFor information about contributing to NoETL or building from source:\n\n- [Development Guide](https://github.com/noetl/noetl/blob/master/docs/development.md) - Setting up a development environment\n- [PyPI Publishing Guide](https://github.com/noetl/noetl/blob/master/docs/pypi_manual.md) - Building and publishing to PyPI\n\n## Community & Support\n\n- **GitHub Issues**: [Report bugs or request features](https://github.com/noetl/noetl/issues)\n- **Documentation**: [Full documentation](https://noetl.io/docs)\n- **Website**: [https://noetl.io](https://noetl.io)\n\n## License\n\nNoETL is released under the MIT License. See the [LICENSE](LICENSE) file for details.\n",
"bugtrack_url": null,
"license": null,
"summary": "A framework to build and run data pipelines and workflows.",
"version": "0.1.24",
"project_urls": {
"Homepage": "https://noetl.io",
"Issues": "https://github.com/noetl/noetl/issues",
"Repository": "https://github.com/noetl/noetl"
},
"split_keywords": [
"etl",
" data",
" pipeline",
" workflow",
" automation"
],
"urls": [
{
"comment_text": null,
"digests": {
"blake2b_256": "52fd5b138d74e7c249857a71ebf79b904e5f893eed17402f2477f43b058b01e4",
"md5": "5bbf3ac03384022c33cd6a75ff061f04",
"sha256": "984d0946118f952546bc44987b43d909abb7594beb8c1ebdeb2ff9d20bdbba2b"
},
"downloads": -1,
"filename": "noetl-0.1.24-py3-none-any.whl",
"has_sig": false,
"md5_digest": "5bbf3ac03384022c33cd6a75ff061f04",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.12",
"size": 279498,
"upload_time": "2025-07-14T06:05:33",
"upload_time_iso_8601": "2025-07-14T06:05:33.530546Z",
"url": "https://files.pythonhosted.org/packages/52/fd/5b138d74e7c249857a71ebf79b904e5f893eed17402f2477f43b058b01e4/noetl-0.1.24-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "7488d7d42a3be982179c119369f310e5e50265c36bc6e9c66dab43e4461fdae3",
"md5": "f12db1989ebe6b1d13a097a7bf496457",
"sha256": "2cc6dddf7bd12621c9abff663f860ebd96d5d3824e9742a296d2df01807260ce"
},
"downloads": -1,
"filename": "noetl-0.1.24.tar.gz",
"has_sig": false,
"md5_digest": "f12db1989ebe6b1d13a097a7bf496457",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.12",
"size": 334337,
"upload_time": "2025-07-14T06:05:35",
"upload_time_iso_8601": "2025-07-14T06:05:35.044507Z",
"url": "https://files.pythonhosted.org/packages/74/88/d7d42a3be982179c119369f310e5e50265c36bc6e9c66dab43e4461fdae3/noetl-0.1.24.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-07-14 06:05:35",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "noetl",
"github_project": "noetl",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "noetl"
}
