| Name | noetl JSON |
| Version |
0.1.37
JSON |
| download |
| home_page | None |
| Summary | A framework to build and run data pipelines and workflows. |
| upload_time | 2025-08-06 03:13:40 |
| maintainer | None |
| docs_url | None |
| author | None |
| requires_python | >=3.12 |
| license | None |
| keywords |
etl
data
pipeline
workflow
automation
|
| VCS |
 |
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
# Not Only ETL
__NoETL__ is an automation framework for data processing and MLOps orchestration.
[](https://badge.fury.io/py/noetl)
[](https://pypi.org/project/noetl/)
[](https://github.com/noetl/noetl/blob/main/LICENSE)
## Quick Start
### Installation
- Install NoETL from PyPI:
```bash
pip install noetl
```
For development or specific versions:
- Install in a virtual environment
```bash
python -m venv .venv
source .venv/bin/activate
pip install noetl
```
- For Windows users (in PowerShell)
```bash
python -m venv .venv
Set-ExecutionPolicy -Scope Process -ExecutionPolicy Bypass
.venv\Scripts\Activate.ps1
pip install noetl
```
- Install a specific version
```bash
pip install noetl==0.1.24
```
### Prerequisites
- Python 3.11+
- For full functionality:
- Postgres database (mandatory, for event log persistent storage and NoETL system metadata)
- Docker (optional, for containerized development and deployment)
## Basic Usage
After installing NoETL:
### 1. Run the NoETL Server
Start the NoETL server to access the web UI and REST API:
```bash
# Start the server with default settings
noetl server
# use the explicit start command with options
noetl server start --host 0.0.0.0 --port 8080 --workers 4 --debug
# Stop the server
noetl server stop
# Force stop without confirmation
noetl server stop --force
```
The server starts on http://localhost:8080 by default. You can customize the host, port, number of workers, and enable debug mode using command options.
### 2. Using the Command Line
NoETL provides a streamlined command-line interface for managing and executing playbooks:
- Register a playbook in the catalog
```bash
noetl register ./path/to/playbook.yaml
```
- List playbooks in the catalog
```bash
noetl catalog list playbook
```
- Execute a registered playbook
```bash
noetl execute my_playbook --version 0.1.0
```
- Register and execute with the catalog command
```bash
noetl catalog register ./path/to/playbook.yaml
noetl catalog execute my_playbook --version 0.1.0
```
### 3. Docker Deployment
For containerized deployment:
```bash
# Pull the latest image
docker pull noetl/noetl:latest
# Start the server
docker run -p 8080:8080 noetl/noetl:latest
# with environment variables
docker run -p 8080:8080 -e NOETL_RUN_MODE=server noetl/noetl:latest
# Stop the server
docker run -e NOETL_RUN_MODE=server-stop -e NOETL_FORCE_STOP=true noetl/noetl:latest
```
### 4. Kubernetes Deployment
For Kubernetes deployment using Kind (Kubernetes in Docker):
```bash
# Follow the instructions in k8s/README.md
# Or use the automated deployment script
./k8s/deploy-kind.sh
# To stop the server in Kubernetes, create a job:
kubectl apply -f - <<EOF
apiVersion: batch/v1
kind: Job
metadata:
name: noetl-server-stop
spec:
template:
spec:
containers:
- name: noetl-stop
image: noetl:latest
env:
- name: NOETL_RUN_MODE
value: "server-stop"
- name: NOETL_FORCE_STOP
value: "true"
restartPolicy: Never
backoffLimit: 1
EOF
```
See [Kubernetes Deployment Guide](k8s/KIND-README.md) for detailed instructions.
## Workflow DSL Structure
NoETL uses a declarative YAML-based Domain Specific Language (DSL) for defining workflows. The key components of a NoETL playbook include:
- **Metadata**: Version, path, and description of the playbook
- **Workload**: Input data and parameters for the workflow
- **Workflow**: A list of steps that make up the workflow, where each step is defined with `step: step_name`, including:
- **Steps**: Individual operations in the workflow
- **Tasks**: Actions performed at each step (HTTP requests, database operations, Python code)
- **Transitions**: Rules for moving between steps
- **Conditions**: Logic for branching the workflow
- **Workbook**: Reusable task definitions that can be called from workflow steps, including:
- **Task Types**: Python, HTTP, DuckDB, PostgreSQL, Secret.
- **Parameters**: Input parameters for the tasks
- **Code**: Implementation of the tasks
For examples of NoETL playbooks and detailed explanations, see the [Examples Guide](https://github.com/noetl/noetl/blob/master/docs/examples.md).
To run a playbook:
```bash
noetl agent -f path/to/playbooks.yaml
```
## Documentation
For more detailed information, please refer to the following documentation:
> **Note:**
> When installed from PyPI, the `docs` folder is included in your local package.
> You can find all documentation files in the `docs/` directory of your installed package.
### Getting Started
- [Installation Guide](https://github.com/noetl/noetl/blob/master/docs/installation.md) - Installation instructions
- [CLI Usage Guide](https://github.com/noetl/noetl/blob/master/docs/cli_usage.md) - Commandline interface usage
- [API Usage Guide](https://github.com/noetl/noetl/blob/master/docs/api_usage.md) - REST API usage
- [Docker Usage Guide](https://github.com/noetl/noetl/blob/master/docs/docker_usage.md) - Docker deployment
### Core Concepts
- [Playbook Structure](https://github.com/noetl/noetl/blob/master/docs/playbook_structure.md) - Structure of NoETL playbooks
- [Workflow Tasks](https://github.com/noetl/noetl/blob/master/docs/action_type.md) - Action types and parameters
- [Environment Configuration](https://github.com/noetl/noetl/blob/master/docs/environment_variables.md) - Setting up environment variables
### Examples
NoETL includes several example playbooks that demonstrate some capabilities:
- **Weather API Integration** - Fetches and processes weather data from external APIs
- **Database Operations** - Demonstrates Postgres and DuckDB integration
- **Google Cloud Storage** - A secure cloud storage operations with Google Cloud
- **Secrets Management** - Illustrates secure handling of credentials and sensitive data
- **Multi-Playbook Workflows** - Complex workflow orchestration
For detailed examples, see the [Examples Guide](https://github.com/noetl/noetl/blob/master/docs/examples.md).
## Development
For information about contributing to NoETL or building from source:
- [Development Guide](https://github.com/noetl/noetl/blob/master/docs/development.md) - Setting up a development environment
- [PyPI Publishing Guide](https://github.com/noetl/noetl/blob/master/docs/pypi_manual.md) - Building and publishing to PyPI
## Community & Support
- **GitHub Issues**: [Report bugs or request features](https://github.com/noetl/noetl/issues)
- **Documentation**: [Full documentation](https://noetl.io/docs)
- **Website**: [https://noetl.io](https://noetl.io)
## License
NoETL is released under the MIT License. See the [LICENSE](LICENSE) file for details.
Raw data
{
"_id": null,
"home_page": null,
"name": "noetl",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.12",
"maintainer_email": null,
"keywords": "etl, data, pipeline, workflow, automation",
"author": null,
"author_email": "Kadyapam <182583029+kadyapam@users.noreply.github.com>",
"download_url": "https://files.pythonhosted.org/packages/00/62/173793c45b56aad706f019124f37227eed64e657f9a68bd3d64dc59d7ee4/noetl-0.1.37.tar.gz",
"platform": null,
"description": "# Not Only ETL\n\n__NoETL__ is an automation framework for data processing and MLOps orchestration.\n\n[](https://badge.fury.io/py/noetl)\n[](https://pypi.org/project/noetl/)\n[](https://github.com/noetl/noetl/blob/main/LICENSE)\n\n## Quick Start\n\n### Installation\n\n- Install NoETL from PyPI:\n ```bash\n pip install noetl\n ```\n\nFor development or specific versions:\n- Install in a virtual environment\n ```bash\n python -m venv .venv\n source .venv/bin/activate\n pip install noetl\n ```\n- For Windows users (in PowerShell)\n ```bash\n python -m venv .venv\n Set-ExecutionPolicy -Scope Process -ExecutionPolicy Bypass\n .venv\\Scripts\\Activate.ps1\n pip install noetl\n ```\n- Install a specific version\n ```bash\n pip install noetl==0.1.24\n ```\n\n### Prerequisites\n\n- Python 3.11+\n- For full functionality:\n - Postgres database (mandatory, for event log persistent storage and NoETL system metadata)\n - Docker (optional, for containerized development and deployment)\n\n## Basic Usage\n\nAfter installing NoETL:\n\n### 1. Run the NoETL Server\n\nStart the NoETL server to access the web UI and REST API:\n\n```bash\n# Start the server with default settings\nnoetl server\n\n# use the explicit start command with options\nnoetl server start --host 0.0.0.0 --port 8080 --workers 4 --debug\n\n# Stop the server\nnoetl server stop\n\n# Force stop without confirmation\nnoetl server stop --force\n```\n\nThe server starts on http://localhost:8080 by default. You can customize the host, port, number of workers, and enable debug mode using command options.\n\n### 2. Using the Command Line\n\nNoETL provides a streamlined command-line interface for managing and executing playbooks:\n\n- Register a playbook in the catalog\n```bash\nnoetl register ./path/to/playbook.yaml\n```\n\n- List playbooks in the catalog\n```bash\nnoetl catalog list playbook\n```\n\n- Execute a registered playbook\n```bash\nnoetl execute my_playbook --version 0.1.0\n```\n\n- Register and execute with the catalog command\n```bash\nnoetl catalog register ./path/to/playbook.yaml\nnoetl catalog execute my_playbook --version 0.1.0\n```\n\n### 3. Docker Deployment\n\nFor containerized deployment:\n\n```bash\n# Pull the latest image\ndocker pull noetl/noetl:latest\n\n# Start the server\ndocker run -p 8080:8080 noetl/noetl:latest\n\n# with environment variables\ndocker run -p 8080:8080 -e NOETL_RUN_MODE=server noetl/noetl:latest\n\n# Stop the server\ndocker run -e NOETL_RUN_MODE=server-stop -e NOETL_FORCE_STOP=true noetl/noetl:latest\n```\n\n### 4. Kubernetes Deployment\n\nFor Kubernetes deployment using Kind (Kubernetes in Docker):\n\n```bash\n# Follow the instructions in k8s/README.md\n# Or use the automated deployment script\n./k8s/deploy-kind.sh\n\n# To stop the server in Kubernetes, create a job:\nkubectl apply -f - <<EOF\napiVersion: batch/v1\nkind: Job\nmetadata:\n name: noetl-server-stop\nspec:\n template:\n spec:\n containers:\n - name: noetl-stop\n image: noetl:latest\n env:\n - name: NOETL_RUN_MODE\n value: \"server-stop\"\n - name: NOETL_FORCE_STOP\n value: \"true\"\n restartPolicy: Never\n backoffLimit: 1\nEOF\n```\n\nSee [Kubernetes Deployment Guide](k8s/KIND-README.md) for detailed instructions.\n\n## Workflow DSL Structure\n\nNoETL uses a declarative YAML-based Domain Specific Language (DSL) for defining workflows. The key components of a NoETL playbook include:\n\n- **Metadata**: Version, path, and description of the playbook\n- **Workload**: Input data and parameters for the workflow\n- **Workflow**: A list of steps that make up the workflow, where each step is defined with `step: step_name`, including:\n - **Steps**: Individual operations in the workflow\n - **Tasks**: Actions performed at each step (HTTP requests, database operations, Python code)\n - **Transitions**: Rules for moving between steps\n - **Conditions**: Logic for branching the workflow\n- **Workbook**: Reusable task definitions that can be called from workflow steps, including:\n - **Task Types**: Python, HTTP, DuckDB, PostgreSQL, Secret.\n - **Parameters**: Input parameters for the tasks\n - **Code**: Implementation of the tasks\n\nFor examples of NoETL playbooks and detailed explanations, see the [Examples Guide](https://github.com/noetl/noetl/blob/master/docs/examples.md).\n\nTo run a playbook:\n\n```bash\nnoetl agent -f path/to/playbooks.yaml\n```\n\n## Documentation\n\nFor more detailed information, please refer to the following documentation:\n\n> **Note:** \n> When installed from PyPI, the `docs` folder is included in your local package. \n> You can find all documentation files in the `docs/` directory of your installed package.\n\n### Getting Started\n- [Installation Guide](https://github.com/noetl/noetl/blob/master/docs/installation.md) - Installation instructions\n- [CLI Usage Guide](https://github.com/noetl/noetl/blob/master/docs/cli_usage.md) - Commandline interface usage\n- [API Usage Guide](https://github.com/noetl/noetl/blob/master/docs/api_usage.md) - REST API usage\n- [Docker Usage Guide](https://github.com/noetl/noetl/blob/master/docs/docker_usage.md) - Docker deployment\n\n### Core Concepts\n- [Playbook Structure](https://github.com/noetl/noetl/blob/master/docs/playbook_structure.md) - Structure of NoETL playbooks\n- [Workflow Tasks](https://github.com/noetl/noetl/blob/master/docs/action_type.md) - Action types and parameters\n- [Environment Configuration](https://github.com/noetl/noetl/blob/master/docs/environment_variables.md) - Setting up environment variables\n\n\n### Examples\n\nNoETL includes several example playbooks that demonstrate some capabilities:\n\n- **Weather API Integration** - Fetches and processes weather data from external APIs\n- **Database Operations** - Demonstrates Postgres and DuckDB integration\n- **Google Cloud Storage** - A secure cloud storage operations with Google Cloud\n- **Secrets Management** - Illustrates secure handling of credentials and sensitive data\n- **Multi-Playbook Workflows** - Complex workflow orchestration\n\nFor detailed examples, see the [Examples Guide](https://github.com/noetl/noetl/blob/master/docs/examples.md).\n\n## Development\n\nFor information about contributing to NoETL or building from source:\n\n- [Development Guide](https://github.com/noetl/noetl/blob/master/docs/development.md) - Setting up a development environment\n- [PyPI Publishing Guide](https://github.com/noetl/noetl/blob/master/docs/pypi_manual.md) - Building and publishing to PyPI\n\n## Community & Support\n\n- **GitHub Issues**: [Report bugs or request features](https://github.com/noetl/noetl/issues)\n- **Documentation**: [Full documentation](https://noetl.io/docs)\n- **Website**: [https://noetl.io](https://noetl.io)\n\n## License\n\nNoETL is released under the MIT License. See the [LICENSE](LICENSE) file for details.\n",
"bugtrack_url": null,
"license": null,
"summary": "A framework to build and run data pipelines and workflows.",
"version": "0.1.37",
"project_urls": {
"Homepage": "https://noetl.io",
"Issues": "https://github.com/noetl/noetl/issues",
"Repository": "https://github.com/noetl/noetl"
},
"split_keywords": [
"etl",
" data",
" pipeline",
" workflow",
" automation"
],
"urls": [
{
"comment_text": null,
"digests": {
"blake2b_256": "13e17f1295ead6380d2c2141c662cee00ef0d2a4f42467519cbebcc2a9489367",
"md5": "75211bdd8a8e21ad233d346fa192dcb6",
"sha256": "80cca7bd41d330e65a36d6acfddef826080edc0ce6dc40b3ccda1ca233fbbdd9"
},
"downloads": -1,
"filename": "noetl-0.1.37-py3-none-any.whl",
"has_sig": false,
"md5_digest": "75211bdd8a8e21ad233d346fa192dcb6",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.12",
"size": 1812079,
"upload_time": "2025-08-06T03:13:36",
"upload_time_iso_8601": "2025-08-06T03:13:36.913286Z",
"url": "https://files.pythonhosted.org/packages/13/e1/7f1295ead6380d2c2141c662cee00ef0d2a4f42467519cbebcc2a9489367/noetl-0.1.37-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "0062173793c45b56aad706f019124f37227eed64e657f9a68bd3d64dc59d7ee4",
"md5": "e04e79bcee8c990bee1c0dae08463d13",
"sha256": "f8442c3bff02dbda1c6ebc2a2055e487b38b48879c7065fe5901e7cf74449c23"
},
"downloads": -1,
"filename": "noetl-0.1.37.tar.gz",
"has_sig": false,
"md5_digest": "e04e79bcee8c990bee1c0dae08463d13",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.12",
"size": 2918921,
"upload_time": "2025-08-06T03:13:40",
"upload_time_iso_8601": "2025-08-06T03:13:40.036804Z",
"url": "https://files.pythonhosted.org/packages/00/62/173793c45b56aad706f019124f37227eed64e657f9a68bd3d64dc59d7ee4/noetl-0.1.37.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-08-06 03:13:40",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "noetl",
"github_project": "noetl",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "noetl"
}
