flowtask


Nameflowtask JSON
Version 5.8.18 PyPI version JSON
download
home_pagehttps://github.com/phenobarbital/flowtask
SummaryFramework for running Tasks and from CLI and API for orchestation. Component-based Task builder/Runner for non-programmers.
upload_time2025-07-16 15:36:26
maintainerNone
docs_urlNone
authorJesus Lara
requires_python>=3.9.16
licenseApache-2.0
keywords dataintegration task orchestation task-runner pipelines data-pipelines
VCS
bugtrack_url
requirements No requirements were recorded.
Travis-CI No Travis.
coveralls test coverage No coveralls.
            # FlowTask DataIntegration #

FlowTask DataIntegration is a plugin-based, component-driven task execution framework for create complex Tasks.

FlowTask runs Tasks defined in JSON, YAML or TOML files, any Task is a combination of Components,
and every component in the Task run sequentially or depend of others, like a DAG.

Can create a Task combining Commands, Shell scripts and other specific Components (as TableInput: Open a Table using a datasource, DownloadFromIMAP: Download a File from a IMAP Folder, and so on), any Python Callable can be a Component inside a Task, or can extends UserComponent to build your own componets.

Every designed Task can run from CLI, programmatically, via RESTful API (using our aioHTTP-based Handler), called by WebHooks or even dispatched to a external Worker using our built-in Scheduler.

## Quickstart ##

```console
pip install flowtask
```

Tasks can organizated into directory structure like this:

tasks /
    ├── programs /
      ├── test /
           ├── tasks /

The main reason of this structure, is maintain organized several tasks by tenant/program, avoiding filling a directory with several task files.

FlowTask support "TaskStorage", a Task Storage is the main repository for tasks, main Task Storage is a directory in any filesystem path (optionally you can syncronize that path using git), but Tasks can be saved onto a Database or a S3 bucket.

## Dependencies ##

 * aiohttp (Asyncio Web Framework and Server) (required by navigator)
 * AsyncDB
 * QuerySource
 * Navigator-api
 * (Optional) Qworker (for distributing asyncio Tasks on distributed workers).

## Features ##

* Component-based Task execution framework with several components covering several actions (download files, create pandas dataframes from files, mapping dataframe columns to a json-dictionary, etc)
* Built-in API for execution of Tasks.

### How I run a Task? ###

Can run a Task from CLI:
```console
task --program=test --task=example
```

on CLI, you can pass an ENV (enviroment) to change the environment file on task execution.
```console
ENV=dev task --program=test --task=example
```

or Programmatically:
```python
from flowtask import Task
import asyncio

task = Task(program='test', task='example')
results = asyncio.run(task.run())
# we can alternatively, using the execution mode of task object:
results = asyncio.run(task())
```

### Requirements ###

* Python >= 3.9
* asyncio (https://pypi.python.org/pypi/asyncio/)
* aiohttp >= 3.6.2

### Contribution guidelines ###

Please have a look at the Contribution Guide

* Writing tests
* Code review
* Other guidelines

### Who do I talk to? ###

* Repo owner or admin
* Other community or team contact

### License ###

Navigator is licensed under Apache 2.0 License. See the LICENSE file for more details.

            

Raw data

            {
    "_id": null,
    "home_page": "https://github.com/phenobarbital/flowtask",
    "name": "flowtask",
    "maintainer": null,
    "docs_url": null,
    "requires_python": ">=3.9.16",
    "maintainer_email": null,
    "keywords": "DataIntegration, Task, Orchestation, Task-Runner, Pipelines, Data-Pipelines",
    "author": "Jesus Lara",
    "author_email": "\"Jesus Lara G.\" <jesuslarag@gmail.com>",
    "download_url": null,
    "platform": "*nix",
    "description": "# FlowTask DataIntegration #\n\nFlowTask DataIntegration is a plugin-based, component-driven task execution framework for create complex Tasks.\n\nFlowTask runs Tasks defined in JSON, YAML or TOML files, any Task is a combination of Components,\nand every component in the Task run sequentially or depend of others, like a DAG.\n\nCan create a Task combining Commands, Shell scripts and other specific Components (as TableInput: Open a Table using a datasource, DownloadFromIMAP: Download a File from a IMAP Folder, and so on), any Python Callable can be a Component inside a Task, or can extends UserComponent to build your own componets.\n\nEvery designed Task can run from CLI, programmatically, via RESTful API (using our aioHTTP-based Handler), called by WebHooks or even dispatched to a external Worker using our built-in Scheduler.\n\n## Quickstart ##\n\n```console\npip install flowtask\n```\n\nTasks can organizated into directory structure like this:\n\ntasks /\n    \u251c\u2500\u2500 programs /\n      \u251c\u2500\u2500 test /\n           \u251c\u2500\u2500 tasks /\n\nThe main reason of this structure, is maintain organized several tasks by tenant/program, avoiding filling a directory with several task files.\n\nFlowTask support \"TaskStorage\", a Task Storage is the main repository for tasks, main Task Storage is a directory in any filesystem path (optionally you can syncronize that path using git), but Tasks can be saved onto a Database or a S3 bucket.\n\n## Dependencies ##\n\n * aiohttp (Asyncio Web Framework and Server) (required by navigator)\n * AsyncDB\n * QuerySource\n * Navigator-api\n * (Optional) Qworker (for distributing asyncio Tasks on distributed workers).\n\n## Features ##\n\n* Component-based Task execution framework with several components covering several actions (download files, create pandas dataframes from files, mapping dataframe columns to a json-dictionary, etc)\n* Built-in API for execution of Tasks.\n\n### How I run a Task? ###\n\nCan run a Task from CLI:\n```console\ntask --program=test --task=example\n```\n\non CLI, you can pass an ENV (enviroment) to change the environment file on task execution.\n```console\nENV=dev task --program=test --task=example\n```\n\nor Programmatically:\n```python\nfrom flowtask import Task\nimport asyncio\n\ntask = Task(program='test', task='example')\nresults = asyncio.run(task.run())\n# we can alternatively, using the execution mode of task object:\nresults = asyncio.run(task())\n```\n\n### Requirements ###\n\n* Python >= 3.9\n* asyncio (https://pypi.python.org/pypi/asyncio/)\n* aiohttp >= 3.6.2\n\n### Contribution guidelines ###\n\nPlease have a look at the Contribution Guide\n\n* Writing tests\n* Code review\n* Other guidelines\n\n### Who do I talk to? ###\n\n* Repo owner or admin\n* Other community or team contact\n\n### License ###\n\nNavigator is licensed under Apache 2.0 License. See the LICENSE file for more details.\n",
    "bugtrack_url": null,
    "license": "Apache-2.0",
    "summary": "Framework for running Tasks and from CLI and API for orchestation. Component-based Task builder/Runner for non-programmers.",
    "version": "5.8.18",
    "project_urls": {
        "Funding": "https://paypal.me/phenobarbital",
        "Homepage": "https://github.com/phenobarbital/flowtask",
        "Say Thanks!": "https://saythanks.io/to/phenobarbital",
        "Source": "https://github.com/phenobarbital/flowtask"
    },
    "split_keywords": [
        "dataintegration",
        " task",
        " orchestation",
        " task-runner",
        " pipelines",
        " data-pipelines"
    ],
    "urls": [
        {
            "comment_text": null,
            "digests": {
                "blake2b_256": "a8f7577662041f50c145c501ba1cae13b853f7fd4116137ebe53557e460c6ef7",
                "md5": "b3df6b9f835669489f039f3e23eaefdf",
                "sha256": "32c66746430bd6b35f4747c006ad082087346772aacf48ea847af73e12ef8fb2"
            },
            "downloads": -1,
            "filename": "flowtask-5.8.18-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.whl",
            "has_sig": false,
            "md5_digest": "b3df6b9f835669489f039f3e23eaefdf",
            "packagetype": "bdist_wheel",
            "python_version": "cp310",
            "requires_python": ">=3.9.16",
            "size": 3728606,
            "upload_time": "2025-07-16T15:36:26",
            "upload_time_iso_8601": "2025-07-16T15:36:26.037313Z",
            "url": "https://files.pythonhosted.org/packages/a8/f7/577662041f50c145c501ba1cae13b853f7fd4116137ebe53557e460c6ef7/flowtask-5.8.18-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.whl",
            "yanked": false,
            "yanked_reason": null
        },
        {
            "comment_text": null,
            "digests": {
                "blake2b_256": "adebde5dfeb5fa0a604717f459039623eb41c61571db9664952de199672edd83",
                "md5": "4c5b4587b330884a65a34807a5714b1f",
                "sha256": "3d6f8367efa6b9769cb58e452d9baa598841c1b87d356c7e2d2bae6481c87be4"
            },
            "downloads": -1,
            "filename": "flowtask-5.8.18-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.whl",
            "has_sig": false,
            "md5_digest": "4c5b4587b330884a65a34807a5714b1f",
            "packagetype": "bdist_wheel",
            "python_version": "cp311",
            "requires_python": ">=3.9.16",
            "size": 3905846,
            "upload_time": "2025-07-16T15:36:28",
            "upload_time_iso_8601": "2025-07-16T15:36:28.605838Z",
            "url": "https://files.pythonhosted.org/packages/ad/eb/de5dfeb5fa0a604717f459039623eb41c61571db9664952de199672edd83/flowtask-5.8.18-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.whl",
            "yanked": false,
            "yanked_reason": null
        },
        {
            "comment_text": null,
            "digests": {
                "blake2b_256": "e2d7c6ce7049b4c051781d720b2286d7ab0e3974b2b1ee84cd8bbc4aaaaebe92",
                "md5": "b6c88e89ec2f0d367cc78eff80b7c28e",
                "sha256": "5ba035a776ed5eb3dbdcb2a6b7c99c58c1a2a1a0d5d52fd46983f1ad753ce2f6"
            },
            "downloads": -1,
            "filename": "flowtask-5.8.18-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl",
            "has_sig": false,
            "md5_digest": "b6c88e89ec2f0d367cc78eff80b7c28e",
            "packagetype": "bdist_wheel",
            "python_version": "cp312",
            "requires_python": ">=3.9.16",
            "size": 4071419,
            "upload_time": "2025-07-16T15:36:30",
            "upload_time_iso_8601": "2025-07-16T15:36:30.141055Z",
            "url": "https://files.pythonhosted.org/packages/e2/d7/c6ce7049b4c051781d720b2286d7ab0e3974b2b1ee84cd8bbc4aaaaebe92/flowtask-5.8.18-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl",
            "yanked": false,
            "yanked_reason": null
        },
        {
            "comment_text": null,
            "digests": {
                "blake2b_256": "cc6f4c36037393a72f6c0fc261ba16f30dda0c39908577801efdb10f42a4c0d3",
                "md5": "9fe9f50af15e031d711b65172320a4d8",
                "sha256": "0c11116a264f0dc12a2d80e8e8be00224d3563da9a34ec1eb762e08480550e10"
            },
            "downloads": -1,
            "filename": "flowtask-5.8.18-cp39-cp39-manylinux2014_x86_64.manylinux_2_17_x86_64.whl",
            "has_sig": false,
            "md5_digest": "9fe9f50af15e031d711b65172320a4d8",
            "packagetype": "bdist_wheel",
            "python_version": "cp39",
            "requires_python": ">=3.9.16",
            "size": 3734981,
            "upload_time": "2025-07-16T15:36:31",
            "upload_time_iso_8601": "2025-07-16T15:36:31.434306Z",
            "url": "https://files.pythonhosted.org/packages/cc/6f/4c36037393a72f6c0fc261ba16f30dda0c39908577801efdb10f42a4c0d3/flowtask-5.8.18-cp39-cp39-manylinux2014_x86_64.manylinux_2_17_x86_64.whl",
            "yanked": false,
            "yanked_reason": null
        }
    ],
    "upload_time": "2025-07-16 15:36:26",
    "github": true,
    "gitlab": false,
    "bitbucket": false,
    "codeberg": false,
    "github_user": "phenobarbital",
    "github_project": "flowtask",
    "github_not_found": true,
    "lcname": "flowtask"
}
        
Elapsed time: 1.55274s