# FlowTask DataIntegration #
FlowTask DataIntegration is a plugin-based, component-driven task execution framework for create complex Tasks.
FlowTask runs Tasks defined in JSON, YAML or TOML files, any Task is a combination of Components,
and every component in the Task run sequentially or depend of others, like a DAG.
Can create a Task combining Commands, Shell scripts and other specific Components (as TableInput: Open a Table using a datasource, DownloadFromIMAP: Download a File from a IMAP Folder, and so on), any Python Callable can be a Component inside a Task, or can extends UserComponent to build your own componets.
Every designed Task can run from CLI, programmatically, via RESTful API (using our aioHTTP-based Handler), called by WebHooks or even dispatched to a external Worker using our built-in Scheduler.
## Quickstart ##
```console
pip install flowtask
```
Tasks can organizated into directory structure like this:
tasks /
├── programs /
├── test /
├── tasks /
The main reason of this structure, is maintain organized several tasks by tenant/program, avoiding filling a directory with several task files.
FlowTask support "TaskStorage", a Task Storage is the main repository for tasks, main Task Storage is a directory in any filesystem path (optionally you can syncronize that path using git), but Tasks can be saved onto a Database or a S3 bucket.
## Dependencies ##
* aiohttp (Asyncio Web Framework and Server) (required by navigator)
* AsyncDB
* QuerySource
* Navigator-api
* (Optional) Qworker (for distributing asyncio Tasks on distributed workers).
## Features ##
* Component-based Task execution framework with several components covering several actions (download files, create pandas dataframes from files, mapping dataframe columns to a json-dictionary, etc)
* Built-in API for execution of Tasks.
### How I run a Task? ###
Can run a Task from CLI:
```console
task --program=test --task=example
```
on CLI, you can pass an ENV (enviroment) to change the environment file on task execution.
```console
ENV=dev task --program=test --task=example
```
or Programmatically:
```python
from flowtask import Task
import asyncio
task = Task(program='test', task='example')
results = asyncio.run(task.run())
# we can alternatively, using the execution mode of task object:
results = asyncio.run(task())
```
### Requirements ###
* Python >= 3.9
* asyncio (https://pypi.python.org/pypi/asyncio/)
* aiohttp >= 3.6.2
### Contribution guidelines ###
Please have a look at the Contribution Guide
* Writing tests
* Code review
* Other guidelines
### Who do I talk to? ###
* Repo owner or admin
* Other community or team contact
### License ###
Navigator is licensed under Apache 2.0 License. See the LICENSE file for more details.
Raw data
{
"_id": null,
"home_page": "https://github.com/phenobarbital/flowtask",
"name": "flowtask",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.9.16",
"maintainer_email": null,
"keywords": "DataIntegration, Task, Orchestation, Task-Runner, Pipelines, Data-Pipelines",
"author": "Jesus Lara",
"author_email": "\"Jesus Lara G.\" <jesuslarag@gmail.com>",
"download_url": null,
"platform": "*nix",
"description": "# FlowTask DataIntegration #\n\nFlowTask DataIntegration is a plugin-based, component-driven task execution framework for create complex Tasks.\n\nFlowTask runs Tasks defined in JSON, YAML or TOML files, any Task is a combination of Components,\nand every component in the Task run sequentially or depend of others, like a DAG.\n\nCan create a Task combining Commands, Shell scripts and other specific Components (as TableInput: Open a Table using a datasource, DownloadFromIMAP: Download a File from a IMAP Folder, and so on), any Python Callable can be a Component inside a Task, or can extends UserComponent to build your own componets.\n\nEvery designed Task can run from CLI, programmatically, via RESTful API (using our aioHTTP-based Handler), called by WebHooks or even dispatched to a external Worker using our built-in Scheduler.\n\n## Quickstart ##\n\n```console\npip install flowtask\n```\n\nTasks can organizated into directory structure like this:\n\ntasks /\n \u251c\u2500\u2500 programs /\n \u251c\u2500\u2500 test /\n \u251c\u2500\u2500 tasks /\n\nThe main reason of this structure, is maintain organized several tasks by tenant/program, avoiding filling a directory with several task files.\n\nFlowTask support \"TaskStorage\", a Task Storage is the main repository for tasks, main Task Storage is a directory in any filesystem path (optionally you can syncronize that path using git), but Tasks can be saved onto a Database or a S3 bucket.\n\n## Dependencies ##\n\n * aiohttp (Asyncio Web Framework and Server) (required by navigator)\n * AsyncDB\n * QuerySource\n * Navigator-api\n * (Optional) Qworker (for distributing asyncio Tasks on distributed workers).\n\n## Features ##\n\n* Component-based Task execution framework with several components covering several actions (download files, create pandas dataframes from files, mapping dataframe columns to a json-dictionary, etc)\n* Built-in API for execution of Tasks.\n\n### How I run a Task? ###\n\nCan run a Task from CLI:\n```console\ntask --program=test --task=example\n```\n\non CLI, you can pass an ENV (enviroment) to change the environment file on task execution.\n```console\nENV=dev task --program=test --task=example\n```\n\nor Programmatically:\n```python\nfrom flowtask import Task\nimport asyncio\n\ntask = Task(program='test', task='example')\nresults = asyncio.run(task.run())\n# we can alternatively, using the execution mode of task object:\nresults = asyncio.run(task())\n```\n\n### Requirements ###\n\n* Python >= 3.9\n* asyncio (https://pypi.python.org/pypi/asyncio/)\n* aiohttp >= 3.6.2\n\n### Contribution guidelines ###\n\nPlease have a look at the Contribution Guide\n\n* Writing tests\n* Code review\n* Other guidelines\n\n### Who do I talk to? ###\n\n* Repo owner or admin\n* Other community or team contact\n\n### License ###\n\nNavigator is licensed under Apache 2.0 License. See the LICENSE file for more details.\n",
"bugtrack_url": null,
"license": "Apache-2.0",
"summary": "Framework for running Tasks and from CLI and API for orchestation. Component-based Task builder/Runner for non-programmers.",
"version": "5.6.9",
"project_urls": {
"Funding": "https://paypal.me/phenobarbital",
"Homepage": "https://github.com/phenobarbital/flowtask",
"Say Thanks!": "https://saythanks.io/to/phenobarbital",
"Source": "https://github.com/phenobarbital/flowtask"
},
"split_keywords": [
"dataintegration",
" task",
" orchestation",
" task-runner",
" pipelines",
" data-pipelines"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "63c60bc3c00c36468d64af50ea23b654d49b48eef79d1f78bf55a5aac82a5bbd",
"md5": "c8c6bec042de760de5c339f44cc2db83",
"sha256": "5cee873227bcde1e0c300e5f3788147ce4b9c1ac7b28d2aaed11f4286cd5b133"
},
"downloads": -1,
"filename": "flowtask-5.6.9-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl",
"has_sig": false,
"md5_digest": "c8c6bec042de760de5c339f44cc2db83",
"packagetype": "bdist_wheel",
"python_version": "cp310",
"requires_python": ">=3.9.16",
"size": 3441425,
"upload_time": "2024-12-17T22:22:44",
"upload_time_iso_8601": "2024-12-17T22:22:44.204382Z",
"url": "https://files.pythonhosted.org/packages/63/c6/0bc3c00c36468d64af50ea23b654d49b48eef79d1f78bf55a5aac82a5bbd/flowtask-5.6.9-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "692cc9e7791f0d21dda53b44bd6879ae3562235838b0c16fcb021831b68182e4",
"md5": "9422b5fde0ec97d000fa1c171a1c008d",
"sha256": "4990d0cd02148d3b135fc071990822a8b011ab09c7570076390dfa663186b531"
},
"downloads": -1,
"filename": "flowtask-5.6.9-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl",
"has_sig": false,
"md5_digest": "9422b5fde0ec97d000fa1c171a1c008d",
"packagetype": "bdist_wheel",
"python_version": "cp311",
"requires_python": ">=3.9.16",
"size": 3623667,
"upload_time": "2024-12-17T22:22:47",
"upload_time_iso_8601": "2024-12-17T22:22:47.785339Z",
"url": "https://files.pythonhosted.org/packages/69/2c/c9e7791f0d21dda53b44bd6879ae3562235838b0c16fcb021831b68182e4/flowtask-5.6.9-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "b09dfe70e316170d9a16bb70b03f1668f1cdc3c7f9b49ffe08b9c8a429ee82d6",
"md5": "1a36c5442fa59a88d9e5218b263d12d8",
"sha256": "33f2f95e05e042b7f56f51e4eca0ea0765a87ab4e83590e65b528863ce07287f"
},
"downloads": -1,
"filename": "flowtask-5.6.9-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl",
"has_sig": false,
"md5_digest": "1a36c5442fa59a88d9e5218b263d12d8",
"packagetype": "bdist_wheel",
"python_version": "cp312",
"requires_python": ">=3.9.16",
"size": 3786787,
"upload_time": "2024-12-17T22:22:49",
"upload_time_iso_8601": "2024-12-17T22:22:49.369485Z",
"url": "https://files.pythonhosted.org/packages/b0/9d/fe70e316170d9a16bb70b03f1668f1cdc3c7f9b49ffe08b9c8a429ee82d6/flowtask-5.6.9-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "606942c703ef3e3b2361d6a141a8ddb54a1de6dbeaa0313928b1277378037326",
"md5": "00fe60faf81d9211f1bc21afb6c55e91",
"sha256": "4e1b67366d133d4f231e2f1676a31f0347f7252489fc58daedaad6c8ae9a85b1"
},
"downloads": -1,
"filename": "flowtask-5.6.9-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl",
"has_sig": false,
"md5_digest": "00fe60faf81d9211f1bc21afb6c55e91",
"packagetype": "bdist_wheel",
"python_version": "cp39",
"requires_python": ">=3.9.16",
"size": 3458416,
"upload_time": "2024-12-17T22:22:51",
"upload_time_iso_8601": "2024-12-17T22:22:51.065958Z",
"url": "https://files.pythonhosted.org/packages/60/69/42c703ef3e3b2361d6a141a8ddb54a1de6dbeaa0313928b1277378037326/flowtask-5.6.9-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-12-17 22:22:44",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "phenobarbital",
"github_project": "flowtask",
"github_not_found": true,
"lcname": "flowtask"
}
JSON
