| Name | gator-eda JSON |
| Version |
0.1
JSON |
| download |
| home_page | |
| Summary | Hierarchical job execution and logging |
| upload_time | 2023-06-18 14:41:42 |
| maintainer | |
| docs_url | None |
| author | Peter Birch |
| requires_python | >=3.8,<4.0 |
| license | Apache-2.0 |
| keywords |
|
| VCS |
|
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
# 🐊 Gator
Gator is a combination of a task runner and a logging system. Every job is
managed by a wrapper layer that monitors its progress, resource usage, and
captures the messages that it emits via STDOUT or STDERR. Execution is managed
in a hierarchical structure, with jobs at the leaves reporting back to layers of
parent processes.
**NOTE** This project is under development is not feature complete, nor has it
been battle tested.
## Setting Up
Gator has been developed using `poetry` for package management:
```bash
$> python3 -m pip install poetry
$> poetry install
```
## YAML Specification
Tasks to perform are specified in a custom YAML syntax, arranged into two
distinct object types:
* `!Job` that describes a single task to perform;
* `!JobGroup` that describes a set of tasks to perform, also supporting layers
of nested groups;
* `!JobArray` similar to a `!JobGroup`, but repeats the set of described tasks
a specified number of times.
A simple specification may look like this:
```yaml
!JobGroup
id : top
jobs:
# Nested layer
- !JobGroup
id : inner
jobs:
- !Job
id : say_hi
command: echo
args : ["hi"]
# Arrayed job - waits for 'say_hi' to complete
- !JobArray
id : counting
on_pass:
- say_hi
repeats: 4
jobs :
- !Job
id : echo_count
command: echo
args : ["$GATOR_ARRAY_INDEX"]
# Directly attached to root - waits for 'counting' to complete
- !Job
id : say_bye
on_pass:
- counting
command: echo
args : ["bye"]
```
## Executing a Job Specification
To run a given job specification, use the Gator CLI:
```bash
$> python3 -m gator examples/job.yaml
[17:58:50] Starting Gator 🐊
[INFO ] Launching task: echo hey there you
[INFO ] Monitoring task
[INFO ] hey there you
[INFO ] Task completed with return code 0
[INFO ] Recorded 0 warnings and 0 errors
```
## Hub
To run the hub:
```bash
$> poe hub
```
## TODO
* [ ] Get hub working
* [ ] Pass artefacts between jobs and form artefact based dependencies
* [ ] Arbitrary metrics gathering - replace warning and error counts with a generalised mechanism that supports aggregation while summarising min, max, mean, sum, and count of metrics recorded
* [ ] Random number seeding
* [ ] Hooks
* [ ] Tool based log parsers
* [ ] Custom runners - currently everything is shell, perhaps support other things?
* [ ] Non-environment variable based parameters
Raw data
{
"_id": null,
"home_page": "",
"name": "gator-eda",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.8,<4.0",
"maintainer_email": "",
"keywords": "",
"author": "Peter Birch",
"author_email": "peter@lightlogic.co.uk",
"download_url": "https://files.pythonhosted.org/packages/4b/7a/c0d722298f67013b288549d3ea76ae02c76183c77bb419702e2f77609ae5/gator_eda-0.1.tar.gz",
"platform": null,
"description": "# \ud83d\udc0a Gator\n\nGator is a combination of a task runner and a logging system. Every job is\nmanaged by a wrapper layer that monitors its progress, resource usage, and\ncaptures the messages that it emits via STDOUT or STDERR. Execution is managed\nin a hierarchical structure, with jobs at the leaves reporting back to layers of\nparent processes.\n\n**NOTE** This project is under development is not feature complete, nor has it\nbeen battle tested.\n\n## Setting Up\n\nGator has been developed using `poetry` for package management:\n\n```bash\n$> python3 -m pip install poetry\n$> poetry install\n```\n\n## YAML Specification\n\nTasks to perform are specified in a custom YAML syntax, arranged into two\ndistinct object types:\n\n * `!Job` that describes a single task to perform;\n * `!JobGroup` that describes a set of tasks to perform, also supporting layers\n of nested groups;\n * `!JobArray` similar to a `!JobGroup`, but repeats the set of described tasks\n a specified number of times.\n\nA simple specification may look like this:\n\n```yaml\n!JobGroup\n id : top\n jobs:\n # Nested layer\n - !JobGroup\n id : inner\n jobs:\n - !Job\n id : say_hi\n command: echo\n args : [\"hi\"]\n # Arrayed job - waits for 'say_hi' to complete\n - !JobArray\n id : counting\n on_pass:\n - say_hi\n repeats: 4\n jobs :\n - !Job\n id : echo_count\n command: echo\n args : [\"$GATOR_ARRAY_INDEX\"]\n # Directly attached to root - waits for 'counting' to complete\n - !Job\n id : say_bye\n on_pass:\n - counting\n command: echo\n args : [\"bye\"]\n```\n\n## Executing a Job Specification\n\nTo run a given job specification, use the Gator CLI:\n\n```bash\n$> python3 -m gator examples/job.yaml\n[17:58:50] Starting Gator \ud83d\udc0a\n [INFO ] Launching task: echo hey there you\n [INFO ] Monitoring task\n [INFO ] hey there you\n [INFO ] Task completed with return code 0\n [INFO ] Recorded 0 warnings and 0 errors\n```\n\n## Hub\n\nTo run the hub:\n\n```bash\n$> poe hub\n```\n\n## TODO\n\n * [ ] Get hub working\n * [ ] Pass artefacts between jobs and form artefact based dependencies\n * [ ] Arbitrary metrics gathering - replace warning and error counts with a generalised mechanism that supports aggregation while summarising min, max, mean, sum, and count of metrics recorded\n * [ ] Random number seeding\n * [ ] Hooks\n * [ ] Tool based log parsers\n * [ ] Custom runners - currently everything is shell, perhaps support other things?\n * [ ] Non-environment variable based parameters\n",
"bugtrack_url": null,
"license": "Apache-2.0",
"summary": "Hierarchical job execution and logging",
"version": "0.1",
"project_urls": null,
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "9a03526f1f3db530245444965a2cbd983a37419e68ee31211c8c0db8e31241c7",
"md5": "f3824b6b8bcfa01baadadb62f95219d0",
"sha256": "3d334604cf64e5cf4ea5c070a799f1b9ac55652c00115501c4f14346cb15d327"
},
"downloads": -1,
"filename": "gator_eda-0.1-py3-none-any.whl",
"has_sig": false,
"md5_digest": "f3824b6b8bcfa01baadadb62f95219d0",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8,<4.0",
"size": 50128,
"upload_time": "2023-06-18T14:41:40",
"upload_time_iso_8601": "2023-06-18T14:41:40.528969Z",
"url": "https://files.pythonhosted.org/packages/9a/03/526f1f3db530245444965a2cbd983a37419e68ee31211c8c0db8e31241c7/gator_eda-0.1-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "4b7ac0d722298f67013b288549d3ea76ae02c76183c77bb419702e2f77609ae5",
"md5": "f571efaecef621fc0b32300de9b84278",
"sha256": "cdd22e774529c09dedbdec698f97e3c6c4385a9c7a7d98d6408a239a5a54efb2"
},
"downloads": -1,
"filename": "gator_eda-0.1.tar.gz",
"has_sig": false,
"md5_digest": "f571efaecef621fc0b32300de9b84278",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8,<4.0",
"size": 33766,
"upload_time": "2023-06-18T14:41:42",
"upload_time_iso_8601": "2023-06-18T14:41:42.153005Z",
"url": "https://files.pythonhosted.org/packages/4b/7a/c0d722298f67013b288549d3ea76ae02c76183c77bb419702e2f77609ae5/gator_eda-0.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-06-18 14:41:42",
"github": false,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"lcname": "gator-eda"
}
