# Nuvla Job Engine
[](https://github.com/nuvla/job-engine/actions/workflows/devel.yml)
This repository contains the code and configuration for the Job engine,
packaged as a Docker container.
Nuvla job engine use cimi job resource and zookeeper as a locking queue.
It's done in a way to be horizontally scalled on different nodes.
## Artifacts
- `nuvla/job:<version>`. A Docker container that can be obtained from
the [nuvla/job repository](https://hub.docker.com/r/nuvla/job)
in Docker Hub. The tags indicate the release number.
## Contributing
### Source Code Changes
To contribute code to this repository, please follow these steps:
1. Create a branch from main with a descriptive, kebab-cased name
to hold all your changes.
2. Follow the developer guidelines concerning formatting, etc. when
modifying the code.
3. Once the changes are ready to be reviewed, create a GitHub pull
request. With the pull request, provide a description of the
changes and links to any relevant issues (in this repository or
others).
4. Ensure that the triggered CI checks all pass. These are triggered
automatically with the results shown directly in the pull request.
5. Once the checks pass, assign the pull request to the repository
coordinator (who may then assign it to someone else).
6. Interact with the reviewer to address any comments.
When the reviewer is happy with the pull request, he/she will "squash
& merge" the pull request and delete the corresponding branch.
### Code Formatting
The bulk of the code in this repository is written in Python.
The formatting follows the coding style in defined in PEP 8.
### Job engine behavior
- Each action should be distributed by a standalone distributor
- More than one distributor for the same action can be started but only
one will be elected to distribute the job
- Executor load actions dynamically at startup time
- Zookeeper is used as a Locking queue containing only job uuid in
/job/entries
- Running jobs are put in zookeeper under /job/taken
- If executor is unable to communicate with CIMI, the job in running
state is released (put back in zookeeper queue).
- The action implementation should take care to continue or to make the
cleanup of a running interrupted job
- If the connection break with zookeeper, job in exection will be
released automatically. This is because /job/taken entries
are ephemeral nodes.
- Stopping the executor will try to make a proper shuttdown by waiting
2 minutes before killing the process. Each thread that terminate his
running action will not take a new one.
## Running unit tests
Before running unit tests with `tox` you need to generate requirements file out. To do so, we first need to
generate the test requirements file. This is done using poetry requirements exporter which can be installed as follows:
```shell
pip install poetry poetry-plugin-export
```
Then run export command:
```shell
poetry export -f requirements.txt -o requirements.test.txt --without-hashes --without-urls --with test --with server
```
Then run the unit tests with:
```shell
tox
```
## Locally Building Job Engine
Before running `docker build` you need to generate requirements file and the wheel package.
To do, install poetry and poetry-plugin-export as described above and then run:
```shell
poetry build --format=wheel
poetry export -f requirements.txt -o dist/requirements.txt --without-hashes --without-urls --with server
```
The order of the commands might be important since the requirements file output goes into dist/ directory which is created by the first command.
Then building the image requires the following command:
```shell
export JOB_ENGINE_VERSION=$(poetry version -s)
docker build --build-arg="PACKAGE_TAG=${JOB_ENGINE_VERSION}" -t local/job:${JOB_ENGINE_VERSION} .
```
## Copyright
Copyright © 2019-2024, SixSq SA
## License
Licensed under the Apache License, Version 2.0 (the "License"); you
may not use this file except in compliance with the License. You may
obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied. See the License for the specific language governing
permissions and limitations under the License.
Raw data
{
"_id": null,
"home_page": "https://github.com/nuvla/job-engine",
"name": "nuvla-job-engine",
"maintainer": "Ignacio Penas",
"docs_url": null,
"requires_python": "<3.12,>=3.11",
"maintainer_email": "nacho@sixsq.com",
"keywords": null,
"author": "SixSq SA",
"author_email": "support@sixsq.com",
"download_url": "https://files.pythonhosted.org/packages/1e/46/8aea0cf89bbd29fbc11d1f9a53900045e30f4e18f1051dc80d43b1039753/nuvla_job_engine-4.0.9.tar.gz",
"platform": null,
"description": "# Nuvla Job Engine\n\n[](https://github.com/nuvla/job-engine/actions/workflows/devel.yml)\n\nThis repository contains the code and configuration for the Job engine, \npackaged as a Docker container. \n\nNuvla job engine use cimi job resource and zookeeper as a locking queue. \nIt's done in a way to be horizontally scalled on different nodes.\n\n## Artifacts\n\n - `nuvla/job:<version>`. A Docker container that can be obtained from\n the [nuvla/job repository](https://hub.docker.com/r/nuvla/job)\n in Docker Hub. The tags indicate the release number.\n\n## Contributing\n\n### Source Code Changes\n\nTo contribute code to this repository, please follow these steps:\n\n 1. Create a branch from main with a descriptive, kebab-cased name\n to hold all your changes.\n\n 2. Follow the developer guidelines concerning formatting, etc. when\n modifying the code.\n \n 3. Once the changes are ready to be reviewed, create a GitHub pull\n request. With the pull request, provide a description of the\n changes and links to any relevant issues (in this repository or\n others). \n \n 4. Ensure that the triggered CI checks all pass. These are triggered\n automatically with the results shown directly in the pull request.\n\n 5. Once the checks pass, assign the pull request to the repository\n coordinator (who may then assign it to someone else).\n\n 6. Interact with the reviewer to address any comments.\n\nWhen the reviewer is happy with the pull request, he/she will \"squash\n& merge\" the pull request and delete the corresponding branch.\n\n\n### Code Formatting\n\nThe bulk of the code in this repository is written in Python.\n\nThe formatting follows the coding style in defined in PEP 8.\n\n\n### Job engine behavior\n\n - Each action should be distributed by a standalone distributor\n - More than one distributor for the same action can be started but only\n one will be elected to distribute the job\n - Executor load actions dynamically at startup time\n - Zookeeper is used as a Locking queue containing only job uuid in \n /job/entries\n - Running jobs are put in zookeeper under /job/taken\n - If executor is unable to communicate with CIMI, the job in running \n state is released (put back in zookeeper queue).\n - The action implementation should take care to continue or to make the \n cleanup of a running interrupted job\n - If the connection break with zookeeper, job in exection will be \n released automatically. This is because /job/taken entries \n are ephemeral nodes.\n - Stopping the executor will try to make a proper shuttdown by waiting \n 2 minutes before killing the process. Each thread that terminate his \n running action will not take a new one.\n\n## Running unit tests\nBefore running unit tests with `tox` you need to generate requirements file out. To do so, we first need to\ngenerate the test requirements file. This is done using poetry requirements exporter which can be installed as follows:\n\n```shell\npip install poetry poetry-plugin-export\n```\n\nThen run export command:\n\n```shell\npoetry export -f requirements.txt -o requirements.test.txt --without-hashes --without-urls --with test --with server\n```\n\nThen run the unit tests with:\n\n```shell\ntox\n```\n\n## Locally Building Job Engine\nBefore running `docker build` you need to generate requirements file and the wheel package.\nTo do, install poetry and poetry-plugin-export as described above and then run:\n\n```shell\npoetry build --format=wheel\npoetry export -f requirements.txt -o dist/requirements.txt --without-hashes --without-urls --with server\n```\nThe order of the commands might be important since the requirements file output goes into dist/ directory which is created by the first command.\n\n\n\nThen building the image requires the following command:\n```shell\nexport JOB_ENGINE_VERSION=$(poetry version -s)\ndocker build --build-arg=\"PACKAGE_TAG=${JOB_ENGINE_VERSION}\" -t local/job:${JOB_ENGINE_VERSION} .\n```\n\n\n## Copyright\n\nCopyright © 2019-2024, SixSq SA\n\n## License\n\nLicensed under the Apache License, Version 2.0 (the \"License\"); you\nmay not use this file except in compliance with the License. You may\nobtain a copy of the License at\n\nhttp://www.apache.org/licenses/LICENSE-2.0\n\nUnless required by applicable law or agreed to in writing, software\ndistributed under the License is distributed on an \"AS IS\" BASIS,\nWITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or\nimplied. See the License for the specific language governing\npermissions and limitations under the License.\n",
"bugtrack_url": null,
"license": "Apache-2.0",
"summary": "Nuvla Job Engine.",
"version": "4.0.9",
"project_urls": {
"Homepage": "https://github.com/nuvla/job-engine",
"Repository": "https://github.com/nuvla/job-engine"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "ca2193b47a63e89a7f5454afc2dee95f3a06e38cdc6c7998c961e6c4e2b6b43d",
"md5": "1937e246635dc7341507ece4fb224147",
"sha256": "0f7e7440e03e1d1a995a34b4fc4d45c5d7a6102cf5ed2fb1718d6b56576d4fa8"
},
"downloads": -1,
"filename": "nuvla_job_engine-4.0.9-py3-none-any.whl",
"has_sig": false,
"md5_digest": "1937e246635dc7341507ece4fb224147",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": "<3.12,>=3.11",
"size": 141567,
"upload_time": "2024-05-19T18:23:04",
"upload_time_iso_8601": "2024-05-19T18:23:04.662794Z",
"url": "https://files.pythonhosted.org/packages/ca/21/93b47a63e89a7f5454afc2dee95f3a06e38cdc6c7998c961e6c4e2b6b43d/nuvla_job_engine-4.0.9-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "1e468aea0cf89bbd29fbc11d1f9a53900045e30f4e18f1051dc80d43b1039753",
"md5": "38aac15529448f9d904f9ac9abf83349",
"sha256": "5f49280cf0d2d2f6cc852b33d77d82e7e300887545b52915b7e293b44d66e8f8"
},
"downloads": -1,
"filename": "nuvla_job_engine-4.0.9.tar.gz",
"has_sig": false,
"md5_digest": "38aac15529448f9d904f9ac9abf83349",
"packagetype": "sdist",
"python_version": "source",
"requires_python": "<3.12,>=3.11",
"size": 95166,
"upload_time": "2024-05-19T18:23:06",
"upload_time_iso_8601": "2024-05-19T18:23:06.711721Z",
"url": "https://files.pythonhosted.org/packages/1e/46/8aea0cf89bbd29fbc11d1f9a53900045e30f4e18f1051dc80d43b1039753/nuvla_job_engine-4.0.9.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-05-19 18:23:06",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "nuvla",
"github_project": "job-engine",
"travis_ci": false,
"coveralls": true,
"github_actions": true,
"tox": true,
"lcname": "nuvla-job-engine"
}
JSON
