| Name | nomad-tools JSON |
| Version |
0.5.3
JSON |
| download |
| home_page | None |
| Summary | Set of tools and utilities to ease interacting with Hashicorp Nomad scheduling solution. |
| upload_time | 2024-11-11 12:14:54 |
| maintainer | None |
| docs_url | None |
| author | Kamil Cukrowski |
| requires_python | >3.7 |
| license | GPL-3.0-or-later |
| keywords |
|
| VCS |
 |
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
# nomadtools
Set of tools and utilities to ease interacting with HashiCorp Nomad scheduling solution.
## Table of Contents
<!-- mtoc-start -->
* [Installation](#installation)
* [Shell completion](#shell-completion)
* [Usage](#usage)
* [`watch` - get all logs and events of a Nomad job in terminal](#watch---get-all-logs-and-events-of-a-nomad-job-in-terminal)
* [`go` - create, run and watch a Nomad job from command line](#go---create-run-and-watch-a-nomad-job-from-command-line)
* [`constrainteval` - evaluate constraint in command line](#constrainteval---evaluate-constraint-in-command-line)
* [`listattributes` - List nodes with given attributes or show all available attributes.](#listattributes---list-nodes-with-given-attributes-or-show-all-available-attributes)
* [`listnodeattributes` - list all available node attributes](#listnodeattributes---list-all-available-node-attributes)
* [`port` - list ports allocated by job or allocation](#port---list-ports-allocated-by-job-or-allocation)
* [`vardir` - manipulate nomad variable keys as files](#vardir---manipulate-nomad-variable-keys-as-files)
* [`cp` - copy files to/from/between nomad allocations](#cp---copy-files-tofrombetween-nomad-allocations)
* [`gitlab-runner` - dynamically schedule gitlab CI jobs](#gitlab-runner---dynamically-schedule-gitlab-ci-jobs)
* [`githubrunner` - dynamically schedule GitHub actions jobs](#githubrunner---dynamically-schedule-github-actions-jobs)
* [`dockers` - list docker images referenced by Nomad job](#dockers---list-docker-images-referenced-by-nomad-job)
* [`downloadrelease` - download specific Nomad executable version](#downloadrelease---download-specific-nomad-executable-version)
* [`task` - find task and allocation and execute action on it](#task---find-task-and-allocation-and-execute-action-on-it)
* [`task exec` - execute a command inside the allocation](#task-exec---execute-a-command-inside-the-allocation)
* [`task json` - output found allocations and task names in json form](#task-json---output-found-allocations-and-task-names-in-json-form)
* [`task path` - output path in the form properly escaped for use with nomadtools cp](#task-path---output-path-in-the-form-properly-escaped-for-use-with-nomadtools-cp)
* [`task xargs` - output in the form -task <task> <allocid> that is usable with xargs nomad alloc](#task-xargs---output-in-the-form--task-task-allocid-that-is-usable-with-xargs-nomad-alloc)
* [`nodenametoid` - convert node name to id](#nodenametoid---convert-node-name-to-id)
* [`info topology` - show some information about Nomad node usage](#info-topology---show-some-information-about-nomad-node-usage)
* [import nomad_tools](#import-nomad_tools)
* [History](#history)
* [Contributing](#contributing)
* [Running tests](#running-tests)
* [License](#license)
<!-- mtoc-end -->
If you use this package, wheather you like it or not, would you want something to improve,
or you feel like talking, consider leaving your opinion on github discussions
https://github.com/Kamilcuk/nomad-tools/discussions .
If there is a feature request of bug, create a github issue
https://github.com/Kamilcuk/nomad-tools/issues .
In the world of burning open source developers your comment
will help brighten or darken my day and also serve to prioritize_is_correct
this tool development.
# Installation
You can use docker image:
```
docker run -t --rm -e NOMAD_ADDR ghcr.io/kamilcuk/nomad-tools:0.4.0 --help
```
Install the project using `pipx` project.
```
pipx install nomad-tools
```
After installation the executable `nomadtools` should be available.
```
nomadtools --help
```
## Shell completion
After installation, see `nomadtools watch --autocomplete-info` for shell
completion installation instruction.
# Usage
This module installs command line tool `nomadtools` with several modes of
operation:
## `watch` - get all logs and events of a Nomad job in terminal
`nomadtools watch` is meant to watch over a job change that you type in
terminal. It prints all relevant messages - messages about allocation,
evaluation, deployment and stdout and stderr logs from all the
processes. Depending on the mode of operation, the tool waits until an
action is finished.
I primarily use `watch` to deploy new versions of services. I was always
frustrated that I start something from terminal and then I have to check the
logs of the service in multiple tabs in the Nomad web interface. For example,
you can use `watch start ./postgres.nomad.hcl` to update PostgreSQL
container and watch it's logs in your terminal.
An example terminal session deploying a HTTP server job with canary and health
check. Note that while the new version is deployed, the old one still prints
the logs.
Another usage of the job is to run an one-shot batch jobs to do something and
wait until they are finished and collect the exit status and logs, for example
as an airflow or cron job. In this case `run` mode will wait for the job to be
finished. For example `watch --purge run ./compute.nomad.hcl` will run
a calculation job, purge after it is done and exit with calculate job exit
status (if there is one task).
Internally, watch uses Nomad event stream to get the events in real time.
## `go` - create, run and watch a Nomad job from command line
Mimics operation of `docker run`, it is built on top of `watch` mode to
execute a single Nomad job created dynamically from command line arguments.
It creates a Nomad job specification from command line arguments and then
"watches" over the execution of the job.
```
$ nomadtools go --rm alpine apk add bash
INFO:nomad_tools.nomad_watch:Watching job nomad_tools_go_5305da8f-b376-4c35-9a05-71027aadd587@default until it is finished
Allocation 70c4ac9d-0e03-53d7-6e34-9c86cf8ee768 started on leonidas
Received Task received by client
Task Setup Building Task Directory
Driver Downloading image
Started Task started by client
INFO:nomad_tools.nomad_watch:Job nomad_tools_go_5305da8f-b376-4c35-9a05-71027aadd587#0@default started allocations 70c4ac running group 'nomad_tools_go_5305da8f-b376-4c35-9a05-71027aadd587' with 1 main tasks.
fetch https://dl-cdn.alpinelinux.org/alpine/v3.19/main/x86_64/APKINDEX.tar.gz
fetch https://dl-cdn.alpinelinux.org/alpine/v3.19/community/x86_64/APKINDEX.tar.gz
(1/4) Installing ncurses-terminfo-base (6.4_p20231125-r0)
(2/4) Installing libncursesw (6.4_p20231125-r0)
(3/4) Installing readline (8.2.1-r2)
(4/4) Installing bash (5.2.21-r0)
Terminated Exit Code: 0
Allocation 70c4ac9d-0e03-53d7-6e34-9c86cf8ee768 finished
Executing bash-5.2.21-r0.post-install
Executing busybox-1.36.1-r15.trigger
OK: 10 MiB in 19 packages
INFO:nomad_tools.nomad_watch:Purging job nomad_tools_go_5305da8f-b376-4c35-9a05-71027aadd587
INFO:nomad_tools.nomad_watch:Job nomad_tools_go_5305da8f-b376-4c35-9a05-71027aadd587#0@default purged with no active allocations, evaluations nor deployments. Exiting.
INFO:nomad_tools.nomad_watch:Single task exited with 0 exit status. Exit code is 0.
```
## `constrainteval` - evaluate constraint in command line
Evaluate a constraint and show all nodes that match the constraint. In
addition to the node names, it also shows all attributes referenced while
evaluating the constraint given on command line arguments. Useful for
searching for which hosts contain what value of a attribute.
```
$ nomadtools constrainteval '${attr.cpu.arch}' = amd64
name attr.cpu.arch
node1 amd64
node2 amd64
```
This mode uses a cache in `~/.cache/nomadtools/nodes.json` for caching all the
attributes of nodes downloaded from Nomad. This is used to speed up. The
program needs to make one query for every single node in Nomad, which for a
lot of nodes is costly.
## `listattributes` - List nodes with given attributes or show all available attributes.
```
$ nomadtools listattributes attr.cpu.arch
name attr.cpu.arch
node1 amd64
node2 amd64
```
Bash completion works.
## `listnodeattributes` - list all available node attributes
Lists all node attributes or attribute of given nodes.
Additionally with a leading dot lists the whole json return from Nomad API.
This is meant to be used with `grep` and other unix tools for easy parsing.
```
$ nomadtools listnodeattributes node1 | grep datacenter
dc.datacenter dc
```
It uses the same cache and option as `constrainteval` option.
## `port` - list ports allocated by job or allocation
Prints out the ports allocated for a particular Nomad job or
allocation. It is meant to mimic `docker port` command.
```
$ nomadtools port httpd
192.168.0.5:31076
$ nomadtools port -l httpd
192.168.0.5 31076 http httpd.listen[0] d409e855-bf13-a342-fe7a-6fb579d2de85
$ nomadtools port --alloc d409e855
192.168.0.5:31076
```
Further argument allows to filter for port label.
```
$ nomadtools port httpd http
192.168.0.5:31076
```
## `vardir` - manipulate nomad variable keys as files
I was frustrated with how Nomad variables look like. It is really hard to
incrementally modify Nomad variables. The API is at one go. You either update
all variables or nothing. Most often I wanted to update a single key
from a Nomad variable at a time and the variable value was usually a file content.
Example execution of putting a `passwordfile.txt` into `nomad/jobs/nginx`
Nomad variable:
```
$ nomadtools vardir -j nginx put ./passwordfile.txt
nomad_vardir: Putting var nomad/jobs/nginx@default with keys: passwordfile.txt
$ nomadtools vardir -j nginx cat passwordfile.txt
secretpassword
$ nomadtools vardir -j nginx ls
nomad_vardir: Listing Nomad variable at nomad/jobs/nginx@default
key size
passwordfile.txt 15
```
You can then remove the `passwordfile.txt` key from the Nomad variable:
```
$ nomadtools vardir -j nginx rm passwordfile.txt
nomad_vardir: Removing passwordfile.txt
nomad_vardir: Removing empty var nomad/jobs/nginx@default
$ nomadtools vardir -j nginx ls
nomad_vardir: Nomad variable not found at nomad/jobs/nginx@default
```
## `cp` - copy files to/from/between nomad allocations
This is a copy of the `docker cp` command. The syntax is meant to be the
same with docker. The rules of copying a file vs directory are meant to be
in-line with `docker cp` documentation.
`nomadtools cp` uses some special syntax for specifying from which allocation/task
exactly do you want to copy by using colon `:`. The number of colons in the
arguments determines the format. The colon can be escaped with slash `\` in
the path if needed.
Both `SRC` and `DST` addresses can be specified as follows:
```
# Search a task matching specific URL query:
task://JOB[@NAMESPACE]/PATH[?group=GROUP][&alloc=ALLOC][&task=TASK][&hostname=HOSTNAME][&node=NODE]
# or
:ALLOCATION:PATH copy path from this allocation having one job
:ALLOCATION::TASK:PATH copy path from this task inside allocation
:ALLOCATION:GROUP:TASK:PATH like above, but filter by group name
JOB:PATH copy path from one task inside specified job
JOB::TASK:PATH copy path from the task inside this job
JOB:GROUP:TASK:PATH like above, but filter also by group name
PATH copy local path
- copy stdin or stdout TAR stream
```
`cp` depends on `sh` and `tar` command line utility to be available
inside the allocation it is coping to/from. It has to be available there.
Example:
```
$ nomadtools cp -v nginx:/etc/nginx/nginx.conf ./nginx.conf
INFO nomad_cp.py:copy_mode:487: File :d409e855-bf13-a342-fe7a-6fb579d2de85:listen:/etc/nginx/nginx.conf -> ./nginx.conf
$ nomadtools cp -v alpine:/etc/. ./etc/
INFO nomad_cp.py:copy_mode:512: New mkdir :d409e855-bf13-a342-fe7a-6fb579d2de85:listen:/etc/. -> /home/kamil/tmp/etc2/
```
Nomad does not have the capability of accessing any file inside the
allocation file system. Instead, `nomad-cp` executes several `nomad exec`
calls to execute a `tar` pipe to stream the data from or to the allocation
context to or from the local host using stdout and stdin forwarded by
`nomad exec`.
## `gitlab-runner` - dynamically schedule gitlab CI jobs
An implementation of custom Gitlab executor driver that runs Gitlab CI/CD jobs
using Nomad.
This program does _not_ run the `gitlab-runner` itself in Nomad. Rather, the
`gitlab-runner` is running on (any) one host. That `gitlab-runner` will then
schedule Nomad jobs to execute using the script as an executor. These jobs will
execute the CI/CD from Gitlab inside Nomad cluster.
More on it can be read on github wiki
[https://github.com/Kamilcuk/nomad-tools/wiki/gitlab%E2%80%90runner](https://github.com/Kamilcuk/nomad-tools/wiki/gitlab%E2%80%90runner) .
## `githubrunner` - dynamically schedule GitHub actions jobs
Schedule Github actions runners as Nomad job for user repositories.
See documentation in [doc/githubrunner.md](doc/githubrunner.md) .
## `dockers` - list docker images referenced by Nomad job
Lists docker images referenced in Nomad job file or a running Nomad job.
```
$ nomadtools dockers ./httpd.nomad.hcl
busybox:stable
$ nomadtools dockers --job httpd
busybox:stable
```
## `downloadrelease` - download specific Nomad executable version
Program for downloading specific Nomad release binary from their release page.
I use it for testing and checking new Nomad versions.
```
$ nomadtools downloadrelease nomad
INFO:nomad_tools.nomad_downloadrelease:Downloading https://releases.hashicorp.com/nomad/1.7.3/nomad_1.7.3_linux_amd64.zip to nomad
INFO:nomad_tools.nomad_downloadrelease:https://releases.hashicorp.com/nomad/1.7.3/nomad_1.7.3_linux_amd64.zip -> -rwxr-xr-x 105.7MB nomad
$ nomadtools downloadrelease consul
INFO:nomad_tools.nomad_downloadrelease:Downloading https://releases.hashicorp.com/consul/1.9.9/consul_1.9.9_linux_amd64.zip to consul
INFO:nomad_tools.nomad_downloadrelease:https://releases.hashicorp.com/consul/1.9.9/consul_1.9.9_linux_amd64.zip -> -rwxr-xr-x 105.8MB consul
$ nomadtools downloadrelease -p 1.6.3 nomad ./nomad1.6.3
INFO:nomad_tools.nomad_downloadrelease:Downloading https://releases.hashicorp.com/nomad/1.6.3/nomad_1.6.3_linux_amd64.zip to nomad1.6.3
INFO:nomad_tools.nomad_downloadrelease:https://releases.hashicorp.com/nomad/1.6.3/nomad_1.6.3_linux_amd64.zip -> -rwxr-xr-x 101.8MB nomad1.6.3
```
## `task` - find task and allocation and execute action on it
`nomad alloc exec` didn't met my expectations. It is boring and tedious to find an allocation ID
and then copy it in the terminal. There is also no bash completion for task names.
I decided I want to have advanced filtering, where given a job name and node name some tool
automatically finds an allocation and task to copy from.
Given a system job promtail running on all machines, `nomadtools task -j promtail -n host1 exec bash -l`
will drop into a bash shell inside promtail job allocation running on host1.
This tool can be used to get a shell to the chosen allocation.
### `task exec` - execute a command inside the allocation
This internally calls `nomad alloc exec`.
```
$ nomadtools task -j mail exec bash -l
root@main:/#
```
### `task json` - output found allocations and task names in json form
### `task path` - output path in the form properly escaped for use with nomadtools cp
```
$ nomadtools cp "$(nomadtools task -j mail path /etc/fstab)" ./fstab
INFO entry_cp.py:copy_mode:650: File :0c18e9aa-f053-cc3e-6fe3-3d23f159c2e5:mail:/etc/fstab -> ./fstab
37.0 B 0:00:00 [73.3 B/s] [ <=>
```
### `task xargs` - output in the form -task <task> <allocid> that is usable with xargs nomad alloc
```
$ nomadtools task -j mail xargs -0 logs -- -stderr | xargs -0 nomad alloc
$ nomadtools task -j mail xargs logs -- -stderr | xargs nomad alloc
$ nomad alloc logs $(nomadtools task -j mail xargs) -stderr
```
## `nodenametoid` - convert node name to id
```
$ nomadtools nodenametoid node1
3e50c2ef-16bd-0253-6635-1a55c25e74ca
```
## `info topology` - show some information about Nomad node usage
```
$ nomadtools info topology
dc node1 ready 2000B/15988MB 1000MHz/12000MHz 1allocs [mariadb.mariadb[0] mariadb services 2000MB 1000MHz]
```
## import nomad_tools
This project is licensed under GPL. The internal API of this project can be
used, however it is not stable at all and is an implementation detail.
Internally, `nomad_tools.nomadlib` is a Python class definitions which
represents models for Nomad API data documentation.
# History
This module once installed bunch of separate tools, like `nomad-watch` or
`nomad-gitlab-runner`. That became unmaintainable. It is one `nomadtools`
executable with several sub-commands.
# Contributing
Kindly make a issue or pull request on GitHub.
I should be fast to respond and contributions are super welcome.
## Running tests
I want to support Python 3.7 with the project.
To test first install editable package locally with test dependencies:
```
pip install -e '.[test]'
```
You can run unit tests always without any external tools:
```
./unit_tests.sh
```
To run integration tests, you have to be able to connect to Nomad server.
```
./integration_tests.sh
./integration_tests.sh -k nomad_vardir
```
# License
GPL
Raw data
{
"_id": null,
"home_page": null,
"name": "nomad-tools",
"maintainer": null,
"docs_url": null,
"requires_python": ">3.7",
"maintainer_email": null,
"keywords": null,
"author": "Kamil Cukrowski",
"author_email": null,
"download_url": "https://files.pythonhosted.org/packages/df/dc/a3146bbe9efbc86c3a6dc915aab9cbea3f1ebd97d7b37c84255ed9c476b1/nomad_tools-0.5.3.tar.gz",
"platform": null,
"description": "# nomadtools\n\nSet of tools and utilities to ease interacting with HashiCorp Nomad scheduling solution.\n\n## Table of Contents\n\n\n<!-- mtoc-start -->\n\n* [Installation](#installation)\n * [Shell completion](#shell-completion)\n* [Usage](#usage)\n * [`watch` - get all logs and events of a Nomad job in terminal](#watch---get-all-logs-and-events-of-a-nomad-job-in-terminal)\n * [`go` - create, run and watch a Nomad job from command line](#go---create-run-and-watch-a-nomad-job-from-command-line)\n * [`constrainteval` - evaluate constraint in command line](#constrainteval---evaluate-constraint-in-command-line)\n * [`listattributes` - List nodes with given attributes or show all available attributes.](#listattributes---list-nodes-with-given-attributes-or-show-all-available-attributes)\n * [`listnodeattributes` - list all available node attributes](#listnodeattributes---list-all-available-node-attributes)\n * [`port` - list ports allocated by job or allocation](#port---list-ports-allocated-by-job-or-allocation)\n * [`vardir` - manipulate nomad variable keys as files](#vardir---manipulate-nomad-variable-keys-as-files)\n * [`cp` - copy files to/from/between nomad allocations](#cp---copy-files-tofrombetween-nomad-allocations)\n * [`gitlab-runner` - dynamically schedule gitlab CI jobs](#gitlab-runner---dynamically-schedule-gitlab-ci-jobs)\n * [`githubrunner` - dynamically schedule GitHub actions jobs](#githubrunner---dynamically-schedule-github-actions-jobs)\n * [`dockers` - list docker images referenced by Nomad job](#dockers---list-docker-images-referenced-by-nomad-job)\n * [`downloadrelease` - download specific Nomad executable version](#downloadrelease---download-specific-nomad-executable-version)\n * [`task` - find task and allocation and execute action on it](#task---find-task-and-allocation-and-execute-action-on-it)\n * [`task exec` - execute a command inside the allocation](#task-exec---execute-a-command-inside-the-allocation)\n * [`task json` - output found allocations and task names in json form](#task-json---output-found-allocations-and-task-names-in-json-form)\n * [`task path` - output path in the form properly escaped for use with nomadtools cp](#task-path---output-path-in-the-form-properly-escaped-for-use-with-nomadtools-cp)\n * [`task xargs` - output in the form -task <task> <allocid> that is usable with xargs nomad alloc](#task-xargs---output-in-the-form--task-task-allocid-that-is-usable-with-xargs-nomad-alloc)\n * [`nodenametoid` - convert node name to id](#nodenametoid---convert-node-name-to-id)\n * [`info topology` - show some information about Nomad node usage](#info-topology---show-some-information-about-nomad-node-usage)\n * [import nomad_tools](#import-nomad_tools)\n* [History](#history)\n* [Contributing](#contributing)\n * [Running tests](#running-tests)\n* [License](#license)\n\n<!-- mtoc-end -->\n\nIf you use this package, wheather you like it or not, would you want something to improve,\nor you feel like talking, consider leaving your opinion on github discussions\nhttps://github.com/Kamilcuk/nomad-tools/discussions .\n\nIf there is a feature request of bug, create a github issue\nhttps://github.com/Kamilcuk/nomad-tools/issues .\n\nIn the world of burning open source developers your comment\nwill help brighten or darken my day and also serve to prioritize_is_correct\nthis tool development.\n\n# Installation\n\nYou can use docker image:\n\n```\ndocker run -t --rm -e NOMAD_ADDR ghcr.io/kamilcuk/nomad-tools:0.4.0 --help\n```\n\nInstall the project using `pipx` project.\n\n```\npipx install nomad-tools\n```\n\nAfter installation the executable `nomadtools` should be available.\n\n```\nnomadtools --help\n```\n\n## Shell completion\n\nAfter installation, see `nomadtools watch --autocomplete-info` for shell\ncompletion installation instruction.\n\n# Usage\n\nThis module installs command line tool `nomadtools` with several modes of\noperation:\n\n## `watch` - get all logs and events of a Nomad job in terminal\n\n`nomadtools watch` is meant to watch over a job change that you type in\nterminal. It prints all relevant messages - messages about allocation,\nevaluation, deployment and stdout and stderr logs from all the\nprocesses. Depending on the mode of operation, the tool waits until an\naction is finished.\n\nI primarily use `watch` to deploy new versions of services. I was always\nfrustrated that I start something from terminal and then I have to check the\nlogs of the service in multiple tabs in the Nomad web interface. For example,\nyou can use `watch start ./postgres.nomad.hcl` to update PostgreSQL\ncontainer and watch it's logs in your terminal.\n\nAn example terminal session deploying a HTTP server job with canary and health\ncheck. Note that while the new version is deployed, the old one still prints\nthe logs.\n\n\n\nAnother usage of the job is to run an one-shot batch jobs to do something and\nwait until they are finished and collect the exit status and logs, for example\nas an airflow or cron job. In this case `run` mode will wait for the job to be\nfinished. For example `watch --purge run ./compute.nomad.hcl` will run\na calculation job, purge after it is done and exit with calculate job exit\nstatus (if there is one task).\n\n\n\nInternally, watch uses Nomad event stream to get the events in real time.\n\n## `go` - create, run and watch a Nomad job from command line\n\nMimics operation of `docker run`, it is built on top of `watch` mode to\nexecute a single Nomad job created dynamically from command line arguments.\nIt creates a Nomad job specification from command line arguments and then\n\"watches\" over the execution of the job.\n\n```\n$ nomadtools go --rm alpine apk add bash\nINFO:nomad_tools.nomad_watch:Watching job nomad_tools_go_5305da8f-b376-4c35-9a05-71027aadd587@default until it is finished\nAllocation 70c4ac9d-0e03-53d7-6e34-9c86cf8ee768 started on leonidas\nReceived Task received by client\nTask Setup Building Task Directory\nDriver Downloading image\nStarted Task started by client\nINFO:nomad_tools.nomad_watch:Job nomad_tools_go_5305da8f-b376-4c35-9a05-71027aadd587#0@default started allocations 70c4ac running group 'nomad_tools_go_5305da8f-b376-4c35-9a05-71027aadd587' with 1 main tasks.\nfetch https://dl-cdn.alpinelinux.org/alpine/v3.19/main/x86_64/APKINDEX.tar.gz\nfetch https://dl-cdn.alpinelinux.org/alpine/v3.19/community/x86_64/APKINDEX.tar.gz\n(1/4) Installing ncurses-terminfo-base (6.4_p20231125-r0)\n(2/4) Installing libncursesw (6.4_p20231125-r0)\n(3/4) Installing readline (8.2.1-r2)\n(4/4) Installing bash (5.2.21-r0)\nTerminated Exit Code: 0\nAllocation 70c4ac9d-0e03-53d7-6e34-9c86cf8ee768 finished\nExecuting bash-5.2.21-r0.post-install\nExecuting busybox-1.36.1-r15.trigger\nOK: 10 MiB in 19 packages\nINFO:nomad_tools.nomad_watch:Purging job nomad_tools_go_5305da8f-b376-4c35-9a05-71027aadd587\nINFO:nomad_tools.nomad_watch:Job nomad_tools_go_5305da8f-b376-4c35-9a05-71027aadd587#0@default purged with no active allocations, evaluations nor deployments. Exiting.\nINFO:nomad_tools.nomad_watch:Single task exited with 0 exit status. Exit code is 0.\n```\n\n## `constrainteval` - evaluate constraint in command line\n\nEvaluate a constraint and show all nodes that match the constraint. In\naddition to the node names, it also shows all attributes referenced while\nevaluating the constraint given on command line arguments. Useful for\nsearching for which hosts contain what value of a attribute.\n\n```\n$ nomadtools constrainteval '${attr.cpu.arch}' = amd64\nname attr.cpu.arch\nnode1 amd64\nnode2 amd64\n```\n\nThis mode uses a cache in `~/.cache/nomadtools/nodes.json` for caching all the\nattributes of nodes downloaded from Nomad. This is used to speed up. The\nprogram needs to make one query for every single node in Nomad, which for a\nlot of nodes is costly.\n\n## `listattributes` - List nodes with given attributes or show all available attributes.\n\n```\n$ nomadtools listattributes attr.cpu.arch\nname attr.cpu.arch\nnode1 amd64\nnode2 amd64\n```\n\nBash completion works.\n\n## `listnodeattributes` - list all available node attributes\n\nLists all node attributes or attribute of given nodes.\nAdditionally with a leading dot lists the whole json return from Nomad API.\n\nThis is meant to be used with `grep` and other unix tools for easy parsing.\n\n```\n$ nomadtools listnodeattributes node1 | grep datacenter\ndc.datacenter dc\n```\n\nIt uses the same cache and option as `constrainteval` option.\n\n## `port` - list ports allocated by job or allocation\n\nPrints out the ports allocated for a particular Nomad job or\nallocation. It is meant to mimic `docker port` command.\n\n```\n$ nomadtools port httpd\n192.168.0.5:31076\n$ nomadtools port -l httpd\n192.168.0.5 31076 http httpd.listen[0] d409e855-bf13-a342-fe7a-6fb579d2de85\n$ nomadtools port --alloc d409e855\n192.168.0.5:31076\n```\n\nFurther argument allows to filter for port label.\n\n```\n$ nomadtools port httpd http\n192.168.0.5:31076\n```\n\n## `vardir` - manipulate nomad variable keys as files\n\nI was frustrated with how Nomad variables look like. It is really hard to\nincrementally modify Nomad variables. The API is at one go. You either update\nall variables or nothing. Most often I wanted to update a single key\nfrom a Nomad variable at a time and the variable value was usually a file content.\n\nExample execution of putting a `passwordfile.txt` into `nomad/jobs/nginx`\nNomad variable:\n\n```\n$ nomadtools vardir -j nginx put ./passwordfile.txt \nnomad_vardir: Putting var nomad/jobs/nginx@default with keys: passwordfile.txt\n$ nomadtools vardir -j nginx cat passwordfile.txt \nsecretpassword\n$ nomadtools vardir -j nginx ls\nnomad_vardir: Listing Nomad variable at nomad/jobs/nginx@default\nkey size\npasswordfile.txt 15\n```\n\nYou can then remove the `passwordfile.txt` key from the Nomad variable:\n\n```\n$ nomadtools vardir -j nginx rm passwordfile.txt \nnomad_vardir: Removing passwordfile.txt\nnomad_vardir: Removing empty var nomad/jobs/nginx@default\n$ nomadtools vardir -j nginx ls\nnomad_vardir: Nomad variable not found at nomad/jobs/nginx@default\n```\n\n## `cp` - copy files to/from/between nomad allocations\n\nThis is a copy of the `docker cp` command. The syntax is meant to be the\nsame with docker. The rules of copying a file vs directory are meant to be\nin-line with `docker cp` documentation.\n\n`nomadtools cp` uses some special syntax for specifying from which allocation/task\nexactly do you want to copy by using colon `:`. The number of colons in the\narguments determines the format. The colon can be escaped with slash `\\` in\nthe path if needed.\n\nBoth `SRC` and `DST` addresses can be specified as follows:\n\n```\n# Search a task matching specific URL query:\ntask://JOB[@NAMESPACE]/PATH[?group=GROUP][&alloc=ALLOC][&task=TASK][&hostname=HOSTNAME][&node=NODE]\n# or\n:ALLOCATION:PATH copy path from this allocation having one job\n:ALLOCATION::TASK:PATH copy path from this task inside allocation\n:ALLOCATION:GROUP:TASK:PATH like above, but filter by group name\nJOB:PATH copy path from one task inside specified job\nJOB::TASK:PATH copy path from the task inside this job\nJOB:GROUP:TASK:PATH like above, but filter also by group name\nPATH copy local path\n- copy stdin or stdout TAR stream\n```\n\n`cp` depends on `sh` and `tar` command line utility to be available\ninside the allocation it is coping to/from. It has to be available there.\n\nExample:\n\n```\n$ nomadtools cp -v nginx:/etc/nginx/nginx.conf ./nginx.conf\nINFO nomad_cp.py:copy_mode:487: File :d409e855-bf13-a342-fe7a-6fb579d2de85:listen:/etc/nginx/nginx.conf -> ./nginx.conf\n$ nomadtools cp -v alpine:/etc/. ./etc/\nINFO nomad_cp.py:copy_mode:512: New mkdir :d409e855-bf13-a342-fe7a-6fb579d2de85:listen:/etc/. -> /home/kamil/tmp/etc2/\n```\n\nNomad does not have the capability of accessing any file inside the\nallocation file system. Instead, `nomad-cp` executes several `nomad exec`\ncalls to execute a `tar` pipe to stream the data from or to the allocation\ncontext to or from the local host using stdout and stdin forwarded by\n`nomad exec`.\n\n## `gitlab-runner` - dynamically schedule gitlab CI jobs\n\nAn implementation of custom Gitlab executor driver that runs Gitlab CI/CD jobs\nusing Nomad.\n\nThis program does _not_ run the `gitlab-runner` itself in Nomad. Rather, the\n`gitlab-runner` is running on (any) one host. That `gitlab-runner` will then\nschedule Nomad jobs to execute using the script as an executor. These jobs will\nexecute the CI/CD from Gitlab inside Nomad cluster.\n\nMore on it can be read on github wiki\n[https://github.com/Kamilcuk/nomad-tools/wiki/gitlab%E2%80%90runner](https://github.com/Kamilcuk/nomad-tools/wiki/gitlab%E2%80%90runner) .\n\n## `githubrunner` - dynamically schedule GitHub actions jobs\n\nSchedule Github actions runners as Nomad job for user repositories.\n\nSee documentation in [doc/githubrunner.md](doc/githubrunner.md) .\n\n## `dockers` - list docker images referenced by Nomad job\n\nLists docker images referenced in Nomad job file or a running Nomad job.\n\n```\n$ nomadtools dockers ./httpd.nomad.hcl\nbusybox:stable\n$ nomadtools dockers --job httpd\nbusybox:stable\n```\n\n## `downloadrelease` - download specific Nomad executable version\n\nProgram for downloading specific Nomad release binary from their release page.\nI use it for testing and checking new Nomad versions.\n\n```\n$ nomadtools downloadrelease nomad\nINFO:nomad_tools.nomad_downloadrelease:Downloading https://releases.hashicorp.com/nomad/1.7.3/nomad_1.7.3_linux_amd64.zip to nomad\nINFO:nomad_tools.nomad_downloadrelease:https://releases.hashicorp.com/nomad/1.7.3/nomad_1.7.3_linux_amd64.zip -> -rwxr-xr-x 105.7MB nomad\n$ nomadtools downloadrelease consul\nINFO:nomad_tools.nomad_downloadrelease:Downloading https://releases.hashicorp.com/consul/1.9.9/consul_1.9.9_linux_amd64.zip to consul\nINFO:nomad_tools.nomad_downloadrelease:https://releases.hashicorp.com/consul/1.9.9/consul_1.9.9_linux_amd64.zip -> -rwxr-xr-x 105.8MB consul\n$ nomadtools downloadrelease -p 1.6.3 nomad ./nomad1.6.3\nINFO:nomad_tools.nomad_downloadrelease:Downloading https://releases.hashicorp.com/nomad/1.6.3/nomad_1.6.3_linux_amd64.zip to nomad1.6.3\nINFO:nomad_tools.nomad_downloadrelease:https://releases.hashicorp.com/nomad/1.6.3/nomad_1.6.3_linux_amd64.zip -> -rwxr-xr-x 101.8MB nomad1.6.3\n```\n\n## `task` - find task and allocation and execute action on it\n\n`nomad alloc exec` didn't met my expectations. It is boring and tedious to find an allocation ID\nand then copy it in the terminal. There is also no bash completion for task names.\n\nI decided I want to have advanced filtering, where given a job name and node name some tool\nautomatically finds an allocation and task to copy from.\n\nGiven a system job promtail running on all machines, `nomadtools task -j promtail -n host1 exec bash -l`\nwill drop into a bash shell inside promtail job allocation running on host1.\n\nThis tool can be used to get a shell to the chosen allocation.\n\n### `task exec` - execute a command inside the allocation\n\nThis internally calls `nomad alloc exec`.\n\n```\n$ nomadtools task -j mail exec bash -l\nroot@main:/#\n```\n\n### `task json` - output found allocations and task names in json form\n\n### `task path` - output path in the form properly escaped for use with nomadtools cp\n\n```\n$ nomadtools cp \"$(nomadtools task -j mail path /etc/fstab)\" ./fstab\nINFO entry_cp.py:copy_mode:650: File :0c18e9aa-f053-cc3e-6fe3-3d23f159c2e5:mail:/etc/fstab -> ./fstab\n37.0 B 0:00:00 [73.3 B/s] [ <=>\n```\n\n### `task xargs` - output in the form -task <task> <allocid> that is usable with xargs nomad alloc\n\n```\n$ nomadtools task -j mail xargs -0 logs -- -stderr | xargs -0 nomad alloc\n$ nomadtools task -j mail xargs logs -- -stderr | xargs nomad alloc\n$ nomad alloc logs $(nomadtools task -j mail xargs) -stderr\n```\n\n## `nodenametoid` - convert node name to id\n\n```\n$ nomadtools nodenametoid node1\n3e50c2ef-16bd-0253-6635-1a55c25e74ca\n```\n\n## `info topology` - show some information about Nomad node usage\n\n```\n$ nomadtools info topology\ndc node1 ready 2000B/15988MB 1000MHz/12000MHz 1allocs [mariadb.mariadb[0] mariadb services 2000MB 1000MHz]\n```\n\n## import nomad_tools\n\nThis project is licensed under GPL. The internal API of this project can be\nused, however it is not stable at all and is an implementation detail.\n\nInternally, `nomad_tools.nomadlib` is a Python class definitions which\nrepresents models for Nomad API data documentation.\n\n# History\n\nThis module once installed bunch of separate tools, like `nomad-watch` or\n`nomad-gitlab-runner`. That became unmaintainable. It is one `nomadtools`\nexecutable with several sub-commands.\n\n# Contributing\n\nKindly make a issue or pull request on GitHub.\nI should be fast to respond and contributions are super welcome.\n\n## Running tests\n\nI want to support Python 3.7 with the project.\n\nTo test first install editable package locally with test dependencies:\n\n```\npip install -e '.[test]'\n```\n\nYou can run unit tests always without any external tools:\n\n```\n./unit_tests.sh\n```\n\nTo run integration tests, you have to be able to connect to Nomad server.\n\n```\n./integration_tests.sh\n./integration_tests.sh -k nomad_vardir\n```\n\n# License\n\nGPL\n\n",
"bugtrack_url": null,
"license": "GPL-3.0-or-later",
"summary": "Set of tools and utilities to ease interacting with Hashicorp Nomad scheduling solution.",
"version": "0.5.3",
"project_urls": {
"documentation": "https://github.com/Kamilcuk/nomad-tools",
"homepage": "https://github.com/Kamilcuk/nomad-tools",
"repository": "https://github.com/Kamilcuk/nomad-tools"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "a2c6d7e464f1c8e6439d039f9d4dc66d619f0180a7225348164d7a5604e5a442",
"md5": "38d1fda5eef14e3ae0bea7dfab06424a",
"sha256": "637f24b1227942294717c7c337cbcadc130db5826882ed2bea7f34ae94c78246"
},
"downloads": -1,
"filename": "nomad_tools-0.5.3-py3-none-any.whl",
"has_sig": false,
"md5_digest": "38d1fda5eef14e3ae0bea7dfab06424a",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">3.7",
"size": 145018,
"upload_time": "2024-11-11T12:14:52",
"upload_time_iso_8601": "2024-11-11T12:14:52.947597Z",
"url": "https://files.pythonhosted.org/packages/a2/c6/d7e464f1c8e6439d039f9d4dc66d619f0180a7225348164d7a5604e5a442/nomad_tools-0.5.3-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "dfdca3146bbe9efbc86c3a6dc915aab9cbea3f1ebd97d7b37c84255ed9c476b1",
"md5": "2ac4f425487e221f0542e4eecd04b235",
"sha256": "8ce29a239d90658479f162f8e71dd92695c6ebeeaf0d5f39ed82ee78ac0b8ba9"
},
"downloads": -1,
"filename": "nomad_tools-0.5.3.tar.gz",
"has_sig": false,
"md5_digest": "2ac4f425487e221f0542e4eecd04b235",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">3.7",
"size": 137856,
"upload_time": "2024-11-11T12:14:54",
"upload_time_iso_8601": "2024-11-11T12:14:54.876742Z",
"url": "https://files.pythonhosted.org/packages/df/dc/a3146bbe9efbc86c3a6dc915aab9cbea3f1ebd97d7b37c84255ed9c476b1/nomad_tools-0.5.3.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-11-11 12:14:54",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "Kamilcuk",
"github_project": "nomad-tools",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"requirements": [],
"tox": true,
"lcname": "nomad-tools"
}
