# batou_ext - a library of components for batou
`batou_ext` master is now supporting Python3 and is depending on batou2. If you still want to use batou_ext with batou 1.x running Python2 you still can use the [batou1-py2](https://github.com/flyingcircusio/batou_ext/tree/batou1-py2) branch.
To add `batou_ext` to your deployment, add a like to the `requirements.txt` of your batou deployment::
```
batou_ext>=2.4
```
## Development and release process
* Changes should be accompanied with a changelog entry. Use `./changelog.sh` to create one.
* Releasing will create a tag and publishes the package to pypi. Use `./release-this.sh` to create a release.
## Bootstrapping of S3 buckets
Only applicable for administrators of the Flying Circus.
Install the `s3-bootstrap` feature:
```
batou_ext[s3-bootstrap]>=2.4.6
```
Then run
```
./appenv update-lockfile
./appenv run s3_bootstrap
```
The script will interactively walk you through the creation of
creating an [S3 bucket](https://wiki.flyingcircus.io/S3) and - if needed -
an access keypair and lifecycle rules.
On an activated virtualenv this can be tested with `python -m batou_ext.s3_bootstrap`.
## 2.4.32 (2025-07-10)
- Fix UserEnv: Jinja2 variables in the final template were no longer substituted due to a regression.
- Drop Python 3.8 support to match reality since 2.4.31.
## 2.4.31 (2025-07-07)
- Remove deprecated pkg_resources and replace with importlib.resources
- Fixed compatibility of `batou_ext.oci` with `podman` backend and NixOS 25.05.
## 2.4.30 (2025-05-14)
- Added `batou_ext.fcio.Maintenance{Start,End}`: with these components it's possible to mark the RG
as "in maintenance". Components can be scheduled between these two by doing
`self.provide("needs-maintenance", self)`.
- The components `batou_ext.fcio.{MaintenanceStart,MaintenanceEnd,DirectoryXMLRPC}` were added to
put a resource group into maintenance for a deployment. Please read the doc comment
of `batou_ext.fcio.MaintenanceStart` for further reference.
- Fix S3 integration to be compatible with custom S3 services.
## 2.4.29 (2025-04-29)
- oci/podman: don't set `PrivateTmp` in the systemd unit anymore. This has the opposite effect of what's intended, it breaks rootless containers.
## 2.4.28 (2025-03-31)
- oci: make `user@uid.service` wait until all containers have exited and clear `/tmp` of containers.
Without those, unclean shutdowns were observed that prevented the containers from getting back
up on a reboot.
## 2.4.27 (2025-03-20)
- Add a new parameter for `nix.UserEnv` components to ignore collisions between file names from different derivations or outputs of the same derivation
## 2.4.26 (2025-02-28)
- `batou_ext.oci.Container`: set `backend` explicitly in Nix expression.
Otherwise this depends on the state version having varying results depending on whether
the machine was installed with a NixOS older or newer than 22.05.
## 2.4.25 (2025-02-21)
- `batou_ext.oci.Container`: allow to use `podman` as backend instead of `docker`.
This also enables the following features:
* Rootless containers: by setting the `user` option to a different user. By default,
the service user of the deployment is used.
* Only mark services as `active` if the container is up. This requires that the
container has a healthcheck. Alternatively, a healthcheck can be configured
with the health_cmd attribute.
## 2.4.24 (2025-02-17)
- `batou_ext.file.SymlinkAndCleanup`: add option `etag_suffix`.
This contains a suffix that each symlinked file may have.
For instance, when doing `SymlinkAndCleanup` on a file downloaded with
`batou_ext.s3.Download`, the pattern `*.tar.gz` doesn't clean up the
`.etag` files. However, `*.tar.gz*` (or an equivalent) would also remove
the etag files of the files that are symlinked to `current` & `last`.
## 2.4.23 (2025-01-07)
- Fix using multiple DeploymentTrash Component on a single host breaking the Nix rebuild, especially across different deployments to the same machine.
This was because the values for the IOPS read and write Limits in the Systemd serviceConfig attribute set were defined as strings (which cannot be merged unless identical) instead of lists (which can always be merged).
## 2.4.22 (2024-12-27)
- Allow to configure the name of the `.nix` file created by `batou_ext.file.DeploymentTrash`.
## 2.4.21 (2024-12-13)
- Fix interactive version select.
- Change the releaser defaults to actually release
## 2.4.20 (2024-12-10)
- Correctness fix for `jenkins set-version`: if a tag is resolved, make sure it's _always_
resolved to the rev of the tagged commit (instead of the tag's rev) or fail hard to avoid
incorrect revs.
- Improve documentation of `batou_ext.file.DeploymentTrash`
## 2.4.19 (2024-11-28)
- make the DeploymentTrash's trash directory configurable and output a potentially helpful message on OSErrors which could indicate that the trash directory and the directory that is being trashed are on different devices
## 2.4.18 (2024-11-15)
- nixos.NixOSModule: Mark generated context file as sensitive (Fixes #167)
- add an env argument for the `Run` component to support running commands with specific environment variables
- The component `batou_ext.python.FixELFRunPath` now uses a patched version of patchelf to make sure that the
dynamic libraries don't get larger per deploy.
When a certain threshold is exceeded, Python will fail to import these.
If the component got regularly executed in deployments, you may want to consider recreating
the virtualenv once.
## 2.4.17 (2024-10-28)
- A new component `batou_ext.systemd.ScalableService` has been added. This provides configurations
for a service that can exist multiple times (e.g. queue consumers). Detailed usage instructions
and further information can be found in the component's docstring.
## 2.4.16 (2024-09-11)
- A new component `batou_ext.mail.Mailpit` has been added.
Mailpit is an alternative for Mailhog which is not maintained anymore.
- fix a mysterious regression that cause a test to fail
- redis.Redis: Allow to set provide name
* The `SymlinkAndCleanup` internally uses the `DeploymentTrash` component internally which
deletes old code using `systemd-tmpfiles` and throttles the operation with `IOReadIOPSMax`
and `IOWriteIOPSMax`.
This didn't have any effect before because these settings were wrongly placed in `[Unit]`
instead of `[Service]`.
## 2.4.15 (2024-08-26)
- fix a regression in the versions update script where the environment was not loaded correctly
## 2.4.14 (2024-08-12)
- `ssl.Certificate`: Set proper ACL for non-let's encrypt certificates.
## 2.4.13 (2024-08-12)
- Fix setting acl for `ssl.Certificate` during ceritificate renewal.
## 2.4.12 (2024-08-12)
- Set correct acl for `ssl.Certificates` on certificate renew.
## 2.4.11 (2024-08-09)
- `oci.Container`: Add option to disable OCI container monitoring.
This is mainly useful for containers which are not running all the time.
- `oci.Container`: make rebuild optional
This is useful, when there are multiple container deployed which should be activated at once.
- Fix a bug in the version update script where multiple environments sharing the same branch would not be updated correctly
- the `SymlinkAndCleanup` component was adjusted to clean up asynchronously using systemd's tmpfiles instead of deleting all candidates immediately
## 2.4.10 (2024-06-11)
- oci.Container: Fix a bug where containers were not restarted properly even though their image digest was out of sync after the remote tag has been updated
- oci.Container: Fix a typo in the oci container component's verify method
## 2.4.9 (2024-06-04)
- `batou_ext.python.FixELFRunPath`: search not only `env_directory`, but also its subdirs for C libraries needed by the libraries to patch.
- Fix `PurgePackage` raising error when package is not found.
- The attribute `public_smtp_name` of `batou_ext.mail.Mailhog` now has a default value. It points to `self.host.fqdn`.
- adjust the certificate expiry check output to be more easily parseable
## 2.4.8 (2024-05-08)
- systemd timers: add an option to enable persistence
breaking change: systemd timers are now non-persistent by default.
The previous default behaviour was a problem for cronjobs that should
not be started immediately following a reboot / downtime
* Added a component `batou_ext.python.FixELFRunPath` which modifies `DT_RUNPATH` & `DT_RPATH` of `.so`-files in a venv to load the correct libraries (from either a Nix env or other Python libraries). Please read the docstring carefully before using it.
- OCI: cache validation result during deployment.
Caching results speeds up deployments where multiple containers with the same image are deployed.
## 2.4.7 (2024-04-29)
* Added component `batou_ext.http.HTTPServiceWatchdog` that adds a check to a systemd unit
whether a given URL is reachable (e.g. a `/health` endpoint). If the URL cannot be reached within
a certain interval, the service will be restarted. Further details are documented in the
docstring.
- Fix `SymlinkAndCleanup` async delete and allow custom extra arguments to `systemd run`.
## 2.4.6 (2024-04-23)
- OCI: Support registries where the docker login is different than the registry used in referencing containers.
- OCI: Improve change detection of remote images (required for docker.io)
- OCI: The nix file does not contain sensitive data, so don’t mark it as such.
- OCI: add support for extraOptions
* Added a script `s3_bootstrap` that interactively creates an S3 bucket (including a radosgw account & keys if needed). Will be installed with `batou_ext` if the `s3-bootstrap` extra is requested.
## 2.4.5 (2024-04-17)
- add an option to move mailhog log output (`stdout` + `stderr`) to a different namespace, e.g. "mailhog". see systemd.exec(5) for more information
- add an option to disable `stdout` logging for the mailhog service
- improve dectection of a versions file for versions updates
- fix the oci.Container verify method not throwing an updaterequired on changes to the docker container's environment file
- Add systemd-run async cleanup option for SymlinkAndCleanup removals
## 2.4.4 (2024-04-05)
- Change the behaviour of the batou_ext.versions updater to allow environments to share a branch
* Added a component `batou_ext.git.Remote` which allows to manipulate remotes of a git repository.
## 2.4.3 (2024-01-17)
- Improve output handling for the `PurgePackage` component. Will not appear like a fatal error in logs anymore when the package has been purged already or is not installed for another reason
## 2.4.2 (2023-12-08)
* Make it possible to add arbitrary additional configuration to a service created by a `SystemdTimer()`.
* Add `nixos.NixOSModule` to inject component attributes into .nix files.
## 2.4.1 (2023-11-16)
* Add `*.md` to the release, so it can actually be used.
## 2.4.0 (2023-11-16)
- Add release process with changelog (FC-33250).
Raw data
{
"_id": null,
"home_page": "https://github.com/flyingcircusio/batou_ext",
"name": "batou-ext",
"maintainer": null,
"docs_url": null,
"requires_python": null,
"maintainer_email": null,
"keywords": "deployment",
"author": "Flying Circus <support@flyingcircus.io>",
"author_email": "support@flyingcircus.io",
"download_url": "https://files.pythonhosted.org/packages/16/e1/c0450f3e9cb87e3e98200e8869cf5a56a5fa5317b3ee86f8780ab2349f13/batou_ext-2.4.32.tar.gz",
"platform": null,
"description": "# batou_ext - a library of components for batou\n\n`batou_ext` master is now supporting Python3 and is depending on batou2. If you still want to use batou_ext with batou 1.x running Python2 you still can use the [batou1-py2](https://github.com/flyingcircusio/batou_ext/tree/batou1-py2) branch.\n\nTo add `batou_ext` to your deployment, add a like to the `requirements.txt` of your batou deployment::\n\n```\nbatou_ext>=2.4\n```\n\n## Development and release process\n\n* Changes should be accompanied with a changelog entry. Use `./changelog.sh` to create one.\n\n* Releasing will create a tag and publishes the package to pypi. Use `./release-this.sh` to create a release.\n\n## Bootstrapping of S3 buckets\n\nOnly applicable for administrators of the Flying Circus.\n\nInstall the `s3-bootstrap` feature:\n\n```\nbatou_ext[s3-bootstrap]>=2.4.6\n```\n\nThen run\n\n```\n./appenv update-lockfile\n./appenv run s3_bootstrap\n```\n\nThe script will interactively walk you through the creation of\ncreating an [S3 bucket](https://wiki.flyingcircus.io/S3) and - if needed -\nan access keypair and lifecycle rules.\n\nOn an activated virtualenv this can be tested with `python -m batou_ext.s3_bootstrap`.\n\n\n\n## 2.4.32 (2025-07-10)\n\n\n- Fix UserEnv: Jinja2 variables in the final template were no longer substituted due to a regression.\n\n- Drop Python 3.8 support to match reality since 2.4.31.\n\n\n## 2.4.31 (2025-07-07)\n\n\n- Remove deprecated pkg_resources and replace with importlib.resources\n\n- Fixed compatibility of `batou_ext.oci` with `podman` backend and NixOS 25.05.\n\n\n## 2.4.30 (2025-05-14)\n\n\n- Added `batou_ext.fcio.Maintenance{Start,End}`: with these components it's possible to mark the RG\n as \"in maintenance\". Components can be scheduled between these two by doing\n `self.provide(\"needs-maintenance\", self)`.\n\n- The components `batou_ext.fcio.{MaintenanceStart,MaintenanceEnd,DirectoryXMLRPC}` were added to\n put a resource group into maintenance for a deployment. Please read the doc comment\n of `batou_ext.fcio.MaintenanceStart` for further reference.\n\n- Fix S3 integration to be compatible with custom S3 services.\n\n\n## 2.4.29 (2025-04-29)\n\n\n- oci/podman: don't set `PrivateTmp` in the systemd unit anymore. This has the opposite effect of what's intended, it breaks rootless containers.\n\n\n## 2.4.28 (2025-03-31)\n\n\n- oci: make `user@uid.service` wait until all containers have exited and clear `/tmp` of containers.\n Without those, unclean shutdowns were observed that prevented the containers from getting back\n up on a reboot.\n\n\n## 2.4.27 (2025-03-20)\n\n\n- Add a new parameter for `nix.UserEnv` components to ignore collisions between file names from different derivations or outputs of the same derivation\n\n\n## 2.4.26 (2025-02-28)\n\n\n- `batou_ext.oci.Container`: set `backend` explicitly in Nix expression.\n\n Otherwise this depends on the state version having varying results depending on whether\n the machine was installed with a NixOS older or newer than 22.05.\n\n\n## 2.4.25 (2025-02-21)\n\n\n- `batou_ext.oci.Container`: allow to use `podman` as backend instead of `docker`.\n This also enables the following features:\n\n * Rootless containers: by setting the `user` option to a different user. By default,\n the service user of the deployment is used.\n\n * Only mark services as `active` if the container is up. This requires that the\n container has a healthcheck. Alternatively, a healthcheck can be configured\n with the health_cmd attribute.\n\n\n## 2.4.24 (2025-02-17)\n\n\n- `batou_ext.file.SymlinkAndCleanup`: add option `etag_suffix`.\n This contains a suffix that each symlinked file may have.\n\n For instance, when doing `SymlinkAndCleanup` on a file downloaded with\n `batou_ext.s3.Download`, the pattern `*.tar.gz` doesn't clean up the\n `.etag` files. However, `*.tar.gz*` (or an equivalent) would also remove\n the etag files of the files that are symlinked to `current` & `last`.\n\n\n## 2.4.23 (2025-01-07)\n\n\n- Fix using multiple DeploymentTrash Component on a single host breaking the Nix rebuild, especially across different deployments to the same machine.\n This was because the values for the IOPS read and write Limits in the Systemd serviceConfig attribute set were defined as strings (which cannot be merged unless identical) instead of lists (which can always be merged).\n\n\n## 2.4.22 (2024-12-27)\n\n\n- Allow to configure the name of the `.nix` file created by `batou_ext.file.DeploymentTrash`.\n\n\n## 2.4.21 (2024-12-13)\n\n\n- Fix interactive version select.\n\n- Change the releaser defaults to actually release\n\n\n## 2.4.20 (2024-12-10)\n\n\n- Correctness fix for `jenkins set-version`: if a tag is resolved, make sure it's _always_\n resolved to the rev of the tagged commit (instead of the tag's rev) or fail hard to avoid\n incorrect revs.\n\n- Improve documentation of `batou_ext.file.DeploymentTrash`\n\n\n## 2.4.19 (2024-11-28)\n\n\n- make the DeploymentTrash's trash directory configurable and output a potentially helpful message on OSErrors which could indicate that the trash directory and the directory that is being trashed are on different devices\n\n\n## 2.4.18 (2024-11-15)\n\n\n- nixos.NixOSModule: Mark generated context file as sensitive (Fixes #167)\n\n- add an env argument for the `Run` component to support running commands with specific environment variables\n\n- The component `batou_ext.python.FixELFRunPath` now uses a patched version of patchelf to make sure that the\n dynamic libraries don't get larger per deploy.\n\n When a certain threshold is exceeded, Python will fail to import these.\n\n If the component got regularly executed in deployments, you may want to consider recreating\n the virtualenv once.\n\n\n## 2.4.17 (2024-10-28)\n\n\n- A new component `batou_ext.systemd.ScalableService` has been added. This provides configurations\n for a service that can exist multiple times (e.g. queue consumers). Detailed usage instructions\n and further information can be found in the component's docstring.\n\n\n## 2.4.16 (2024-09-11)\n\n\n- A new component `batou_ext.mail.Mailpit` has been added.\n Mailpit is an alternative for Mailhog which is not maintained anymore.\n\n- fix a mysterious regression that cause a test to fail\n\n- redis.Redis: Allow to set provide name\n\n* The `SymlinkAndCleanup` internally uses the `DeploymentTrash` component internally which\n deletes old code using `systemd-tmpfiles` and throttles the operation with `IOReadIOPSMax`\n and `IOWriteIOPSMax`.\n\n This didn't have any effect before because these settings were wrongly placed in `[Unit]`\n instead of `[Service]`.\n\n\n## 2.4.15 (2024-08-26)\n\n\n- fix a regression in the versions update script where the environment was not loaded correctly\n\n\n## 2.4.14 (2024-08-12)\n\n\n- `ssl.Certificate`: Set proper ACL for non-let's encrypt certificates.\n\n\n## 2.4.13 (2024-08-12)\n\n\n- Fix setting acl for `ssl.Certificate` during ceritificate renewal.\n\n\n## 2.4.12 (2024-08-12)\n\n\n- Set correct acl for `ssl.Certificates` on certificate renew.\n\n\n## 2.4.11 (2024-08-09)\n\n\n- `oci.Container`: Add option to disable OCI container monitoring.\n\n This is mainly useful for containers which are not running all the time.\n\n- `oci.Container`: make rebuild optional\n\n This is useful, when there are multiple container deployed which should be activated at once.\n\n- Fix a bug in the version update script where multiple environments sharing the same branch would not be updated correctly\n\n- the `SymlinkAndCleanup` component was adjusted to clean up asynchronously using systemd's tmpfiles instead of deleting all candidates immediately\n\n\n## 2.4.10 (2024-06-11)\n\n\n- oci.Container: Fix a bug where containers were not restarted properly even though their image digest was out of sync after the remote tag has been updated\n\n- oci.Container: Fix a typo in the oci container component's verify method\n\n\n## 2.4.9 (2024-06-04)\n\n\n- `batou_ext.python.FixELFRunPath`: search not only `env_directory`, but also its subdirs for C libraries needed by the libraries to patch.\n\n- Fix `PurgePackage` raising error when package is not found.\n\n- The attribute `public_smtp_name` of `batou_ext.mail.Mailhog` now has a default value. It points to `self.host.fqdn`.\n\n- adjust the certificate expiry check output to be more easily parseable\n\n\n## 2.4.8 (2024-05-08)\n\n\n- systemd timers: add an option to enable persistence\n breaking change: systemd timers are now non-persistent by default.\n The previous default behaviour was a problem for cronjobs that should\n not be started immediately following a reboot / downtime\n\n* Added a component `batou_ext.python.FixELFRunPath` which modifies `DT_RUNPATH` & `DT_RPATH` of `.so`-files in a venv to load the correct libraries (from either a Nix env or other Python libraries). Please read the docstring carefully before using it.\n\n- OCI: cache validation result during deployment.\n\n Caching results speeds up deployments where multiple containers with the same image are deployed.\n\n\n## 2.4.7 (2024-04-29)\n\n\n* Added component `batou_ext.http.HTTPServiceWatchdog` that adds a check to a systemd unit\n whether a given URL is reachable (e.g. a `/health` endpoint). If the URL cannot be reached within\n a certain interval, the service will be restarted. Further details are documented in the\n docstring.\n\n- Fix `SymlinkAndCleanup` async delete and allow custom extra arguments to `systemd run`.\n\n\n## 2.4.6 (2024-04-23)\n\n\n- OCI: Support registries where the docker login is different than the registry used in referencing containers.\n\n- OCI: Improve change detection of remote images (required for docker.io)\n\n- OCI: The nix file does not contain sensitive data, so don\u2019t mark it as such.\n\n- OCI: add support for extraOptions\n\n* Added a script `s3_bootstrap` that interactively creates an S3 bucket (including a radosgw account & keys if needed). Will be installed with `batou_ext` if the `s3-bootstrap` extra is requested.\n\n\n## 2.4.5 (2024-04-17)\n\n\n- add an option to move mailhog log output (`stdout` + `stderr`) to a different namespace, e.g. \"mailhog\". see systemd.exec(5) for more information\n\n- add an option to disable `stdout` logging for the mailhog service\n\n- improve dectection of a versions file for versions updates\n\n- fix the oci.Container verify method not throwing an updaterequired on changes to the docker container's environment file\n\n- Add systemd-run async cleanup option for SymlinkAndCleanup removals\n\n\n## 2.4.4 (2024-04-05)\n\n\n- Change the behaviour of the batou_ext.versions updater to allow environments to share a branch\n\n* Added a component `batou_ext.git.Remote` which allows to manipulate remotes of a git repository.\n\n\n## 2.4.3 (2024-01-17)\n\n\n- Improve output handling for the `PurgePackage` component. Will not appear like a fatal error in logs anymore when the package has been purged already or is not installed for another reason\n\n\n## 2.4.2 (2023-12-08)\n\n\n* Make it possible to add arbitrary additional configuration to a service created by a `SystemdTimer()`.\n\n* Add `nixos.NixOSModule` to inject component attributes into .nix files.\n\n\n## 2.4.1 (2023-11-16)\n\n\n* Add `*.md` to the release, so it can actually be used.\n\n\n## 2.4.0 (2023-11-16)\n\n- Add release process with changelog (FC-33250).\n",
"bugtrack_url": null,
"license": "BSD (2-clause)",
"summary": "A library of components for batou.",
"version": "2.4.32",
"project_urls": {
"Homepage": "https://github.com/flyingcircusio/batou_ext"
},
"split_keywords": [
"deployment"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "16e1c0450f3e9cb87e3e98200e8869cf5a56a5fa5317b3ee86f8780ab2349f13",
"md5": "b3041885a29363eba4050bb4b5ae3dda",
"sha256": "f7407edb986eacef5a12cde1ffa6e2a0b7ec814f4d00790686d17ccee360fd20"
},
"downloads": -1,
"filename": "batou_ext-2.4.32.tar.gz",
"has_sig": false,
"md5_digest": "b3041885a29363eba4050bb4b5ae3dda",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 102587,
"upload_time": "2025-07-10T07:41:29",
"upload_time_iso_8601": "2025-07-10T07:41:29.303741Z",
"url": "https://files.pythonhosted.org/packages/16/e1/c0450f3e9cb87e3e98200e8869cf5a56a5fa5317b3ee86f8780ab2349f13/batou_ext-2.4.32.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-07-10 07:41:29",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "flyingcircusio",
"github_project": "batou_ext",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"tox": true,
"lcname": "batou-ext"
}
JSON
