# Artifactory cleanup #
`artifactory-cleanup` is an extended and flexible cleanup tool for JFrog Artifactory.
The tool has simple YAML-defined cleanup configuration and can be extended with your own rules on Python.
Everything must be as a code, even cleanup policies!
# Tables of Contents
<!-- toc -->
- [Artifactory cleanup](#artifactory-cleanup)
- [Tables of Contents](#tables-of-contents)
- [Installation](#installation)
- [Usage](#usage)
- [Notes](#notes)
- [Commands](#commands)
- [Rules](#rules)
- [Common](#common)
- [Delete](#delete)
- [Keep](#keep)
- [Docker](#docker)
- [Filters](#filters)
- [Create your own rule](#create-your-own-rule)
- [How to](#how-to)
- [How to connect self-signed certificates for docker?](#how-to-connect-self-signed-certificates-for-docker)
- [How to clean up Conan repository?](#how-to-clean-up-conan-repository)
- [How to keep latest N docker images?](#how-to-keep-latest-n-docker-images)
- [Release](#release)
<!-- tocstop -->
# Installation
As simple as one command!
```bash
# docker
docker pull devopshq/artifactory-cleanup
docker run --rm devopshq/artifactory-cleanup artifactory-cleanup --help
# python (later we call it 'cli')
python3 -mpip install artifactory-cleanup
artifactory-cleanup --help
```
# Usage
Suppose you want to remove **all artifacts older than N days** from **reponame** repository.
You should take the following steps:
1. Install `artifactory-cleanup` (see above)
2. Create a configuration file `artifactory-cleanup.yaml`. variables.
```yaml
# artifactory-cleanup.yaml
artifactory-cleanup:
server: https://repo.example.com/artifactory
# $VAR is auto populated from environment variables
user: $ARTIFACTORY_USERNAME
password: $ARTIFACTORY_PASSWORD
policies:
- name: Remove all files from repo-name-here older than 7 days
rules:
- rule: Repo
name: "reponame"
- rule: DeleteOlderThan
days: 7
```
3. Run the command **TO SHOW** (not remove) artifacts that will be deleted. By default `artifactory-cleanup` uses "dry
mode".
```bash
# Set the credentials with delete permissions
export ARTIFACTORY_USERNAME=usernamehere
export ARTIFACTORY_PASSWORD=password
# docker
docker run --rm -v "$(pwd)":/app -e ARTIFACTORY_USERNAME -e ARTIFACTORY_PASSWORD devopshq/artifactory-cleanup artifactory-cleanup
# cli
artifactory-cleanup
```
4. Verify that right artifacts will be removed and add `--destroy` flag **TO REMOVE** artifacts:
```bash
# docker
docker run --rm -v "$(pwd)":/app -e ARTIFACTORY_USERNAME -e ARTIFACTORY_PASSWORD devopshq/artifactory-cleanup artifactory-cleanup --destroy
# cli
artifactory-cleanup --destroy
```
Looking for more examples? Check [examples](./examples) folder!
## Notes
- **Always** specify version of `artifactory-cleanup` when using it in the production. `1.0.0` is just an example, find the latest version in pypi: https://pypi.org/project/artifactory-cleanup/
```bash
# docker
docker pull devopshq/artifactory-cleanup:1.0.0
docker run --rm devopshq/artifactory-cleanup:1.0.0 artifactory-cleanup --version
# python (later we call it 'cli')
python3 -mpip install artifactory-cleanup==1.0.0
artifactory-cleanup --help
```
- Use CI servers or cron-like utilities to run `artifactory-cleanup` every day (or every hour). TeamCity and GitHub have
built-in support and show additional logs format
- Do not save credentials in the configuration file, use environment variables.
- Use `--ignore-not-found` flag to ignore errors when the repository is not found. It's useful when you have a
configuration for multiple repositories and some of them are not found.
- Use `--worker-count=<WORKER_NUM>` to increase the number of workers. By default, it's 1. It's useful when you have a lot of
artifacts and you want to speed up the process.
## Commands ##
```bash
# Debug - "dry run" mode by default
# debug run - only print artifacts. it does not delete any artifacts
artifactory-cleanup
# Debug run only for policytestname.
artifactory-cleanup --policy-name policytestname
# REMOVE
# For remove artifacts use --destroy
artifactory-cleanup --destroy
# For remove artifacts use environment variable
export ARTIFACTORY_CLEANUP_DESTROY=True
artifactory-cleanup
# Specify config filename
artifactory-cleanup --config artifactory-cleanup.yaml
# Specify config filename using environment variable
export ARTIFACTORY_CLEANUP_CONFIG_FILE=artifactory-cleanup.yaml
artifactory-cleanup --config artifactory-cleanup.yaml
# Look in the future - shows what the tool WILL remove after 10 days
artifactory-cleanup --days-in-future=10
# Not satisfied with built-in rules? Write your own rules in python and connect them!
artifactory-cleanup --load-rules=myrule.py
docker run -v "$(pwd)":/app devopshq/artifactory-cleanup artifactory-cleanup --load-rules=myrule.py
# Save the table summary in a file
artifactory-cleanup --output=myfile.txt
# Save the summary in a Json file
artifactory-cleanup --output=myfile.txt --output-format=json
# Save the summary in a json file and append a list with removed artifacts
artifactory-cleanup --output=myfile.json --output-format json-with-artifact-list
```
# Rules
## Common
- `Repo` - Apply the rule to one repository. If no name is specified, it is taken from the rule name (in `CleanupPolicy`
definition)
```yaml
- rule: Repo
name: reponame
```
```yaml
# OR - if you have a single policy for the repo - you can name the policy as reponame
# Both configurations are equal
policies:
- name: reponame
rules:
- rule: Repo
```
- `RepoList` - Apply the policy to list of repositories.
```yaml
- rule: RepoList
repos:
- repo1
- repo2
- repo3
```
- `RepoByMask` - Apply rule to repositories matching by mask
```yaml
- rule: RepoByMask
mask: "*.banned"
```
- `PropertyEq`- Delete repository artifacts only with a specific property value (property_key is the name of the
parameter, property_value is the value)
```yaml
- rule: PropertyEq
property_key: key-name
property_value: 1
```
- `PropertyNeq`- Delete repository artifacts only if the value != specified. If there is no value, delete it anyway.
Allows you to specify the deletion flag `do_not_delete = 1`
```yaml
- rule: PropertyNeq
property_key: key-name
property_value: 1
```
## Delete
- `DeleteOlderThan` - deletes artifacts that are older than N days
```yaml
- rule: DeleteOlderThan
days: 1
```
- `DeleteWithoutDownloads` - deletes artifacts that have never been downloaded (DownloadCount=0). Better to use
with `DeleteOlderThan` rule
```yaml
- rule: DeleteWithoutDownloads
```
- `DeleteOlderThanNDaysWithoutDownloads` - deletes artifacts that are older than N days and have not been
downloaded
```yaml
- rule: DeleteOlderThanNDaysWithoutDownloads
days: 1
```
- `DeleteNotUsedSince` - delete artifacts that were downloaded, but for a long time. N days passed. Or not
downloaded at all from the moment of creation and it's been N days
```yaml
- rule: DeleteNotUsedSince
days: 1
```
- `DeleteEmptyFolders` - Clean up empty folders in given repository list
```yaml
- rule: DeleteEmptyFolders
```
- `DeleteByRegexpName` - delete artifacts whose name matches the specified regexp
```yaml
- rule: DeleteByRegexpName
regex_pattern: "\d"
```
## Keep
- `KeepLatestNFiles` - Leaves the last (by creation time) files in the amount of N pieces. WITHOUT accounting
subfolders
```yaml
- rule: KeepLatestNFiles
count: 1
```
- `KeepLatestNFilesInFolder` - Leaves the last (by creation time) files in the number of N pieces in each
folder
```yaml
- rule: KeepLatestNFilesInFolder
count: 1
```
- `KeepLatestVersionNFilesInFolder` - Leaves the latest N (by version) files in each
folder. The definition of the version is using regexp. By default it parses [semver](https://semver.org/) using the regex - `([\d]+\.[\d]+\.[\d]+)")`
```yaml
- rule: KeepLatestVersionNFilesInFolder
count: 1
custom_regexp: "[^\\d][\\._]((\\d+\\.)+\\d+)"
```
- `KeepLatestNupkgNVersions` - Leaves N nupkg (adds `*.nupkg` filter) in release feature builds
```yaml
- rule: KeepLatestNupkgNVersions
count: 1
```
## Docker
- `DeleteDockerImagesOlderThan` - Delete docker images that are older than N days
```yaml
- rule: DeleteDockerImagesOlderThan
days: 1
```
- `DeleteDockerImagesOlderThanNDaysWithoutDownloads` - Deletes docker images that are older than N days and have
not been downloaded
```yaml
- rule: DeleteDockerImagesOlderThanNDaysWithoutDownloads
days: 1
```
- `DeleteDockerImagesNotUsed` - Removes Docker image not downloaded since N days
```yaml
- rule: DeleteDockerImagesNotUsed
days: 1
```
- `IncludeDockerImages` - Apply to docker images with the specified names and tags
```yaml
- rule: IncludeDockerImages
masks: "*singlemask*"
- rule: IncludeDockerImages
masks:
- "*production*"
- "*release*"
```
- `ExcludeDockerImages` - Exclude Docker images by name and tags.
```yaml
- rule: ExcludeDockerImages
masks:
- "*production*"
- "*release*"
```
- `KeepLatestNVersionImagesByProperty(count=N, custom_regexp='some-regexp', number_of_digits_in_version=X)` - Leaves N
Docker images with the same major. `(^\d+\.\d+\.\d+$)` is the default regexp how to determine version which matches semver `1.1.1`. If you
need to add minor then set `number_of_digits_in_version` to 2 or if patch then set to 3 (by default we match major, which 1). Semver tags
prefixed with `v` are supported by updating the regexp to include (an optional) `v` in the expression (e.g., `(^v?\d+\.\d+\.\d+$)`).
```yaml
- rule: KeepLatestNVersionImagesByProperty
count: 1
custom_regexp: "[^\\d][\\._]((\\d+\\.)+\\d+)"
```
- `DeleteDockerImageIfNotContainedInProperties(docker_repo='docker-local', properties_prefix='my-prop', image_prefix=None, full_docker_repo_name=None)`
\- Remove Docker image, if it is not found in the properties of the artifact repository.
- `DeleteDockerImageIfNotContainedInPropertiesValue(docker_repo='docker-local', properties_prefix='my-prop', image_prefix=None, full_docker_repo_name=None)`
\- Remove Docker image, if it is not found in the properties of the artifact repository.
## Filters
- `IncludePath` - Apply to artifacts by path / mask.
```yaml
- rule: IncludePath
masks: "*production*"
- rule: IncludePath
masks:
- "*production*"
- "*develop*"
```
- `IncludeFilename` - Apply to artifacts by name/mask
```yaml
- rule: IncludeFilename
masks:
- "*production*"
- "*develop*"
```
- `ExcludePath` - Exclude artifacts by path/mask
```yaml
- rule: ExcludePath
masks:
- "*production*"
- "*develop*"
```
- `ExcludeFilename` - Exclude artifacts by name/mask
```yaml
- rule: ExcludeFilename
masks:
- "*.tag.gz"
- "*.zip"
```
## Create your own rule
If you want to create your own rule, you can do it!
The basic flow how the tool calls Rules:
1. `Rule.check(*args, **kwargs)` - verify that the Rule configured right. Call other services to get more information.
2. `Rule.aql_add_filter(filters)` - add Artifactory Query Language expressions
3. `Rule.aql_add_text(aql)` - add text to the result aql query
4. `artifactory-cleanup` calls Artifactory with AQL and pass the result to the next step
5. `Rule.filter(artifacts)` - filter out artifacts. The method returns **artifacts that will be removed!**.
- To keep artifacts use `artifacts.keep(artifact)` method
Create `myrule.py` file at the same folder as `artifactory-cleanup.yaml`:
```python
# myrule.py
from typing import List
from artifactory_cleanup import register
from artifactory_cleanup.rules import Rule, ArtifactsList
class MySimpleRule(Rule):
"""
This doc string is used as rule title
For more methods look at Rule source code
"""
def __init__(self, my_param: str, value: int):
self.my_param = my_param
self.value = value
def aql_add_filter(self, filters: List) -> List:
print(f"Today is {self.today}")
print(self.my_param)
print(self.value)
return filters
def filter(self, artifacts: ArtifactsList) -> ArtifactsList:
"""I'm here just to print the list"""
print(self.my_param)
print(self.value)
# You can make requests to artifactory by using self.session:
# url = f"/api/storage/{self.repo}"
# r = self.session.get(url)
# r.raise_for_status()
return artifacts
# Register your rule in the system
register(MySimpleRule)
```
Use `rule: MySimpleRule` in configuration:
```yaml
# artifactory-cleanup.yaml
- rule: MySimpleRule
my_param: "Hello, world!"
value: 42
```
Specify `--load-rules` to the command:
```bash
# docker
docker run -v "$(pwd)":/app devopshq/artifactory-cleanup artifactory-cleanup --load-rules=myrule.py
# cli
artifactory-cleanup --load-rules=myrule.py
```
# How to
## How to connect self-signed certificates for docker?
In case you have set up your Artifactory self-signed certificates, place all certificates of the chain of trust into
the `certificates` folder and add additional argument to the command:
```bash
docker run -v "$(pwd)":/app -v "$(pwd)/certificates":/mnt/self-signed-certs/ devopshq/artifactory-cleanup artifactory-cleanup
```
## How to clean up Conan repository?
We can handle conan's metadata by creating two policies:
1. First one removes files but keep all metadata.
2. Second one look at folders and if it contains only medata files - removes it (because there's no associated with
metadata files)
The idea came from https://github.com/devopshq/artifactory-cleanup/issues/47
```yaml
# artifactory-cleanup.yaml
artifactory-cleanup:
server: https://repo.example.com/artifactory
user: $ARTIFACTORY_USERNAME
password: $ARTIFACTORY_PASSWORD
policies:
- name: Conan - delete files older than 60 days
rules:
- rule: Repo
name: "conan-testing"
- rule: DeleteNotUsedSince
days: 60
- rule: ExcludeFilename
masks:
- ".timestamp"
- "index.json"
- name: Conan - delete empty folders (to fix the index)
rules:
- rule: Repo
name: "conan-testing"
- rule: DeleteEmptyFolders
- rule: ExcludeFilename
masks:
- ".timestamp"
- "index.json"
```
## How to keep latest N docker images?
We can combine docker rules with usual "files" rules!
The idea came from https://github.com/devopshq/artifactory-cleanup/issues/61
```yaml
# artifactory-cleanup.yaml
artifactory-cleanup:
server: https://repo.example.com/artifactory
user: $ARTIFACTORY_USERNAME
password: $ARTIFACTORY_PASSWORD
policies:
- name: Remove docker images, but keep last 3
rules:
# Select repo
- rule: Repo
name: docker-demo
# Delete docker images older than 30 days
- rule: DeleteDockerImagesOlderThan
days: 30
# Keep these tags for all images
- rule: ExcludeDockerImages
masks:
- "*:latest"
- "*:release*"
# Exclude these docker tags
- rule: ExcludePath
masks: "*base-tools*"
# Keep 3 docker tags for all images
- rule: KeepLatestNFilesInFolder
count: 3
```
# Release
In order to provide a new release of `artifactory-cleanup`, there are two steps involved.
1. Bump the version in the [setup.py](setup.py)
2. Bump the version in the [__init__.py](./artifactory_cleanup/__init__.py)
3. Create a Git release tag (in format `1.0.1`) by creating a release on GitHub
Raw data
{
"_id": null,
"home_page": "https://github.com/zkygr/artifactory-cleanup",
"name": "zkygr3-artifactory-cleanup",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.6",
"maintainer_email": null,
"keywords": "artifactory, cleanup",
"author": "Kay Gerlitzki",
"author_email": "git@kygr.net",
"download_url": "https://files.pythonhosted.org/packages/a9/fc/d6d75235d87f924524126152e611c1ccb195042c4dd43bc235f42a1443b8/zkygr3_artifactory_cleanup-1.1.0.tar.gz",
"platform": null,
"description": "# Artifactory cleanup #\n\n`artifactory-cleanup` is an extended and flexible cleanup tool for JFrog Artifactory.\n\nThe tool has simple YAML-defined cleanup configuration and can be extended with your own rules on Python.\nEverything must be as a code, even cleanup policies!\n\n# Tables of Contents\n\n<!-- toc -->\n\n- [Artifactory cleanup](#artifactory-cleanup)\n- [Tables of Contents](#tables-of-contents)\n- [Installation](#installation)\n- [Usage](#usage)\n - [Notes](#notes)\n - [Commands](#commands)\n- [Rules](#rules)\n - [Common](#common)\n - [Delete](#delete)\n - [Keep](#keep)\n - [Docker](#docker)\n - [Filters](#filters)\n - [Create your own rule](#create-your-own-rule)\n- [How to](#how-to)\n - [How to connect self-signed certificates for docker?](#how-to-connect-self-signed-certificates-for-docker)\n - [How to clean up Conan repository?](#how-to-clean-up-conan-repository)\n - [How to keep latest N docker images?](#how-to-keep-latest-n-docker-images)\n- [Release](#release)\n\n<!-- tocstop -->\n\n# Installation\n\nAs simple as one command!\n\n```bash\n# docker\ndocker pull devopshq/artifactory-cleanup\ndocker run --rm devopshq/artifactory-cleanup artifactory-cleanup --help\n\n# python (later we call it 'cli')\npython3 -mpip install artifactory-cleanup\nartifactory-cleanup --help\n```\n\n# Usage\n\nSuppose you want to remove **all artifacts older than N days** from **reponame** repository.\nYou should take the following steps:\n\n1. Install `artifactory-cleanup` (see above)\n2. Create a configuration file `artifactory-cleanup.yaml`. variables.\n\n```yaml\n# artifactory-cleanup.yaml\nartifactory-cleanup:\n server: https://repo.example.com/artifactory\n # $VAR is auto populated from environment variables\n user: $ARTIFACTORY_USERNAME\n password: $ARTIFACTORY_PASSWORD\n\n policies:\n - name: Remove all files from repo-name-here older than 7 days\n rules:\n - rule: Repo\n name: \"reponame\"\n - rule: DeleteOlderThan\n days: 7\n```\n\n3. Run the command **TO SHOW** (not remove) artifacts that will be deleted. By default `artifactory-cleanup` uses \"dry\n mode\".\n\n```bash\n# Set the credentials with delete permissions\nexport ARTIFACTORY_USERNAME=usernamehere\nexport ARTIFACTORY_PASSWORD=password\n\n# docker\ndocker run --rm -v \"$(pwd)\":/app -e ARTIFACTORY_USERNAME -e ARTIFACTORY_PASSWORD devopshq/artifactory-cleanup artifactory-cleanup\n\n# cli\nartifactory-cleanup\n```\n\n4. Verify that right artifacts will be removed and add `--destroy` flag **TO REMOVE** artifacts:\n\n```bash\n# docker\ndocker run --rm -v \"$(pwd)\":/app -e ARTIFACTORY_USERNAME -e ARTIFACTORY_PASSWORD devopshq/artifactory-cleanup artifactory-cleanup --destroy\n\n# cli\nartifactory-cleanup --destroy\n```\n\nLooking for more examples? Check [examples](./examples) folder!\n\n## Notes\n\n- **Always** specify version of `artifactory-cleanup` when using it in the production. `1.0.0` is just an example, find the latest version in pypi: https://pypi.org/project/artifactory-cleanup/\n\n```bash\n# docker\ndocker pull devopshq/artifactory-cleanup:1.0.0\ndocker run --rm devopshq/artifactory-cleanup:1.0.0 artifactory-cleanup --version\n\n# python (later we call it 'cli')\npython3 -mpip install artifactory-cleanup==1.0.0\nartifactory-cleanup --help\n```\n\n- Use CI servers or cron-like utilities to run `artifactory-cleanup` every day (or every hour). TeamCity and GitHub have\n built-in support and show additional logs format\n- Do not save credentials in the configuration file, use environment variables.\n- Use `--ignore-not-found` flag to ignore errors when the repository is not found. It's useful when you have a\n configuration for multiple repositories and some of them are not found.\n- Use `--worker-count=<WORKER_NUM>` to increase the number of workers. By default, it's 1. It's useful when you have a lot of\n artifacts and you want to speed up the process.\n\n## Commands ##\n\n```bash\n# Debug - \"dry run\" mode by default\n# debug run - only print artifacts. it does not delete any artifacts\nartifactory-cleanup\n\n# Debug run only for policytestname.\nartifactory-cleanup --policy-name policytestname\n\n# REMOVE\n# For remove artifacts use --destroy\nartifactory-cleanup --destroy\n\n# For remove artifacts use environment variable\nexport ARTIFACTORY_CLEANUP_DESTROY=True\nartifactory-cleanup\n\n# Specify config filename\nartifactory-cleanup --config artifactory-cleanup.yaml\n\n# Specify config filename using environment variable\nexport ARTIFACTORY_CLEANUP_CONFIG_FILE=artifactory-cleanup.yaml\nartifactory-cleanup --config artifactory-cleanup.yaml\n\n# Look in the future - shows what the tool WILL remove after 10 days\nartifactory-cleanup --days-in-future=10\n\n# Not satisfied with built-in rules? Write your own rules in python and connect them!\nartifactory-cleanup --load-rules=myrule.py\ndocker run -v \"$(pwd)\":/app devopshq/artifactory-cleanup artifactory-cleanup --load-rules=myrule.py\n\n# Save the table summary in a file\nartifactory-cleanup --output=myfile.txt\n\n# Save the summary in a Json file\nartifactory-cleanup --output=myfile.txt --output-format=json\n\n# Save the summary in a json file and append a list with removed artifacts\nartifactory-cleanup --output=myfile.json --output-format json-with-artifact-list\n```\n\n# Rules\n\n## Common\n\n- `Repo` - Apply the rule to one repository. If no name is specified, it is taken from the rule name (in `CleanupPolicy`\n definition)\n\n```yaml\n- rule: Repo\n name: reponame\n```\n\n```yaml\n# OR - if you have a single policy for the repo - you can name the policy as reponame\n# Both configurations are equal\npolicies:\n - name: reponame\n rules:\n - rule: Repo\n```\n\n- `RepoList` - Apply the policy to list of repositories.\n\n```yaml\n- rule: RepoList\n repos:\n - repo1\n - repo2\n - repo3\n```\n\n- `RepoByMask` - Apply rule to repositories matching by mask\n\n```yaml\n- rule: RepoByMask\n mask: \"*.banned\"\n```\n\n- `PropertyEq`- Delete repository artifacts only with a specific property value (property_key is the name of the\n parameter, property_value is the value)\n\n```yaml\n- rule: PropertyEq\n property_key: key-name\n property_value: 1\n```\n\n- `PropertyNeq`- Delete repository artifacts only if the value != specified. If there is no value, delete it anyway.\n Allows you to specify the deletion flag `do_not_delete = 1`\n\n```yaml\n- rule: PropertyNeq\n property_key: key-name\n property_value: 1\n```\n\n## Delete\n\n- `DeleteOlderThan` - deletes artifacts that are older than N days\n\n```yaml\n- rule: DeleteOlderThan\n days: 1\n```\n\n- `DeleteWithoutDownloads` - deletes artifacts that have never been downloaded (DownloadCount=0). Better to use\n with `DeleteOlderThan` rule\n\n```yaml\n- rule: DeleteWithoutDownloads\n```\n\n- `DeleteOlderThanNDaysWithoutDownloads` - deletes artifacts that are older than N days and have not been\n downloaded\n\n```yaml\n- rule: DeleteOlderThanNDaysWithoutDownloads\n days: 1\n```\n\n- `DeleteNotUsedSince` - delete artifacts that were downloaded, but for a long time. N days passed. Or not\n downloaded at all from the moment of creation and it's been N days\n\n```yaml\n- rule: DeleteNotUsedSince\n days: 1\n```\n\n- `DeleteEmptyFolders` - Clean up empty folders in given repository list\n\n```yaml\n- rule: DeleteEmptyFolders\n```\n\n- `DeleteByRegexpName` - delete artifacts whose name matches the specified regexp\n\n```yaml\n- rule: DeleteByRegexpName\n regex_pattern: \"\\d\"\n```\n\n## Keep\n\n- `KeepLatestNFiles` - Leaves the last (by creation time) files in the amount of N pieces. WITHOUT accounting\n subfolders\n\n```yaml\n- rule: KeepLatestNFiles\n count: 1\n```\n\n- `KeepLatestNFilesInFolder` - Leaves the last (by creation time) files in the number of N pieces in each\n folder\n\n```yaml\n- rule: KeepLatestNFilesInFolder\n count: 1\n```\n\n- `KeepLatestVersionNFilesInFolder` - Leaves the latest N (by version) files in each\n folder. The definition of the version is using regexp. By default it parses [semver](https://semver.org/) using the regex - `([\\d]+\\.[\\d]+\\.[\\d]+)\")`\n\n```yaml\n- rule: KeepLatestVersionNFilesInFolder\n count: 1\n custom_regexp: \"[^\\\\d][\\\\._]((\\\\d+\\\\.)+\\\\d+)\"\n```\n\n- `KeepLatestNupkgNVersions` - Leaves N nupkg (adds `*.nupkg` filter) in release feature builds\n\n```yaml\n- rule: KeepLatestNupkgNVersions\n count: 1\n```\n\n## Docker\n\n- `DeleteDockerImagesOlderThan` - Delete docker images that are older than N days\n\n```yaml\n- rule: DeleteDockerImagesOlderThan\n days: 1\n```\n\n- `DeleteDockerImagesOlderThanNDaysWithoutDownloads` - Deletes docker images that are older than N days and have\n not been downloaded\n\n```yaml\n- rule: DeleteDockerImagesOlderThanNDaysWithoutDownloads\n days: 1\n```\n\n- `DeleteDockerImagesNotUsed` - Removes Docker image not downloaded since N days\n\n```yaml\n- rule: DeleteDockerImagesNotUsed\n days: 1\n```\n\n- `IncludeDockerImages` - Apply to docker images with the specified names and tags\n\n```yaml\n- rule: IncludeDockerImages\n masks: \"*singlemask*\"\n- rule: IncludeDockerImages\n masks:\n - \"*production*\"\n - \"*release*\"\n```\n\n- `ExcludeDockerImages` - Exclude Docker images by name and tags.\n\n```yaml\n- rule: ExcludeDockerImages\n masks:\n - \"*production*\"\n - \"*release*\"\n```\n\n- `KeepLatestNVersionImagesByProperty(count=N, custom_regexp='some-regexp', number_of_digits_in_version=X)` - Leaves N\n Docker images with the same major. `(^\\d+\\.\\d+\\.\\d+$)` is the default regexp how to determine version which matches semver `1.1.1`. If you\n need to add minor then set `number_of_digits_in_version` to 2 or if patch then set to 3 (by default we match major, which 1). Semver tags\n prefixed with `v` are supported by updating the regexp to include (an optional) `v` in the expression (e.g., `(^v?\\d+\\.\\d+\\.\\d+$)`).\n\n```yaml\n- rule: KeepLatestNVersionImagesByProperty\n count: 1\n custom_regexp: \"[^\\\\d][\\\\._]((\\\\d+\\\\.)+\\\\d+)\"\n```\n\n- `DeleteDockerImageIfNotContainedInProperties(docker_repo='docker-local', properties_prefix='my-prop', image_prefix=None, full_docker_repo_name=None)`\n \\- Remove Docker image, if it is not found in the properties of the artifact repository.\n\n- `DeleteDockerImageIfNotContainedInPropertiesValue(docker_repo='docker-local', properties_prefix='my-prop', image_prefix=None, full_docker_repo_name=None)`\n \\- Remove Docker image, if it is not found in the properties of the artifact repository.\n\n## Filters\n\n- `IncludePath` - Apply to artifacts by path / mask.\n\n```yaml\n- rule: IncludePath\n masks: \"*production*\"\n- rule: IncludePath\n masks:\n - \"*production*\"\n - \"*develop*\"\n```\n\n- `IncludeFilename` - Apply to artifacts by name/mask\n\n```yaml\n- rule: IncludeFilename\n masks:\n - \"*production*\"\n - \"*develop*\"\n```\n\n- `ExcludePath` - Exclude artifacts by path/mask\n\n```yaml\n- rule: ExcludePath\n masks:\n - \"*production*\"\n - \"*develop*\"\n```\n\n- `ExcludeFilename` - Exclude artifacts by name/mask\n\n```yaml\n- rule: ExcludeFilename\n masks:\n - \"*.tag.gz\"\n - \"*.zip\"\n```\n\n## Create your own rule\n\nIf you want to create your own rule, you can do it!\n\nThe basic flow how the tool calls Rules:\n\n1. `Rule.check(*args, **kwargs)` - verify that the Rule configured right. Call other services to get more information.\n2. `Rule.aql_add_filter(filters)` - add Artifactory Query Language expressions\n3. `Rule.aql_add_text(aql)` - add text to the result aql query\n4. `artifactory-cleanup` calls Artifactory with AQL and pass the result to the next step\n5. `Rule.filter(artifacts)` - filter out artifacts. The method returns **artifacts that will be removed!**.\n - To keep artifacts use `artifacts.keep(artifact)` method\n\nCreate `myrule.py` file at the same folder as `artifactory-cleanup.yaml`:\n\n```python\n# myrule.py\nfrom typing import List\n\nfrom artifactory_cleanup import register\nfrom artifactory_cleanup.rules import Rule, ArtifactsList\n\n\nclass MySimpleRule(Rule):\n \"\"\"\n This doc string is used as rule title\n\n For more methods look at Rule source code\n \"\"\"\n\n def __init__(self, my_param: str, value: int):\n self.my_param = my_param\n self.value = value\n\n def aql_add_filter(self, filters: List) -> List:\n print(f\"Today is {self.today}\")\n print(self.my_param)\n print(self.value)\n return filters\n\n def filter(self, artifacts: ArtifactsList) -> ArtifactsList:\n \"\"\"I'm here just to print the list\"\"\"\n print(self.my_param)\n print(self.value)\n # You can make requests to artifactory by using self.session:\n # url = f\"/api/storage/{self.repo}\"\n # r = self.session.get(url)\n # r.raise_for_status()\n return artifacts\n\n\n# Register your rule in the system\nregister(MySimpleRule)\n```\n\nUse `rule: MySimpleRule` in configuration:\n\n```yaml\n# artifactory-cleanup.yaml\n- rule: MySimpleRule\n my_param: \"Hello, world!\"\n value: 42\n```\n\nSpecify `--load-rules` to the command:\n\n```bash\n# docker\ndocker run -v \"$(pwd)\":/app devopshq/artifactory-cleanup artifactory-cleanup --load-rules=myrule.py\n\n# cli\nartifactory-cleanup --load-rules=myrule.py\n```\n\n# How to\n\n## How to connect self-signed certificates for docker?\n\nIn case you have set up your Artifactory self-signed certificates, place all certificates of the chain of trust into\nthe `certificates` folder and add additional argument to the command:\n\n```bash\ndocker run -v \"$(pwd)\":/app -v \"$(pwd)/certificates\":/mnt/self-signed-certs/ devopshq/artifactory-cleanup artifactory-cleanup\n```\n\n## How to clean up Conan repository?\n\nWe can handle conan's metadata by creating two policies:\n\n1. First one removes files but keep all metadata.\n2. Second one look at folders and if it contains only medata files - removes it (because there's no associated with\n metadata files)\n\nThe idea came from https://github.com/devopshq/artifactory-cleanup/issues/47\n\n```yaml\n# artifactory-cleanup.yaml\nartifactory-cleanup:\n server: https://repo.example.com/artifactory\n user: $ARTIFACTORY_USERNAME\n password: $ARTIFACTORY_PASSWORD\n\n policies:\n - name: Conan - delete files older than 60 days\n rules:\n - rule: Repo\n name: \"conan-testing\"\n - rule: DeleteNotUsedSince\n days: 60\n - rule: ExcludeFilename\n masks:\n - \".timestamp\"\n - \"index.json\"\n - name: Conan - delete empty folders (to fix the index)\n rules:\n - rule: Repo\n name: \"conan-testing\"\n - rule: DeleteEmptyFolders\n - rule: ExcludeFilename\n masks:\n - \".timestamp\"\n - \"index.json\"\n```\n\n## How to keep latest N docker images?\n\nWe can combine docker rules with usual \"files\" rules!\n\nThe idea came from https://github.com/devopshq/artifactory-cleanup/issues/61\n\n```yaml\n# artifactory-cleanup.yaml\nartifactory-cleanup:\n server: https://repo.example.com/artifactory\n user: $ARTIFACTORY_USERNAME\n password: $ARTIFACTORY_PASSWORD\n\n policies:\n - name: Remove docker images, but keep last 3\n rules:\n # Select repo\n - rule: Repo\n name: docker-demo\n # Delete docker images older than 30 days\n - rule: DeleteDockerImagesOlderThan\n days: 30\n # Keep these tags for all images\n - rule: ExcludeDockerImages\n masks:\n - \"*:latest\"\n - \"*:release*\"\n # Exclude these docker tags\n - rule: ExcludePath\n masks: \"*base-tools*\"\n # Keep 3 docker tags for all images\n - rule: KeepLatestNFilesInFolder\n count: 3\n```\n\n# Release\n\nIn order to provide a new release of `artifactory-cleanup`, there are two steps involved.\n\n1. Bump the version in the [setup.py](setup.py)\n2. Bump the version in the [__init__.py](./artifactory_cleanup/__init__.py)\n3. Create a Git release tag (in format `1.0.1`) by creating a release on GitHub\n\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "Rules and cleanup policies for Artifactory",
"version": "1.1.0",
"project_urls": {
"Homepage": "https://github.com/zkygr/artifactory-cleanup"
},
"split_keywords": [
"artifactory",
" cleanup"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "27a6b219de37a7ad6dfcfc37e67015b986777c74718578f47f0008f370c0ff82",
"md5": "d7f67f2edf7dbe1d33e113dbc200102c",
"sha256": "9de82570cf1b6a27010d9200b0baf9d297d4dd5827cee3bd4036850d9b010424"
},
"downloads": -1,
"filename": "zkygr3_artifactory_cleanup-1.1.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "d7f67f2edf7dbe1d33e113dbc200102c",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.6",
"size": 29145,
"upload_time": "2024-07-10T11:41:45",
"upload_time_iso_8601": "2024-07-10T11:41:45.206872Z",
"url": "https://files.pythonhosted.org/packages/27/a6/b219de37a7ad6dfcfc37e67015b986777c74718578f47f0008f370c0ff82/zkygr3_artifactory_cleanup-1.1.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "a9fcd6d75235d87f924524126152e611c1ccb195042c4dd43bc235f42a1443b8",
"md5": "d1bc3255e77827d227644375884a9e4a",
"sha256": "3e0bfa206498539f58a31690bb1c858d8d4becff34cb6540c8b59e7764152f54"
},
"downloads": -1,
"filename": "zkygr3_artifactory_cleanup-1.1.0.tar.gz",
"has_sig": false,
"md5_digest": "d1bc3255e77827d227644375884a9e4a",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.6",
"size": 33005,
"upload_time": "2024-07-10T11:41:49",
"upload_time_iso_8601": "2024-07-10T11:41:49.305280Z",
"url": "https://files.pythonhosted.org/packages/a9/fc/d6d75235d87f924524126152e611c1ccb195042c4dd43bc235f42a1443b8/zkygr3_artifactory_cleanup-1.1.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-07-10 11:41:49",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "zkygr",
"github_project": "artifactory-cleanup",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "zkygr3-artifactory-cleanup"
}
JSON
