# backups-rotate
You run a website with a database, and you want to backup your database every day, and keep the last 7 days of backups. This script will do that for you.
Contrary to logrotate, where contents is aassumed to be **appended** to files, here we assume all the files are **full backups**; and that each backup is independant from the others. So the job is not to rename files, but to remove the old ones.
## What it does
You can define a policy on several frequency levels, e.g.
- one per year - no matter how far back in time
- and/or, keep one backup for each of the last 3 quarters
- and/or, the last 2 months
- and/or, the last 4 weeks
- and/or, the last 5 days
- and/or, the last 6 hours
## Installation
```bash
pip install backups-rotate
```
## Configuration
The tool comes with 6 predefined "periods": `year`, `quarter`, `month`, `week`, `day`, `hour`.
You define your folder and policy in a YAML file, like e.g (for the policy described above):
```yaml
---
# this defines the area named "database" that can be used
# on the command line to specify what needs to be cleaned up
- name: database
folder: /usr/lib/backups
patterns:
- "mydb*.sql"
- "mydb*.sql.gz"
policy:
year: infinite
quarter: 3
month: 2
week: 4
day: 5
hour: 6
# by default the time attaached to each file is its creation time
# optionally, you can specify the format of the timestamp
# in the file name, like so:
# datetime_format: "%Y-%m-%d_%H-%M-%S"
# in that case, files that do not match the format will be ignored
# OR, still optionally, you can specify to use the modification time
# instead of the creation time
# use_modification_time: true
# you define as many areas as you want
- name: builds
folder: /usr/lib/builds
patterns:
- "myapp*"
# type can be either folder, file or symlink
# by default all files are considered
type: folder
policy:
month: 4
week: 3
day: 4
```
## Usage
Here are a few ways to run the tool
```bash
# run on all areas
backups-rotate
# using a config file (default is /etc/backups-rotate.yaml, and ./backups-rotate.yaml)
backups-rotate --config my_policy.yaml database
# on a specific area
backups-rotate database
# on a specific area; list the files considered
# you can use -l instead of --list
backups-rotate database --list
# same in verbose mode
# you can use -v instead of --verbose
backups-rotate database --list --verbose
# same but only shows what would be deleted
# you can use -d instead of --deleted
backups-rotate database --list --deleted
# not doing anything, just showing what would be done
# you can use -n instead of --dry-run
backups-rotate database --dry-run
# do it (i.e. remove the selected files), and be verbose about it
backups-rotate database --verbose
```
## How policies are implemented
When run on a specific area, the tool will:
- gather all matching files in the specified folder
- then for each frequency present in the policy, starting with the shortest one:
- split time into bins of that frequency, starting from now and going backwards
- attach every file to corresponding bin
- then for the number attached to that frequency (can be infinite), and starting from the most recent bin:
- take the most recent file and mark it as kept
- then list (if dry-run) or remove (if not dry-run) all files that are not marked as kept
### Example
Based on `sample2()` in `tests/samples.py`, assume you have one file per hour between
`2024-01-01 00:00` and `2024-11-15 23:00`, then applying the policy above would keep:
```text
2024-06-30 23:00 # for quarter
2024-09-30 23:00 # for quarter
2024-10-27 23:00 # for week
2024-10-31 23:00 # for month
2024-11-03 23:00 # for week
2024-11-10 23:00 # for week
2024-11-11 23:00 # for day
2024-11-12 23:00 # for day
2024-11-13 23:00 # for day
2024-11-14 23:00 # for day
2024-11-15 18:00 # for hour
2024-11-15 19:00 # for hour
2024-11-15 20:00 # for hour
2024-11-15 21:00 # for hour
2024-11-15 22:00 # for hour
2024-11-15 23:00 # for hour day week month quarter year
```
if instead the policy was defined with `month: infinite`, then the policy would also retain
```text
2024-01-31 23:00
2024-02-29 23:00
2024-03-31 23:00
2024-04-30 23:00
2024-05-31 23:00
2024-07-31 23:00
2024-08-31 23:00
```
noting that the following 2 were already kept for quarter:
```text
2024-06-30 23:00
2024-09-30 23:00
```
### Timestamps
By default, time is taken from the file's creation time. If you want to use the
file's modification time instead, you can use the `use-modification-time`
flag in your yaml config.
Also if your files are named with a timestamp, you can use the `datetime-format`
option to specify the format of the timestamp in the file name (using Python's
`datetime` format).
## Tests
```bash
pip install pytest
pytest
```
Raw data
{
"_id": null,
"home_page": null,
"name": "backups-rotate",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.11",
"maintainer_email": null,
"keywords": "backup, database, dump, rotate",
"author": null,
"author_email": "Thierry Parmentelat <thierry.parmentelat@inria.fr>",
"download_url": "https://files.pythonhosted.org/packages/93/0a/d52f8f8a2d4c820044352339e27a015fb59299b2140f4edb128797be5532/backups_rotate-0.0.2.tar.gz",
"platform": null,
"description": "# backups-rotate\n\nYou run a website with a database, and you want to backup your database every day, and keep the last 7 days of backups. This script will do that for you.\n\nContrary to logrotate, where contents is aassumed to be **appended** to files, here we assume all the files are **full backups**; and that each backup is independant from the others. So the job is not to rename files, but to remove the old ones.\n\n## What it does\n\nYou can define a policy on several frequency levels, e.g.\n\n- one per year - no matter how far back in time\n- and/or, keep one backup for each of the last 3 quarters\n- and/or, the last 2 months\n- and/or, the last 4 weeks\n- and/or, the last 5 days\n- and/or, the last 6 hours\n\n## Installation\n\n```bash\npip install backups-rotate\n```\n\n## Configuration\n\nThe tool comes with 6 predefined \"periods\": `year`, `quarter`, `month`, `week`, `day`, `hour`.\n\nYou define your folder and policy in a YAML file, like e.g (for the policy described above):\n\n```yaml\n---\n# this defines the area named \"database\" that can be used\n# on the command line to specify what needs to be cleaned up\n\n- name: database\n folder: /usr/lib/backups\n patterns:\n - \"mydb*.sql\"\n - \"mydb*.sql.gz\"\n policy:\n year: infinite\n quarter: 3\n month: 2\n week: 4\n day: 5\n hour: 6\n\n# by default the time attaached to each file is its creation time\n# optionally, you can specify the format of the timestamp \n# in the file name, like so:\n # datetime_format: \"%Y-%m-%d_%H-%M-%S\"\n# in that case, files that do not match the format will be ignored\n\n# OR, still optionally, you can specify to use the modification time\n# instead of the creation time\n # use_modification_time: true\n\n# you define as many areas as you want\n- name: builds\n folder: /usr/lib/builds\n patterns:\n - \"myapp*\"\n # type can be either folder, file or symlink\n # by default all files are considered\n type: folder\n policy:\n month: 4\n week: 3\n day: 4\n```\n\n## Usage\n\nHere are a few ways to run the tool\n\n```bash\n# run on all areas\nbackups-rotate\n\n# using a config file (default is /etc/backups-rotate.yaml, and ./backups-rotate.yaml)\nbackups-rotate --config my_policy.yaml database\n\n# on a specific area\nbackups-rotate database\n\n# on a specific area; list the files considered\n# you can use -l instead of --list\nbackups-rotate database --list\n\n# same in verbose mode\n# you can use -v instead of --verbose\nbackups-rotate database --list --verbose\n\n# same but only shows what would be deleted\n# you can use -d instead of --deleted\nbackups-rotate database --list --deleted\n\n# not doing anything, just showing what would be done\n# you can use -n instead of --dry-run\nbackups-rotate database --dry-run\n\n# do it (i.e. remove the selected files), and be verbose about it\nbackups-rotate database --verbose\n```\n\n## How policies are implemented\n\nWhen run on a specific area, the tool will:\n\n- gather all matching files in the specified folder\n- then for each frequency present in the policy, starting with the shortest one:\n - split time into bins of that frequency, starting from now and going backwards\n - attach every file to corresponding bin\n - then for the number attached to that frequency (can be infinite), and starting from the most recent bin:\n - take the most recent file and mark it as kept\n - then list (if dry-run) or remove (if not dry-run) all files that are not marked as kept\n\n### Example\n\nBased on `sample2()` in `tests/samples.py`, assume you have one file per hour between\n`2024-01-01 00:00` and `2024-11-15 23:00`, then applying the policy above would keep:\n\n```text\n2024-06-30 23:00 # for quarter\n2024-09-30 23:00 # for quarter\n2024-10-27 23:00 # for week\n2024-10-31 23:00 # for month\n2024-11-03 23:00 # for week\n2024-11-10 23:00 # for week\n2024-11-11 23:00 # for day\n2024-11-12 23:00 # for day\n2024-11-13 23:00 # for day\n2024-11-14 23:00 # for day\n2024-11-15 18:00 # for hour\n2024-11-15 19:00 # for hour\n2024-11-15 20:00 # for hour\n2024-11-15 21:00 # for hour\n2024-11-15 22:00 # for hour\n2024-11-15 23:00 # for hour day week month quarter year\n```\n\nif instead the policy was defined with `month: infinite`, then the policy would also retain\n\n```text\n2024-01-31 23:00\n2024-02-29 23:00\n2024-03-31 23:00\n2024-04-30 23:00\n2024-05-31 23:00\n2024-07-31 23:00\n2024-08-31 23:00\n```\n\nnoting that the following 2 were already kept for quarter:\n\n```text\n2024-06-30 23:00\n2024-09-30 23:00\n```\n\n### Timestamps\n\nBy default, time is taken from the file's creation time. If you want to use the\nfile's modification time instead, you can use the `use-modification-time`\nflag in your yaml config.\n\nAlso if your files are named with a timestamp, you can use the `datetime-format`\noption to specify the format of the timestamp in the file name (using Python's\n`datetime` format).\n\n## Tests\n\n```bash\npip install pytest\npytest\n```\n",
"bugtrack_url": null,
"license": "MIT License",
"summary": "kind of like logrotate, but for backups",
"version": "0.0.2",
"project_urls": {
"Bug Tracker": "https://github.com/parmentelat/backups-rotate/issues",
"Repository": "https://github.com/parmentelat/backups-rotate"
},
"split_keywords": [
"backup",
" database",
" dump",
" rotate"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "2d857dcc3d41fbe7061f953d7319c15d8df15096bde091405a36e620ef7c9822",
"md5": "308d8e10f4f1170047cd4d6b08ad6238",
"sha256": "151db2f69a215adfa99d14a2e83c429f8fe23b2d20af3342bbf25cda17f7db8d"
},
"downloads": -1,
"filename": "backups_rotate-0.0.2-py3-none-any.whl",
"has_sig": false,
"md5_digest": "308d8e10f4f1170047cd4d6b08ad6238",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.11",
"size": 8568,
"upload_time": "2024-11-25T14:37:08",
"upload_time_iso_8601": "2024-11-25T14:37:08.291342Z",
"url": "https://files.pythonhosted.org/packages/2d/85/7dcc3d41fbe7061f953d7319c15d8df15096bde091405a36e620ef7c9822/backups_rotate-0.0.2-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "930ad52f8f8a2d4c820044352339e27a015fb59299b2140f4edb128797be5532",
"md5": "35ccd221b0ff73f654fe5bc6904d6fd0",
"sha256": "6f4deb2349095796d8f44fa1caad5a217b9dd8ba8f189992faa73827c116c1f4"
},
"downloads": -1,
"filename": "backups_rotate-0.0.2.tar.gz",
"has_sig": false,
"md5_digest": "35ccd221b0ff73f654fe5bc6904d6fd0",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.11",
"size": 11175,
"upload_time": "2024-11-25T14:37:09",
"upload_time_iso_8601": "2024-11-25T14:37:09.824472Z",
"url": "https://files.pythonhosted.org/packages/93/0a/d52f8f8a2d4c820044352339e27a015fb59299b2140f4edb128797be5532/backups_rotate-0.0.2.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-11-25 14:37:09",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "parmentelat",
"github_project": "backups-rotate",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"requirements": [],
"lcname": "backups-rotate"
}
JSON
