| Name | haondt-charon JSON |
| Version |
2.1.0
JSON |
| download |
| home_page | None |
| Summary | charon is a utility for backing up data from one location to another at regular intervals. |
| upload_time | 2025-02-26 19:41:23 |
| maintainer | None |
| docs_url | None |
| author | haondt |
| requires_python | None |
| license | None |
| keywords |
backup
recovery
|
| VCS |
|
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
# charon
charon is a utility for automating data backups. charon uses [`restic`](https://restic.net/) for managing the backups.
# table of contents
- [installation](#installation)
- [usage](#usage)
- [configuration](#configuration)
- [sources](#sources)
- [repository](#repository)
- [schedule](#schedule)
- [cli](#cli)
- [tests](#tests)
# installation
charon can be installed as a docker image
```bash
# from docker hub
docker pull haumea/charon
# from gitlab
docker pull registry.gitlab.com/haondt/cicd/registry/charon:latest
```
see `docker-compose.yml` for a sample docker compose setup.
charon can also be installed as a python package
```bash
# from pypi
pip install haondt-charon
# from gitlab
pip install haondt-charon --index-url https://gitlab.com/api/v4/projects/57154225/packages/pypi/simple
```
# usage
start charon with:
```bash
# if installed as a python package, or if running from source
python3 -m charon
# the pypi package also includes a standlone binary
charon
# from the docker image
docker run --rm -it -v ./charon.yml:/config/charon.yml registry.gitlab.com/haondt/cicd/registry/charon:latest
```
charon will look for a config file at `charon.yml`. a different path can be specified with:
```bash
charon -f MY_CONFIG.yml
```
charon uses the `sched` library for scheduling tasks, which introduces some caveats:
- charon will exit when there are no more jobs to run. this is possible depending on the configuration
- all jobs are run on the same thread, sequentially
- the duration of the execution of a job can interfere with it's schedule. if a job misses a run due to the previous run still completing, the run is skipped and rescheduled at the next repetition
# configuration
configuration is given as a yaml file with the following structure:
```yml
jobs:
my_job:
source: # where data is coming from
type: type_of_source
# ...
repository: # configuration for restic repository
password: myresticpassword
# ...
schedule: # how often to run job
# ...
my_job_2:
# ...
```
see `charon.yml` for an example config.
## sources
all sources will have a few shared fields:
```yaml
source:
type: local # determines how to interpret the source config
```
the data from the source will be backed up using `restic`. If the source is coming from somewhere external, like an http request, it will be cached in a temporary directory before being run through `restic`.
below are the possible ways you can configure the source object, based on the `type` key.
**local**
this pulls from a local file
```yml
source:
type: local
path: /path/to/data # path to data to back up. can be a file or a directory. does not use variable expansion
```
**http**
performs an http request, and saves the response body to a file
```yml
source:
type: http
url: http://example.com/ # url to make request to
method: get # optional, request method, defaults to get
ext: json # optional, extension to use for saved file, defaults to txt
auth: # optional, authentication configuration
bearer: eyJhbGc... # optional, bearer token
transform: # optional, list of transforms to perform on the response body
- jq: .[] + 1 # uses the jq library to run `jq.compile(<PATTERN>).input_text(<PAYLOAD>).first()`
```
you can also make multiple requests and save the results to multiple files
```yml
source:
type: http
targets:
com: # will be saved to com.json
url: http://example.com/ # url to make request to
method: get # optional, request method, defaults to get
ext: json # optional, extension to use for saved file, defaults to txt
auth: # optional, authentication configuration
bearer: eyJhbGc... # optional, bearer token
uk: # will be saved to uk.zip
url: http://example.co.uk/
ext: zip
```
`url` and `targets` can both be provided, but at least one _must_ be provided.
**sqlite**
performs a backup on an sqlite3 db
```yml
source:
type: sqlite
db_path: /path/to/db_file.db
```
## repository
the `repository` section is for configuring the `restic` repository.
```yml
repository:
password: my-restic-password # password for repository
create: false # optional, whether or not charon should create the repository if it doesn't exist. default is true
max_snapshots: 3 # optional, prune old snapshots to keep this amount or fewer snapshots in the repository
backend: # configuration for the restic backend
type: local # determines how to interpret the backend config
```
below are the possible ways you can configure the `repository.backend` object, based on the `type` key.
**local**
this pushes to a local directory
```yml
backend:
type: local
path: ./foo/bar # must be a directory
```
**gcs_bucket**
uploads to a google cloud storage bucket
```yml
backend:
type: gcs_bucket
bucket: 9e4376a1-a0ce-4ff4-a67b-8af4a54d15c1-foo # bucket name
credentials: ./credentials.json # path to credentials file for service account with access to bucket
path: /path/to/repo # path to repository inside bucket
```
**rclone**
uses rclone as a target.
the `backend.rclone_config` object will be used to configure rclone. each key in that map will be configured to an environment variable named `f'{RCLONE_CONFIG_{job.upper()}_{key.upper()}'}`. for example, the config `jobs.myjob.backend.rclone_config.host: myhost.com` would be converted to `RCLONE_CONFIG_MYJOB_HOST=myhost.com`.
> [!NOTE]
> the password needs to be stored in its obscured form, **charon will not obscure the password for you**. you can obscure your password using `rclone obscure`
```yml
backend:
type: rclone
path: path/to/repo # path within rclone target for repository
rclone_config: # configuration to pass through to rclone env vars
type: ftp
host: my-host.com
user: my-username
pass: ... # obscured password
port: 21
explicit_tls: "true"
```
## schedule
how often the program is run. there are a few different ways to configure the schedule
**cron**
the schedule can be configured using a cron string.
note: this program uses [croniter](https://github.com/kiorky/croniter) for scheduling with the cron format. Croniter accepts seconds, but they must be at the _end_ (right hand side) of the cron string.
```yml
schedule:
cron: "* * * * * */10" # every 10 seconds
```
**one shot**
this runs once, after the given delay. the delay is given in the `1d2h3m4s` format. numbers must be integers.
```yml
schedule:
after: 1d # wait 1 day, then run once
```
**intervals**
this runs at regular intervals, using the one shot format, starting from the time charon is run.
```yml
schedule:
every: 1h30m # run every hour and a half
```
**combinations**
you can combine schedules, for example to run immediately, and then every other day
```yml
schedule:
after: 0s
every: 2d
```
**timeout**
optionally, you can specify a job timeout. if the job (both the fetch and the upload) do not complete within the timeout, the job will be cancelled.
```yml
schedule:
every: 1d
timeout: 15m
```
# cli
charon provides a cli for manual work. the `apply` command can be used to run a job once, immediately.
```bash
charon apply MY_JOB
```
charon can also run the job in reverse, pulling it from the destination and dumping it to a given directory
```bash
charon revert MY_JOB OUTPUT_DIRECTORY
```
you can specify the config file before running either command
```bash
charon -f MY_CONFIG.yml styx apply MY_JOB
```
see tests for more examples.
## tests
each `test*.sh` file will run some commands (must be run inside the tests folder, with a python environment set up for charon), and has a comment in the file detailing the expected output.
```bash
cd tests
./test.sh
./test2.sh
...
```
Raw data
{
"_id": null,
"home_page": null,
"name": "haondt-charon",
"maintainer": null,
"docs_url": null,
"requires_python": null,
"maintainer_email": null,
"keywords": "backup, recovery",
"author": "haondt",
"author_email": null,
"download_url": "https://files.pythonhosted.org/packages/8c/92/bd9c798a5cf37a00711f45c0a2e588460d1d5010654344fdb1be24206335/haondt_charon-2.1.0.tar.gz",
"platform": null,
"description": "# charon\n\ncharon is a utility for automating data backups. charon uses [`restic`](https://restic.net/) for managing the backups.\n\n# table of contents\n\n- [installation](#installation)\n- [usage](#usage)\n- [configuration](#configuration)\n - [sources](#sources)\n - [repository](#repository)\n - [schedule](#schedule)\n- [cli](#cli)\n- [tests](#tests)\n\n# installation\n\ncharon can be installed as a docker image\n\n```bash\n# from docker hub\ndocker pull haumea/charon\n# from gitlab\ndocker pull registry.gitlab.com/haondt/cicd/registry/charon:latest\n```\n\nsee `docker-compose.yml` for a sample docker compose setup.\n\ncharon can also be installed as a python package\n\n```bash\n# from pypi\npip install haondt-charon\n# from gitlab\npip install haondt-charon --index-url https://gitlab.com/api/v4/projects/57154225/packages/pypi/simple\n```\n\n# usage\n\nstart charon with:\n\n```bash\n# if installed as a python package, or if running from source\npython3 -m charon\n# the pypi package also includes a standlone binary\ncharon\n# from the docker image\ndocker run --rm -it -v ./charon.yml:/config/charon.yml registry.gitlab.com/haondt/cicd/registry/charon:latest\n```\n\ncharon will look for a config file at `charon.yml`. a different path can be specified with:\n\n```bash\ncharon -f MY_CONFIG.yml\n```\n\ncharon uses the `sched` library for scheduling tasks, which introduces some caveats:\n\n- charon will exit when there are no more jobs to run. this is possible depending on the configuration\n- all jobs are run on the same thread, sequentially\n- the duration of the execution of a job can interfere with it's schedule. if a job misses a run due to the previous run still completing, the run is skipped and rescheduled at the next repetition\n\n# configuration\n\nconfiguration is given as a yaml file with the following structure:\n\n```yml\njobs:\n my_job:\n source: # where data is coming from\n type: type_of_source\n # ...\n repository: # configuration for restic repository\n password: myresticpassword\n # ...\n schedule: # how often to run job\n # ...\n my_job_2:\n # ...\n```\n\nsee `charon.yml` for an example config.\n\n## sources\n\nall sources will have a few shared fields:\n\n```yaml\nsource:\n type: local # determines how to interpret the source config\n```\n\nthe data from the source will be backed up using `restic`. If the source is coming from somewhere external, like an http request, it will be cached in a temporary directory before being run through `restic`.\n\nbelow are the possible ways you can configure the source object, based on the `type` key.\n\n**local**\n\nthis pulls from a local file\n\n```yml\nsource:\n type: local\n path: /path/to/data # path to data to back up. can be a file or a directory. does not use variable expansion\n```\n\n**http**\n\nperforms an http request, and saves the response body to a file\n\n```yml\nsource:\n type: http\n url: http://example.com/ # url to make request to\n method: get # optional, request method, defaults to get\n ext: json # optional, extension to use for saved file, defaults to txt\n auth: # optional, authentication configuration\n bearer: eyJhbGc... # optional, bearer token\n transform: # optional, list of transforms to perform on the response body\n - jq: .[] + 1 # uses the jq library to run `jq.compile(<PATTERN>).input_text(<PAYLOAD>).first()`\n```\n\nyou can also make multiple requests and save the results to multiple files\n\n```yml\nsource:\n type: http\n targets:\n com: # will be saved to com.json\n url: http://example.com/ # url to make request to\n method: get # optional, request method, defaults to get\n ext: json # optional, extension to use for saved file, defaults to txt\n auth: # optional, authentication configuration\n bearer: eyJhbGc... # optional, bearer token\n uk: # will be saved to uk.zip\n url: http://example.co.uk/\n ext: zip\n```\n\n`url` and `targets` can both be provided, but at least one _must_ be provided.\n\n\n**sqlite**\n\nperforms a backup on an sqlite3 db\n\n```yml\nsource:\n type: sqlite\n db_path: /path/to/db_file.db\n```\n\n## repository\n\nthe `repository` section is for configuring the `restic` repository.\n\n```yml\nrepository:\n password: my-restic-password # password for repository\n create: false # optional, whether or not charon should create the repository if it doesn't exist. default is true\n max_snapshots: 3 # optional, prune old snapshots to keep this amount or fewer snapshots in the repository\n backend: # configuration for the restic backend\n type: local # determines how to interpret the backend config\n```\n\nbelow are the possible ways you can configure the `repository.backend` object, based on the `type` key.\n\n**local**\n\nthis pushes to a local directory\n\n```yml\nbackend:\n type: local\n path: ./foo/bar # must be a directory\n```\n\n**gcs_bucket**\n\nuploads to a google cloud storage bucket\n\n```yml\nbackend:\n type: gcs_bucket\n bucket: 9e4376a1-a0ce-4ff4-a67b-8af4a54d15c1-foo # bucket name\n credentials: ./credentials.json # path to credentials file for service account with access to bucket\n path: /path/to/repo # path to repository inside bucket\n```\n\n**rclone**\n\nuses rclone as a target.\n\nthe `backend.rclone_config` object will be used to configure rclone. each key in that map will be configured to an environment variable named `f'{RCLONE_CONFIG_{job.upper()}_{key.upper()}'}`. for example, the config `jobs.myjob.backend.rclone_config.host: myhost.com` would be converted to `RCLONE_CONFIG_MYJOB_HOST=myhost.com`.\n\n> [!NOTE]\n> the password needs to be stored in its obscured form, **charon will not obscure the password for you**. you can obscure your password using `rclone obscure`\n\n```yml\nbackend:\n type: rclone\n path: path/to/repo # path within rclone target for repository\n rclone_config: # configuration to pass through to rclone env vars\n type: ftp\n host: my-host.com\n user: my-username\n pass: ... # obscured password\n port: 21\n explicit_tls: \"true\"\n```\n\n## schedule\n\nhow often the program is run. there are a few different ways to configure the schedule\n\n**cron**\n\nthe schedule can be configured using a cron string.\n\nnote: this program uses [croniter](https://github.com/kiorky/croniter) for scheduling with the cron format. Croniter accepts seconds, but they must be at the _end_ (right hand side) of the cron string.\n\n```yml\nschedule:\n cron: \"* * * * * */10\" # every 10 seconds\n```\n\n**one shot**\n\nthis runs once, after the given delay. the delay is given in the `1d2h3m4s` format. numbers must be integers.\n\n```yml\nschedule:\n after: 1d # wait 1 day, then run once\n```\n\n**intervals**\n\nthis runs at regular intervals, using the one shot format, starting from the time charon is run. \n\n```yml\nschedule:\n every: 1h30m # run every hour and a half\n```\n\n**combinations**\n\nyou can combine schedules, for example to run immediately, and then every other day\n\n```yml\nschedule:\n after: 0s\n every: 2d\n```\n\n**timeout**\n\noptionally, you can specify a job timeout. if the job (both the fetch and the upload) do not complete within the timeout, the job will be cancelled.\n\n```yml\nschedule:\n every: 1d\n timeout: 15m\n```\n\n# cli\n\ncharon provides a cli for manual work. the `apply` command can be used to run a job once, immediately.\n\n```bash\ncharon apply MY_JOB\n```\n\ncharon can also run the job in reverse, pulling it from the destination and dumping it to a given directory\n\n```bash\ncharon revert MY_JOB OUTPUT_DIRECTORY\n```\n\nyou can specify the config file before running either command\n\n```bash\ncharon -f MY_CONFIG.yml styx apply MY_JOB\n```\n\nsee tests for more examples.\n\n## tests\n\neach `test*.sh` file will run some commands (must be run inside the tests folder, with a python environment set up for charon), and has a comment in the file detailing the expected output. \n\n```bash\ncd tests\n./test.sh\n./test2.sh\n...\n```\n\n",
"bugtrack_url": null,
"license": null,
"summary": "charon is a utility for backing up data from one location to another at regular intervals.",
"version": "2.1.0",
"project_urls": {
"Homepage": "https://gitlab.com/haondt/charon",
"Repository": "https://gitlab.com/haondt/charon"
},
"split_keywords": [
"backup",
" recovery"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "d6a6151384f31c5fb1b74a6d3d180d3e9d1b34c387cab6f34c88c924787c9874",
"md5": "b69b20d064ebbd87eaf435857b990088",
"sha256": "b9bcf6ad73b99741aa00ee5c7823614257e9f39dfef8585ceb4d365cecb75368"
},
"downloads": -1,
"filename": "haondt_charon-2.1.0-py2.py3-none-any.whl",
"has_sig": false,
"md5_digest": "b69b20d064ebbd87eaf435857b990088",
"packagetype": "bdist_wheel",
"python_version": "py2.py3",
"requires_python": null,
"size": 12840,
"upload_time": "2025-02-26T19:41:22",
"upload_time_iso_8601": "2025-02-26T19:41:22.602191Z",
"url": "https://files.pythonhosted.org/packages/d6/a6/151384f31c5fb1b74a6d3d180d3e9d1b34c387cab6f34c88c924787c9874/haondt_charon-2.1.0-py2.py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "8c92bd9c798a5cf37a00711f45c0a2e588460d1d5010654344fdb1be24206335",
"md5": "2ba26c0ed7943bc7d1c2489902cc18d1",
"sha256": "58ced95f96ae92bc997aaf73dc2d9ad9974976399783e6d760b3893aab6ed8f1"
},
"downloads": -1,
"filename": "haondt_charon-2.1.0.tar.gz",
"has_sig": false,
"md5_digest": "2ba26c0ed7943bc7d1c2489902cc18d1",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 15046,
"upload_time": "2025-02-26T19:41:23",
"upload_time_iso_8601": "2025-02-26T19:41:23.804313Z",
"url": "https://files.pythonhosted.org/packages/8c/92/bd9c798a5cf37a00711f45c0a2e588460d1d5010654344fdb1be24206335/haondt_charon-2.1.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-02-26 19:41:23",
"github": false,
"gitlab": true,
"bitbucket": false,
"codeberg": false,
"gitlab_user": "haondt",
"gitlab_project": "charon",
"lcname": "haondt-charon"
}
