| Name | haondt-charon JSON |
| Version |
1.0.4
JSON |
| download |
| home_page | None |
| Summary | charon is a utility for backing up data from one location to another at regular intervals. |
| upload_time | 2024-05-03 22:31:26 |
| maintainer | None |
| docs_url | None |
| author | haondt |
| requires_python | None |
| license | None |
| keywords |
backup
recovery
|
| VCS |
|
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
# charon
charon is a utility for backing up data from one location to another at regular intervals.
# table of contents
- [installation](#installation)
- [usage](#usage)
- [configuration](#configuration)
- [sources](#sources)
- [destinations](#destinations)
- [schedule](#schedule)
- [styx](#styx)
- [tests](#tests)
# installation
charon can be installed as a docker image
```bash
# from docker hub
docker pull haumea/charon
# from gitlab
docker pull registry.gitlab.com/haondt/cicd/registry/charon:latest
```
see `docker-compose.yml` for a sample docker compose setup.
charon can also be installed as a python package
```bash
# from pypi
pip install haondt-charon
# from gitlab
pip install haondt-charon --index-url https://gitlab.com/api/v4/projects/57154225/packages/pypi/simple
```
# usage
start charon with:
```bash
# if installed as a python package, or if running from source
python3 -m charon
# the pypi package also includes a standlone binary
charon
# from the docker image
docker run --rm -it -v ./charon.yml:/config/charon.yml registry.gitlab.com/haondt/cicd/registry/charon:latest
```
charon will look for a config file at `charon.yml`. a different path can be specified with:
```bash
charon -f MY_CONFIG.yml
```
charon uses the `sched` library for scheduling tasks, meaning charon will exit when there are no more tasks to run. this is possible depending on the configuration.
# configuration
configuration is given as a yaml file with the following structure:
```yml
# NOTE: to use gcp buckets, charon must be run in an environment where GOOGLE_APPLICATION_CREDENTIALS exists
gcp_buckets: # optional, configuration for gcp buckets
my_bucket:
bucket: 1234-name-of-my-bucket-in-gcp
jobs:
my_job:
source: # where data is coming from
type: type_of_source
# ...
destination: # where data is going
type: type_of_destination
# ...
schedule: # how often to run job
# ...
my_job_2:
# ...
```
see `charon.yml` for an example config.
## sources
all sources will have a few shared fields:
```yaml
source:
type: local # determines how to interpret the source config
encrypt: 4B71... # optional, 32 byte hex-encoded encryption key
```
the data from the source will be archived in a gz'd tar file. if an encryption key is provided, the tar file will then be encrypted.
below are the possible ways you can configure the source object, based on the `type` key.
**local**
this pulls from a local file
```yml
source:
type: local
path: /path/to/data # path to data to back up. can be a file or a directory. does not use variable expansion
```
**http**
performs an http request, and saves the response body to a file
```yml
source:
type: http
url: http://example.com/ # url to make request to
method: get # optional, request method, defaults to get
ext: json # optional, extension to use for saved file, defaults to txt
auth: # optional, authentication configuration
bearer: eyJhbGc... # optional, bearer token
```
## destinations
all destinations will also have some shared fields
```yml
destination:
type: local # determines how to interpret the destination config
name: my_output # the name of the output file, can include path seperators (foo/bar)
```
**note**: the name of the file (where applicable) in the destination will be `destination.name` + a file extension determined by the source.
bewlow are the possible ways you can configure the destination object, based on the `type` key.
**local**
this pushes to a local file
```yml
destination:
type: local
path: ./foo # must be a directory, file will be created inside this dir
overwrite: false # optional, whether or not to overwrite an existing output file. defaults to false
```
**gcp_bucket**
uploads to a google cloud storage bucket. requires `gcp_buckets` to be configured, and `GOOGLE_APPLICATION_CREDENTIALS` envrionment variable.
```yml
destination:
type: gcp_bucket
config: my-bucket # name of config in gcp_buckets:
```
## schedule
how often the program is run. there are a few different ways to configure the schedule
**cron**
the schedule can be configured using a cron string.
note: this program uses [croniter](https://github.com/kiorky/croniter) for scheduling with the cron format. Croniter accepts seconds, but they must be at the _end_ (right hand side) of the cron string.
```yml
schedule:
cron: "* * * * * */10" # every 10 seconds
```
**one shot**
this runs once, after the given delay. the delay is given in the `1d2h3m4s` format. numbers must be integers.
```yml
schedule:
after: 1d # wait 1 day, then run once
```
**intervals**
this runs at regular intervals, using the one shot format, starting from the time charon is run.
```yml
schedule:
every: 1h30m # run every hour and a half
```
**combinations**
you can combine schedules, for example to run immediately, and then every other day
```yml
schedule:
after: 0s
every: 2d
```
# styx
charon includes a subcommand, `styx`, that will run a job once, immediately.
```bash
charon styx apply MY_JOB
```
styx can also run the job in reverse, pulling it from the destination and dumping it to a given directory
```bash
charon styx revert MY_JOB OUTPUT_DIRECTORY
```
you can specify the config file before calling styx
```bash
charon -f MY_CONFIG.yml styx apply MY_JOB
```
see tests for more examples.
# tests
each `test*.sh` file will run some commands (must be run inside the tests folder, with a python environment set up for charon), and has a comment in the file detailing the expected output.
```bash
cd tests
./test.sh
./test2.sh
...
```
Raw data
{
"_id": null,
"home_page": null,
"name": "haondt-charon",
"maintainer": null,
"docs_url": null,
"requires_python": null,
"maintainer_email": null,
"keywords": "backup, recovery",
"author": "haondt",
"author_email": null,
"download_url": "https://files.pythonhosted.org/packages/82/7e/01f688b2f73a56bd8522f234b34b3c127c2d3ff52f262e367edd303b09aa/haondt_charon-1.0.4.tar.gz",
"platform": null,
"description": "# charon\n\ncharon is a utility for backing up data from one location to another at regular intervals.\n\n# table of contents\n\n- [installation](#installation)\n- [usage](#usage)\n- [configuration](#configuration)\n - [sources](#sources)\n - [destinations](#destinations)\n - [schedule](#schedule)\n- [styx](#styx)\n- [tests](#tests)\n\n# installation\n\ncharon can be installed as a docker image\n\n```bash\n# from docker hub\ndocker pull haumea/charon\n# from gitlab\ndocker pull registry.gitlab.com/haondt/cicd/registry/charon:latest\n```\n\nsee `docker-compose.yml` for a sample docker compose setup.\n\ncharon can also be installed as a python package\n\n```bash\n# from pypi\npip install haondt-charon\n# from gitlab\npip install haondt-charon --index-url https://gitlab.com/api/v4/projects/57154225/packages/pypi/simple\n```\n\n# usage\n\nstart charon with:\n\n```bash\n# if installed as a python package, or if running from source\npython3 -m charon\n# the pypi package also includes a standlone binary\ncharon\n# from the docker image\ndocker run --rm -it -v ./charon.yml:/config/charon.yml registry.gitlab.com/haondt/cicd/registry/charon:latest\n```\n\ncharon will look for a config file at `charon.yml`. a different path can be specified with:\n\n```bash\ncharon -f MY_CONFIG.yml\n```\n\ncharon uses the `sched` library for scheduling tasks, meaning charon will exit when there are no more tasks to run. this is possible depending on the configuration.\n\n# configuration\n\nconfiguration is given as a yaml file with the following structure:\n\n```yml\n# NOTE: to use gcp buckets, charon must be run in an environment where GOOGLE_APPLICATION_CREDENTIALS exists\ngcp_buckets: # optional, configuration for gcp buckets\n my_bucket:\n bucket: 1234-name-of-my-bucket-in-gcp\n\njobs:\n my_job:\n source: # where data is coming from\n type: type_of_source\n # ...\n destination: # where data is going\n type: type_of_destination\n # ...\n schedule: # how often to run job\n # ...\n my_job_2:\n # ...\n```\n\nsee `charon.yml` for an example config.\n\n## sources\n\nall sources will have a few shared fields:\n\n```yaml\nsource:\n type: local # determines how to interpret the source config\n encrypt: 4B71... # optional, 32 byte hex-encoded encryption key\n\n```\n\nthe data from the source will be archived in a gz'd tar file. if an encryption key is provided, the tar file will then be encrypted.\n\n\nbelow are the possible ways you can configure the source object, based on the `type` key.\n\n**local**\n\nthis pulls from a local file\n\n```yml\nsource:\n type: local\n path: /path/to/data # path to data to back up. can be a file or a directory. does not use variable expansion\n\n```\n\n**http**\n\nperforms an http request, and saves the response body to a file\n\n```yml\nsource:\n type: http\n url: http://example.com/ # url to make request to\n method: get # optional, request method, defaults to get\n ext: json # optional, extension to use for saved file, defaults to txt\n auth: # optional, authentication configuration\n bearer: eyJhbGc... # optional, bearer token\n```\n\n## destinations\n\nall destinations will also have some shared fields\n\n```yml\ndestination:\n type: local # determines how to interpret the destination config\n name: my_output # the name of the output file, can include path seperators (foo/bar)\n```\n\n**note**: the name of the file (where applicable) in the destination will be `destination.name` + a file extension determined by the source.\n\nbewlow are the possible ways you can configure the destination object, based on the `type` key.\n\n**local**\n\nthis pushes to a local file\n\n```yml\ndestination:\n type: local\n path: ./foo # must be a directory, file will be created inside this dir\n overwrite: false # optional, whether or not to overwrite an existing output file. defaults to false\n```\n\n**gcp_bucket**\n\nuploads to a google cloud storage bucket. requires `gcp_buckets` to be configured, and `GOOGLE_APPLICATION_CREDENTIALS` envrionment variable.\n\n\n```yml\ndestination:\n type: gcp_bucket\n config: my-bucket # name of config in gcp_buckets:\n```\n\n## schedule\n\nhow often the program is run. there are a few different ways to configure the schedule\n\n**cron**\n\nthe schedule can be configured using a cron string.\n\nnote: this program uses [croniter](https://github.com/kiorky/croniter) for scheduling with the cron format. Croniter accepts seconds, but they must be at the _end_ (right hand side) of the cron string.\n\n```yml\nschedule:\n cron: \"* * * * * */10\" # every 10 seconds\n```\n\n**one shot**\n\nthis runs once, after the given delay. the delay is given in the `1d2h3m4s` format. numbers must be integers.\n\n```yml\nschedule:\n after: 1d # wait 1 day, then run once\n```\n\n**intervals**\n\nthis runs at regular intervals, using the one shot format, starting from the time charon is run. \n\n```yml\nschedule:\n every: 1h30m # run every hour and a half\n```\n\n**combinations**\n\nyou can combine schedules, for example to run immediately, and then every other day\n\n```yml\nschedule:\n after: 0s\n every: 2d\n```\n\n# styx\n\ncharon includes a subcommand, `styx`, that will run a job once, immediately.\n\n```bash\ncharon styx apply MY_JOB\n```\n\nstyx can also run the job in reverse, pulling it from the destination and dumping it to a given directory\n\n```bash\ncharon styx revert MY_JOB OUTPUT_DIRECTORY\n```\n\nyou can specify the config file before calling styx\n\n```bash\ncharon -f MY_CONFIG.yml styx apply MY_JOB\n```\n\nsee tests for more examples.\n\n# tests\n\neach `test*.sh` file will run some commands (must be run inside the tests folder, with a python environment set up for charon), and has a comment in the file detailing the expected output. \n\n```bash\ncd tests\n./test.sh\n./test2.sh\n...\n```\n\n",
"bugtrack_url": null,
"license": null,
"summary": "charon is a utility for backing up data from one location to another at regular intervals.",
"version": "1.0.4",
"project_urls": {
"Homepage": "https://gitlab.com/haondt/charon",
"Repository": "https://gitlab.com/haondt/charon"
},
"split_keywords": [
"backup",
" recovery"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "347a16cecbf80721f1aaa90d44f3c1cf23452569f0baab0f7e4245279037e8d7",
"md5": "d91e8fd87b3f1216cbd562ce74dc3271",
"sha256": "cc396425f9544ff89c9612aac24bfb1ea6cfa40eb2e132ddf968734ea71e6f90"
},
"downloads": -1,
"filename": "haondt_charon-1.0.4-py2.py3-none-any.whl",
"has_sig": false,
"md5_digest": "d91e8fd87b3f1216cbd562ce74dc3271",
"packagetype": "bdist_wheel",
"python_version": "py2.py3",
"requires_python": null,
"size": 11766,
"upload_time": "2024-05-03T22:31:24",
"upload_time_iso_8601": "2024-05-03T22:31:24.561081Z",
"url": "https://files.pythonhosted.org/packages/34/7a/16cecbf80721f1aaa90d44f3c1cf23452569f0baab0f7e4245279037e8d7/haondt_charon-1.0.4-py2.py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "827e01f688b2f73a56bd8522f234b34b3c127c2d3ff52f262e367edd303b09aa",
"md5": "f9567752bb27d786627b6fd4fff4e307",
"sha256": "231f3aac565b64b431f59cf62a256ae77c22e555854ff78004acca7e5e26f75c"
},
"downloads": -1,
"filename": "haondt_charon-1.0.4.tar.gz",
"has_sig": false,
"md5_digest": "f9567752bb27d786627b6fd4fff4e307",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 12036,
"upload_time": "2024-05-03T22:31:26",
"upload_time_iso_8601": "2024-05-03T22:31:26.123470Z",
"url": "https://files.pythonhosted.org/packages/82/7e/01f688b2f73a56bd8522f234b34b3c127c2d3ff52f262e367edd303b09aa/haondt_charon-1.0.4.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-05-03 22:31:26",
"github": false,
"gitlab": true,
"bitbucket": false,
"codeberg": false,
"gitlab_user": "haondt",
"gitlab_project": "charon",
"lcname": "haondt-charon"
}
