haondt-charon


Namehaondt-charon JSON
Version 1.2.0 PyPI version JSON
download
home_pageNone
Summarycharon is a utility for backing up data from one location to another at regular intervals.
upload_time2024-05-23 12:38:18
maintainerNone
docs_urlNone
authorhaondt
requires_pythonNone
licenseNone
keywords backup recovery
VCS
bugtrack_url
requirements No requirements were recorded.
Travis-CI No Travis.
coveralls test coverage No coveralls.
            # charon

charon is a utility for backing up data from one location to another at regular intervals.

# table of contents

- [installation](#installation)
- [usage](#usage)
- [configuration](#configuration)
    - [sources](#sources)
    - [destinations](#destinations)
    - [schedule](#schedule)
- [styx](#styx)
- [tests](#tests)

# installation

charon can be installed as a docker image

```bash
# from docker hub
docker pull haumea/charon
# from gitlab
docker pull registry.gitlab.com/haondt/cicd/registry/charon:latest
```

see `docker-compose.yml` for a sample docker compose setup.

charon can also be installed as a python package

```bash
# from pypi
pip install haondt-charon
# from gitlab
pip install haondt-charon --index-url https://gitlab.com/api/v4/projects/57154225/packages/pypi/simple
```

# usage

start charon with:

```bash
# if installed as a python package, or if running from source
python3 -m charon
# the pypi package also includes a standlone binary
charon
# from the docker image
docker run --rm -it -v ./charon.yml:/config/charon.yml registry.gitlab.com/haondt/cicd/registry/charon:latest
```

charon will look for a config file at `charon.yml`. a different path can be specified with:

```bash
charon -f MY_CONFIG.yml
```

charon uses the `sched` library for scheduling tasks, meaning charon will exit when there are no more tasks to run. this is possible depending on the configuration.

# configuration

configuration is given as a yaml file with the following structure:

```yml
# NOTE: to use gcp buckets, charon must be run in an environment where GOOGLE_APPLICATION_CREDENTIALS exists
gcp_buckets: # optional, configuration for gcp buckets
    my_bucket:
        bucket: 1234-name-of-my-bucket-in-gcp

jobs:
    my_job:
        source: # where data is coming from
            type: type_of_source
            # ...
        destination: # where data is going
            type: type_of_destination
            # ...
        schedule: # how often to run job
            # ...
    my_job_2:
        # ...
```

see `charon.yml` for an example config.

## sources

all sources will have a few shared fields:

```yaml
source:
    type: local # determines how to interpret the source config
    encrypt: 4B71... # optional, 32 byte hex-encoded encryption key

```

the data from the source will be archived in a gz'd tar file. if an encryption key is provided, the tar file will then be encrypted.


below are the possible ways you can configure the source object, based on the `type` key.

**local**

this pulls from a local file

```yml
source:
    type: local
    path: /path/to/data # path to data to back up. can be a file or a directory. does not use variable expansion

```

**http**

performs an http request, and saves the response body to a file

```yml
source:
    type: http
    url: http://example.com/ # url to make request to
    method: get # optional, request method, defaults to get
    ext: json # optional, extension to use for saved file, defaults to txt
    auth:  # optional, authentication configuration
        bearer: eyJhbGc... # optional, bearer token
```

**sqlite**

performs a backup on an sqlite3 db

```yml
source:
    type: sqlite
    db_path: /path/to/db_file.db
```

## destinations

all destinations will also have some shared fields

```yml
destination:
    type: local # determines how to interpret the destination config
    name: my_output # the name of the output file, can include path seperators (foo/bar)
```

**note**: the name of the file (where applicable) in the destination will be `destination.name` + a file extension determined by the source.

bewlow are the possible ways you can configure the destination object, based on the `type` key.

**local**

this pushes to a local file

```yml
destination:
    type: local
    path: ./foo # must be a directory, file will be created inside this dir
    overwrite: false # optional, whether or not to overwrite an existing output file. defaults to false
```

**gcp_bucket**

uploads to a google cloud storage bucket. requires `gcp_buckets` to be configured, and `GOOGLE_APPLICATION_CREDENTIALS` envrionment variable.


```yml
destination:
    type: gcp_bucket
    config: my-bucket # name of config in gcp_buckets:
```

## schedule

how often the program is run. there are a few different ways to configure the schedule

**cron**

the schedule can be configured using a cron string.

note: this program uses [croniter](https://github.com/kiorky/croniter) for scheduling with the cron format. Croniter accepts seconds, but they must be at the _end_ (right hand side) of the cron string.

```yml
schedule:
    cron: "* * * * * */10" # every 10 seconds
```

**one shot**

this runs once, after the given delay. the delay is given in the `1d2h3m4s` format. numbers must be integers.

```yml
schedule:
    after: 1d # wait 1 day, then run once
```

**intervals**

this runs at regular intervals, using the one shot format, starting from the time charon is run. 

```yml
schedule:
    every: 1h30m # run every hour and a half
```

**combinations**

you can combine schedules, for example to run immediately, and then every other day

```yml
schedule:
    after: 0s
    every: 2d
```

# styx

charon includes a subcommand, `styx`, that will run a job once, immediately.

```bash
charon styx apply MY_JOB
```

styx can also run the job in reverse, pulling it from the destination and dumping it to a given directory

```bash
charon styx revert MY_JOB OUTPUT_DIRECTORY
```

you can specify the config file before calling styx

```bash
charon -f MY_CONFIG.yml styx apply MY_JOB
```

see tests for more examples.

# tests

each `test*.sh` file will run some commands (must be run inside the tests folder, with a python environment set up for charon), and has a comment in the file detailing the expected output. 

```bash
cd tests
./test.sh
./test2.sh
...
```


            

Raw data

            {
    "_id": null,
    "home_page": null,
    "name": "haondt-charon",
    "maintainer": null,
    "docs_url": null,
    "requires_python": null,
    "maintainer_email": null,
    "keywords": "backup, recovery",
    "author": "haondt",
    "author_email": null,
    "download_url": "https://files.pythonhosted.org/packages/01/04/3b97ea4c6d6f104ca51018af2b1cfd28ef572a20b8c1696c7cdf4fb5bf05/haondt_charon-1.2.0.tar.gz",
    "platform": null,
    "description": "# charon\n\ncharon is a utility for backing up data from one location to another at regular intervals.\n\n# table of contents\n\n- [installation](#installation)\n- [usage](#usage)\n- [configuration](#configuration)\n    - [sources](#sources)\n    - [destinations](#destinations)\n    - [schedule](#schedule)\n- [styx](#styx)\n- [tests](#tests)\n\n# installation\n\ncharon can be installed as a docker image\n\n```bash\n# from docker hub\ndocker pull haumea/charon\n# from gitlab\ndocker pull registry.gitlab.com/haondt/cicd/registry/charon:latest\n```\n\nsee `docker-compose.yml` for a sample docker compose setup.\n\ncharon can also be installed as a python package\n\n```bash\n# from pypi\npip install haondt-charon\n# from gitlab\npip install haondt-charon --index-url https://gitlab.com/api/v4/projects/57154225/packages/pypi/simple\n```\n\n# usage\n\nstart charon with:\n\n```bash\n# if installed as a python package, or if running from source\npython3 -m charon\n# the pypi package also includes a standlone binary\ncharon\n# from the docker image\ndocker run --rm -it -v ./charon.yml:/config/charon.yml registry.gitlab.com/haondt/cicd/registry/charon:latest\n```\n\ncharon will look for a config file at `charon.yml`. a different path can be specified with:\n\n```bash\ncharon -f MY_CONFIG.yml\n```\n\ncharon uses the `sched` library for scheduling tasks, meaning charon will exit when there are no more tasks to run. this is possible depending on the configuration.\n\n# configuration\n\nconfiguration is given as a yaml file with the following structure:\n\n```yml\n# NOTE: to use gcp buckets, charon must be run in an environment where GOOGLE_APPLICATION_CREDENTIALS exists\ngcp_buckets: # optional, configuration for gcp buckets\n    my_bucket:\n        bucket: 1234-name-of-my-bucket-in-gcp\n\njobs:\n    my_job:\n        source: # where data is coming from\n            type: type_of_source\n            # ...\n        destination: # where data is going\n            type: type_of_destination\n            # ...\n        schedule: # how often to run job\n            # ...\n    my_job_2:\n        # ...\n```\n\nsee `charon.yml` for an example config.\n\n## sources\n\nall sources will have a few shared fields:\n\n```yaml\nsource:\n    type: local # determines how to interpret the source config\n    encrypt: 4B71... # optional, 32 byte hex-encoded encryption key\n\n```\n\nthe data from the source will be archived in a gz'd tar file. if an encryption key is provided, the tar file will then be encrypted.\n\n\nbelow are the possible ways you can configure the source object, based on the `type` key.\n\n**local**\n\nthis pulls from a local file\n\n```yml\nsource:\n    type: local\n    path: /path/to/data # path to data to back up. can be a file or a directory. does not use variable expansion\n\n```\n\n**http**\n\nperforms an http request, and saves the response body to a file\n\n```yml\nsource:\n    type: http\n    url: http://example.com/ # url to make request to\n    method: get # optional, request method, defaults to get\n    ext: json # optional, extension to use for saved file, defaults to txt\n    auth:  # optional, authentication configuration\n        bearer: eyJhbGc... # optional, bearer token\n```\n\n**sqlite**\n\nperforms a backup on an sqlite3 db\n\n```yml\nsource:\n    type: sqlite\n    db_path: /path/to/db_file.db\n```\n\n## destinations\n\nall destinations will also have some shared fields\n\n```yml\ndestination:\n    type: local # determines how to interpret the destination config\n    name: my_output # the name of the output file, can include path seperators (foo/bar)\n```\n\n**note**: the name of the file (where applicable) in the destination will be `destination.name` + a file extension determined by the source.\n\nbewlow are the possible ways you can configure the destination object, based on the `type` key.\n\n**local**\n\nthis pushes to a local file\n\n```yml\ndestination:\n    type: local\n    path: ./foo # must be a directory, file will be created inside this dir\n    overwrite: false # optional, whether or not to overwrite an existing output file. defaults to false\n```\n\n**gcp_bucket**\n\nuploads to a google cloud storage bucket. requires `gcp_buckets` to be configured, and `GOOGLE_APPLICATION_CREDENTIALS` envrionment variable.\n\n\n```yml\ndestination:\n    type: gcp_bucket\n    config: my-bucket # name of config in gcp_buckets:\n```\n\n## schedule\n\nhow often the program is run. there are a few different ways to configure the schedule\n\n**cron**\n\nthe schedule can be configured using a cron string.\n\nnote: this program uses [croniter](https://github.com/kiorky/croniter) for scheduling with the cron format. Croniter accepts seconds, but they must be at the _end_ (right hand side) of the cron string.\n\n```yml\nschedule:\n    cron: \"* * * * * */10\" # every 10 seconds\n```\n\n**one shot**\n\nthis runs once, after the given delay. the delay is given in the `1d2h3m4s` format. numbers must be integers.\n\n```yml\nschedule:\n    after: 1d # wait 1 day, then run once\n```\n\n**intervals**\n\nthis runs at regular intervals, using the one shot format, starting from the time charon is run. \n\n```yml\nschedule:\n    every: 1h30m # run every hour and a half\n```\n\n**combinations**\n\nyou can combine schedules, for example to run immediately, and then every other day\n\n```yml\nschedule:\n    after: 0s\n    every: 2d\n```\n\n# styx\n\ncharon includes a subcommand, `styx`, that will run a job once, immediately.\n\n```bash\ncharon styx apply MY_JOB\n```\n\nstyx can also run the job in reverse, pulling it from the destination and dumping it to a given directory\n\n```bash\ncharon styx revert MY_JOB OUTPUT_DIRECTORY\n```\n\nyou can specify the config file before calling styx\n\n```bash\ncharon -f MY_CONFIG.yml styx apply MY_JOB\n```\n\nsee tests for more examples.\n\n# tests\n\neach `test*.sh` file will run some commands (must be run inside the tests folder, with a python environment set up for charon), and has a comment in the file detailing the expected output. \n\n```bash\ncd tests\n./test.sh\n./test2.sh\n...\n```\n\n",
    "bugtrack_url": null,
    "license": null,
    "summary": "charon is a utility for backing up data from one location to another at regular intervals.",
    "version": "1.2.0",
    "project_urls": {
        "Homepage": "https://gitlab.com/haondt/charon",
        "Repository": "https://gitlab.com/haondt/charon"
    },
    "split_keywords": [
        "backup",
        " recovery"
    ],
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "blake2b_256": "8f8a749692cd889930f08afaf1366da5f2e1443c190b1c22c918e313a921ee0f",
                "md5": "bb62ae0d7fab6ba0cb77dbe5baa551bf",
                "sha256": "6ea665a0f5ae327ba8bc3b49aef2f85ddedd6177860a4a02300f59f5651d03ef"
            },
            "downloads": -1,
            "filename": "haondt_charon-1.2.0-py2.py3-none-any.whl",
            "has_sig": false,
            "md5_digest": "bb62ae0d7fab6ba0cb77dbe5baa551bf",
            "packagetype": "bdist_wheel",
            "python_version": "py2.py3",
            "requires_python": null,
            "size": 12797,
            "upload_time": "2024-05-23T12:38:17",
            "upload_time_iso_8601": "2024-05-23T12:38:17.225239Z",
            "url": "https://files.pythonhosted.org/packages/8f/8a/749692cd889930f08afaf1366da5f2e1443c190b1c22c918e313a921ee0f/haondt_charon-1.2.0-py2.py3-none-any.whl",
            "yanked": false,
            "yanked_reason": null
        },
        {
            "comment_text": "",
            "digests": {
                "blake2b_256": "01043b97ea4c6d6f104ca51018af2b1cfd28ef572a20b8c1696c7cdf4fb5bf05",
                "md5": "e16bc819fb2ff596dd1714985cb3db4f",
                "sha256": "04ec8155098cbad5d144cde382c27d03f8e87d0682b96f98e4f25b2991e3575e"
            },
            "downloads": -1,
            "filename": "haondt_charon-1.2.0.tar.gz",
            "has_sig": false,
            "md5_digest": "e16bc819fb2ff596dd1714985cb3db4f",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": null,
            "size": 13132,
            "upload_time": "2024-05-23T12:38:18",
            "upload_time_iso_8601": "2024-05-23T12:38:18.338017Z",
            "url": "https://files.pythonhosted.org/packages/01/04/3b97ea4c6d6f104ca51018af2b1cfd28ef572a20b8c1696c7cdf4fb5bf05/haondt_charon-1.2.0.tar.gz",
            "yanked": false,
            "yanked_reason": null
        }
    ],
    "upload_time": "2024-05-23 12:38:18",
    "github": false,
    "gitlab": true,
    "bitbucket": false,
    "codeberg": false,
    "gitlab_user": "haondt",
    "gitlab_project": "charon",
    "lcname": "haondt-charon"
}
        
Elapsed time: 0.24237s