# leveled-hotbackup-s3-sync
A Python 3.8 tool which can backup and restore leveled hotbackups with Amazon S3, and optionally retrieve individual objects from a backup in Amazon S3.
## Quick Start
For backup and restore with S3.
```
python3.8 -m venv .venv
source .venv/bin/activate
pip install leveled-hotbackup-s3-sync
# Usage
s3sync [backup|restore] [tag] --config config.cfg
[backup|restore] - specify operation to perform
[tag] - alphanumeric string to tag backup, or to select which backup to restore from
--config config.cfg - filename for the config file, see example config.cfg below
# Backup example
s3sync backup 123 --config config.cfg
# Restore example
s3sync restore 123 --config config.cfg
```
To use the object retrieval functionality, the python package must be installed with extras, shown below.
```
python3.8 -m venv .venv
source .venv/bin/activate
pip install 'leveled-hotbackup-s3-sync[retrieve]'
# Usage
s3retrieve [tag] --config config.cfg --bucket bucketName --key keyName [--buckettype type] [--output filename]
[tag] - alphanumeric string to select which backup to retrive object from
--config config.cfg - filename for the config file, see example config.cfg below
--bucket - name of bucket to retrieve object from
--key - name of object key to retrieve
[--buckettype type] - optional, bucket type to retrieve object from
[--output filename] - optional, filename to write out the object. If omitted, object value will be printed to screen.
```
## About
This tool will backup and restore LevelEd (https://github.com/martinsumner/leveled) hotbackups to/from Amazon S3.
Intended to be used with Riak KV.
Backup will use the local Riak ring data at `ring_path` to determine which partitions are owned by the local node, then upload each hotbackup from the `hotbackup_path` to `s3_path`. When backup is being uploaded to S3, the manifest files are updated to reference the new S3 URIs for the journal files. Specifiy a unique `tag` for each backup (this is then used by restore).
`hints_files = true` option in config will also create a hints file for every journal file. The hints file is a CDB of Bucket/Key to sequence number.
The hints file is required for single object retrieval directly from S3. If a backup has been saved to S3 without this option, then attempting to perform a single object retrieval will result in an error.
To restore to local filesystem, the same `tag` used during backup must be used.
Restore will use the local Riak ring data at `ring_path` to determine which partitions are owned by the local node. Restore will then download the relevant tagged manifest from `s3_path` to the local `leveled_path` and download journal files if needed.
New manifests are then written locally with updated references to the new journal file locations.
## Example config.cfg
```
# hotbackup_path is the local filesystem path to the
# directory containing LevelEd hotbackups.
# e.g. what is passed to riak_client:hotbackup()
hotbackup_path = "/riak/hotbackup"
# ring_path is the local filesystem path to the
# Riak ring data directory. This is used to determine partition ownership
# of the current node.
ring_path = "/riak/data/ring"
# leveled_path is the local filesystem path to the
# Riak leveled data directory. This is used during restore.
leveled_path = "/riak/data/leveled"
# s3_path is the S3 URI to store/retrive the hotbackup
s3_path = "s3://bucket-name/hotbackup/"
# hints_file when set to true will create a hints file
# for each journal file. This is only needed for object level
# retrieval from S3
# valid values: true|false
# Optional
hints_files = true
# s3_endpoint can be used when the target S3 service
# is not the standard AWS S3 (e.g. localstack)
# Optional. Omit to use standard AWS URLs
# s3_endpoint = "http://localhost:4566"
```
## Testing
To aid testing a `docker-compose.yml` is included in the `localstack` directory which will start Localstack to use as a local S3 service.
```
make localstack
```
The `s3_endpoint` config parameter can then be set in config.cfg to use the localstack S3 endpoint URL (e.g. "http://localhost:4566").
## Riak backup example
To perform a Riak LevelEd hotbackup.
```
sudo riak remote_console
{ok, C} = riak:local_client().
riak_client:hotbackup("/riak/hotbackup",3,3,C).
```
Raw data
{
"_id": null,
"home_page": "https://github.com/yorkshire-steve/leveled-hotbackup-s3-sync",
"name": "leveled-hotbackup-s3-sync",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.8,<4.0",
"maintainer_email": "",
"keywords": "riak,leveled,backup,s3",
"author": "yorkshire-steve",
"author_email": "",
"download_url": "",
"platform": null,
"description": "# leveled-hotbackup-s3-sync\nA Python 3.8 tool which can backup and restore leveled hotbackups with Amazon S3, and optionally retrieve individual objects from a backup in Amazon S3.\n\n## Quick Start\nFor backup and restore with S3.\n```\npython3.8 -m venv .venv\nsource .venv/bin/activate\npip install leveled-hotbackup-s3-sync\n\n# Usage\ns3sync [backup|restore] [tag] --config config.cfg\n\n[backup|restore] - specify operation to perform\n[tag] - alphanumeric string to tag backup, or to select which backup to restore from\n--config config.cfg - filename for the config file, see example config.cfg below\n\n# Backup example\ns3sync backup 123 --config config.cfg\n\n# Restore example\ns3sync restore 123 --config config.cfg\n```\n\nTo use the object retrieval functionality, the python package must be installed with extras, shown below.\n```\npython3.8 -m venv .venv\nsource .venv/bin/activate\npip install 'leveled-hotbackup-s3-sync[retrieve]'\n\n# Usage\ns3retrieve [tag] --config config.cfg --bucket bucketName --key keyName [--buckettype type] [--output filename]\n\n[tag] - alphanumeric string to select which backup to retrive object from\n--config config.cfg - filename for the config file, see example config.cfg below\n--bucket - name of bucket to retrieve object from\n--key - name of object key to retrieve\n[--buckettype type] - optional, bucket type to retrieve object from\n[--output filename] - optional, filename to write out the object. If omitted, object value will be printed to screen.\n```\n\n## About\nThis tool will backup and restore LevelEd (https://github.com/martinsumner/leveled) hotbackups to/from Amazon S3.\n\nIntended to be used with Riak KV.\n\nBackup will use the local Riak ring data at `ring_path` to determine which partitions are owned by the local node, then upload each hotbackup from the `hotbackup_path` to `s3_path`. When backup is being uploaded to S3, the manifest files are updated to reference the new S3 URIs for the journal files. Specifiy a unique `tag` for each backup (this is then used by restore).\n\n`hints_files = true` option in config will also create a hints file for every journal file. The hints file is a CDB of Bucket/Key to sequence number. \nThe hints file is required for single object retrieval directly from S3. If a backup has been saved to S3 without this option, then attempting to perform a single object retrieval will result in an error.\n\nTo restore to local filesystem, the same `tag` used during backup must be used.\nRestore will use the local Riak ring data at `ring_path` to determine which partitions are owned by the local node. Restore will then download the relevant tagged manifest from `s3_path` to the local `leveled_path` and download journal files if needed.\nNew manifests are then written locally with updated references to the new journal file locations.\n\n## Example config.cfg\n```\n# hotbackup_path is the local filesystem path to the\n# directory containing LevelEd hotbackups.\n# e.g. what is passed to riak_client:hotbackup()\nhotbackup_path = \"/riak/hotbackup\"\n\n# ring_path is the local filesystem path to the\n# Riak ring data directory. This is used to determine partition ownership\n# of the current node.\nring_path = \"/riak/data/ring\"\n\n# leveled_path is the local filesystem path to the\n# Riak leveled data directory. This is used during restore.\nleveled_path = \"/riak/data/leveled\"\n\n# s3_path is the S3 URI to store/retrive the hotbackup\ns3_path = \"s3://bucket-name/hotbackup/\"\n\n# hints_file when set to true will create a hints file\n# for each journal file. This is only needed for object level\n# retrieval from S3\n# valid values: true|false\n# Optional\nhints_files = true\n\n# s3_endpoint can be used when the target S3 service\n# is not the standard AWS S3 (e.g. localstack)\n# Optional. Omit to use standard AWS URLs\n# s3_endpoint = \"http://localhost:4566\"\n```\n\n## Testing\nTo aid testing a `docker-compose.yml` is included in the `localstack` directory which will start Localstack to use as a local S3 service.\n```\nmake localstack\n```\n\nThe `s3_endpoint` config parameter can then be set in config.cfg to use the localstack S3 endpoint URL (e.g. \"http://localhost:4566\").\n\n## Riak backup example\nTo perform a Riak LevelEd hotbackup.\n```\nsudo riak remote_console\n{ok, C} = riak:local_client().\nriak_client:hotbackup(\"/riak/hotbackup\",3,3,C).\n```\n",
"bugtrack_url": null,
"license": "Apache-2.0",
"summary": "Tool to syncrhonise LevelEd hotbackup files to Amazon S3",
"version": "1.0.2",
"project_urls": {
"Homepage": "https://github.com/yorkshire-steve/leveled-hotbackup-s3-sync",
"Repository": "https://github.com/yorkshire-steve/leveled-hotbackup-s3-sync"
},
"split_keywords": [
"riak",
"leveled",
"backup",
"s3"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "efa94081747a4ae9b656967e2326876041ce651450f38c02c0f1285e955626ff",
"md5": "afcb818e13d878ebbc6d847e4562cc50",
"sha256": "fab163fa832bc8403fc9782d96d5617ee51b8cac521ce58df0d93729054a2e92"
},
"downloads": -1,
"filename": "leveled_hotbackup_s3_sync-1.0.2-py3-none-any.whl",
"has_sig": false,
"md5_digest": "afcb818e13d878ebbc6d847e4562cc50",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8,<4.0",
"size": 55414,
"upload_time": "2024-02-16T12:32:17",
"upload_time_iso_8601": "2024-02-16T12:32:17.147724Z",
"url": "https://files.pythonhosted.org/packages/ef/a9/4081747a4ae9b656967e2326876041ce651450f38c02c0f1285e955626ff/leveled_hotbackup_s3_sync-1.0.2-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-02-16 12:32:17",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "yorkshire-steve",
"github_project": "leveled-hotbackup-s3-sync",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "leveled-hotbackup-s3-sync"
}
JSON
