# Create RPM and DEB repositories in S3
`mkrepo` is a repository generator with pluggable backends,
which allows you to maintain an RPM or DEB repository on various
storages, like local filesystem or S3, and periodically regenerate metadata.
Use it in tandem with your favourite CI system to produce a better pipeline.
`mkrepo` helps you to get rid of ad-hoc cron jobs.
As a bonus, `mkrepo` supports on-premises S3 servers like [Minio](http://minio.io).
Works on Linux and OS X. Should also work on BSD and Windows, but I haven't checked.
## Quickstart
Create an s3 bucket named e.g. `builds` and put a sample package `package.rpm` to `s3://builds/rpmrepo/Packages`. Then do the following:
``` bash
./mkrepo.py s3://builds/rpmrepo
```
After this, you will find all metadata generated in `s3://builds/rpmrepo/repodata`
## Run tests
To run the tests, use the following command::
``` bash
make test
```
## Dependencies
Python libraries:
* boto3
## Command-line reference
`mkrepo` parses your `~/.aws/config` and reads secret key and region settings.
So you may skip them in command line invocation in case you have aws config.
``` bash
mkrepo.py [-h]
[--temp-dir TEMP_DIR]
[--s3-access-key-id S3_ACCESS_KEY_ID]
[--s3-secret-access-key S3_SECRET_ACCESS_KEY]
[--s3-endpoint S3_ENDPOINT]
[--s3-region S3_REGION]
[--s3-public-read]
[--sign]
[--force]
path [path ...]
```
* `--temp-dir` - /(optional)/directory used to store temporary artifacts (default is .mkrepo)
* `--s3-access-key-id` - /(optional)/ specify S3 access key ID
* `--s3-secret-access-key` - /(optional)/ specify S3 secret key
* `--s3-endpoint` - /(optional)/ specify S3 server URI
* `--s3-region` - /(optional)/ specify S3 region (default is us-east-1)
* `--s3-public-read` - /(optional)/ set read-only permission on files uploaded
to S3 for anonymous users
* `--sign` - /(optional) sign package metadata
* `--force` - /(optional) when adding packages to the index, the malformed one
will be skipped. By default, a malformed package will cause the utility to
stop working. The malformed_list.txt file will also be added to the repository
* `path` - specify list of path to scan for repositories
## Environment variables reference
* `GPG_SIGN_KEY` - the name of the key that will be used to sign package metadata.
<details><summary>Tips for working with GPG keys</summary>
* Create a new key:
``` bash
gpg --full-generate-key
```
* To view all your keys, you can use:
``` bash
gpg --list-secret-keys --keyid-format LONG
```
* Scripts can use something like this to get the Key ID:
``` bash
export GPG_SIGN_KEY="$(gpg --list-secret-keys --with-colons | grep ^sec: | cut -d: -f5)"
```
* Export the key in ASCII armored format:
``` bash
gpg --armor --export-secret-keys MYKEYID > mykeys.asc
```
* Import the key:
``` bash
cat mykeys.asc | gpg --batch --import
```
</details>
* `MKREPO_DEB_ORIGIN` - the value of the ["Origin"](https://wiki.debian.org/DebianRepository/Format#Origin)
field of the "Release" file.
* `MKREPO_DEB_LABEL` - the value of the ["Label"](https://wiki.debian.org/DebianRepository/Format#Label)
field of the "Release" file.
* `MKREPO_DEB_DESCRIPTION` - the value of the "Description" field of the "Release" file.
## How it works
`mkrepo` searches the supplied path for either `Packages` or `pool` subdir. If
it finds `Packages`, it assumes an rpm repo. If it finds `pool`, it assumes a
deb repo.
Then it parses existing metadata files (if any) and compares timestamps recorded
there with timestamps of all package files in the repo. Any packages that have
different timestamps or that don't exist in metadata, are parsed and added to
metadata.
Then new metadata is uploaded to S3, replacing previous one.
## Credits
Thanks to [Cyril Rohr](https://github.com/crohr) and [Ken Robertson](https://github.com/krobertson), authors of the following awesome tools:
* [rpm-s3](https://github.com/crohr/rpm-s3)
* [deb-s3](https://github.com/krobertson/deb-s3)
Unfortunately, we needed a solution that is completely decoupled from CI pipeline,
and the mentioned tools only support package push mode, when you have to use a
tool to actually push packages to s3, insted of native s3 clients.
Raw data
{
"_id": null,
"home_page": "https://github.com/tarantool/mkrepo",
"name": "mkrepo",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.6.*",
"maintainer_email": "",
"keywords": "rpm,deb",
"author": "Konstantin Nazarov",
"author_email": "mail@kn.am",
"download_url": "https://files.pythonhosted.org/packages/85/ed/616e994dd0a130e60b69ca13b05650bc2e2aa35d0f9dbcdeac54d5e61e78/mkrepo-1.0.2.tar.gz",
"platform": null,
"description": "# Create RPM and DEB repositories in S3\n\n`mkrepo` is a repository generator with pluggable backends,\nwhich allows you to maintain an RPM or DEB repository on various\nstorages, like local filesystem or S3, and periodically regenerate metadata.\n\nUse it in tandem with your favourite CI system to produce a better pipeline.\n`mkrepo` helps you to get rid of ad-hoc cron jobs.\n\nAs a bonus, `mkrepo` supports on-premises S3 servers like [Minio](http://minio.io).\n\nWorks on Linux and OS X. Should also work on BSD and Windows, but I haven't checked.\n\n## Quickstart\n\nCreate an s3 bucket named e.g. `builds` and put a sample package `package.rpm` to `s3://builds/rpmrepo/Packages`. Then do the following:\n\n``` bash\n./mkrepo.py s3://builds/rpmrepo\n```\n\nAfter this, you will find all metadata generated in `s3://builds/rpmrepo/repodata`\n\n## Run tests\n\nTo run the tests, use the following command::\n\n``` bash\nmake test\n```\n\n## Dependencies\n\nPython libraries:\n\n* boto3\n\n## Command-line reference\n\n`mkrepo` parses your `~/.aws/config` and reads secret key and region settings.\nSo you may skip them in command line invocation in case you have aws config.\n\n``` bash\n mkrepo.py [-h] \n [--temp-dir TEMP_DIR]\n [--s3-access-key-id S3_ACCESS_KEY_ID]\n [--s3-secret-access-key S3_SECRET_ACCESS_KEY]\n [--s3-endpoint S3_ENDPOINT]\n [--s3-region S3_REGION]\n [--s3-public-read]\n [--sign]\n [--force]\n path [path ...]\n```\n\n* `--temp-dir` - /(optional)/directory used to store temporary artifacts (default is .mkrepo)\n* `--s3-access-key-id` - /(optional)/ specify S3 access key ID\n* `--s3-secret-access-key` - /(optional)/ specify S3 secret key\n* `--s3-endpoint` - /(optional)/ specify S3 server URI\n* `--s3-region` - /(optional)/ specify S3 region (default is us-east-1)\n* `--s3-public-read` - /(optional)/ set read-only permission on files uploaded\n to S3 for anonymous users\n* `--sign` - /(optional) sign package metadata\n* `--force` - /(optional) when adding packages to the index, the malformed one\n will be skipped. By default, a malformed package will cause the utility to\n stop working. The malformed_list.txt file will also be added to the repository\n* `path` - specify list of path to scan for repositories\n\n## Environment variables reference\n\n* `GPG_SIGN_KEY` - the name of the key that will be used to sign package metadata.\n\n<details><summary>Tips for working with GPG keys</summary>\n\n * Create a new key:\n ``` bash\n gpg --full-generate-key\n ```\n * To view all your keys, you can use:\n ``` bash\n gpg --list-secret-keys --keyid-format LONG\n ```\n * Scripts can use something like this to get the Key ID:\n ``` bash\n export GPG_SIGN_KEY=\"$(gpg --list-secret-keys --with-colons | grep ^sec: | cut -d: -f5)\"\n ```\n * Export the key in ASCII armored format:\n ``` bash\n gpg --armor --export-secret-keys MYKEYID > mykeys.asc\n ```\n * Import the key:\n ``` bash\n cat mykeys.asc | gpg --batch --import\n ```\n\n</details>\n\n* `MKREPO_DEB_ORIGIN` - the value of the [\"Origin\"](https://wiki.debian.org/DebianRepository/Format#Origin)\n field of the \"Release\" file.\n* `MKREPO_DEB_LABEL` - the value of the [\"Label\"](https://wiki.debian.org/DebianRepository/Format#Label)\n field of the \"Release\" file.\n* `MKREPO_DEB_DESCRIPTION` - the value of the \"Description\" field of the \"Release\" file.\n\n## How it works\n\n`mkrepo` searches the supplied path for either `Packages` or `pool` subdir. If\nit finds `Packages`, it assumes an rpm repo. If it finds `pool`, it assumes a\ndeb repo.\n\nThen it parses existing metadata files (if any) and compares timestamps recorded\nthere with timestamps of all package files in the repo. Any packages that have\ndifferent timestamps or that don't exist in metadata, are parsed and added to\nmetadata.\n\nThen new metadata is uploaded to S3, replacing previous one.\n\n## Credits\n\nThanks to [Cyril Rohr](https://github.com/crohr) and [Ken Robertson](https://github.com/krobertson), authors of the following awesome tools:\n\n* [rpm-s3](https://github.com/crohr/rpm-s3)\n* [deb-s3](https://github.com/krobertson/deb-s3)\n\nUnfortunately, we needed a solution that is completely decoupled from CI pipeline,\nand the mentioned tools only support package push mode, when you have to use a\ntool to actually push packages to s3, insted of native s3 clients.\n",
"bugtrack_url": null,
"license": "",
"summary": "Maintain DEB and RPM repositories on S3",
"version": "1.0.2",
"split_keywords": [
"rpm",
"deb"
],
"urls": [
{
"comment_text": "",
"digests": {
"md5": "9e7cc355f954b5c66fff094d7fe91ddd",
"sha256": "e1d0fefc8c36574f5ea4125d94d9a346991f3ed28ae143a4b98bfe5a7ffc2931"
},
"downloads": -1,
"filename": "mkrepo-1.0.2-py3-none-any.whl",
"has_sig": false,
"md5_digest": "9e7cc355f954b5c66fff094d7fe91ddd",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.6.*",
"size": 27319,
"upload_time": "2022-12-07T12:34:15",
"upload_time_iso_8601": "2022-12-07T12:34:15.832390Z",
"url": "https://files.pythonhosted.org/packages/ee/ae/c8ce9486f71054ff9a4664381f7b9e3e26d542669a33e307a3a77f0ae70a/mkrepo-1.0.2-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"md5": "e25a4136e57f18dccd74b674cf31a110",
"sha256": "994cdd684ff95e67c3d255518a61851df82dd13cc4f85f2b480dd8000f6511c9"
},
"downloads": -1,
"filename": "mkrepo-1.0.2.tar.gz",
"has_sig": false,
"md5_digest": "e25a4136e57f18dccd74b674cf31a110",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.6.*",
"size": 35069,
"upload_time": "2022-12-07T12:34:17",
"upload_time_iso_8601": "2022-12-07T12:34:17.778194Z",
"url": "https://files.pythonhosted.org/packages/85/ed/616e994dd0a130e60b69ca13b05650bc2e2aa35d0f9dbcdeac54d5e61e78/mkrepo-1.0.2.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2022-12-07 12:34:17",
"github": true,
"gitlab": false,
"bitbucket": false,
"github_user": "tarantool",
"github_project": "mkrepo",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"requirements": [],
"lcname": "mkrepo"
}
JSON
