# conda-mirror
[](https://travis-ci.org/regro/conda-mirror)
[](https://badge.fury.io/py/conda-mirror)
[](https://codecov.io/gh/regro/conda-mirror)
Mirrors an upstream conda channel to a local directory.
## Install
`conda-mirror` is available on PyPI and conda-forge.
Install with:
`pip install conda-mirror`
or:
`conda install conda-mirror -c conda-forge`
## Compatibility
`conda-mirror` is intentionally a py3 only package
## CLI
CLI interface for `conda-mirror.py`
```
usage: conda-mirror [-h] [--upstream-channel UPSTREAM_CHANNEL]
[--target-directory TARGET_DIRECTORY]
[--temp-directory TEMP_DIRECTORY] [--platform PLATFORM]
[-D] [-v] [--config CONFIG] [--pdb]
[--num-threads NUM_THREADS] [--version] [--dry-run]
[--no-validate-target]
[--minimum-free-space MINIMUM_FREE_SPACE] [--proxy PROXY]
[--ssl-verify SSL_VERIFY] [-k]
[--max-retries MAX_RETRIES] [--no-progress]
CLI interface for conda-mirror.py
optional arguments:
-h, --help show this help message and exit
--upstream-channel UPSTREAM_CHANNEL
The target channel to mirror. Can be a channel on
anaconda.org like "conda-forge" or a full qualified
channel like "https://repo.continuum.io/pkgs/free/"
--target-directory TARGET_DIRECTORY
The place where packages should be mirrored to
--temp-directory TEMP_DIRECTORY
Temporary download location for the packages.
Defaults to a randomly selected temporary directory.
Note that you might need to specify a different
location if your default temp directory has less
available space than your mirroring target
--platform PLATFORM The OS platform(s) to mirror. one of: {'linux-64',
'linux-32','osx-64', 'win-32', 'win-64'}
-D, --include-depends
Include packages matching any dependencies of
packages in whitelist.
-v, --verbose logging defaults to error/exception only. Takes up to
three '-v' flags. '-v': warning. '-vv': info. '-vvv':
debug.
--config CONFIG Path to the yaml config file
--pdb Enable PDB debugging on exception
--num-threads NUM_THREADS
Num of threads for validation. 1: Serial mode. 0: All
available.
--version Print version and quit
--dry-run Show what will be downloaded and what will be
removed. Will not validate existing packages
--no-validate-target Skip validation of files already present in target-
directory
--minimum-free-space MINIMUM_FREE_SPACE
Threshold for free diskspace. Given in megabytes.
--proxy PROXY Proxy URL to access internet if needed
--ssl-verify SSL_VERIFY, --ssl_verify SSL_VERIFY
Path to a CA_BUNDLE file with certificates of trusted
CAs, this may be "False" to disable verification as
per the requests API.
-k, --insecure Allow conda to perform "insecure" SSL connections and
transfers. Equivalent to setting 'ssl_verify' to
'false'.
--max-retries MAX_RETRIES
Maximum number of retries before a download error is
reraised, defaults to 100
--no-progress Do not display progress bars.
```
## Example Usage
WARNING: Invoking this command will pull ~10GB and take at least an hour
`conda-mirror --upstream-channel conda-forge --target-directory local_mirror --platform linux-64`
## More Details
### blacklist/whitelist configuration
example-conf.yaml:
```yaml
blacklist:
- license: "*agpl*"
- license: None
- license: ""
whitelist:
- name: system
```
`blacklist` removes package(s) that match the condition(s) listed from the
upstream repodata.
`whitelist` re-includes any package(s) from blacklist that match the
whitelist conditions.
blacklist and whitelist both take lists of dictionaries. The keys in the
dictionary need to be values in the `repodata.json` metadata. The values are
(unix) globs to match on, but in the case of the `version` attribute,
[conda package match version specifications](https://docs.conda.io/projects/conda/en/latest/user-guide/concepts/pkg-specs.html#package-match-specifications)
may also be used.
Go here for the full repodata of the upstream
"defaults" channel:
http://conda.anaconda.org/anaconda/linux-64/repodata.json
Here are the contents of one of the entries in repodata['packages']
```python
{'botocore-1.4.10-py34_0.tar.bz2': {'arch': 'x86_64',
'binstar': {'channel': 'main',
'owner_id': '55fc8527d3234d09d4951c71',
'package_id': '56b88ea1be1cc95a362b218e'},
'build': 'py34_0',
'build_number': 0,
'date': '2016-04-11',
'depends': ['docutils >=0.10',
'jmespath >=0.7.1,<1.0.0',
'python 3.4*',
'python-dateutil >=2.1,<3.0.0'],
'license': 'Apache',
'md5': 'b35a5c1240ba672e0d9d1296141e383c',
'name': 'botocore',
'platform': 'linux',
'requires': [],
'size': 1831799,
'version': '1.4.10'}}
```
See implementation details in the `conda_mirror:match` function for more
information.
#### Common usage patterns
##### Mirror **only** one specific package
If you wanted to match exactly the botocore package listed above with your
config, then you could use the following configuration to first blacklist
**all** packages and then include just the botocore packages:
```yaml
blacklist:
- name: "*"
whitelist:
- name: botocore
version: 1.4.10
build: py34_0
```
you can use standard conda package version specifiers to filter a range of versions:
```yaml
blacklist:
- name: "*"
whitelist:
- name: botocore
version: ">=1.4.10,<1.5"
```
##### Mirror everything but agpl licenses
```yaml
blacklist:
- license: "*agpl*"
```
##### Mirror only python 3 packages
```yaml
blacklist:
- name: "*"
whitelist:
- build: "*py3*"
```
##### Mirror specified packages and their dependencies
This will include all instances of botocore with at least
version 1.4.10 along with any packages that match its
dependencies (and likewise for dependencies of those packages).
```yaml
blacklist:
- name: "*"
whitelist:
- name: botocore
version: ">=1.4.10"
include_depends: True
```
If this includes too many packages versions, you can add additional
entries to the whitelist to limit what will be included.
## Testing
### Install test requirements
Note: Will install packages from pip
```
$ pip install -r test-requirements.txt
Requirement already satisfied: pytest in /home/edill/miniconda/lib/python3.5/site-packages (from -r test-requirements.txt (line 1))
Requirement already satisfied: coverage in /home/edill/miniconda/lib/python3.5/site-packages (from -r test-requirements.txt (line 2))
Requirement already satisfied: pytest-ordering in /home/edill/miniconda/lib/python3.5/site-packages (from -r test-requirements.txt (line 3))
Requirement already satisfied: py>=1.4.29 in /home/edill/miniconda/lib/python3.5/site-packages (from pytest->-r test-requirements.txt (line 1))
```
### Run the tests, invoking with the `coverage` tool.
```
$ coverage run run_tests.py
sys.argv=['run_tests.py']
========================================= test session starts ==========================================
platform linux -- Python 3.5.3, pytest-3.0.6, py-1.4.31, pluggy-0.4.0 -- /home/edill/miniconda/bin/python
cachedir: .cache
rootdir: /home/edill/dev/maxpoint/github/conda-mirror, inifile:
plugins: xonsh-0.5.2, ordering-0.4
collected 4 items
test/test_conda_mirror.py::test_match PASSED
test/test_conda_mirror.py::test_cli[https://repo.continuum.io/pkgs/free-linux-64] PASSED
test/test_conda_mirror.py::test_cli[conda-forge-linux-64] PASSED
test/test_conda_mirror.py::test_handling_bad_package PASSED
======================================= 4 passed in 4.41 seconds =======================================
```
### Show the coverage statistics
```
$ coverage report -m
Name Stmts Miss Cover Missing
------------------------------------------------------------
conda_mirror/__init__.py 3 0 100%
conda_mirror/conda_mirror.py 236 20 92% 203-205, 209-210, 214, 240, 249-254, 262-264, 303, 366, 497, 542-543, 629
------------------------------------------------------------
TOTAL 239 20 92%
```
## Other
After a new contributor makes a pull-request that is approved, we will reach out
and invite you to be a maintainer of the project.
## Releasing
To release you need three things
1. Commit rights to conda-mirror
2. A github token
3. The version number that you want to use for the new tag
After you have all three of these things, run the release.sh script (on a unix machine) and
pass it the tag that you want to use and your github token:
```bash
GITHUB_TOKEN=<github_token> ./release.sh <tag>
```
Raw data
{
"_id": null,
"home_page": "https://github.com/regro/conda-mirror",
"name": "conda-mirror",
"maintainer": "",
"docs_url": null,
"requires_python": "",
"maintainer_email": "",
"keywords": "",
"author": "Eric Dill",
"author_email": "eric.dill@maxpoint.com",
"download_url": "https://files.pythonhosted.org/packages/e3/f4/030f80640521f08efcae9741823f90edbbdff58598a86c0bf1c6637bdc85/conda_mirror-0.9.0.tar.gz",
"platform": "Linux",
"description": "# conda-mirror\n\n[](https://travis-ci.org/regro/conda-mirror)\n[](https://badge.fury.io/py/conda-mirror)\n[](https://codecov.io/gh/regro/conda-mirror)\n\nMirrors an upstream conda channel to a local directory.\n\n## Install\n\n`conda-mirror` is available on PyPI and conda-forge.\n\nInstall with:\n\n`pip install conda-mirror`\n\nor:\n\n`conda install conda-mirror -c conda-forge`\n\n## Compatibility\n\n`conda-mirror` is intentionally a py3 only package\n\n## CLI\n\nCLI interface for `conda-mirror.py`\n\n```\nusage: conda-mirror [-h] [--upstream-channel UPSTREAM_CHANNEL]\n [--target-directory TARGET_DIRECTORY]\n [--temp-directory TEMP_DIRECTORY] [--platform PLATFORM]\n [-D] [-v] [--config CONFIG] [--pdb]\n [--num-threads NUM_THREADS] [--version] [--dry-run]\n [--no-validate-target]\n [--minimum-free-space MINIMUM_FREE_SPACE] [--proxy PROXY]\n [--ssl-verify SSL_VERIFY] [-k]\n [--max-retries MAX_RETRIES] [--no-progress]\n\nCLI interface for conda-mirror.py\n\noptional arguments:\n -h, --help show this help message and exit\n --upstream-channel UPSTREAM_CHANNEL\n The target channel to mirror. Can be a channel on\n anaconda.org like \"conda-forge\" or a full qualified\n channel like \"https://repo.continuum.io/pkgs/free/\"\n --target-directory TARGET_DIRECTORY\n The place where packages should be mirrored to\n --temp-directory TEMP_DIRECTORY\n Temporary download location for the packages.\n Defaults to a randomly selected temporary directory.\n Note that you might need to specify a different\n location if your default temp directory has less\n available space than your mirroring target\n --platform PLATFORM The OS platform(s) to mirror. one of: {'linux-64',\n 'linux-32','osx-64', 'win-32', 'win-64'}\n -D, --include-depends\n Include packages matching any dependencies of\n packages in whitelist.\n -v, --verbose logging defaults to error/exception only. Takes up to\n three '-v' flags. '-v': warning. '-vv': info. '-vvv':\n debug.\n --config CONFIG Path to the yaml config file\n --pdb Enable PDB debugging on exception\n --num-threads NUM_THREADS\n Num of threads for validation. 1: Serial mode. 0: All\n available.\n --version Print version and quit\n --dry-run Show what will be downloaded and what will be\n removed. Will not validate existing packages\n --no-validate-target Skip validation of files already present in target-\n directory\n --minimum-free-space MINIMUM_FREE_SPACE\n Threshold for free diskspace. Given in megabytes.\n --proxy PROXY Proxy URL to access internet if needed\n --ssl-verify SSL_VERIFY, --ssl_verify SSL_VERIFY\n Path to a CA_BUNDLE file with certificates of trusted\n CAs, this may be \"False\" to disable verification as\n per the requests API.\n -k, --insecure Allow conda to perform \"insecure\" SSL connections and\n transfers. Equivalent to setting 'ssl_verify' to\n 'false'.\n --max-retries MAX_RETRIES\n Maximum number of retries before a download error is\n reraised, defaults to 100\n --no-progress Do not display progress bars.\n```\n\n## Example Usage\n\nWARNING: Invoking this command will pull ~10GB and take at least an hour\n\n`conda-mirror --upstream-channel conda-forge --target-directory local_mirror --platform linux-64`\n\n## More Details\n\n### blacklist/whitelist configuration\n\nexample-conf.yaml:\n\n```yaml\nblacklist:\n - license: \"*agpl*\"\n - license: None\n - license: \"\"\n\nwhitelist:\n - name: system\n```\n\n`blacklist` removes package(s) that match the condition(s) listed from the\nupstream repodata.\n\n`whitelist` re-includes any package(s) from blacklist that match the\nwhitelist conditions.\n\nblacklist and whitelist both take lists of dictionaries. The keys in the\ndictionary need to be values in the `repodata.json` metadata. The values are\n(unix) globs to match on, but in the case of the `version` attribute,\n[conda package match version specifications](https://docs.conda.io/projects/conda/en/latest/user-guide/concepts/pkg-specs.html#package-match-specifications)\nmay also be used.\n\nGo here for the full repodata of the upstream\n\"defaults\" channel:\nhttp://conda.anaconda.org/anaconda/linux-64/repodata.json\n\nHere are the contents of one of the entries in repodata['packages']\n\n```python\n{'botocore-1.4.10-py34_0.tar.bz2': {'arch': 'x86_64',\n 'binstar': {'channel': 'main',\n 'owner_id': '55fc8527d3234d09d4951c71',\n 'package_id': '56b88ea1be1cc95a362b218e'},\n 'build': 'py34_0',\n 'build_number': 0,\n 'date': '2016-04-11',\n 'depends': ['docutils >=0.10',\n 'jmespath >=0.7.1,<1.0.0',\n 'python 3.4*',\n 'python-dateutil >=2.1,<3.0.0'],\n 'license': 'Apache',\n 'md5': 'b35a5c1240ba672e0d9d1296141e383c',\n 'name': 'botocore',\n 'platform': 'linux',\n 'requires': [],\n 'size': 1831799,\n 'version': '1.4.10'}}\n```\n\nSee implementation details in the `conda_mirror:match` function for more\ninformation.\n\n#### Common usage patterns\n##### Mirror **only** one specific package\nIf you wanted to match exactly the botocore package listed above with your\nconfig, then you could use the following configuration to first blacklist\n**all** packages and then include just the botocore packages:\n\n```yaml\nblacklist:\n - name: \"*\"\nwhitelist:\n - name: botocore\n version: 1.4.10\n build: py34_0\n```\n\nyou can use standard conda package version specifiers to filter a range of versions:\n\n```yaml\nblacklist:\n - name: \"*\"\nwhitelist:\n - name: botocore\n version: \">=1.4.10,<1.5\"\n```\n\n##### Mirror everything but agpl licenses\n```yaml\nblacklist:\n - license: \"*agpl*\"\n```\n\n##### Mirror only python 3 packages\n```yaml\nblacklist:\n - name: \"*\"\nwhitelist:\n - build: \"*py3*\"\n```\n\n##### Mirror specified packages and their dependencies\n\nThis will include all instances of botocore with at least\nversion 1.4.10 along with any packages that match its\ndependencies (and likewise for dependencies of those packages).\n\n```yaml\nblacklist:\n - name: \"*\"\nwhitelist:\n - name: botocore\n version: \">=1.4.10\"\ninclude_depends: True\n```\n\nIf this includes too many packages versions, you can add additional\nentries to the whitelist to limit what will be included.\n\n## Testing\n\n### Install test requirements\n\nNote: Will install packages from pip\n\n```\n$ pip install -r test-requirements.txt\nRequirement already satisfied: pytest in /home/edill/miniconda/lib/python3.5/site-packages (from -r test-requirements.txt (line 1))\nRequirement already satisfied: coverage in /home/edill/miniconda/lib/python3.5/site-packages (from -r test-requirements.txt (line 2))\nRequirement already satisfied: pytest-ordering in /home/edill/miniconda/lib/python3.5/site-packages (from -r test-requirements.txt (line 3))\nRequirement already satisfied: py>=1.4.29 in /home/edill/miniconda/lib/python3.5/site-packages (from pytest->-r test-requirements.txt (line 1))\n```\n\n### Run the tests, invoking with the `coverage` tool.\n\n```\n$ coverage run run_tests.py\nsys.argv=['run_tests.py']\n========================================= test session starts ==========================================\nplatform linux -- Python 3.5.3, pytest-3.0.6, py-1.4.31, pluggy-0.4.0 -- /home/edill/miniconda/bin/python\ncachedir: .cache\nrootdir: /home/edill/dev/maxpoint/github/conda-mirror, inifile:\nplugins: xonsh-0.5.2, ordering-0.4\ncollected 4 items\n\ntest/test_conda_mirror.py::test_match PASSED\ntest/test_conda_mirror.py::test_cli[https://repo.continuum.io/pkgs/free-linux-64] PASSED\ntest/test_conda_mirror.py::test_cli[conda-forge-linux-64] PASSED\ntest/test_conda_mirror.py::test_handling_bad_package PASSED\n\n======================================= 4 passed in 4.41 seconds =======================================\n```\n\n### Show the coverage statistics\n\n```\n$ coverage report -m\nName Stmts Miss Cover Missing\n------------------------------------------------------------\nconda_mirror/__init__.py 3 0 100%\nconda_mirror/conda_mirror.py 236 20 92% 203-205, 209-210, 214, 240, 249-254, 262-264, 303, 366, 497, 542-543, 629\n------------------------------------------------------------\nTOTAL 239 20 92%\n```\n\n## Other\n\nAfter a new contributor makes a pull-request that is approved, we will reach out\nand invite you to be a maintainer of the project.\n\n\n## Releasing\n\nTo release you need three things\n\n1. Commit rights to conda-mirror\n2. A github token\n3. The version number that you want to use for the new tag\n\nAfter you have all three of these things, run the release.sh script (on a unix machine) and\npass it the tag that you want to use and your github token:\n\n```bash\nGITHUB_TOKEN=<github_token> ./release.sh <tag>\n```\n",
"bugtrack_url": null,
"license": "BSD 3-Clause",
"summary": "Mirror an upstream conda channel to a local directory",
"version": "0.9.0",
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "3557ab1738bdf0c20c1df6d1121e666fdd15bddd848a8b22b31d0a34706768af",
"md5": "7aabfae62ad4287abe3a8422fee57be5",
"sha256": "46fdf64e71254f20bf9eaf7198e702546a6daa8813e81dd8d4312f6d22a15c81"
},
"downloads": -1,
"filename": "conda_mirror-0.9.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "7aabfae62ad4287abe3a8422fee57be5",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": null,
"size": 27948,
"upload_time": "2023-01-13T09:18:31",
"upload_time_iso_8601": "2023-01-13T09:18:31.551989Z",
"url": "https://files.pythonhosted.org/packages/35/57/ab1738bdf0c20c1df6d1121e666fdd15bddd848a8b22b31d0a34706768af/conda_mirror-0.9.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "e3f4030f80640521f08efcae9741823f90edbbdff58598a86c0bf1c6637bdc85",
"md5": "0d82aef65539b8dbbba2cf9490f645ee",
"sha256": "32500b6ecabd22beffb4a62f208da261afc39b431c88f265adbd06f1805a32d3"
},
"downloads": -1,
"filename": "conda_mirror-0.9.0.tar.gz",
"has_sig": false,
"md5_digest": "0d82aef65539b8dbbba2cf9490f645ee",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 51546,
"upload_time": "2023-01-13T09:18:33",
"upload_time_iso_8601": "2023-01-13T09:18:33.895689Z",
"url": "https://files.pythonhosted.org/packages/e3/f4/030f80640521f08efcae9741823f90edbbdff58598a86c0bf1c6637bdc85/conda_mirror-0.9.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-01-13 09:18:33",
"github": true,
"gitlab": false,
"bitbucket": false,
"github_user": "regro",
"github_project": "conda-mirror",
"travis_ci": false,
"coveralls": true,
"github_actions": true,
"requirements": [],
"lcname": "conda-mirror"
}
JSON
