ghcloneall
==========
.. image:: https://github.com/mgedmin/ghcloneall/workflows/build/badge.svg?branch=master
:target: https://github.com/mgedmin/ghcloneall/actions
.. image:: https://ci.appveyor.com/api/projects/status/github/mgedmin/ghcloneall?branch=master&svg=true
:target: https://ci.appveyor.com/project/mgedmin/ghcloneall
It's a script to clone/update all repos for a user/organization from GitHub.
Target audience: maintainers of large collections of projects (for example,
ZopeFoundation members).
Usage examples
--------------
First ``pip install ghcloneall``.
Clone all mgedmin's vim plugins::
mkdir ~/src/vim-plugins
cd ~/src/vim-plugins
ghcloneall --init --user mgedmin --pattern '*.vim'
ghcloneall
Clone all mgedmin's gists::
mkdir ~/src/gists
cd ~/src/gists
ghcloneall --init --user mgedmin --gists
ghcloneall
Clone all ZopeFoundation repositories::
mkdir ~/src/zf
cd ~/src/zf
ghcloneall --init --org ZopeFoundation
ghcloneall
Here's a screencast of the above (running a slightly older version so the
script name differs):
.. image:: https://asciinema.org/a/29651.png
:alt: asciicast
:width: 582
:height: 380
:align: center
:target: https://asciinema.org/a/29651
Details
-------
What it does:
- clones repositories you don't have locally
- pulls changes for repositories you already have locally
- warns you about local changes and other unexpected situations:
- unknown files in the tree (in --verbose mode only)
- staged but not committed changes
- uncommitted (and unstaged changes)
- non-default branch checked out
- committed changes that haven't been pushed to default branch
- remote URL pointing to an unexpected location (in --verbose mode only)
You can ask it to not change any files on disk and just look for pending
changes by running ``ghcloneall --dry-run``. This will also make the
check faster!
Synopsis
--------
.. [[[cog
.. import cog, subprocess, textwrap, os
.. os.environ['COLUMNS'] = '80' # consistent line wrapping
.. helptext = subprocess.run(['ghcloneall', '--help'],
.. capture_output=True, text=True).stdout
.. cog.outl('\nOther command-line options::\n')
.. cog.outl(' $ ghcloneall --help')
.. cog.outl(textwrap.indent(helptext, ' '))
.. ]]]
Other command-line options::
$ ghcloneall --help
usage: ghcloneall [-h] [--version] [-c CONCURRENCY] [-n] [-q] [-v]
[--start-from REPO] [--organization ORGANIZATION]
[--user USER] [--github-token GITHUB_TOKEN] [--gists]
[--repositories] [--pattern PATTERN] [--include-forks]
[--exclude-forks] [--include-archived] [--exclude-archived]
[--include-private] [--exclude-private] [--include-disabled]
[--exclude-disabled] [--init] [--http-cache DBNAME]
[--no-http-cache]
Clone/update all user/org repositories from GitHub.
options:
-h, --help show this help message and exit
--version show program's version number and exit
-c CONCURRENCY, --concurrency CONCURRENCY
set concurrency level (default: 4)
-n, --dry-run don't pull/clone, just print what would be done
-q, --quiet terser output
-v, --verbose perform additional checks
--start-from REPO skip all repositories that come before REPO
alphabetically
--organization ORGANIZATION
specify the GitHub organization
--user USER specify the GitHub user
--github-token GITHUB_TOKEN
specify the GitHub token
--gists clone user's gists
--repositories clone user's or organisation's repositories (default)
--pattern PATTERN specify repository name glob pattern to filter
--include-forks include repositories forked from other users/orgs
--exclude-forks exclude repositories forked from other users/orgs
(default)
--include-archived include archived repositories
--exclude-archived exclude archived repositories (default)
--include-private include private repositories (default when a github
token is provided)
--exclude-private exclude private repositories
--include-disabled include disabled repositories (default)
--exclude-disabled exclude disabled repositories
--init create a .ghcloneallrc from command-line arguments
--http-cache DBNAME cache HTTP requests on disk in an sqlite database for
5 minutes (default: .httpcache)
--no-http-cache disable HTTP disk caching
.. [[[end]]]
Configuration file
------------------
The script looks for ``.ghcloneallrc`` in the current working directory, which
should look like this::
[ghcloneall]
# Provide either github_user or github_org, but not both
# github_org = ZopeFoundation
github_user = mgedmin
pattern = *.vim
# Provide github token for authentication
# github_token = <my-github-token>
# You can also uncomment and change these if you wish
# gists = False
# include_forks = False
# include_archived = False
# Listing private repositories requires a valid github_token
# include_private = True
# include_disabled = True
You can create one with ``ghcloneall --init --{user,org} X [--pattern Y]
[--{include,exclude}-{forks,archived,private,disabled}] [--gists|--repos]``.
Tips
----
For best results configure SSH persistence to speed up git pulls -- in your
``~/.ssh/config``::
Host github.com
ControlMaster auto
ControlPersist yes
ControlPath ~/.ssh/control-%r@%h-%p
It takes about 80 seconds to run ``git pull`` on all 382 ZopeFoundation
repos on my laptop with this kind of setup.
Changelog
=========
1.12.0 (2024-10-09)
-------------------
- Add support for Python 3.12 and 3.13.
- Drop support for Python 2.7.
1.11.0 (2022-10-27)
-------------------
- Add support for Python 3.10 and 3.11.
- Drop support for Python 3.6.
- Fix ``ghcloneall --user ... --github-token ... --include-private`` not
including any private repositories (GH: #16).
1.10.1 (2021-05-26)
-------------------
- When determining if a repository is dirty, use the repository's
configured default branch from GitHub instead of assuming that the
default is "master".
1.10.0 (2021-04-10)
-------------------
- Allow authentication with GitHub token.
- Depend on requests-cache < 0.6 on Python 2.7.
- Add support for Python 3.9.
- Drop support for Python 3.5.
1.9.2 (2019-10-15)
------------------
- Add support for Python 3.8.
1.9.1 (2019-10-07)
------------------
- Reuse HTTP connections for GitHub API requests.
1.9.0 (2019-09-06)
------------------
- Can now clone all user's gists.
- Command line args: --gists, --repos.
1.8.0 (2019-08-28)
------------------
- Skip forks and archived repositories by default.
- Command-line args: --include-forks, --exclude-forks.
- Command-line args: --include-archived, --exclude-archived.
- Command-line args: --include-private, --exclude-private.
- Command-line args: --include-disabled, --exclude-disabled.
- Use a custom User-Agent header when talking to GitHub.
1.7.1 (2019-08-14)
------------------
- Drop support for Python 3.3 and 3.4.
- Add a test suite.
- Fix AttributeError: 'str' object has no attribute 'format_map' on Python 2.
1.7.0 (2018-12-19)
------------------
- Command line args: -q, --quiet
- Fix display corruption on ^C
1.6.1 (2018-10-19)
------------------
- Fix TypeError: get() got an unexpected keyword argument 'fallback' on
Python 2.
1.6 (2016-12-29)
----------------
- Comprehensive rebranding:
- Rename the GitHub repository to https://github.com/mgedmin/ghcloneall
- Rename ``cloneall.py`` to ``ghcloneall.py``
- Rename the config file to ``.ghcloneallrc``, and rename the config
section to ``[ghcloneall]``.
- Don't print tracebacks on ^C (this regressed in 1.5).
1.5 (2016-12-29)
----------------
- Released to PyPI as ``ghcloneall``
- Added Python 2.7 support
1.4 (2016-12-28)
----------------
- Command line args: --user, --pattern, --init
- Load (some) options from a ``.cloneallrc``
- Stop using ``--organization=ZopeFoundation`` by default, require an
explicit option (or config file)
- Rename clone_all_zf_repos.py to cloneall.py
1.3 (2016-12-28)
----------------
- Command line args: -c
- Show progress while fetching the list of repositories from GitHub
- Update repositories concurrently by default
- Highlight items in progress
- Highlight failed items in red
- Tweak progress bar style from ``[=== ]`` to ``[###..]``
- Clear the progress bar on ^C
- Handle git errors nicely
- Bugfix: -vv could fail with NameError if unknown files were present in a
working tree
- Bugfix: correctly show progress when using --start-from
- Bugfix: script would hang (for 10 minutes) if you didn't already have an
SSH control master process running
- Bugfix: --dry-run didn't show which repos were new
1.2 (2016-11-09)
----------------
- Command line args: --dry-run, --verbose
- Cache HTTP responses on disk for 10 minutes to avoid GitHub API rate limits
- Report about forgotten uncommitted and staged changes
- Warn about local (unpushed) commits too
- Warn about other branches being checked out
- Default to SSH URLs again (faster when using SSH's ControlPersist)
1.1 (2015-11-07)
----------------
- Command line args: --version, --start-from, --organization
- Output formatting: shorter repository names, totals at the end
- Use ANSI colors to indicate changes
- Don't print tracebacks on ^C
- Default to HTTPS URLs
1.0 (2015-11-07)
----------------
- Moved from a gist to a proper GitHub repository.
Raw data
{
"_id": null,
"home_page": "https://github.com/mgedmin/ghcloneall",
"name": "ghcloneall",
"maintainer": null,
"docs_url": null,
"requires_python": null,
"maintainer_email": null,
"keywords": "github git clone automation",
"author": "Marius Gedminas",
"author_email": "marius@gedmin.as",
"download_url": "https://files.pythonhosted.org/packages/0b/50/f81688ee13c68f1c3167754f74c0167507403ce567514af806b8cf2f3533/ghcloneall-1.12.0.tar.gz",
"platform": null,
"description": "ghcloneall\n==========\n\n.. image:: https://github.com/mgedmin/ghcloneall/workflows/build/badge.svg?branch=master\n :target: https://github.com/mgedmin/ghcloneall/actions\n\n.. image:: https://ci.appveyor.com/api/projects/status/github/mgedmin/ghcloneall?branch=master&svg=true\n :target: https://ci.appveyor.com/project/mgedmin/ghcloneall\n\n\nIt's a script to clone/update all repos for a user/organization from GitHub.\n\nTarget audience: maintainers of large collections of projects (for example,\nZopeFoundation members).\n\n\nUsage examples\n--------------\n\nFirst ``pip install ghcloneall``.\n\nClone all mgedmin's vim plugins::\n\n mkdir ~/src/vim-plugins\n cd ~/src/vim-plugins\n ghcloneall --init --user mgedmin --pattern '*.vim'\n ghcloneall\n\nClone all mgedmin's gists::\n\n mkdir ~/src/gists\n cd ~/src/gists\n ghcloneall --init --user mgedmin --gists\n ghcloneall\n\nClone all ZopeFoundation repositories::\n\n mkdir ~/src/zf\n cd ~/src/zf\n ghcloneall --init --org ZopeFoundation\n ghcloneall\n\nHere's a screencast of the above (running a slightly older version so the\nscript name differs):\n\n.. image:: https://asciinema.org/a/29651.png\n :alt: asciicast\n :width: 582\n :height: 380\n :align: center\n :target: https://asciinema.org/a/29651\n\n\nDetails\n-------\n\nWhat it does:\n\n- clones repositories you don't have locally\n- pulls changes for repositories you already have locally\n- warns you about local changes and other unexpected situations:\n\n - unknown files in the tree (in --verbose mode only)\n - staged but not committed changes\n - uncommitted (and unstaged changes)\n - non-default branch checked out\n - committed changes that haven't been pushed to default branch\n - remote URL pointing to an unexpected location (in --verbose mode only)\n\nYou can ask it to not change any files on disk and just look for pending\nchanges by running ``ghcloneall --dry-run``. This will also make the\ncheck faster!\n\n\nSynopsis\n--------\n\n.. [[[cog\n.. import cog, subprocess, textwrap, os\n.. os.environ['COLUMNS'] = '80' # consistent line wrapping\n.. helptext = subprocess.run(['ghcloneall', '--help'],\n.. capture_output=True, text=True).stdout\n.. cog.outl('\\nOther command-line options::\\n')\n.. cog.outl(' $ ghcloneall --help')\n.. cog.outl(textwrap.indent(helptext, ' '))\n.. ]]]\n\nOther command-line options::\n\n $ ghcloneall --help\n usage: ghcloneall [-h] [--version] [-c CONCURRENCY] [-n] [-q] [-v]\n [--start-from REPO] [--organization ORGANIZATION]\n [--user USER] [--github-token GITHUB_TOKEN] [--gists]\n [--repositories] [--pattern PATTERN] [--include-forks]\n [--exclude-forks] [--include-archived] [--exclude-archived]\n [--include-private] [--exclude-private] [--include-disabled]\n [--exclude-disabled] [--init] [--http-cache DBNAME]\n [--no-http-cache]\n\n Clone/update all user/org repositories from GitHub.\n\n options:\n -h, --help show this help message and exit\n --version show program's version number and exit\n -c CONCURRENCY, --concurrency CONCURRENCY\n set concurrency level (default: 4)\n -n, --dry-run don't pull/clone, just print what would be done\n -q, --quiet terser output\n -v, --verbose perform additional checks\n --start-from REPO skip all repositories that come before REPO\n alphabetically\n --organization ORGANIZATION\n specify the GitHub organization\n --user USER specify the GitHub user\n --github-token GITHUB_TOKEN\n specify the GitHub token\n --gists clone user's gists\n --repositories clone user's or organisation's repositories (default)\n --pattern PATTERN specify repository name glob pattern to filter\n --include-forks include repositories forked from other users/orgs\n --exclude-forks exclude repositories forked from other users/orgs\n (default)\n --include-archived include archived repositories\n --exclude-archived exclude archived repositories (default)\n --include-private include private repositories (default when a github\n token is provided)\n --exclude-private exclude private repositories\n --include-disabled include disabled repositories (default)\n --exclude-disabled exclude disabled repositories\n --init create a .ghcloneallrc from command-line arguments\n --http-cache DBNAME cache HTTP requests on disk in an sqlite database for\n 5 minutes (default: .httpcache)\n --no-http-cache disable HTTP disk caching\n\n.. [[[end]]]\n\n\nConfiguration file\n------------------\n\nThe script looks for ``.ghcloneallrc`` in the current working directory, which\nshould look like this::\n\n [ghcloneall]\n # Provide either github_user or github_org, but not both\n # github_org = ZopeFoundation\n github_user = mgedmin\n pattern = *.vim\n # Provide github token for authentication\n # github_token = <my-github-token>\n # You can also uncomment and change these if you wish\n # gists = False\n # include_forks = False\n # include_archived = False\n # Listing private repositories requires a valid github_token\n # include_private = True\n # include_disabled = True\n\nYou can create one with ``ghcloneall --init --{user,org} X [--pattern Y]\n[--{include,exclude}-{forks,archived,private,disabled}] [--gists|--repos]``.\n\n\nTips\n----\n\nFor best results configure SSH persistence to speed up git pulls -- in your\n``~/.ssh/config``::\n\n Host github.com\n ControlMaster auto\n ControlPersist yes\n ControlPath ~/.ssh/control-%r@%h-%p\n\nIt takes about 80 seconds to run ``git pull`` on all 382 ZopeFoundation\nrepos on my laptop with this kind of setup.\n\n\nChangelog\n=========\n\n\n1.12.0 (2024-10-09)\n-------------------\n\n- Add support for Python 3.12 and 3.13.\n- Drop support for Python 2.7.\n\n\n1.11.0 (2022-10-27)\n-------------------\n\n- Add support for Python 3.10 and 3.11.\n- Drop support for Python 3.6.\n- Fix ``ghcloneall --user ... --github-token ... --include-private`` not\n including any private repositories (GH: #16).\n\n\n1.10.1 (2021-05-26)\n-------------------\n\n- When determining if a repository is dirty, use the repository's\n configured default branch from GitHub instead of assuming that the\n default is \"master\".\n\n\n1.10.0 (2021-04-10)\n-------------------\n\n- Allow authentication with GitHub token.\n- Depend on requests-cache < 0.6 on Python 2.7.\n- Add support for Python 3.9.\n- Drop support for Python 3.5.\n\n\n1.9.2 (2019-10-15)\n------------------\n\n- Add support for Python 3.8.\n\n\n1.9.1 (2019-10-07)\n------------------\n\n- Reuse HTTP connections for GitHub API requests.\n\n\n1.9.0 (2019-09-06)\n------------------\n\n- Can now clone all user's gists.\n- Command line args: --gists, --repos.\n\n\n1.8.0 (2019-08-28)\n------------------\n\n- Skip forks and archived repositories by default.\n- Command-line args: --include-forks, --exclude-forks.\n- Command-line args: --include-archived, --exclude-archived.\n- Command-line args: --include-private, --exclude-private.\n- Command-line args: --include-disabled, --exclude-disabled.\n- Use a custom User-Agent header when talking to GitHub.\n\n\n1.7.1 (2019-08-14)\n------------------\n\n- Drop support for Python 3.3 and 3.4.\n- Add a test suite.\n- Fix AttributeError: 'str' object has no attribute 'format_map' on Python 2.\n\n\n1.7.0 (2018-12-19)\n------------------\n\n- Command line args: -q, --quiet\n- Fix display corruption on ^C\n\n\n1.6.1 (2018-10-19)\n------------------\n\n- Fix TypeError: get() got an unexpected keyword argument 'fallback' on\n Python 2.\n\n\n1.6 (2016-12-29)\n----------------\n\n- Comprehensive rebranding:\n\n - Rename the GitHub repository to https://github.com/mgedmin/ghcloneall\n - Rename ``cloneall.py`` to ``ghcloneall.py``\n - Rename the config file to ``.ghcloneallrc``, and rename the config\n section to ``[ghcloneall]``.\n\n- Don't print tracebacks on ^C (this regressed in 1.5).\n\n\n1.5 (2016-12-29)\n----------------\n\n- Released to PyPI as ``ghcloneall``\n- Added Python 2.7 support\n\n\n1.4 (2016-12-28)\n----------------\n\n- Command line args: --user, --pattern, --init\n- Load (some) options from a ``.cloneallrc``\n- Stop using ``--organization=ZopeFoundation`` by default, require an\n explicit option (or config file)\n- Rename clone_all_zf_repos.py to cloneall.py\n\n\n1.3 (2016-12-28)\n----------------\n\n- Command line args: -c\n- Show progress while fetching the list of repositories from GitHub\n- Update repositories concurrently by default\n- Highlight items in progress\n- Highlight failed items in red\n- Tweak progress bar style from ``[=== ]`` to ``[###..]``\n- Clear the progress bar on ^C\n- Handle git errors nicely\n- Bugfix: -vv could fail with NameError if unknown files were present in a\n working tree\n- Bugfix: correctly show progress when using --start-from\n- Bugfix: script would hang (for 10 minutes) if you didn't already have an\n SSH control master process running\n- Bugfix: --dry-run didn't show which repos were new\n\n\n1.2 (2016-11-09)\n----------------\n\n- Command line args: --dry-run, --verbose\n- Cache HTTP responses on disk for 10 minutes to avoid GitHub API rate limits\n- Report about forgotten uncommitted and staged changes\n- Warn about local (unpushed) commits too\n- Warn about other branches being checked out\n- Default to SSH URLs again (faster when using SSH's ControlPersist)\n\n\n1.1 (2015-11-07)\n----------------\n\n- Command line args: --version, --start-from, --organization\n- Output formatting: shorter repository names, totals at the end\n- Use ANSI colors to indicate changes\n- Don't print tracebacks on ^C\n- Default to HTTPS URLs\n\n\n1.0 (2015-11-07)\n----------------\n\n- Moved from a gist to a proper GitHub repository.\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "Clone/update all user/organization GitHub repositories",
"version": "1.12.0",
"project_urls": {
"Homepage": "https://github.com/mgedmin/ghcloneall"
},
"split_keywords": [
"github",
"git",
"clone",
"automation"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "2ece41ca0baa23cfa29fa9eee22e5ecfd4bcfa7500a16719d84cb675c30fa27f",
"md5": "6a66638d332ac6cec4ec3324297bde1b",
"sha256": "d8ea1e26fd6ed7830a1ad3f99e5865a45d76c221a3f21caa1069a892defe76b0"
},
"downloads": -1,
"filename": "ghcloneall-1.12.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "6a66638d332ac6cec4ec3324297bde1b",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": null,
"size": 15403,
"upload_time": "2024-10-09T10:53:01",
"upload_time_iso_8601": "2024-10-09T10:53:01.354782Z",
"url": "https://files.pythonhosted.org/packages/2e/ce/41ca0baa23cfa29fa9eee22e5ecfd4bcfa7500a16719d84cb675c30fa27f/ghcloneall-1.12.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "0b50f81688ee13c68f1c3167754f74c0167507403ce567514af806b8cf2f3533",
"md5": "4f4c3726f0efbda106a0cb93b160a039",
"sha256": "2cbcbed8d52727081b7fe9b4c6d0a54406f512fb0d595edb0d70a639e1853809"
},
"downloads": -1,
"filename": "ghcloneall-1.12.0.tar.gz",
"has_sig": false,
"md5_digest": "4f4c3726f0efbda106a0cb93b160a039",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 26350,
"upload_time": "2024-10-09T10:53:02",
"upload_time_iso_8601": "2024-10-09T10:53:02.660835Z",
"url": "https://files.pythonhosted.org/packages/0b/50/f81688ee13c68f1c3167754f74c0167507403ce567514af806b8cf2f3533/ghcloneall-1.12.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-10-09 10:53:02",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "mgedmin",
"github_project": "ghcloneall",
"travis_ci": false,
"coveralls": true,
"github_actions": true,
"appveyor": true,
"tox": true,
"lcname": "ghcloneall"
}
JSON
