[](https://github.com/asottile/all-repos/actions/workflows/main.yml)
[](https://results.pre-commit.ci/latest/github/asottile/all-repos/main)
all-repos
=========
Clone all your repositories and apply sweeping changes.
## Installation
```bash
pip install all-repos
```
## CLI
All command line interfaces provided by `all-repos` provide the following
options:
- `-h` / `--help`: show usage information
- `-C CONFIG_FILENAME` / `--config-filename CONFIG_FILENAME`: use a non-default
config file (the default `all-repos.json` can be changed with the environment
variable `ALL_REPOS_CONFIG_FILENAME`).
- `--color {auto,always,never}`: use color in output (default `auto`).
### `all-repos-complete [options]`
Add `git clone` tab completion for all-repos repositories.
Requires [jq](https://stedolan.github.io/jq/) to function.
Add to `.bash_profile`:
```bash
eval "$(all-repos-complete -C ~/.../all-repos.json --bash)"
```
### `all-repos-clone [options]`
Clone all the repositories into the `output_dir`. If run again, this command
will update existing repositories.
Options:
- `-j JOBS` / `--jobs JOBS`: how many concurrent jobs will be used to complete
the operation. Specify 0 or -1 to match the number of cpus. (default `8`).
Sample invocations:
- `all-repos-clone`: clone the repositories specified in `all-repos.json`
- `all-repos-clone -C all-repos2.json`: clone using a non-default config
filename.
### `all-repos-find-files [options] PATTERN`
Similar to a distributed `git ls-files | grep -P PATTERN`.
Arguments:
- `PATTERN`: the [python regex](https://docs.python.org/3/library/re.html)
to match.
Options:
- `--repos-with-matches`: only print repositories with matches.
Sample invocations:
- `all-repos-find-files setup.py`: find all `setup.py` files.
- `all-repos-find-files --repos setup.py`: find all repositories containing
a `setup.py`.
### `all-repos-grep [options] [GIT_GREP_OPTIONS]`
Similar to a distributed `git grep ...`.
Options:
- `--repos-with-matches`: only print repositories with matches.
- `GIT_GREP_OPTIONS`: additional arguments will be passed on to `git grep`.
see `git grep --help` for available options.
Sample invocations:
- `all-repos-grep pre-commit -- 'requirements*.txt'`: find all repositories
which have `pre-commit` listed in a requirements file.
- `all-repos-grep -L six -- setup.py`: find setup.py files which do not
contain `six`.
### `all-repos-list-repos [options]`
List all cloned repository names.
### `all-repos-manual [options]`
Interactively apply a manual change across repos.
_note_: `all-repos-manual` will always run in `--interactive` autofixing mode.
_note_: `all-repos-manual` _requires_ the `--repos` autofixer option.
Options:
- [autofix options](#all_reposautofix_libadd_fixer_args): `all-repos-manual` is
an autofixer and supports all of the autofixer options.
- `--branch-name BRANCH_NAME`: override the autofixer branch name (default
`all-repos-manual`).
- `--commit-msg COMMIT_MSG` (required): set the autofixer commit message.
### `all-repos-sed [options] EXPRESSION FILENAMES`
Similar to a distributed
`git ls-files -z -- FILENAMES | xargs -0 sed -i EXPRESSION`.
_note_: this assumes GNU sed. If you're on macOS, install `gnu-sed` with Homebrew:
```bash
brew install gnu-sed
# Add to .bashrc / .zshrc
export PATH="$(brew --prefix)/opt/gnu-sed/libexec/gnubin:$PATH"
```
Arguments:
- `EXPRESSION`: sed program. For example: `s/hi/hello/g`.
- `FILENAMES`: filenames glob (passed to `git ls-files`).
Options:
- [autofix options](#all_reposautofix_libadd_fixer_args): `all-repos-sed` is
an autofixer and supports all of the autofixer options.
- `-r` / `--regexp-extended`: use extended regular expressions in the script.
See `man sed` for further details.
- `--branch-name BRANCH_NAME` override the autofixer branch name (default
`all-repos-sed`).
- `--commit-msg COMMIT_MSG` override the autofixer commit message. (default
`git ls-files -z -- FILENAMES | xargs -0 sed -i ... EXPRESSION`).
Sample invocations:
- `all-repos-sed 's/foo/bar/g' -- '*'`: replace `foo` with `bar` in all files.
## Configuring
A configuration file looks roughly like this:
```json
{
"output_dir": "output",
"source": "all_repos.source.github",
"source_settings": {
"api_key": "...",
"username": "asottile"
},
"push": "all_repos.push.github_pull_request",
"push_settings": {
"api_key": "...",
"username": "asottile"
}
}
```
- `output_dir`: where repositories will be cloned to when `all-repos-clone` is
run.
- `source`: the module import path to a `source`, see below for builtin
source modules as well as directions for writing your own.
- `source_settings`: the source-type-specific settings, the source module's
documentation will explain the various possible values.
- `push`: the module import path to a `push`, see below for builtin push
modules as well as directions for writing your own.
- `push_settings`: the push-type-specific settings, the push module's
documentation will explain the various possible values.
- `include` (default `""`): python regex for selecting repositories. Only
repository names which match this regex will be included.
- `exclude` (default `"^$"`): python regex for excluding repositories.
Repository names which match this regex will be excluded.
- `all_branches` (default `false`): whether to clone all of the branches or
just the default upstream branch.
## Source modules
### `all_repos.source.json_file`
Clones all repositories listed in a file. The file must be formatted as
follows:
```json
{
"example/repo1": "https://git.example.com/example/repo1",
"repo2": "https://git.example.com/repo2"
}
```
#### Required `source_settings`
- `filename`: file containing repositories one-per-line.
#### Directory location
```
output/
+--- repos.json
+--- repos_filtered.json
+--- {repo_key1}/
+--- {repo_key2}/
+--- {repo_key3}/
```
### `all_repos.source.github`
Clones all repositories available to a user on github.
#### Required `source_settings`
- `api_key`: the api key which the user will log in as.
- Use [the settings tab](//github.com/settings/tokens/new) to create a
personal access token.
- The minimum scope required to function is `public_repo`, though you'll
need `repo` to access private repositories.
- `api_key_env`: alternatively API key can also be passed via an environment variable
- `username`: the github username you will log in as.
#### Optional `source_settings`
- `collaborator` (default `false`): whether to include repositories which are
not owned but can be contributed to as a collaborator.
- `forks` (default `false`): whether to include repositories which are forks.
- `private` (default `false`): whether to include private repositories.
- `archived` (default: `false`): whether to include archived repositories.
- `base_url` (default: `https://api.github.com`) is the base URL to the Github
API to use (for Github Enterprise support set this to `https://{your_domain}/api/v3`).
#### Directory location
```
output/
+--- repos.json
+--- repos_filtered.json
+--- {username1}/
+--- {repo1}/
+--- {repo2}/
+--- {username2}/
+--- {repo3}/
```
### `all_repos.source.github_forks`
Clones all repositories forked from a repository on github.
#### Required `source_settings`
- `api_key`: the api key which the user will log in as.
- Use [the settings tab](//github.com/settings/tokens/new) to create a
personal access token.
- The minimum scope required to function is `public_repo`.
- `api_key_env`: alternatively API key can also be passed via an environment variable
- `repo`: the repo which has forks
#### Optional `source_settings`
- `collaborator` (default `true`): whether to include repositories which are
not owned but can be contributed to as a collaborator.
- `forks` (default `true`): whether to include repositories which are forks.
- `private` (default `false`): whether to include private repositories.
- `archived` (default: `false`): whether to include archived repositories.
- `base_url` (default: `https://api.github.com`) is the base URL to the Github
API to use (for Github Enterprise support set this to `https://{your_domain}/api/v3`).
#### Directory location
See the directory structure for
[`all_repos.source.github`](#all_repossourcegithub).
### `all_repos.source.github_org`
Clones all repositories from an organization on github.
#### Required `source_settings`
- `api_key`: the api key which the user will log in as.
- Use [the settings tab](//github.com/settings/tokens/new) to create a
personal access token.
- The minimum scope required to function is `public_repo`, though you'll
need `repo` to access private repositories.
- `api_key_env`: alternatively API key can also be passed via an environment variable
- `org`: the organization to clone from
#### Optional `source_settings`
- `collaborator` (default `true`): whether to include repositories which are
not owned but can be contributed to as a collaborator.
- `forks` (default `false`): whether to include repositories which are forks.
- `private` (default `false`): whether to include private repositories.
- `archived` (default: `false`): whether to include archived repositories.
- `base_url` (default: `https://api.github.com`) is the base URL to the Github
API to use (for Github Enterprise support set this to `https://{your_domain}/api/v3`).
#### Directory location
See the directory structure for
[`all_repos.source.github`](#all_repossourcegithub).
### `all_repos.source.gitolite`
Clones all repositories available to a user on a
[gitolite](http://gitolite.com/gitolite/index.html) host.
#### Required `source_settings`
- `username`: the user to SSH to the server as (usually `git`)
- `hostname`: the hostname of your gitolite server (e.g. `git.mycompany.com`)
The gitolite API is served over SSH. It is assumed that when `all-repos-clone`
is called, it's possible to make SSH connections with the username and hostname
configured here in order to query that API.
#### Optional `source_settings`
- `mirror_path` (default `None`): an optional mirror to clone repositories from.
This is a Python format string, and can use the variable `repo_name`.
This can be anything git understands, such as another remote server (e.g.
`gitmirror.mycompany.com:{repo_name}`) or a local path (e.g.
`/gitolite/git/{repo_name}.git`).
#### Directory location
```
output/
+--- repos.json
+--- repos_filtered.json
+--- {repo_name1}.git/
+--- {repo_name2}.git/
+--- {repo_name3}.git/
```
### `all_repos.source.bitbucket`
Clones all repositories available to a user on Bitbucket Cloud.
#### Required `source_settings`
- `username`: the Bitbucket username you will log in as.
- `app_password`: the authentication method for the above user to login with
- Create an application password within your [account settings](https://bitbucket.org/account/admin/app-passwords).
- We need the scope: Repositories -> Read
### `all_repos.source.bitbucket_server`
Clones all repositories available to a user on Bitbucket Server.
#### Required `source_settings`
- `base_url`: the bitbucket server URL (eg `bitbucket.domain.com`)
- `username`: the Bitbucket username you will log in as.
- `app_password`: the authentication method for the above user to login with
- Create an application password within your [account settings](https://bitbucket.domain.com/plugins/servlet/access-tokens/manage).
- We need the scope: Repositories -> Read
#### Optional `source_settings`
- `project` (default `None`): an optional project to restrict the search for repositories.
#### Directory location
```
output/
+--- repos.json
+--- repos_filtered.json
+--- {username1}/
+--- {repo1}/
+--- {repo2}/
+--- {username2}/
+--- {repo3}/
```
### `all_repos.source.gitlab_org`
Clones all repositories from an organization on gitlab.
#### Required `source_settings`
- `api_key`: the api key which the user will log in as.
- Use the settings tab (eg https://{gitlab.domain.com}/-/profile/personal_access_tokens) to create a
personal access token.
- We need the scope: `read_api`, `read_repository`.
- `api_key_env`: alternatively API key can also be passed via an environment variable
- `org`: the organization to clone from
#### Optional `source_settings`
- `base_url`: (default `https://gitlab.com/api/v4`) the gitlab server URL
- `archived` (default: `false`): whether to include archived repositories.
#### Directory location
```
output/
+--- repos.json
+--- repos_filtered.json
+--- {org}/
+--- {subpgroup1}/
+--- {subpgroup2}/
+--- {repo1}/
+--- {repo2}/
+--- {repo3}/
+--- {repo4}/
```
## Writing your own source
First create a module. This module must have the following api:
### A `Settings` class
This class will receive keyword arguments for all values in the
`source_settings` dictionary.
An easy way to implement the `Settings` class is by using a `namedtuple`:
```python
Settings = collections.namedtuple('Settings', ('required_thing', 'optional'))
Settings.__new__.__defaults__ = ('optional default value',)
```
In this example, the `required_thing` setting is a **required** setting
whereas `optional` may be omitted (and will get a default value of
`'optional default value'`).
### `def list_repos(settings: Settings) -> Dict[str, str]:` callable
This callable will be passed an instance of your `Settings` class. It must
return a mapping from `{repo_name: repository_url}`. The `repo_name` is the
directory name inside the `output_dir`.
## Push modules
### `all_repos.push.merge_to_master`
Merges the branch directly to the default branch and pushes. The commands it
runs look roughly like this:
```bash
git checkout main
git pull
git merge --no-ff $BRANCH
git push origin HEAD
```
#### Optional `push_settings`
- `fast_forward` (default: `false`): if `true`, perform a fast-forward
merge (`--ff-only`). If `false`, create a merge commit (`--no-ff`).
### `all_repos.push.github_pull_request`
Pushes the branch to `origin` and then creates a github pull request for the
branch.
#### Required `push_settings`
- `api_key`: the api key which the user will log in as.
- Use [the settings tab](//github.com/settings/tokens/new) to create a
personal access token.
- The minimum scope required to function is `public_repo`, though you'll
need `repo` to access private repositories.
- `api_key_env`: alternatively API key can also be passed via an environment variable
- `username`: the github username you will log in as.
#### Optional `push_settings`
- `fork` (default: `false`): (if applicable) a fork will be created and pushed
to instead of the upstream repository. The pull request will then be made
to the upstream repository.
- `base_url` (default: `https://api.github.com`) is the base URL to the Github
API to use (for Github Enterprise support set this to `https://{your_domain}/api/v3`).
### `all_repos.push.bitbucket_server_pull_request`
Pushes the branch to `origin` and then creates a Bitbucket pull request for the branch.
#### Required `push_settings`
- `base_url`: the Bitbucket server URL (eg `bitbucket.domain.com`)
- `username`: the Bitbucket username you will log in as.
- `app_password`: the authentication method for the above user to login with
- Create an application password within your [account settings](https://bitbucket.domain.com/plugins/servlet/access-tokens/manage).
- We need the scope: Repositories -> Read
### `all_repos.push.gitlab_pull_request`
Pushes the branch to `origin` and then creates a GitLab pull request for the branch.
#### Required `push_settings`
- `base_url`: the GitLab server URL (eg `https://{gitlab.domain.com}/api/v4`)
- `api_key`: the api key which the user will log in as.
- Use the settings tab (eg https://{gitlab.domain.com}/-/profile/personal_access_tokens) to create a
personal access token.
- We need the scope: `write_repository`.
- `api_key_env`: alternatively API key can also be passed via an environment variable
### `all_repos.push.readonly`
Does nothing.
#### `push_settings`
There are no configurable settings for `readonly`.
## Writing your own push module
First create a module. This module must have the following api:
### A `Settings` class
This class will receive keyword arguments for all values in the `push_settings`
dictionary.
### `def push(settings: Settings, branch_name: str) -> None:`
This callable will be passed an instance of your `Settings` class. It should
deploy the branch. The function will be called with the root of the
repository as the `cwd`.
## Writing an autofixer
An autofixer applies a change over all repositories.
`all-repos` provides several api functions to write your autofixers with:
### `all_repos.autofix_lib.add_fixer_args`
```python
def add_fixer_args(parser):
```
Adds the autofixer cli options.
Options:
- `--dry-run`: show what would happen but do not push.
- `-i` / `--interactive`: interactively approve / deny fixes.
- `-j JOBS` / `--jobs JOBS`: how many concurrent jobs will be used to complete
the operation. Specify 0 or -1 to match the number of cpus. (default `1`).
- `--limit LIMIT`: maximum number of repos to process (default: unlimited).
- `--author AUTHOR`: override commit author. This is passed directly to
`git commit`. An example: `--author='Herp Derp <herp.derp@umich.edu>'`.
- `--repos [REPOS [REPOS ...]]`: run against specific repositories instead.
This is especially useful with `xargs autofixer ... --repos`. This can be
used to specify repositories which are not managed by `all-repos`.
### `all_repos.autofix_lib.from_cli`
```python
def from_cli(args, *, find_repos, msg, branch_name):
```
Parse cli arguments and produce `autofix_lib` primitives. Returns
`(repos, config, commit, autofix_settings)`. This is handled separately from
`fix` to allow for fixers to adjust arguments.
- `find_repos`: callback taking `Config` as a positional argument.
- `msg`: commit message.
- `branch_name`: identifier used to construct the branch name.
### `all_repos.autofix_lib.fix`
```python
def fix(
repos, *,
apply_fix,
check_fix=_noop_check_fix,
config: Config,
commit: Commit,
autofix_settings: AutofixSettings,
):
```
Apply the fix.
- `apply_fix`: callback which will be called once per repository. The `cwd`
when the function is called will be the root of the repository.
### `all_repos.autofix_lib.run`
```python
def run(*cmd, **kwargs):
```
Wrapper around `subprocess.run` which prints the command it will run. Unlike
`subprocess.run`, this defaults `check=True` unless explicitly disabled.
### Example autofixer
The trivial autofixer is as follows:
```python
import argparse
from all_repos import autofix_lib
def find_repos(config):
return []
def apply_fix():
pass
def main(argv=None):
parser = argparse.ArgumentParser()
autofix_lib.add_fixer_args(parser)
args = parser.parse_args(argv)
repos, config, commit, autofix_settings = autofix_lib.from_cli(
args, find_repos=find_repos, msg='msg', branch_name='branch-name',
)
autofix_lib.fix(
repos, apply_fix=apply_fix, config=config, commit=commit,
autofix_settings=autofix_settings,
)
if __name__ == '__main__':
raise SystemExit(main())
```
You can find some more involved examples in [all_repos/autofix](https://github.com/asottile/all-repos/tree/main/all_repos/autofix):
- `all_repos.autofix.azure_pipelines_autoupdate`: upgrade pinned azure
pipelines template repository references.
- `all_repos.autofix.pre_commit_autoupdate`: runs `pre-commit autoupdate`.
- `all_repos.autofix.pre_commit_autopep8_migrate`: migrates `autopep8-wrapper`
from [pre-commit/pre-commit-hooks] to [mirrors-autopep8].
- `all_repos.autofix.pre_commit_cache_dir`: updates the cache directory
for travis-ci / appveyor for pre-commit 1.x.
- `all_repos.autofix.pre_commit_flake8_migrate`: migrates `flake8` from
[pre-commit/pre-commit-hooks] to [pycqa/flake8].
- `all_repos.autofix.pre_commit_migrate_config`: runs
`pre-commit migrate-config`.
- `all_repos.autofix.setup_py_upgrade`: runs [setup-py-upgrade] and then
[setup-cfg-fmt] to migrate `setup.py` to `setup.cfg`.
[pre-commit/pre-commit-hooks]: https://github.com/pre-commit/pre-commit-hooks
[mirrors-autopep8]: https://github.com/pre-commit/mirrors-autopep8
[pycqa/flake8]: https://gitlab.com/pycqa/flake8
[setup-py-upgrade]: https://github.com/asottile/setup-py-upgrade
[setup-cfg-fmt]: https://github.com/asottile/setup-cfg-fmt
Raw data
{
"_id": null,
"home_page": "https://github.com/asottile/all-repos",
"name": "all-repos",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": "",
"keywords": "",
"author": "Anthony Sottile",
"author_email": "asottile@umich.edu",
"download_url": "https://files.pythonhosted.org/packages/a6/56/29006be2546b897a5c62a3d4a7e613abf5a3533554d948b0e0af27546f1b/all_repos-1.27.0.tar.gz",
"platform": null,
"description": "[](https://github.com/asottile/all-repos/actions/workflows/main.yml)\n[](https://results.pre-commit.ci/latest/github/asottile/all-repos/main)\n\nall-repos\n=========\n\nClone all your repositories and apply sweeping changes.\n\n## Installation\n\n```bash\npip install all-repos\n```\n\n\n## CLI\n\nAll command line interfaces provided by `all-repos` provide the following\noptions:\n\n- `-h` / `--help`: show usage information\n- `-C CONFIG_FILENAME` / `--config-filename CONFIG_FILENAME`: use a non-default\n config file (the default `all-repos.json` can be changed with the environment\n variable `ALL_REPOS_CONFIG_FILENAME`).\n- `--color {auto,always,never}`: use color in output (default `auto`).\n\n\n### `all-repos-complete [options]`\n\nAdd `git clone` tab completion for all-repos repositories.\n\nRequires [jq](https://stedolan.github.io/jq/) to function.\n\nAdd to `.bash_profile`:\n\n```bash\neval \"$(all-repos-complete -C ~/.../all-repos.json --bash)\"\n```\n\n### `all-repos-clone [options]`\n\nClone all the repositories into the `output_dir`. If run again, this command\nwill update existing repositories.\n\nOptions:\n\n- `-j JOBS` / `--jobs JOBS`: how many concurrent jobs will be used to complete\n the operation. Specify 0 or -1 to match the number of cpus. (default `8`).\n\nSample invocations:\n\n- `all-repos-clone`: clone the repositories specified in `all-repos.json`\n- `all-repos-clone -C all-repos2.json`: clone using a non-default config\n filename.\n\n### `all-repos-find-files [options] PATTERN`\n\nSimilar to a distributed `git ls-files | grep -P PATTERN`.\n\nArguments:\n- `PATTERN`: the [python regex](https://docs.python.org/3/library/re.html)\n to match.\n\nOptions:\n\n- `--repos-with-matches`: only print repositories with matches.\n\nSample invocations:\n\n- `all-repos-find-files setup.py`: find all `setup.py` files.\n- `all-repos-find-files --repos setup.py`: find all repositories containing\n a `setup.py`.\n\n### `all-repos-grep [options] [GIT_GREP_OPTIONS]`\n\nSimilar to a distributed `git grep ...`.\n\nOptions:\n\n- `--repos-with-matches`: only print repositories with matches.\n- `GIT_GREP_OPTIONS`: additional arguments will be passed on to `git grep`.\n see `git grep --help` for available options.\n\nSample invocations:\n\n- `all-repos-grep pre-commit -- 'requirements*.txt'`: find all repositories\n which have `pre-commit` listed in a requirements file.\n- `all-repos-grep -L six -- setup.py`: find setup.py files which do not\n contain `six`.\n\n### `all-repos-list-repos [options]`\n\nList all cloned repository names.\n\n### `all-repos-manual [options]`\n\nInteractively apply a manual change across repos.\n\n_note_: `all-repos-manual` will always run in `--interactive` autofixing mode.\n\n_note_: `all-repos-manual` _requires_ the `--repos` autofixer option.\n\nOptions:\n\n- [autofix options](#all_reposautofix_libadd_fixer_args): `all-repos-manual` is\n an autofixer and supports all of the autofixer options.\n- `--branch-name BRANCH_NAME`: override the autofixer branch name (default\n `all-repos-manual`).\n- `--commit-msg COMMIT_MSG` (required): set the autofixer commit message.\n\n\n### `all-repos-sed [options] EXPRESSION FILENAMES`\n\nSimilar to a distributed\n`git ls-files -z -- FILENAMES | xargs -0 sed -i EXPRESSION`.\n\n_note_: this assumes GNU sed. If you're on macOS, install `gnu-sed` with Homebrew:\n\n```bash\nbrew install gnu-sed\n\n# Add to .bashrc / .zshrc\nexport PATH=\"$(brew --prefix)/opt/gnu-sed/libexec/gnubin:$PATH\"\n```\n\nArguments:\n\n- `EXPRESSION`: sed program. For example: `s/hi/hello/g`.\n- `FILENAMES`: filenames glob (passed to `git ls-files`).\n\nOptions:\n\n- [autofix options](#all_reposautofix_libadd_fixer_args): `all-repos-sed` is\n an autofixer and supports all of the autofixer options.\n- `-r` / `--regexp-extended`: use extended regular expressions in the script.\n See `man sed` for further details.\n- `--branch-name BRANCH_NAME` override the autofixer branch name (default\n `all-repos-sed`).\n- `--commit-msg COMMIT_MSG` override the autofixer commit message. (default\n `git ls-files -z -- FILENAMES | xargs -0 sed -i ... EXPRESSION`).\n\nSample invocations:\n\n- `all-repos-sed 's/foo/bar/g' -- '*'`: replace `foo` with `bar` in all files.\n\n## Configuring\n\nA configuration file looks roughly like this:\n\n```json\n{\n \"output_dir\": \"output\",\n \"source\": \"all_repos.source.github\",\n \"source_settings\": {\n \"api_key\": \"...\",\n \"username\": \"asottile\"\n },\n \"push\": \"all_repos.push.github_pull_request\",\n \"push_settings\": {\n \"api_key\": \"...\",\n \"username\": \"asottile\"\n }\n}\n```\n\n- `output_dir`: where repositories will be cloned to when `all-repos-clone` is\n run.\n- `source`: the module import path to a `source`, see below for builtin\n source modules as well as directions for writing your own.\n- `source_settings`: the source-type-specific settings, the source module's\n documentation will explain the various possible values.\n- `push`: the module import path to a `push`, see below for builtin push\n modules as well as directions for writing your own.\n- `push_settings`: the push-type-specific settings, the push module's\n documentation will explain the various possible values.\n- `include` (default `\"\"`): python regex for selecting repositories. Only\n repository names which match this regex will be included.\n- `exclude` (default `\"^$\"`): python regex for excluding repositories.\n Repository names which match this regex will be excluded.\n- `all_branches` (default `false`): whether to clone all of the branches or\n just the default upstream branch.\n\n## Source modules\n\n### `all_repos.source.json_file`\n\nClones all repositories listed in a file. The file must be formatted as\nfollows:\n\n```json\n{\n \"example/repo1\": \"https://git.example.com/example/repo1\",\n \"repo2\": \"https://git.example.com/repo2\"\n}\n```\n\n#### Required `source_settings`\n\n- `filename`: file containing repositories one-per-line.\n\n#### Directory location\n\n```\noutput/\n+--- repos.json\n+--- repos_filtered.json\n+--- {repo_key1}/\n+--- {repo_key2}/\n+--- {repo_key3}/\n```\n\n### `all_repos.source.github`\n\nClones all repositories available to a user on github.\n\n#### Required `source_settings`\n\n- `api_key`: the api key which the user will log in as.\n - Use [the settings tab](//github.com/settings/tokens/new) to create a\n personal access token.\n - The minimum scope required to function is `public_repo`, though you'll\n need `repo` to access private repositories.\n- `api_key_env`: alternatively API key can also be passed via an environment variable\n- `username`: the github username you will log in as.\n\n#### Optional `source_settings`\n\n- `collaborator` (default `false`): whether to include repositories which are\n not owned but can be contributed to as a collaborator.\n- `forks` (default `false`): whether to include repositories which are forks.\n- `private` (default `false`): whether to include private repositories.\n- `archived` (default: `false`): whether to include archived repositories.\n- `base_url` (default: `https://api.github.com`) is the base URL to the Github\n API to use (for Github Enterprise support set this to `https://{your_domain}/api/v3`).\n\n#### Directory location\n\n```\noutput/\n+--- repos.json\n+--- repos_filtered.json\n+--- {username1}/\n +--- {repo1}/\n +--- {repo2}/\n+--- {username2}/\n +--- {repo3}/\n```\n\n### `all_repos.source.github_forks`\n\nClones all repositories forked from a repository on github.\n\n#### Required `source_settings`\n\n- `api_key`: the api key which the user will log in as.\n - Use [the settings tab](//github.com/settings/tokens/new) to create a\n personal access token.\n - The minimum scope required to function is `public_repo`.\n- `api_key_env`: alternatively API key can also be passed via an environment variable\n- `repo`: the repo which has forks\n\n#### Optional `source_settings`\n\n- `collaborator` (default `true`): whether to include repositories which are\n not owned but can be contributed to as a collaborator.\n- `forks` (default `true`): whether to include repositories which are forks.\n- `private` (default `false`): whether to include private repositories.\n- `archived` (default: `false`): whether to include archived repositories.\n- `base_url` (default: `https://api.github.com`) is the base URL to the Github\n API to use (for Github Enterprise support set this to `https://{your_domain}/api/v3`).\n\n#### Directory location\n\nSee the directory structure for\n[`all_repos.source.github`](#all_repossourcegithub).\n\n\n### `all_repos.source.github_org`\n\nClones all repositories from an organization on github.\n\n#### Required `source_settings`\n\n- `api_key`: the api key which the user will log in as.\n - Use [the settings tab](//github.com/settings/tokens/new) to create a\n personal access token.\n - The minimum scope required to function is `public_repo`, though you'll\n need `repo` to access private repositories.\n- `api_key_env`: alternatively API key can also be passed via an environment variable\n- `org`: the organization to clone from\n\n#### Optional `source_settings`\n\n- `collaborator` (default `true`): whether to include repositories which are\n not owned but can be contributed to as a collaborator.\n- `forks` (default `false`): whether to include repositories which are forks.\n- `private` (default `false`): whether to include private repositories.\n- `archived` (default: `false`): whether to include archived repositories.\n- `base_url` (default: `https://api.github.com`) is the base URL to the Github\n API to use (for Github Enterprise support set this to `https://{your_domain}/api/v3`).\n\n#### Directory location\n\nSee the directory structure for\n[`all_repos.source.github`](#all_repossourcegithub).\n\n### `all_repos.source.gitolite`\n\nClones all repositories available to a user on a\n[gitolite](http://gitolite.com/gitolite/index.html) host.\n\n#### Required `source_settings`\n\n- `username`: the user to SSH to the server as (usually `git`)\n- `hostname`: the hostname of your gitolite server (e.g. `git.mycompany.com`)\n\nThe gitolite API is served over SSH. It is assumed that when `all-repos-clone`\nis called, it's possible to make SSH connections with the username and hostname\nconfigured here in order to query that API.\n\n#### Optional `source_settings`\n\n- `mirror_path` (default `None`): an optional mirror to clone repositories from.\n This is a Python format string, and can use the variable `repo_name`.\n\n This can be anything git understands, such as another remote server (e.g.\n `gitmirror.mycompany.com:{repo_name}`) or a local path (e.g.\n `/gitolite/git/{repo_name}.git`).\n\n#### Directory location\n\n```\noutput/\n+--- repos.json\n+--- repos_filtered.json\n+--- {repo_name1}.git/\n+--- {repo_name2}.git/\n+--- {repo_name3}.git/\n```\n\n\n### `all_repos.source.bitbucket`\n\nClones all repositories available to a user on Bitbucket Cloud.\n\n#### Required `source_settings`\n\n- `username`: the Bitbucket username you will log in as.\n- `app_password`: the authentication method for the above user to login with\n - Create an application password within your [account settings](https://bitbucket.org/account/admin/app-passwords).\n - We need the scope: Repositories -> Read\n\n\n### `all_repos.source.bitbucket_server`\n\nClones all repositories available to a user on Bitbucket Server.\n\n#### Required `source_settings`\n\n- `base_url`: the bitbucket server URL (eg `bitbucket.domain.com`)\n- `username`: the Bitbucket username you will log in as.\n- `app_password`: the authentication method for the above user to login with\n - Create an application password within your [account settings](https://bitbucket.domain.com/plugins/servlet/access-tokens/manage).\n - We need the scope: Repositories -> Read\n\n#### Optional `source_settings`\n\n- `project` (default `None`): an optional project to restrict the search for repositories.\n\n\n#### Directory location\n\n```\noutput/\n+--- repos.json\n+--- repos_filtered.json\n+--- {username1}/\n +--- {repo1}/\n +--- {repo2}/\n+--- {username2}/\n +--- {repo3}/\n```\n\n### `all_repos.source.gitlab_org`\n\nClones all repositories from an organization on gitlab.\n\n#### Required `source_settings`\n\n- `api_key`: the api key which the user will log in as.\n - Use the settings tab (eg https://{gitlab.domain.com}/-/profile/personal_access_tokens) to create a\n personal access token.\n - We need the scope: `read_api`, `read_repository`.\n- `api_key_env`: alternatively API key can also be passed via an environment variable\n- `org`: the organization to clone from\n\n#### Optional `source_settings`\n\n- `base_url`: (default `https://gitlab.com/api/v4`) the gitlab server URL\n- `archived` (default: `false`): whether to include archived repositories.\n\n#### Directory location\n\n```\noutput/\n+--- repos.json\n+--- repos_filtered.json\n+--- {org}/\n +--- {subpgroup1}/\n +--- {subpgroup2}/\n +--- {repo1}/\n +--- {repo2}/\n +--- {repo3}/\n +--- {repo4}/\n```\n\n\n## Writing your own source\n\nFirst create a module. This module must have the following api:\n\n### A `Settings` class\n\nThis class will receive keyword arguments for all values in the\n`source_settings` dictionary.\n\nAn easy way to implement the `Settings` class is by using a `namedtuple`:\n\n```python\nSettings = collections.namedtuple('Settings', ('required_thing', 'optional'))\nSettings.__new__.__defaults__ = ('optional default value',)\n```\n\nIn this example, the `required_thing` setting is a **required** setting\nwhereas `optional` may be omitted (and will get a default value of\n`'optional default value'`).\n\n### `def list_repos(settings: Settings) -> Dict[str, str]:` callable\n\nThis callable will be passed an instance of your `Settings` class. It must\nreturn a mapping from `{repo_name: repository_url}`. The `repo_name` is the\ndirectory name inside the `output_dir`.\n\n## Push modules\n\n### `all_repos.push.merge_to_master`\n\nMerges the branch directly to the default branch and pushes. The commands it\nruns look roughly like this:\n\n```bash\ngit checkout main\ngit pull\ngit merge --no-ff $BRANCH\ngit push origin HEAD\n```\n\n#### Optional `push_settings`\n\n- `fast_forward` (default: `false`): if `true`, perform a fast-forward\n merge (`--ff-only`). If `false`, create a merge commit (`--no-ff`).\n\n### `all_repos.push.github_pull_request`\n\nPushes the branch to `origin` and then creates a github pull request for the\nbranch.\n\n#### Required `push_settings`\n\n- `api_key`: the api key which the user will log in as.\n - Use [the settings tab](//github.com/settings/tokens/new) to create a\n personal access token.\n - The minimum scope required to function is `public_repo`, though you'll\n need `repo` to access private repositories.\n- `api_key_env`: alternatively API key can also be passed via an environment variable\n- `username`: the github username you will log in as.\n\n#### Optional `push_settings`\n\n- `fork` (default: `false`): (if applicable) a fork will be created and pushed\n to instead of the upstream repository. The pull request will then be made\n to the upstream repository.\n- `base_url` (default: `https://api.github.com`) is the base URL to the Github\n API to use (for Github Enterprise support set this to `https://{your_domain}/api/v3`).\n\n### `all_repos.push.bitbucket_server_pull_request`\n\nPushes the branch to `origin` and then creates a Bitbucket pull request for the branch.\n\n#### Required `push_settings`\n\n- `base_url`: the Bitbucket server URL (eg `bitbucket.domain.com`)\n- `username`: the Bitbucket username you will log in as.\n- `app_password`: the authentication method for the above user to login with\n - Create an application password within your [account settings](https://bitbucket.domain.com/plugins/servlet/access-tokens/manage).\n - We need the scope: Repositories -> Read\n\n### `all_repos.push.gitlab_pull_request`\n\nPushes the branch to `origin` and then creates a GitLab pull request for the branch.\n\n#### Required `push_settings`\n\n- `base_url`: the GitLab server URL (eg `https://{gitlab.domain.com}/api/v4`)\n- `api_key`: the api key which the user will log in as.\n - Use the settings tab (eg https://{gitlab.domain.com}/-/profile/personal_access_tokens) to create a\n personal access token.\n - We need the scope: `write_repository`.\n- `api_key_env`: alternatively API key can also be passed via an environment variable\n\n### `all_repos.push.readonly`\n\nDoes nothing.\n\n#### `push_settings`\n\nThere are no configurable settings for `readonly`.\n\n## Writing your own push module\n\nFirst create a module. This module must have the following api:\n\n### A `Settings` class\n\nThis class will receive keyword arguments for all values in the `push_settings`\ndictionary.\n\n### `def push(settings: Settings, branch_name: str) -> None:`\n\nThis callable will be passed an instance of your `Settings` class. It should\ndeploy the branch. The function will be called with the root of the\nrepository as the `cwd`.\n\n## Writing an autofixer\n\nAn autofixer applies a change over all repositories.\n\n`all-repos` provides several api functions to write your autofixers with:\n\n### `all_repos.autofix_lib.add_fixer_args`\n\n```python\ndef add_fixer_args(parser):\n```\n\nAdds the autofixer cli options.\n\nOptions:\n\n- `--dry-run`: show what would happen but do not push.\n- `-i` / `--interactive`: interactively approve / deny fixes.\n- `-j JOBS` / `--jobs JOBS`: how many concurrent jobs will be used to complete\n the operation. Specify 0 or -1 to match the number of cpus. (default `1`).\n- `--limit LIMIT`: maximum number of repos to process (default: unlimited).\n- `--author AUTHOR`: override commit author. This is passed directly to\n `git commit`. An example: `--author='Herp Derp <herp.derp@umich.edu>'`.\n- `--repos [REPOS [REPOS ...]]`: run against specific repositories instead.\n This is especially useful with `xargs autofixer ... --repos`. This can be\n used to specify repositories which are not managed by `all-repos`.\n\n### `all_repos.autofix_lib.from_cli`\n\n```python\ndef from_cli(args, *, find_repos, msg, branch_name):\n```\n\nParse cli arguments and produce `autofix_lib` primitives. Returns\n`(repos, config, commit, autofix_settings)`. This is handled separately from\n`fix` to allow for fixers to adjust arguments.\n\n- `find_repos`: callback taking `Config` as a positional argument.\n- `msg`: commit message.\n- `branch_name`: identifier used to construct the branch name.\n\n### `all_repos.autofix_lib.fix`\n\n```python\ndef fix(\n repos, *,\n apply_fix,\n check_fix=_noop_check_fix,\n config: Config,\n commit: Commit,\n autofix_settings: AutofixSettings,\n):\n```\n\nApply the fix.\n\n- `apply_fix`: callback which will be called once per repository. The `cwd`\n when the function is called will be the root of the repository.\n\n\n### `all_repos.autofix_lib.run`\n\n```python\ndef run(*cmd, **kwargs):\n```\n\nWrapper around `subprocess.run` which prints the command it will run. Unlike\n`subprocess.run`, this defaults `check=True` unless explicitly disabled.\n\n### Example autofixer\n\nThe trivial autofixer is as follows:\n\n```python\nimport argparse\n\nfrom all_repos import autofix_lib\n\ndef find_repos(config):\n return []\n\ndef apply_fix():\n pass\n\ndef main(argv=None):\n parser = argparse.ArgumentParser()\n autofix_lib.add_fixer_args(parser)\n args = parser.parse_args(argv)\n\n repos, config, commit, autofix_settings = autofix_lib.from_cli(\n args, find_repos=find_repos, msg='msg', branch_name='branch-name',\n )\n autofix_lib.fix(\n repos, apply_fix=apply_fix, config=config, commit=commit,\n autofix_settings=autofix_settings,\n )\n\nif __name__ == '__main__':\n raise SystemExit(main())\n```\n\nYou can find some more involved examples in [all_repos/autofix](https://github.com/asottile/all-repos/tree/main/all_repos/autofix):\n- `all_repos.autofix.azure_pipelines_autoupdate`: upgrade pinned azure\n pipelines template repository references.\n- `all_repos.autofix.pre_commit_autoupdate`: runs `pre-commit autoupdate`.\n- `all_repos.autofix.pre_commit_autopep8_migrate`: migrates `autopep8-wrapper`\n from [pre-commit/pre-commit-hooks] to [mirrors-autopep8].\n- `all_repos.autofix.pre_commit_cache_dir`: updates the cache directory\n for travis-ci / appveyor for pre-commit 1.x.\n- `all_repos.autofix.pre_commit_flake8_migrate`: migrates `flake8` from\n [pre-commit/pre-commit-hooks] to [pycqa/flake8].\n- `all_repos.autofix.pre_commit_migrate_config`: runs\n `pre-commit migrate-config`.\n- `all_repos.autofix.setup_py_upgrade`: runs [setup-py-upgrade] and then\n [setup-cfg-fmt] to migrate `setup.py` to `setup.cfg`.\n\n[pre-commit/pre-commit-hooks]: https://github.com/pre-commit/pre-commit-hooks\n[mirrors-autopep8]: https://github.com/pre-commit/mirrors-autopep8\n[pycqa/flake8]: https://gitlab.com/pycqa/flake8\n[setup-py-upgrade]: https://github.com/asottile/setup-py-upgrade\n[setup-cfg-fmt]: https://github.com/asottile/setup-cfg-fmt\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "Clone all your repositories and apply sweeping changes.",
"version": "1.27.0",
"project_urls": {
"Homepage": "https://github.com/asottile/all-repos"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "68597c442f3d38292ca55b0d1c8d4b22fe988d667ad71acf5fee4835e2876dce",
"md5": "ff2e2804d3b727a5aa2271d6f67634e8",
"sha256": "2632e2a02615f7d3899aa199081b6f6c7ab38395427919886056a947ffa6e2c8"
},
"downloads": -1,
"filename": "all_repos-1.27.0-py2.py3-none-any.whl",
"has_sig": false,
"md5_digest": "ff2e2804d3b727a5aa2271d6f67634e8",
"packagetype": "bdist_wheel",
"python_version": "py2.py3",
"requires_python": ">=3.8",
"size": 44563,
"upload_time": "2024-02-10T18:14:37",
"upload_time_iso_8601": "2024-02-10T18:14:37.096026Z",
"url": "https://files.pythonhosted.org/packages/68/59/7c442f3d38292ca55b0d1c8d4b22fe988d667ad71acf5fee4835e2876dce/all_repos-1.27.0-py2.py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "a65629006be2546b897a5c62a3d4a7e613abf5a3533554d948b0e0af27546f1b",
"md5": "c6a124cd503fd55ed74bb7f373b1d28b",
"sha256": "96fea3e34caa004b0770501e6efb93dc49cbca05fb56c2b8b2a85d06fb3a4573"
},
"downloads": -1,
"filename": "all_repos-1.27.0.tar.gz",
"has_sig": false,
"md5_digest": "c6a124cd503fd55ed74bb7f373b1d28b",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8",
"size": 30939,
"upload_time": "2024-02-10T18:14:38",
"upload_time_iso_8601": "2024-02-10T18:14:38.992077Z",
"url": "https://files.pythonhosted.org/packages/a6/56/29006be2546b897a5c62a3d4a7e613abf5a3533554d948b0e0af27546f1b/all_repos-1.27.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-02-10 18:14:38",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "asottile",
"github_project": "all-repos",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"tox": true,
"lcname": "all-repos"
}
JSON
