| Name | gitutils JSON |
| Version |
1.12.1
JSON |
| download |
| home_page | |
| Summary | concurrent, pipelined, platform-agnostic Git utilities for managing a large number of Git repositories |
| upload_time | 2024-01-13 23:12:21 |
| maintainer | |
| docs_url | None |
| author | |
| requires_python | >=3.9 |
| license | |
| keywords |
concurrent
asynchronous
asyncio
git
github
ssh
|
| VCS |
|
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
# GitMC -- concurrent asynchronous Git Utilities for operations on massive numbers of Git repos
[](https://doi.org/10.5281/zenodo.3339891)
[](https://github.com/scivision/gitMC/actions/workflows/ci.yml)
[](http://pepy.tech/project/gitutils)
Platform-independent (Linux/Mac/Windows) Git utilities, useful for managing large (100+) numbers of Git repos.
Speed is an emphasis throughout, with concurrency via Python stdlib
[asyncio](https://docs.python.org/3/library/asyncio.html)
via
[asyncio.create_subprocess_exec](https://docs.python.org/3/library/asyncio-subprocess.html#asyncio.create_subprocess_exec)
and pipelining makes operations in effect 100x faster overall as the coroutines simultaneously wait for Git operations (particularly remote operations like "fetch" and "pull").
We have implemented individual concurrent subprocess timeout using
[asyncio.wait_for](https://docs.python.org/3/library/asyncio-task.html#asyncio.wait_for)
so that one Git operation hanging doesn't cause other Git operations to fail--this is good for when a Git login popup may go unnoticed by the human.
GitMC uses command-line Git because PyGit also requires command-line Git installed, and we don't need the advanced functionality.
---
Also see
[PyGit-bulk](https://github.com/scivision/pygit-bulk)
for managing large (100+) numbers of users / teams.
This repo contains a
[Git pre-commit script](./scripts/pre-commit)
with
[explanation](https://www.scivision.dev/git-commit-precheck-pep8/).
## Install
Install Git in a way accessible from the command line line
* Mac: `brew install git`
* Linux: `apt install git`
* Windows: command line [Git](https://git-scm.com/download/win).
```sh
python -m pip install -e .
```
## Usage
`gitbranch`
: Tells of any non-master branches under directory ~/code
`python -m gitutils.email`
: list all contributor email addresses. To fix unwanted emails use [Git-filter-repo](https://www.scivision.dev/git-update-email-address/)
`find_missing_file`
: find directories missing exact fullpath to file
`find_matching_file`
: find directories matching exact fullpath to file
### Sync large number of git repos
These assume numerous subdirectories under `~/code`.
They work very quickly for large numbers (100+) of repos.
* `gitstat` check if any local repos have pending changes
* `gitpull` Git pulls all repos (suggest `gitfetch` instead)
* `gitfetch` Git fetches all repos, printing a summary of files changed on remote
Place an empty file `.nogit` in a subdirectory to skip it.
#### [optional] speedup with https pull
For public repos, to make the Git remote checking go at least twice as fast, and significantly reduce the computational burden when SSH is used for `git push` (as is recommended), consider the "pushInsteadOf" global Git config.
To do this, when cloning a public repo (including ones you're a collaborator on), use `git clone https://`.
This global SSH push config one-time does SSH push for HTTPS-cloned repos:
```sh
git config --global url."ssh://github.com/".pushInsteadOf https://github.com/
```
The pattern matching can be made for all sites by omitting `github.com` from the command above, or you can refine it for each site, or even for specific usernames by editing the command above.
For private repos, simply clone with SSH as usual
### Preview all changed Jekyll files
This is for a website made using
[Jekyll](https://www.scivision.dev/create-jekyll-github-pages-website)
or
[Hugo](https://github.com/scivision/hugo-flex-example):
```sh
ActOnChanged . -p
```
It shows web page previews of all pages changed locally--start the Jekyll or Hugo debug server first e.g. `hugo serve`
Raw data
{
"_id": null,
"home_page": "",
"name": "gitutils",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.9",
"maintainer_email": "",
"keywords": "concurrent,asynchronous,asyncio,git,GitHub,ssh",
"author": "",
"author_email": "",
"download_url": "https://files.pythonhosted.org/packages/85/13/08407945a432aa54777696c1e8934b383a447f9e036334f9c2c8c7a4c11b/gitutils-1.12.1.tar.gz",
"platform": null,
"description": "# GitMC -- concurrent asynchronous Git Utilities for operations on massive numbers of Git repos\n\n[](https://doi.org/10.5281/zenodo.3339891)\n[](https://github.com/scivision/gitMC/actions/workflows/ci.yml)\n[](http://pepy.tech/project/gitutils)\n\nPlatform-independent (Linux/Mac/Windows) Git utilities, useful for managing large (100+) numbers of Git repos.\nSpeed is an emphasis throughout, with concurrency via Python stdlib\n[asyncio](https://docs.python.org/3/library/asyncio.html)\nvia\n[asyncio.create_subprocess_exec](https://docs.python.org/3/library/asyncio-subprocess.html#asyncio.create_subprocess_exec)\nand pipelining makes operations in effect 100x faster overall as the coroutines simultaneously wait for Git operations (particularly remote operations like \"fetch\" and \"pull\").\nWe have implemented individual concurrent subprocess timeout using\n[asyncio.wait_for](https://docs.python.org/3/library/asyncio-task.html#asyncio.wait_for)\nso that one Git operation hanging doesn't cause other Git operations to fail--this is good for when a Git login popup may go unnoticed by the human.\n\nGitMC uses command-line Git because PyGit also requires command-line Git installed, and we don't need the advanced functionality.\n\n---\n\nAlso see\n[PyGit-bulk](https://github.com/scivision/pygit-bulk)\nfor managing large (100+) numbers of users / teams.\n\nThis repo contains a\n[Git pre-commit script](./scripts/pre-commit)\nwith\n[explanation](https://www.scivision.dev/git-commit-precheck-pep8/).\n\n## Install\n\nInstall Git in a way accessible from the command line line\n\n* Mac: `brew install git`\n* Linux: `apt install git`\n* Windows: command line [Git](https://git-scm.com/download/win).\n\n```sh\npython -m pip install -e .\n```\n\n## Usage\n\n`gitbranch`\n: Tells of any non-master branches under directory ~/code\n\n`python -m gitutils.email`\n: list all contributor email addresses. To fix unwanted emails use [Git-filter-repo](https://www.scivision.dev/git-update-email-address/)\n\n`find_missing_file`\n: find directories missing exact fullpath to file\n\n`find_matching_file`\n: find directories matching exact fullpath to file\n\n### Sync large number of git repos\n\nThese assume numerous subdirectories under `~/code`.\nThey work very quickly for large numbers (100+) of repos.\n\n* `gitstat` check if any local repos have pending changes\n* `gitpull` Git pulls all repos (suggest `gitfetch` instead)\n* `gitfetch` Git fetches all repos, printing a summary of files changed on remote\n\nPlace an empty file `.nogit` in a subdirectory to skip it.\n\n#### [optional] speedup with https pull\n\nFor public repos, to make the Git remote checking go at least twice as fast, and significantly reduce the computational burden when SSH is used for `git push` (as is recommended), consider the \"pushInsteadOf\" global Git config.\nTo do this, when cloning a public repo (including ones you're a collaborator on), use `git clone https://`.\nThis global SSH push config one-time does SSH push for HTTPS-cloned repos:\n\n```sh\ngit config --global url.\"ssh://github.com/\".pushInsteadOf https://github.com/\n```\n\nThe pattern matching can be made for all sites by omitting `github.com` from the command above, or you can refine it for each site, or even for specific usernames by editing the command above.\nFor private repos, simply clone with SSH as usual\n\n### Preview all changed Jekyll files\n\nThis is for a website made using\n[Jekyll](https://www.scivision.dev/create-jekyll-github-pages-website)\nor\n[Hugo](https://github.com/scivision/hugo-flex-example):\n\n```sh\nActOnChanged . -p\n```\n\nIt shows web page previews of all pages changed locally--start the Jekyll or Hugo debug server first e.g. `hugo serve`\n",
"bugtrack_url": null,
"license": "",
"summary": "concurrent, pipelined, platform-agnostic Git utilities for managing a large number of Git repositories",
"version": "1.12.1",
"project_urls": null,
"split_keywords": [
"concurrent",
"asynchronous",
"asyncio",
"git",
"github",
"ssh"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "40e1c572fc4762fc1def4b7c3a09d48e8fb58810d0ec223c1698c1382a4bf5d6",
"md5": "7b53da21bca74b89fdccf669c7f974f2",
"sha256": "9fd61b9328535316cedd5bea6b4633205d558a5ec94edb6836ecf2c69ef83b30"
},
"downloads": -1,
"filename": "gitutils-1.12.1-py3-none-any.whl",
"has_sig": false,
"md5_digest": "7b53da21bca74b89fdccf669c7f974f2",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.9",
"size": 16844,
"upload_time": "2024-01-13T23:12:19",
"upload_time_iso_8601": "2024-01-13T23:12:19.459398Z",
"url": "https://files.pythonhosted.org/packages/40/e1/c572fc4762fc1def4b7c3a09d48e8fb58810d0ec223c1698c1382a4bf5d6/gitutils-1.12.1-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "851308407945a432aa54777696c1e8934b383a447f9e036334f9c2c8c7a4c11b",
"md5": "f4d168c928ed240da930503daa799ced",
"sha256": "413d1fc33d3405dae0eb4b08f83d7d0dae7cb5368abe88025a8336c380be0389"
},
"downloads": -1,
"filename": "gitutils-1.12.1.tar.gz",
"has_sig": false,
"md5_digest": "f4d168c928ed240da930503daa799ced",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.9",
"size": 13802,
"upload_time": "2024-01-13T23:12:21",
"upload_time_iso_8601": "2024-01-13T23:12:21.195512Z",
"url": "https://files.pythonhosted.org/packages/85/13/08407945a432aa54777696c1e8934b383a447f9e036334f9c2c8c7a4c11b/gitutils-1.12.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-01-13 23:12:21",
"github": false,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"lcname": "gitutils"
}
