# toolforge-cli
CLI to run toolforge related commands
## Local development environment (guideline)
### Requirements
You need to have [Poetry](https://github.com/python-poetry/poetry) installed globally. While you can install it with package managers such as `pip` or `homebrew`, it's highly recommended to use the official installer:
https://python-poetry.org/docs/#installing-with-the-official-installer
### Testing with tox on debian
Clone the repo (instructions here https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli).
Install tox (this is the only debian-specific part):
```
~:$ apt install tox
```
Move to the directory where you cloned the repo, and run tox:
```
/path/to/repo/toolforge-cli:$ tox
```
That will run the tests and create a virtualenv that you can use to manually debug anything you need, to enter it:
```
/path/to/repo/toolforge-cli:$ source .tox/py3-tests/bin/activate
```
## Building the debian packages
The process will be:
* Create new branch
* Bump the version
* Update the `debian/changelog` and `pyproject.toml` (done by `bump_version.sh`)
* Create a patch, get it reviewed and merged
* Create a tag (`debian/<new_version>`) and push
* Build the package (done by `build_deb.sh`)
* Upload the package to the toolforge repositories
Let's get started!
### Create new branch
To get started, create a new branch from main:
```
~:$ git checkout -b <new-branch-name>
```
### Update the changelog and pyproject.toml
To do so, you can run the script:
```
~:$ utils/bump_version.sh
```
That will:
* create an entry in `debian/changelog` from the git log since the last `debian/*` tag
* bump the version in `pyproject.toml` too
At this point, you should create a commit and send it for review, and continue once merged.
```
~:$ git commit -m "Bumped version to <new_version>" --signoff
~:$ git push origin <new-branch-name>
```
### Get the version bump commit merged
Review the `changelog` and the `pyproject.toml` changes to make sure it's what you want (it uses your name, email, etc.), and ask
for reviews.
### Create and upload the debian tag
Once merged, you can create a tag named `debian/<new_version>` locally and push it to the repository (ex. `git push origin debian/<new_version>`).
### Build the package
#### With containers
This is the recommended way of building the package, as it's agnostic of the OS/distro you are using.
It will not allow you to sign your package though, so if you need that try using the manual process.
Now you can build the package with:
```
~:$ utils/build_deb.sh
```
The first time it might take a bit more time as it will build the core image to build packages, downloading many
dependencies. The next run it will not need to download all those dependencies, so it will be way faster.
**NOTE**: If it failed when installing packages, try passing `--no-cache` to force rebuilding the cached layers.
#### wmcs-package-build script
An alternative is using the wmcs-package-build.py script that you can find in
the operations/puppet repo at modules/toolforge/files
```
$ ./wmcs-package-build.py --git-repo https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli -a buster-toolsbeta -a bullseye-toolsbeta --git-branch main --build-dist=bullseye
```
The script will SSH into a build server, build the package there, and publish it
to two repos: `buster-toolsbeta` and `bullseye-tooslbeta`.
The additional params `--backports, --toolforge-repo=tools
--build-dist=bullseye` are necessary because the build requires Poetry and other
build tools not available in the buster repos.
If that command is successful, you should then copy the package from the
"toolsbeta" to the "tools" distribution:
```
ssh tools-services-05.tools.eqiad1.wikimedia.cloud
$ sudo -i
# aptly repo copy buster-toolsbeta buster-tools toolforge-cli_VERSION_all
# aptly repo copy bullseye-toolsbeta bullseye-tools toolforge-cli_VERSION_all
# aptly publish --skip-signing update buster-tools
# aptly publish --skip-signing update bullseye-tools
```
Additional documentation on the wmcs-package-build script is available at
https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Packaging#wmcs-package-build
#### Manual process (only on debian)
For this you'll need debuild installed:
```
~:$ sudo apt install debuild
```
Install the build dependencies, this requires devscripts and equivs:
```
~:$ sudo apt install devscripts equivs
...
/path/to/repo/toolforge-cli:$ sudo mk-build-deps --install debian/control
```
Or just manually check the `debian/control` file `Build-Dependencies` and install them manually.
Note that it will build a debian package right there, and install it, you can remove it to clean up the dependencies any time.
Now for the actuall build:
```
/path/to/repo/toolforge-cli:$ debuild -uc -us
```
That will end up creating an unsigned package under `../toolforge-cli.*.deb`.
If you want to sign it, you will have to do something like:
```
/path/to/repo/toolforge-cli:$ debuild -kmy@key.org
```
### Uploading to the toolforge repository
Once you have built the package you want, you can uploade it following:
https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Packaging#Uploading_a_package
Raw data
{
"_id": null,
"home_page": "https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli",
"name": "toolforge-cli",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.7,<4.0",
"maintainer_email": "",
"keywords": "",
"author": "David Caro",
"author_email": "dcaro@wikimedia.org",
"download_url": "https://files.pythonhosted.org/packages/1b/fe/0f5e7566ad3e919b84839a8fae3ca342c196a8ca6844ba7b4d7c74e81799/toolforge_cli-0.3.4.tar.gz",
"platform": null,
"description": "# toolforge-cli\n\nCLI to run toolforge related commands\n\n## Local development environment (guideline)\n\n### Requirements\n\nYou need to have [Poetry](https://github.com/python-poetry/poetry) installed globally. While you can install it with package managers such as `pip` or `homebrew`, it's highly recommended to use the official installer:\nhttps://python-poetry.org/docs/#installing-with-the-official-installer\n\n### Testing with tox on debian\n\nClone the repo (instructions here https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli).\n\nInstall tox (this is the only debian-specific part):\n```\n~:$ apt install tox\n```\n\nMove to the directory where you cloned the repo, and run tox:\n```\n/path/to/repo/toolforge-cli:$ tox\n```\n\nThat will run the tests and create a virtualenv that you can use to manually debug anything you need, to enter it:\n```\n/path/to/repo/toolforge-cli:$ source .tox/py3-tests/bin/activate\n```\n\n## Building the debian packages\n\nThe process will be:\n* Create new branch\n* Bump the version\n * Update the `debian/changelog` and `pyproject.toml` (done by `bump_version.sh`)\n * Create a patch, get it reviewed and merged\n* Create a tag (`debian/<new_version>`) and push\n* Build the package (done by `build_deb.sh`)\n* Upload the package to the toolforge repositories\n\nLet's get started!\n\n### Create new branch\nTo get started, create a new branch from main:\n```\n~:$ git checkout -b <new-branch-name>\n```\n\n### Update the changelog and pyproject.toml\nTo do so, you can run the script:\n```\n~:$ utils/bump_version.sh\n```\n\nThat will:\n\n* create an entry in `debian/changelog` from the git log since the last `debian/*` tag\n* bump the version in `pyproject.toml` too\n\nAt this point, you should create a commit and send it for review, and continue once merged.\n\n```\n~:$ git commit -m \"Bumped version to <new_version>\" --signoff\n~:$ git push origin <new-branch-name>\n```\n\n### Get the version bump commit merged\n\nReview the `changelog` and the `pyproject.toml` changes to make sure it's what you want (it uses your name, email, etc.), and ask\nfor reviews.\n\n### Create and upload the debian tag\n\nOnce merged, you can create a tag named `debian/<new_version>` locally and push it to the repository (ex. `git push origin debian/<new_version>`).\n\n### Build the package\n#### With containers\nThis is the recommended way of building the package, as it's agnostic of the OS/distro you are using.\n\nIt will not allow you to sign your package though, so if you need that try using the manual process.\n\nNow you can build the package with:\n\n```\n~:$ utils/build_deb.sh\n```\n\nThe first time it might take a bit more time as it will build the core image to build packages, downloading many\ndependencies. The next run it will not need to download all those dependencies, so it will be way faster.\n\n**NOTE**: If it failed when installing packages, try passing `--no-cache` to force rebuilding the cached layers.\n\n#### wmcs-package-build script\nAn alternative is using the wmcs-package-build.py script that you can find in\nthe operations/puppet repo at modules/toolforge/files\n\n```\n$ ./wmcs-package-build.py --git-repo https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli -a buster-toolsbeta -a bullseye-toolsbeta --git-branch main --build-dist=bullseye\n```\n\nThe script will SSH into a build server, build the package there, and publish it\nto two repos: `buster-toolsbeta` and `bullseye-tooslbeta`.\n\nThe additional params `--backports, --toolforge-repo=tools\n--build-dist=bullseye` are necessary because the build requires Poetry and other\nbuild tools not available in the buster repos.\n\nIf that command is successful, you should then copy the package from the\n\"toolsbeta\" to the \"tools\" distribution:\n\n```\nssh tools-services-05.tools.eqiad1.wikimedia.cloud\n$ sudo -i\n# aptly repo copy buster-toolsbeta buster-tools toolforge-cli_VERSION_all\n# aptly repo copy bullseye-toolsbeta bullseye-tools toolforge-cli_VERSION_all\n# aptly publish --skip-signing update buster-tools\n# aptly publish --skip-signing update bullseye-tools\n```\n\nAdditional documentation on the wmcs-package-build script is available at\nhttps://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Packaging#wmcs-package-build\n\n#### Manual process (only on debian)\nFor this you'll need debuild installed:\n```\n~:$ sudo apt install debuild\n```\n\nInstall the build dependencies, this requires devscripts and equivs:\n```\n~:$ sudo apt install devscripts equivs\n...\n/path/to/repo/toolforge-cli:$ sudo mk-build-deps --install debian/control\n```\n\nOr just manually check the `debian/control` file `Build-Dependencies` and install them manually.\n\nNote that it will build a debian package right there, and install it, you can remove it to clean up the dependencies any time.\n\n\nNow for the actuall build:\n```\n/path/to/repo/toolforge-cli:$ debuild -uc -us\n```\n\nThat will end up creating an unsigned package under `../toolforge-cli.*.deb`.\nIf you want to sign it, you will have to do something like:\n```\n/path/to/repo/toolforge-cli:$ debuild -kmy@key.org\n```\n\n### Uploading to the toolforge repository\n\nOnce you have built the package you want, you can uploade it following:\nhttps://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Packaging#Uploading_a_package\n\n",
"bugtrack_url": null,
"license": "GPL-3.0-or-later",
"summary": "Toolforge client",
"version": "0.3.4",
"project_urls": {
"Documentation": "https://wikitech.wikimedia.org/wiki/Help:Toolforge",
"Homepage": "https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "223128bc91ce43a8173f98402163f3e25b2bd3b4a25dda5fe0fb02f02292ef4f",
"md5": "36dfe0e6b7a8fbd2cce024cba8070752",
"sha256": "2f9e3ad6f555c815a17170bed1d9742e1a4111970cb409d0a19bb540e0b85b1b"
},
"downloads": -1,
"filename": "toolforge_cli-0.3.4-py3-none-any.whl",
"has_sig": false,
"md5_digest": "36dfe0e6b7a8fbd2cce024cba8070752",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.7,<4.0",
"size": 17996,
"upload_time": "2023-10-02T07:54:31",
"upload_time_iso_8601": "2023-10-02T07:54:31.729694Z",
"url": "https://files.pythonhosted.org/packages/22/31/28bc91ce43a8173f98402163f3e25b2bd3b4a25dda5fe0fb02f02292ef4f/toolforge_cli-0.3.4-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "1bfe0f5e7566ad3e919b84839a8fae3ca342c196a8ca6844ba7b4d7c74e81799",
"md5": "e963deed02ff2675e88a2ed6702d71dd",
"sha256": "f99391d3851cfee17b7f71c47d1384201c00053ef3d21af176cf61254604be7a"
},
"downloads": -1,
"filename": "toolforge_cli-0.3.4.tar.gz",
"has_sig": false,
"md5_digest": "e963deed02ff2675e88a2ed6702d71dd",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.7,<4.0",
"size": 17237,
"upload_time": "2023-10-02T07:54:33",
"upload_time_iso_8601": "2023-10-02T07:54:33.435940Z",
"url": "https://files.pythonhosted.org/packages/1b/fe/0f5e7566ad3e919b84839a8fae3ca342c196a8ca6844ba7b4d7c74e81799/toolforge_cli-0.3.4.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-10-02 07:54:33",
"github": false,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"lcname": "toolforge-cli"
}
JSON