# HGitaly
HGitaly is Gitaly server for Mercurial.
It implements the subset of the Gitaly gRPC protocol that is relevant for
Mercurial repositories, as well as its own HGitaly protocol, with methods
that are specific to Mercurial.
It comes in two overlapping variants:
- HGitaly proper is written in Python, using the `grpcio` official library.
- RHGitaly is a high-performance partial implementation written in Rust, and
based on the [`tonic`](https://crates.io/crates/tonic) gRPC framework.
As of this writing, RHGitaly implements a strict subset of the methods
implemented in HGitaly, but it is possible that some methods would be
implemented in RHGitaly only in the future.
## Installation
### HGitaly (Python)
In what follows, `$PYTHON` is often the Python interpreter in a virtualenv,
but it can be a system-wide one (typical case in containers, strongly
discouraged on user systemes).
1. Install Mercurial with Rust parts (for the exact version, refer to the
requirements file in the Heptapod main repository sources)
```
$PYTHON -m pip install --no-use-pep517 --global-option --rust Mercurial==6.6.2
```
2. Install HGitaly itself (check that it does not reinstall Mercurial)
```
$PYTHON -m pip install hgitaly
```
### RHGitaly
We distribute a self-contained source tarball. It includes the appropriate
`hg-core` Rust sources.
1. Fetch the tarball
```
wget https://download.heptapod.net/rhgitaly/rhgitaly-x.y.z.tgz
```
2. Fetch and verify the GPG signature
```
wget https://download.heptapod.net/rhgitaly/rhgitaly-x.y.z.tgz.asc
gpg --verify rhgitaly-x.y.z.tgz.asc
```
3. Build
```
tar xzf rhgitaly-x.y.z.tgz
cd rhgitaly-x.y.z/rust
cargo build --locked --release
```
4. Install wherever you want. Example given for a system-wide installation
```
sudo install -o root -g root target/release/rhgitaly /usr/local/bin
```
5. Define a service. Example given for systemd, to be adjusted for your needs.
Make sure in particular that user and all directories exist, with
appropriate permissions.
```
[Unit]
Description=Heptapod RHGitaly Server
[Service]
User=hgitaly
# HGRCPATH not needed yet but probably will be at some point
Environment=HGRCPATH=/etc/heptapod/heptapod.hgrc
Environment=RHGITALY_LISTEN_URL=unix:/run/heptapod/rhgitaly.socket
Environment=RHGITALY_REPOSITORIES_ROOT=/home/hg/repositories
ExecStartPre=rm -f /run/heptapod/rhgitaly.socket
ExecStart=/user/local/bin/rhgitaly
Restart=on-failure
[Install]
WantedBy=default.target
```
### External executables
HGitaly needs several other programs to be installed and will run them
as separate processes.
By default, it expects to find them on `$PATH`, but the actual path to
each executable can be configured.
#### Tokei
[Tokei](https://crates.io/crates/tokei) is a programming languages analysis
tool written in Rust. It is used by the [CommitLanguages](protos/commit.proto)
method.
Tokei is available in several Linux distributions.
As of this writing, HGitaly supports versions 12.0 and 12.1
#### Go license-detector
Usually installed as `license-detector`, this standalone executable is
part of the `go-enry` suite. Its library version is also used by Gitaly.
It is used in the [FindLicense](protos/repository.proto) method.
#### Git
HGitaly can make use of some Git commands that do not involve repositories!
This is for example the case of [GetPatchID](protos/diff.proto): the
`git patch-id` command does not access any repository. Instead it computes any
patch into an identifier.
#### Mercurial
In forthcoming versions, it is probable that HGitaly and/or RHGitaly will
invoke Mercurial subprocesses.
This is not yet the case as of this writing (HGitaly 1.1 / Heptapod 1.1).
### Configuration
HGitaly's configuration is done the standard way in the Mercurial world:
through HGRC files.
In a typical Heptapod installation, these are split into a managed file, for
consistency with other components and another one for edit by the systems
administrator (`/etc/gitlab/heptapod.hgrc` in Omnibus/Docker instances).
Many Mercurial tweaks are interpreted simply because HGitaly internally
calls into Mercurial, but HGitaly also gets its own section. Here are the
settings available as of HGitaly 1.1
```
[hgitaly]
# paths to external executables
tokei-executable = tokei
license-detector-executable = license-detector
git-executable = git
# The number of workers process default value is one plus half the CPU count.
# It can be explicitly set this way:
#workers = 4
# Time to let a worker finish treating its current request, if any, when
# gracefully restarted. Default is high because of backup requests.
worker.graceful-shutdown-timeout-seconds = 300
# Maximum allowed resident size for worker processes (MiB).
# They get gracefully restarted if they cross that threshold
worker.max_rss_mib = 1024
# Interval between memory monitoring of workers (results dumped in logs)
worker.monitoring-interval-seconds = 60
```
Also `heptapod.repositories-root` is used if `--repositories-root` is
not passed on the command line.
## Operation
### Logging
HGitaly is using the standard `logging` Python module, and the
`loggingmod` Mercurial extension to emit logs from the Mercurial core
and other extensions. Therefore, the logging configuration is done
from the Mercurial configuration, typically from one of the Heptapod
HGRC files.
The general convention is that all logs emitted by `hgitaly.service`
provide GitLab's `correlation_id` in the `extra` dict, making it
available in the format string. Here is a minimal example:
```
[correlation_id=%(correlation_id)s] [%(levelname)s] [%(name)s] %(message)s"
```
Conversely, the format strings for logs emitted outside of
`hgitaly.service` must not use `correlation_id`, as subpackages such as
`hgitaly.branch`, `hgitaly.message`, etc. cannnot provide a
value: it is a hard error to use a format that relies on some
extra if the emitter does not provide it.
To summarize the resulting policy:
- in `hgitaly.service`, all logging must be done through
`hgitaly.logging.LoggerAdapter`. Using `correlation_id` in the
format is strongly encouraged.
- outside of `hgitaly.service`, logging should be self-contained
useful without an obvious link to the calling gRPC method. For
instance a repository inconsistency should be logged at `WARNING`
level, with a message including the path.
## Development
### Automated tests and Continuous Integration
#### How to run the tests
Usually, that would be in a virtualenv, but it's not necessary.
```
python3 -m pip install -r test-requirements.txt
./run-all-tests
```
Hint: Check the contents of `run-all-tests`, it's just `pytest` with
a standard set of options (mostly for coverage, see below).
#### Unit and Mercurial integration tests
These are the main tests. They lie inside the `hgitaly`
and `hgext3rd.hgitaly` Python packages. The layout follows the style where
each subpackage has its own tests package, to facilitate future refactorings.
The Mercurial integration tests are written with the [mercurial-testhelpers]
library. Their duty is to assert that HGitaly works as expected and maintains
compatibility with several versions of Mercurial and possibly other
dependencies, such as [grpcio].
The implicit assumption with these tests is that the test authors actually
knew what was expected. HGitaly being meant to be a direct replacement, or
rather a translation of Gitaly in Mercurial terms, those expectation are
actually a mix of:
- Design choices, such as mapping rules between branch/topic combinations
and GitLab branches.
- Gitaly documentation and source code.
- sampling of Gitaly responses.
#### Gitaly comparison tests
If an appropriate Gitaly installation is found, `run-all-tests` will also
run the tests from the `tests_with_gitaly` package. This happens automatically
from within a [HDK] workspace.
These are precisely meant for what the Mercurial integration tests can't do:
check that HGitaly responses take the form expected by the various Gitaly
clients, by comparing directly with the reference Gitaly implementation.
The comparisons work by using the conversions to Git provided by
`py-heptapod`, which are precisely what HGitaly aims to replace as a mean
to expose Mercurial content to GitLab.
Once there is no ambiguity with what Gitaly clients expect, the correctness
of the implementation, with its various corner cases,
should be left to the Mercurial integration tests.
#### Test coverage
This project is being developed with a strong test coverage policy, enforced by
CI: without the Gitaly comparison tests, the coverage has to stay at 100%.
This does not mean that a contribution has to meet this goal to be worthwile,
or even considered. Contributors can expect Maintainers to help them
achieving the required 100% coverage mark, especially if they are newcomers.
Of course, Contributors cannot expect Maintainers to go
as far as write missing tests for them, even if that can still happen
for critical urgent issues.
Selected statements can of course be excluded for good reasons, using
`# pragma no cover`.
Coverage exclusions depending on the Mercurial version are
provided by the coverage plugin of [mercurial-testhelpers].
Unexpected drop of coverage in different Mercurial versions is a powerful
warning system that something not obvious is getting wrong, but the
Gitaly comparison tests are run in CI against a fixed set of
dependencies, hence 100% coverage must be achieved without the Gitaly
comparison tests.
On the other hand, Gitaly comparison tests will warn us when we bump upstream
GitLab if some critical behaviour has changed.
#### Tests Q&A and development hints
##### Doesn't the 100% coverage rule without the Gitaly comparison tests mean writing the same tests twice?
In some cases, yes, but it's limited.
For example, the comparison tests
can tell us that the `FindAllBranchNames` is actually expected to return
GitLab refs (`refs/heads/some-branch`), not GitLab branch names. That can
be settled with a few, very basic, test cases. There is no need to test all
the mapping rules for topics, and even less the various related corner cases
in the comparison tests. These, on the other hand depend strongly on Mercurial
internals, and absolutely have to be fully tested continuously against various
Mercurial versions.
Also, it is possible to deduplicate scenarios that are almost identical in
Mercurial integration tests and Gitaly comparison tests: factorize out the
common code in a helper function made available for both. The question is if
it is worth the effort.
Finally, comparison tests should focus on the fact that Gitaly and HGitaly
results agree, not on what they contain. In the above example,
a comparison for `FindAllBranchNames` could simply assert equality of the
returned sets of branch names. This is a bit less cumbersome, and easier
to maintain.
#### How to reproduce a drop in coverage found by the `compat` CI stage?
These are often due to statements being covered by the Gitaly comparison
tests only, leading to 100% coverage in the `main` stage, but not in the
`compat` stage.
The first thing to do is to run without the Gitaly comparison tests:
```
SKIP_GITALY_COMPARISON_TESTS=yes ./run-all-tests
```
(any non empty value in that environment variable, even `no` or `false` will
trigger the skipping)
In some rare cases, the drop in coverage could be due to an actual change
between Mercurial versions. If that happens, there are good chances that an
actual bug is lurking around.
#### How to run the tests with coverage of the Gitaly comparison tests
```
./run-all-tests --cov tests_with_gitaly --cov-report html
```
The HTML report will be nice if you don't have 100% coverage. To display it,
just do
```
firefox htmlcov/index.html
```
By default, the Gitaly comparison tests themselves are not covered, indeed.
This is because `run-all-tests` does not know whether they will be skipped for
lack of a Gitaly installation – which would be legitimate.
But they *are* covered in the CI jobs that launch them, because Gitaly is
assumed to be available. For these, the coverage would tell us that something
was broken, preventing the tests to run.
#### How to poke into Gitaly protocol?
The Gitaly comparison tests provide exactly a harness for that: take a test,
modify it as needed, insert a `pdb` breakpoint, and get going.
The big advantage here is that startup of the Gitaly comparison tests is
almost instantaneous, especially compared with RSpec, wich takes about a minute
to start even a completely trivial test.
Of course that will raise the question whether it'll be useful to make true
tests of these experiments.
#### When is a Gitaly comparison test required?
Each time there's a need to be sure of what's expected and it can help answer
that question. It doesn't have to do more than that.
#### When to prefer writing RSpec tests in Heptapod Rails over Gitaly comparison tests in HGitaly?
If you need to make sure that Heptapod Rails, as a Gitaly client, sends
the proper requests, because that can depend on specific dispatch code.
For instance, we are currently still converting to Git on the Rails side.
A source of bugs would be to send Git commit ids to HGitaly.
Apart from that, it is expected to be vastly more efficient to use
Gitaly comparison tests.
The more Heptapod progresses, the less complicated all of this should be.
## Updating the Gitaly gRPC protocol
The virtualenv has to be activated
1. `pip install -r dev-requirements.txt`
2. Copy the new `proto` files from a Gitaly checkout with
version matching the wanted GitLab upstream version.
Example in a HDK context:
```
cp ../gitaly/proto/*.proto protos/ # we dont want the `go` subdir
```
3. run `./generate-stubs`
4. run the tests: `./run-all-tests`
5. perform necessary `hg add` after close inspection of `hg status`
### Updating the HGitaly specific gRPC protocol
This package defines and implements an additional gRPC protocol, with
gRPC services and methods that are specific to Mercurial, or more generally
Heptapod.
#### Protocol specification
The sources are `proto` files in the `protos/` directory, same as for the
Gitaly protocol.
They distinguish themselves by this declaration:
```
package hgitaly;
```
Each time a change is made to the protocol, the libraries for all
provided programming languages have to be regenerated and committed, ideally
together with the protocol change.
#### Python library
It has a special status, being versioned together with the protocol and the
server implementation. It is provided as the [hgitaly.stub](hgitaly/stub)
package.
The Python stubs are produced by the same script that takes care of Gitaly
`proto` files:
```
./generate-stubs
```
#### Ruby library
See [the separate documentation](ruby/README.md)
#### Other languages
A Go library will probably be necessary quite soon for Workhorse or perhaps
Heptapod Shell.
A Rust library would be nice to have
[mercurial-testhelpers]: https://pypi.org/project/mercurial-testhelpers/
[grpcio]: https://pypi.org/project/grpcio/
[HDK]: https://foss.heptapod.net/heptapod/heptapod-development-kit
Raw data
{
"_id": null,
"home_page": "https://foss.heptapod.net/heptapod/hgitaly",
"name": "hgitaly",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": null,
"keywords": "hg mercurial heptapod gitlab",
"author": "Georges Racinet",
"author_email": "georges.racinet@octobus.net",
"download_url": "https://files.pythonhosted.org/packages/99/78/8feec022ac38deee6e6265b2217afdd4192c80e26e4fcf73e032f7300533/hgitaly-1.4.0.tar.gz",
"platform": null,
"description": "# HGitaly\n\nHGitaly is Gitaly server for Mercurial.\n\nIt implements the subset of the Gitaly gRPC protocol that is relevant for\nMercurial repositories, as well as its own HGitaly protocol, with methods\nthat are specific to Mercurial.\n\nIt comes in two overlapping variants:\n\n- HGitaly proper is written in Python, using the `grpcio` official library.\n- RHGitaly is a high-performance partial implementation written in Rust, and\n based on the [`tonic`](https://crates.io/crates/tonic) gRPC framework.\n\n As of this writing, RHGitaly implements a strict subset of the methods\n implemented in HGitaly, but it is possible that some methods would be\n implemented in RHGitaly only in the future.\n\n## Installation\n\n### HGitaly (Python)\n\nIn what follows, `$PYTHON` is often the Python interpreter in a virtualenv,\nbut it can be a system-wide one (typical case in containers, strongly\ndiscouraged on user systemes).\n\n1. Install Mercurial with Rust parts (for the exact version, refer to the\n requirements file in the Heptapod main repository sources)\n\n ```\n $PYTHON -m pip install --no-use-pep517 --global-option --rust Mercurial==6.6.2\n ```\n\n2. Install HGitaly itself (check that it does not reinstall Mercurial)\n\n ```\n $PYTHON -m pip install hgitaly\n ```\n\n### RHGitaly\n\nWe distribute a self-contained source tarball. It includes the appropriate\n`hg-core` Rust sources.\n\n1. Fetch the tarball\n\n ```\n wget https://download.heptapod.net/rhgitaly/rhgitaly-x.y.z.tgz\n ```\n\n2. Fetch and verify the GPG signature\n\n ```\n wget https://download.heptapod.net/rhgitaly/rhgitaly-x.y.z.tgz.asc\n gpg --verify rhgitaly-x.y.z.tgz.asc\n ```\n\n3. Build\n\n ```\n tar xzf rhgitaly-x.y.z.tgz\n cd rhgitaly-x.y.z/rust\n cargo build --locked --release\n ```\n\n4. Install wherever you want. Example given for a system-wide installation\n\n ```\n sudo install -o root -g root target/release/rhgitaly /usr/local/bin\n ```\n\n5. Define a service. Example given for systemd, to be adjusted for your needs.\n Make sure in particular that user and all directories exist, with\n appropriate permissions.\n\n ```\n [Unit]\n Description=Heptapod RHGitaly Server\n\n [Service]\n User=hgitaly\n # HGRCPATH not needed yet but probably will be at some point\n Environment=HGRCPATH=/etc/heptapod/heptapod.hgrc\n Environment=RHGITALY_LISTEN_URL=unix:/run/heptapod/rhgitaly.socket\n Environment=RHGITALY_REPOSITORIES_ROOT=/home/hg/repositories\n ExecStartPre=rm -f /run/heptapod/rhgitaly.socket\n ExecStart=/user/local/bin/rhgitaly\n Restart=on-failure\n\n [Install]\n WantedBy=default.target\n ```\n\n### External executables\n\nHGitaly needs several other programs to be installed and will run them\nas separate processes.\n\nBy default, it expects to find them on `$PATH`, but the actual path to\neach executable can be configured.\n\n#### Tokei\n\n[Tokei](https://crates.io/crates/tokei) is a programming languages analysis\ntool written in Rust. It is used by the [CommitLanguages](protos/commit.proto)\nmethod.\n\nTokei is available in several Linux distributions.\n\nAs of this writing, HGitaly supports versions 12.0 and 12.1\n\n#### Go license-detector\n\nUsually installed as `license-detector`, this standalone executable is\npart of the `go-enry` suite. Its library version is also used by Gitaly.\n\nIt is used in the [FindLicense](protos/repository.proto) method.\n\n#### Git\n\nHGitaly can make use of some Git commands that do not involve repositories!\nThis is for example the case of [GetPatchID](protos/diff.proto): the\n`git patch-id` command does not access any repository. Instead it computes any\npatch into an identifier.\n\n#### Mercurial\n\nIn forthcoming versions, it is probable that HGitaly and/or RHGitaly will\ninvoke Mercurial subprocesses.\n\nThis is not yet the case as of this writing (HGitaly 1.1 / Heptapod 1.1).\n\n### Configuration\n\nHGitaly's configuration is done the standard way in the Mercurial world:\nthrough HGRC files.\n\nIn a typical Heptapod installation, these are split into a managed file, for\nconsistency with other components and another one for edit by the systems\nadministrator (`/etc/gitlab/heptapod.hgrc` in Omnibus/Docker instances).\n\nMany Mercurial tweaks are interpreted simply because HGitaly internally\ncalls into Mercurial, but HGitaly also gets its own section. Here are the\nsettings available as of HGitaly 1.1\n\n```\n[hgitaly]\n# paths to external executables\ntokei-executable = tokei\nlicense-detector-executable = license-detector\ngit-executable = git\n\n# The number of workers process default value is one plus half the CPU count.\n# It can be explicitly set this way:\n#workers = 4\n\n# Time to let a worker finish treating its current request, if any, when\n# gracefully restarted. Default is high because of backup requests.\nworker.graceful-shutdown-timeout-seconds = 300\n# Maximum allowed resident size for worker processes (MiB).\n# They get gracefully restarted if they cross that threshold\nworker.max_rss_mib = 1024\n# Interval between memory monitoring of workers (results dumped in logs)\nworker.monitoring-interval-seconds = 60\n```\n\nAlso `heptapod.repositories-root` is used if `--repositories-root` is\nnot passed on the command line.\n\n## Operation\n\n### Logging\n\nHGitaly is using the standard `logging` Python module, and the\n`loggingmod` Mercurial extension to emit logs from the Mercurial core\nand other extensions. Therefore, the logging configuration is done\nfrom the Mercurial configuration, typically from one of the Heptapod\nHGRC files.\n\nThe general convention is that all logs emitted by `hgitaly.service`\nprovide GitLab's `correlation_id` in the `extra` dict, making it\navailable in the format string. Here is a minimal example:\n\n```\n[correlation_id=%(correlation_id)s] [%(levelname)s] [%(name)s] %(message)s\"\n```\n\nConversely, the format strings for logs emitted outside of\n`hgitaly.service` must not use `correlation_id`, as subpackages such as\n`hgitaly.branch`, `hgitaly.message`, etc. cannnot provide a\nvalue: it is a hard error to use a format that relies on some\nextra if the emitter does not provide it.\n\nTo summarize the resulting policy:\n\n- in `hgitaly.service`, all logging must be done through\n `hgitaly.logging.LoggerAdapter`. Using `correlation_id` in the\n format is strongly encouraged.\n- outside of `hgitaly.service`, logging should be self-contained\n useful without an obvious link to the calling gRPC method. For\n instance a repository inconsistency should be logged at `WARNING`\n level, with a message including the path.\n\n## Development\n\n### Automated tests and Continuous Integration\n\n#### How to run the tests\n\nUsually, that would be in a virtualenv, but it's not necessary.\n\n```\n python3 -m pip install -r test-requirements.txt\n ./run-all-tests\n```\n\nHint: Check the contents of `run-all-tests`, it's just `pytest` with\na standard set of options (mostly for coverage, see below).\n\n#### Unit and Mercurial integration tests\n\nThese are the main tests. They lie inside the `hgitaly`\nand `hgext3rd.hgitaly` Python packages. The layout follows the style where\neach subpackage has its own tests package, to facilitate future refactorings.\n\nThe Mercurial integration tests are written with the [mercurial-testhelpers]\nlibrary. Their duty is to assert that HGitaly works as expected and maintains\ncompatibility with several versions of Mercurial and possibly other\ndependencies, such as [grpcio].\n\nThe implicit assumption with these tests is that the test authors actually\nknew what was expected. HGitaly being meant to be a direct replacement, or\nrather a translation of Gitaly in Mercurial terms, those expectation are\nactually a mix of:\n\n- Design choices, such as mapping rules between branch/topic combinations\n and GitLab branches.\n- Gitaly documentation and source code.\n- sampling of Gitaly responses.\n\n#### Gitaly comparison tests\n\nIf an appropriate Gitaly installation is found, `run-all-tests` will also\nrun the tests from the `tests_with_gitaly` package. This happens automatically\nfrom within a [HDK] workspace.\n\nThese are precisely meant for what the Mercurial integration tests can't do:\ncheck that HGitaly responses take the form expected by the various Gitaly\nclients, by comparing directly with the reference Gitaly implementation.\n\nThe comparisons work by using the conversions to Git provided by\n`py-heptapod`, which are precisely what HGitaly aims to replace as a mean\nto expose Mercurial content to GitLab.\n\nOnce there is no ambiguity with what Gitaly clients expect, the correctness\nof the implementation, with its various corner cases,\nshould be left to the Mercurial integration tests.\n\n#### Test coverage\n\nThis project is being developed with a strong test coverage policy, enforced by\nCI: without the Gitaly comparison tests, the coverage has to stay at 100%.\n\nThis does not mean that a contribution has to meet this goal to be worthwile,\nor even considered. Contributors can expect Maintainers to help them\nachieving the required 100% coverage mark, especially if they are newcomers.\nOf course, Contributors cannot expect Maintainers to go\nas far as write missing tests for them, even if that can still happen\nfor critical urgent issues.\n\nSelected statements can of course be excluded for good reasons, using\n`# pragma no cover`.\n\nCoverage exclusions depending on the Mercurial version are\nprovided by the coverage plugin of [mercurial-testhelpers].\n\nUnexpected drop of coverage in different Mercurial versions is a powerful\nwarning system that something not obvious is getting wrong, but the\nGitaly comparison tests are run in CI against a fixed set of\ndependencies, hence 100% coverage must be achieved without the Gitaly\ncomparison tests.\n\nOn the other hand, Gitaly comparison tests will warn us when we bump upstream\nGitLab if some critical behaviour has changed.\n\n#### Tests Q&A and development hints\n\n##### Doesn't the 100% coverage rule without the Gitaly comparison tests mean writing the same tests twice?\n\nIn some cases, yes, but it's limited.\n\nFor example, the comparison tests\ncan tell us that the `FindAllBranchNames` is actually expected to return\nGitLab refs (`refs/heads/some-branch`), not GitLab branch names. That can\nbe settled with a few, very basic, test cases. There is no need to test all\nthe mapping rules for topics, and even less the various related corner cases\nin the comparison tests. These, on the other hand depend strongly on Mercurial\ninternals, and absolutely have to be fully tested continuously against various\nMercurial versions.\n\nAlso, it is possible to deduplicate scenarios that are almost identical in\nMercurial integration tests and Gitaly comparison tests: factorize out the\ncommon code in a helper function made available for both. The question is if\nit is worth the effort.\n\nFinally, comparison tests should focus on the fact that Gitaly and HGitaly\nresults agree, not on what they contain. In the above example,\na comparison for `FindAllBranchNames` could simply assert equality of the\nreturned sets of branch names. This is a bit less cumbersome, and easier\nto maintain.\n\n#### How to reproduce a drop in coverage found by the `compat` CI stage?\n\nThese are often due to statements being covered by the Gitaly comparison\ntests only, leading to 100% coverage in the `main` stage, but not in the\n`compat` stage.\n\nThe first thing to do is to run without the Gitaly comparison tests:\n\n```\nSKIP_GITALY_COMPARISON_TESTS=yes ./run-all-tests\n```\n\n(any non empty value in that environment variable, even `no` or `false` will\ntrigger the skipping)\n\nIn some rare cases, the drop in coverage could be due to an actual change\nbetween Mercurial versions. If that happens, there are good chances that an\nactual bug is lurking around.\n\n#### How to run the tests with coverage of the Gitaly comparison tests\n\n```\n./run-all-tests --cov tests_with_gitaly --cov-report html\n```\n\nThe HTML report will be nice if you don't have 100% coverage. To display it,\njust do\n\n```\nfirefox htmlcov/index.html\n```\n\nBy default, the Gitaly comparison tests themselves are not covered, indeed.\nThis is because `run-all-tests` does not know whether they will be skipped for\nlack of a Gitaly installation \u2013 which would be legitimate.\n\nBut they *are* covered in the CI jobs that launch them, because Gitaly is\nassumed to be available. For these, the coverage would tell us that something\nwas broken, preventing the tests to run.\n\n#### How to poke into Gitaly protocol?\n\nThe Gitaly comparison tests provide exactly a harness for that: take a test,\nmodify it as needed, insert a `pdb` breakpoint, and get going.\n\nThe big advantage here is that startup of the Gitaly comparison tests is\nalmost instantaneous, especially compared with RSpec, wich takes about a minute\nto start even a completely trivial test.\n\nOf course that will raise the question whether it'll be useful to make true\ntests of these experiments.\n\n#### When is a Gitaly comparison test required?\n\nEach time there's a need to be sure of what's expected and it can help answer\nthat question. It doesn't have to do more than that.\n\n#### When to prefer writing RSpec tests in Heptapod Rails over Gitaly comparison tests in HGitaly?\n\nIf you need to make sure that Heptapod Rails, as a Gitaly client, sends\nthe proper requests, because that can depend on specific dispatch code.\n\nFor instance, we are currently still converting to Git on the Rails side.\nA source of bugs would be to send Git commit ids to HGitaly.\n\nApart from that, it is expected to be vastly more efficient to use\nGitaly comparison tests.\n\nThe more Heptapod progresses, the less complicated all of this should be.\n\n## Updating the Gitaly gRPC protocol\n\nThe virtualenv has to be activated\n\n1. `pip install -r dev-requirements.txt`\n\n2. Copy the new `proto` files from a Gitaly checkout with\n version matching the wanted GitLab upstream version.\n Example in a HDK context:\n\n ```\n cp ../gitaly/proto/*.proto protos/ # we dont want the `go` subdir\n ```\n\n3. run `./generate-stubs`\n\n4. run the tests: `./run-all-tests`\n\n5. perform necessary `hg add` after close inspection of `hg status`\n\n### Updating the HGitaly specific gRPC protocol\n\nThis package defines and implements an additional gRPC protocol, with\ngRPC services and methods that are specific to Mercurial, or more generally\nHeptapod.\n\n#### Protocol specification\n\nThe sources are `proto` files in the `protos/` directory, same as for the\nGitaly protocol.\n\nThey distinguish themselves by this declaration:\n\n```\npackage hgitaly;\n```\n\nEach time a change is made to the protocol, the libraries for all\nprovided programming languages have to be regenerated and committed, ideally\ntogether with the protocol change.\n\n#### Python library\n\nIt has a special status, being versioned together with the protocol and the\nserver implementation. It is provided as the [hgitaly.stub](hgitaly/stub)\npackage.\n\nThe Python stubs are produced by the same script that takes care of Gitaly\n`proto` files:\n\n```\n./generate-stubs\n```\n\n#### Ruby library\n\nSee [the separate documentation](ruby/README.md)\n\n#### Other languages\n\nA Go library will probably be necessary quite soon for Workhorse or perhaps\nHeptapod Shell.\n\nA Rust library would be nice to have\n\n[mercurial-testhelpers]: https://pypi.org/project/mercurial-testhelpers/\n[grpcio]: https://pypi.org/project/grpcio/\n[HDK]: https://foss.heptapod.net/heptapod/heptapod-development-kit\n\n\n",
"bugtrack_url": null,
"license": "GPLv2+",
"summary": "Server-side implementation of Gitaly protocol for Mercurial",
"version": "1.4.0",
"project_urls": {
"Homepage": "https://foss.heptapod.net/heptapod/hgitaly"
},
"split_keywords": [
"hg",
"mercurial",
"heptapod",
"gitlab"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "99788feec022ac38deee6e6265b2217afdd4192c80e26e4fcf73e032f7300533",
"md5": "b0ff60242c9f35d8142b85cf1d397c8b",
"sha256": "efb75e85dc8bdebaa463d63e85b830970bb1625c7cd5f74983fe640c84452ab1"
},
"downloads": -1,
"filename": "hgitaly-1.4.0.tar.gz",
"has_sig": false,
"md5_digest": "b0ff60242c9f35d8142b85cf1d397c8b",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8",
"size": 342448,
"upload_time": "2024-05-01T15:45:35",
"upload_time_iso_8601": "2024-05-01T15:45:35.709835Z",
"url": "https://files.pythonhosted.org/packages/99/78/8feec022ac38deee6e6265b2217afdd4192c80e26e4fcf73e032f7300533/hgitaly-1.4.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-05-01 15:45:35",
"github": false,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"lcname": "hgitaly"
}
JSON