<img src="https://i.imgur.com/6uh3Gh4.png" width="550">
# GitLab Watchman
## About GitLab Watchman
GitLab Watchman is an application that uses the GitLab API to detect exposed secrets and personal data. It also enumerates the GitLab instance for any useful information.
### Features
#### Secrets Detection
It searches GitLab for internally shared projects and looks at:
- Code
- Commits
- Wiki pages
- Issues
- Merge requests
- Milestones
- Notes
- Snippets
For the following data:
- GCP keys and service account files
- AWS keys
- Azure keys and service account files
- Google API keys
- Slack API tokens & webhooks
- Private keys (SSH, PGP, any other misc private key)
- Exposed tokens (Bearer tokens, access tokens, client_secret etc.)
- S3 config files
- Tokens for services such as Heroku, PayPal and more
- Passwords in plaintext
- and more
##### Time based searching
You can run GitLab Watchman to look for results going back as far as:
- 24 hours
- 7 days
- 30 days
- All time
This means after one deep scan, you can schedule GitLab Watchman to run regularly and only return results from your chosen timeframe.
#### Enumeration
GitLab Watchman can enumerate potentially useful information from a GitLab instance:
- Instance metadata
- Information on the calling user/token being used
- Output all users to CSV file
- Output all projects to CSV file
- Output all groups to CSV file
### Signatures
GitLab Watchman uses custom YAML signatures to detect matches in GitLab. These signatures are pulled from the central [Watchman Signatures repository](https://github.com/PaperMtn/watchman-signatures). Slack Watchman automatically updates its signature base at runtime to ensure its using the latest signatures to detect secrets.
### Logging
GitLab Watchman gives the following logging options:
- Terminal-friendly Stdout
- JSON to Stdout
GitLab Watchman defaults to terminal-friendly stdout logging if no option is given. This is designed to be easier for humans to read.
JSON logging is also available, which is perfect for ingesting into a SIEM or other log analysis platforms.
JSON formatted logging can be easily redirected to a file as below:
```commandline
gitlab-watchman --timeframe a --all --output json >> gitlab_watchman_log.json
```
## Requirements
### GitLab versions
GitLab Watchman uses the v4 API, and works with GitLab Enterprise Edition versions:
- 13.0 and above - Yes
- GitLab.com - Yes
- 12.0 - 12.10 - Maybe, untested but if using v4 of the API then it could work
### GitLab Licence & Elasticsearch
To search the scopes:
- blobs
- wiki_blobs
- commits
The GitLab instance must have [Elasticsearch](https://docs.gitlab.com/ee/integration/elasticsearch.html) configured, and be running Enterprise Edition with a minimum GitLab Starter or Bronze Licence.
### GitLab personal access token
To run GitLab Watchman, you will need a GitLab personal access token.
You can create a personal access token in the GitLab GUI via Settings -> Access Tokens -> Add a personal access token
The token needs permission for the following scopes:
```
api
```
**Note**: Personal access tokens act on behalf of the user who creates them, so I would suggest you create a token using a service account, otherwise the app will have access to your private repositories.
### GitLab URL
You also need to provide the URL of your GitLab instance.
#### Providing token & URL
GitLab Watchman will get the GitLab token and URL from the environment variables `GITLAB_WATCHMAN_TOKEN` and `GITLAB_WATCHMAN_URL`.
## Installation
You can install the latest stable version via pip:
`python3 -m pip install gitlab-watchman`
Or build from source yourself. Download the release source files, then from the top level repository run:
```shell
python3 -m build
python3 -m pip install --force-reinstall dist/*.whl
```
## Docker Image
GitLab Watchman is also available from the Docker hub as a Docker image:
`docker pull papermountain/gitlab-watchman:latest`
You can then run GitLab Watchman in a container, making sure you pass the required environment variables:
```
// help
docker run --rm papermountain/gitlab-watchman -h
// scan all
docker run --rm -e GITLAB_WATCHMAN_TOKEN=abc123 -e GITLAB_WATCHMAN_URL=https://example.gitlab.com papermountain/gitlab-watchman --timeframe a --all
docker run --rm --env-file .env papermountain/gitlab-watchman --timeframe a --all
```
## Usage
GitLab Watchman will be installed as a global command, use as follows:
```
usage: gitlab-watchman [-h] --timeframe {d,w,m,a} [--output {json,stdout}] [--version] [--all] [--blobs] [--commits] [--wiki-blobs] [--issues]
[--merge-requests] [--milestones] [--notes] [--snippets] [--enumerate] [--debug] [--verbose]
Finding exposed secrets and personal data in GitLab
options:
-h, --help show this help message and exit
--output {json,stdout}, -o {json,stdout}
Where to send results
--version, -v show program's version number and exit
--all, -a Find everything
--blobs, -b Search code blobs
--commits, -c Search commits
--wiki-blobs, -w Search wiki blobs
--issues, -i Search issues
--merge-requests, -mr
Search merge requests
--milestones, -m Search milestones
--notes, -n Search notes
--snippets, -s Search snippets
--enumerate, -e Enumerate this GitLab instance for users, groups, projects.Output will be saved to CSV files
--debug, -d Turn on debug level logging
--verbose, -V Turn on more verbose output for JSON logging. This includes more fields, but is larger
required arguments:
--timeframe {d,w,m,a}
How far back to search: d = 24 hours w = 7 days, m = 30 days, a = all time
```
## Other Watchman apps
You may be interested in the other apps in the Watchman family:
- [Slack Watchman](https://github.com/PaperMtn/slack-watchman)
- [Slack Watchman for Enterprise Grid](https://github.com/PaperMtn/slack-watchman-enterprise-grid)
- [GitHub Watchman](https://github.com/PaperMtn/github-watchman)
## License
The source code for this project is released under the [GNU General Public Licence](https://www.gnu.org/licenses/licenses.html#GPL). This project is not associated with GitLab.
## [3.0.0] - 2023-05-15
This major version release brings multiple updates to GitLab Watchman in usability, functionality and behind the scenes improvements.
### Added
- Support for centralised signatures from the [Watchman Signatures repository](https://github.com/PaperMtn/watchman-signatures)
- This makes it much easier to keep the signature base for all Watchman applications up to date, and to add functionality to GitLab Watchman with new signatures. New signatures are downloaded, and updates to existing signatures are applied, at runtime, meaning GitLab Watchman will always be using the most up to date signatures.
- Major UI overhaul
- A lot of feedback said GitLab Watchman was hard to read. This version introduces new terminal optimised logging as a logging option, as well as JSON formatting. This formatting is now the default when running with no output option selected, and is a lot easier for humans to read. Also, colours!
- Enumeration options added
- GitLab Watchman now gathers more information from an instance. Useful if your use case is more red than blue...
- Instance metadata output to terminal
- Information on the user you are authenticated as, and the token you are using, including what permissions it has.
- All instance users output to CSV
- All instance projects output to CSV
- All instance groups output to CSV
- Option choose between verbose or succinct logging when using JSON output. Default is succinct.
- Debug logging option
### Removed
- Local/custom signatures - Centralised signatures mean that user-created custom signatures can't be used with GitLab Watchman for Enterprise Grid anymore. If you have made a signature you think would be good for sharing with the community, feel free to add it to the Watchman Signatures repository, so it can be used in all Watchman applications
## [2.0.0] - 2022-03-22
### Added:
- New scopes for finding exposed data in:
- notes
- snippets
- Docker image now available from the Docker hub, or by building from source. (Credit [@adioss](https://github.com/adioss) for the inspiration)
- Complete rewrite of the codebase to make searching faster and more efficient.
- More modern packaging and distribution.
- Logs now include more data
- Additional signatures added to find more leaked data
- Updated logo to play nicely with dark mode displays
### Removed:
- Logging to file and TCP stream - logs to stdout like a true 12 factor app. Reroute stdout as you see fit. --output
- .conf file for configuration options. Pass the environment variables `GITLAB_WATCHMAN_TOKEN` and `GITLAB_WATCHMAN_URL`
**Breaking changes:**
- The --output flag is no longer required, and therefore not supported
## [1.4.0] - 2020-12-24
### Added:
- Refactor of rules into directories for easier management
- Multiprocessing implemented for searching for matches. GitLab Watchman now splits regex filtering between the cores available on the device, meaning the more cores you have, the faster searching should run.
- Handling for GitLab API rate limiting, backing off when the rate limit is hit. The rate limit may be more likely to come into effect with multiprocessing
- Rules added to search for:
- Cloudflare tokens
- Facebook API tokens
- GitHub API tokens
- Mailchimp API tokens
- Mailgun API tokens
- Shodan API tokens
- Stripe API tokens
- Twilio API tokens
- Microsoft NuGet keys
## [1.3.0] - 2020-12-12
### Added:
- Add more information about the namespaces a project is in to logs
- Added owner details of that namespace, for groups and users
- Time based searching now looks at the time a file was committed, not when a project was active, which greatly reduces multiples of the same detection because a project is active but a file has not been modified.
- Rules added:
- SSH private keys
- Mastercard datacash tokens
- Heroku tokens
- PagerDuty tokens
### Removed:
- Enhanced logging that includes nested information, such as namespace owners, means that CSV logging is no longer practical. CSV logging has been removed and JSON via STDOUT is now the default option.
## [1.2.0] - 2020-11-16
### Added:
- More data on namespaces added to logs
- Better search queries for existing rules to filter out false positives
### Removed:
- CICD variable search no longer works due to GitLab API now only allowing owners of a project to search it for variables. It has been removed.
### Fixed:
- Bug on outputting match string for blobs/wiki-blobs
## [1.1.0] - 2020-11-14
### Fixed
- Retry added for occasional Requests HTTPSConnectionPool error
### Added
- Exact regex string match added to output from message searches
## [1.0.0] - 2020-10-01
Release
Raw data
{
"_id": null,
"home_page": "https://github.com/PaperMtn/gitlab-watchman",
"name": "gitlab-watchman",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.10",
"maintainer_email": "",
"keywords": "audit,gitlab,gitlab-watchman,watchman,blue-team,red-team,threat-hunting",
"author": "PaperMtn",
"author_email": "papermtn@protonmail.com",
"download_url": "https://files.pythonhosted.org/packages/bb/81/1d42c14ef2887c435556812ae6c3cdf34c0d10ce56c08e4604016919d330/gitlab-watchman-3.0.0.tar.gz",
"platform": null,
"description": "<img src=\"https://i.imgur.com/6uh3Gh4.png\" width=\"550\">\n\n# GitLab Watchman\n\n\n\n\n## About GitLab Watchman\n\nGitLab Watchman is an application that uses the GitLab API to detect exposed secrets and personal data. It also enumerates the GitLab instance for any useful information.\n\n### Features\n\n#### Secrets Detection\nIt searches GitLab for internally shared projects and looks at:\n- Code\n- Commits\n- Wiki pages\n- Issues\n- Merge requests\n- Milestones\n- Notes\n- Snippets\n\nFor the following data:\n- GCP keys and service account files\n- AWS keys\n- Azure keys and service account files\n- Google API keys\n- Slack API tokens & webhooks\n- Private keys (SSH, PGP, any other misc private key)\n- Exposed tokens (Bearer tokens, access tokens, client_secret etc.)\n- S3 config files\n- Tokens for services such as Heroku, PayPal and more\n- Passwords in plaintext\n- and more\n\n##### Time based searching\nYou can run GitLab Watchman to look for results going back as far as:\n- 24 hours\n- 7 days\n- 30 days\n- All time\n\nThis means after one deep scan, you can schedule GitLab Watchman to run regularly and only return results from your chosen timeframe.\n\n#### Enumeration\nGitLab Watchman can enumerate potentially useful information from a GitLab instance:\n- Instance metadata\n- Information on the calling user/token being used\n- Output all users to CSV file\n- Output all projects to CSV file\n- Output all groups to CSV file\n\n### Signatures\nGitLab Watchman uses custom YAML signatures to detect matches in GitLab. These signatures are pulled from the central [Watchman Signatures repository](https://github.com/PaperMtn/watchman-signatures). Slack Watchman automatically updates its signature base at runtime to ensure its using the latest signatures to detect secrets.\n\n### Logging\n\nGitLab Watchman gives the following logging options:\n- Terminal-friendly Stdout\n- JSON to Stdout\n\nGitLab Watchman defaults to terminal-friendly stdout logging if no option is given. This is designed to be easier for humans to read.\n\nJSON logging is also available, which is perfect for ingesting into a SIEM or other log analysis platforms.\n\nJSON formatted logging can be easily redirected to a file as below:\n```commandline\ngitlab-watchman --timeframe a --all --output json >> gitlab_watchman_log.json \n```\n\n## Requirements\n\n### GitLab versions\nGitLab Watchman uses the v4 API, and works with GitLab Enterprise Edition versions:\n- 13.0 and above - Yes\n\n- GitLab.com - Yes\n- 12.0 - 12.10 - Maybe, untested but if using v4 of the API then it could work\n\n### GitLab Licence & Elasticsearch\nTo search the scopes:\n- blobs\n- wiki_blobs\n- commits\n\nThe GitLab instance must have [Elasticsearch](https://docs.gitlab.com/ee/integration/elasticsearch.html) configured, and be running Enterprise Edition with a minimum GitLab Starter or Bronze Licence.\n\n### GitLab personal access token\nTo run GitLab Watchman, you will need a GitLab personal access token.\n\nYou can create a personal access token in the GitLab GUI via Settings -> Access Tokens -> Add a personal access token\n\nThe token needs permission for the following scopes:\n```\napi\n```\n\n**Note**: Personal access tokens act on behalf of the user who creates them, so I would suggest you create a token using a service account, otherwise the app will have access to your private repositories.\n\n### GitLab URL\n\nYou also need to provide the URL of your GitLab instance.\n\n#### Providing token & URL\nGitLab Watchman will get the GitLab token and URL from the environment variables `GITLAB_WATCHMAN_TOKEN` and `GITLAB_WATCHMAN_URL`.\n\n## Installation\nYou can install the latest stable version via pip:\n\n`python3 -m pip install gitlab-watchman`\n\nOr build from source yourself. Download the release source files, then from the top level repository run:\n```shell\npython3 -m build\npython3 -m pip install --force-reinstall dist/*.whl\n```\n\n## Docker Image\n\nGitLab Watchman is also available from the Docker hub as a Docker image:\n\n`docker pull papermountain/gitlab-watchman:latest`\n\nYou can then run GitLab Watchman in a container, making sure you pass the required environment variables:\n\n```\n// help\ndocker run --rm papermountain/gitlab-watchman -h\n\n// scan all\ndocker run --rm -e GITLAB_WATCHMAN_TOKEN=abc123 -e GITLAB_WATCHMAN_URL=https://example.gitlab.com papermountain/gitlab-watchman --timeframe a --all\ndocker run --rm --env-file .env papermountain/gitlab-watchman --timeframe a --all\n```\n\n## Usage\nGitLab Watchman will be installed as a global command, use as follows:\n```\nusage: gitlab-watchman [-h] --timeframe {d,w,m,a} [--output {json,stdout}] [--version] [--all] [--blobs] [--commits] [--wiki-blobs] [--issues]\n [--merge-requests] [--milestones] [--notes] [--snippets] [--enumerate] [--debug] [--verbose]\n\nFinding exposed secrets and personal data in GitLab\n\noptions:\n -h, --help show this help message and exit\n --output {json,stdout}, -o {json,stdout}\n Where to send results\n --version, -v show program's version number and exit\n --all, -a Find everything\n --blobs, -b Search code blobs\n --commits, -c Search commits\n --wiki-blobs, -w Search wiki blobs\n --issues, -i Search issues\n --merge-requests, -mr\n Search merge requests\n --milestones, -m Search milestones\n --notes, -n Search notes\n --snippets, -s Search snippets\n --enumerate, -e Enumerate this GitLab instance for users, groups, projects.Output will be saved to CSV files\n --debug, -d Turn on debug level logging\n --verbose, -V Turn on more verbose output for JSON logging. This includes more fields, but is larger\n\nrequired arguments:\n --timeframe {d,w,m,a}\n How far back to search: d = 24 hours w = 7 days, m = 30 days, a = all time\n\n ```\n\n## Other Watchman apps\nYou may be interested in the other apps in the Watchman family:\n- [Slack Watchman](https://github.com/PaperMtn/slack-watchman)\n- [Slack Watchman for Enterprise Grid](https://github.com/PaperMtn/slack-watchman-enterprise-grid)\n- [GitHub Watchman](https://github.com/PaperMtn/github-watchman)\n\n## License\nThe source code for this project is released under the [GNU General Public Licence](https://www.gnu.org/licenses/licenses.html#GPL). This project is not associated with GitLab.\n\n## [3.0.0] - 2023-05-15\nThis major version release brings multiple updates to GitLab Watchman in usability, functionality and behind the scenes improvements.\n### Added\n- Support for centralised signatures from the [Watchman Signatures repository](https://github.com/PaperMtn/watchman-signatures)\n - This makes it much easier to keep the signature base for all Watchman applications up to date, and to add functionality to GitLab Watchman with new signatures. New signatures are downloaded, and updates to existing signatures are applied, at runtime, meaning GitLab Watchman will always be using the most up to date signatures.\n- Major UI overhaul\n - A lot of feedback said GitLab Watchman was hard to read. This version introduces new terminal optimised logging as a logging option, as well as JSON formatting. This formatting is now the default when running with no output option selected, and is a lot easier for humans to read. Also, colours!\n- Enumeration options added\n - GitLab Watchman now gathers more information from an instance. Useful if your use case is more red than blue...\n - Instance metadata output to terminal \n - Information on the user you are authenticated as, and the token you are using, including what permissions it has.\n - All instance users output to CSV\n - All instance projects output to CSV\n - All instance groups output to CSV\n- Option choose between verbose or succinct logging when using JSON output. Default is succinct.\n- Debug logging option\n### Removed\n- Local/custom signatures - Centralised signatures mean that user-created custom signatures can't be used with GitLab Watchman for Enterprise Grid anymore. If you have made a signature you think would be good for sharing with the community, feel free to add it to the Watchman Signatures repository, so it can be used in all Watchman applications\n\n## [2.0.0] - 2022-03-22\n### Added:\n- New scopes for finding exposed data in:\n - notes\n - snippets\n- Docker image now available from the Docker hub, or by building from source. (Credit [@adioss](https://github.com/adioss) for the inspiration)\n- Complete rewrite of the codebase to make searching faster and more efficient.\n - More modern packaging and distribution.\n- Logs now include more data\n- Additional signatures added to find more leaked data\n- Updated logo to play nicely with dark mode displays\n\n### Removed:\n\n- Logging to file and TCP stream - logs to stdout like a true 12 factor app. Reroute stdout as you see fit. --output \n- .conf file for configuration options. Pass the environment variables `GITLAB_WATCHMAN_TOKEN` and `GITLAB_WATCHMAN_URL`\n\n**Breaking changes:**\n- The --output flag is no longer required, and therefore not supported\n\n\n## [1.4.0] - 2020-12-24\n### Added:\n- Refactor of rules into directories for easier management\n- Multiprocessing implemented for searching for matches. GitLab Watchman now splits regex filtering between the cores available on the device, meaning the more cores you have, the faster searching should run.\n- Handling for GitLab API rate limiting, backing off when the rate limit is hit. The rate limit may be more likely to come into effect with multiprocessing\n- Rules added to search for:\n - Cloudflare tokens\n - Facebook API tokens\n - GitHub API tokens\n - Mailchimp API tokens\n - Mailgun API tokens\n - Shodan API tokens\n - Stripe API tokens\n - Twilio API tokens\n - Microsoft NuGet keys\n\n\n## [1.3.0] - 2020-12-12\n### Added:\n- Add more information about the namespaces a project is in to logs\n- Added owner details of that namespace, for groups and users\n- Time based searching now looks at the time a file was committed, not when a project was active, which greatly reduces multiples of the same detection because a project is active but a file has not been modified.\n- Rules added:\n - SSH private keys\n - Mastercard datacash tokens\n - Heroku tokens\n - PagerDuty tokens\n\n### Removed:\n- Enhanced logging that includes nested information, such as namespace owners, means that CSV logging is no longer practical. CSV logging has been removed and JSON via STDOUT is now the default option.\n\n## [1.2.0] - 2020-11-16\n### Added:\n- More data on namespaces added to logs\n- Better search queries for existing rules to filter out false positives\n\n### Removed:\n- CICD variable search no longer works due to GitLab API now only allowing owners of a project to search it for variables. It has been removed.\n\n### Fixed:\n- Bug on outputting match string for blobs/wiki-blobs\n\n## [1.1.0] - 2020-11-14\n### Fixed\n- Retry added for occasional Requests HTTPSConnectionPool error\n\n### Added\n- Exact regex string match added to output from message searches\n\n## [1.0.0] - 2020-10-01\nRelease\n",
"bugtrack_url": null,
"license": "GPL-3.0",
"summary": "Finding exposed secrets and personal data in GitLab",
"version": "3.0.0",
"project_urls": {
"Homepage": "https://github.com/PaperMtn/gitlab-watchman"
},
"split_keywords": [
"audit",
"gitlab",
"gitlab-watchman",
"watchman",
"blue-team",
"red-team",
"threat-hunting"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "3b0da0afa3f00d749230ec458c5ae0d7a64928b37f1026f2b3f3c138ce0c5b3a",
"md5": "8c45e0d4b9aef2206ee7248eb6803491",
"sha256": "b740453775bb89464fa4a9db2dd0a8085495a5f5d8e9e851be2f4f6b77b3de3a"
},
"downloads": -1,
"filename": "gitlab_watchman-3.0.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "8c45e0d4b9aef2206ee7248eb6803491",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.10",
"size": 41491,
"upload_time": "2023-05-20T13:52:19",
"upload_time_iso_8601": "2023-05-20T13:52:19.084034Z",
"url": "https://files.pythonhosted.org/packages/3b/0d/a0afa3f00d749230ec458c5ae0d7a64928b37f1026f2b3f3c138ce0c5b3a/gitlab_watchman-3.0.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "bb811d42c14ef2887c435556812ae6c3cdf34c0d10ce56c08e4604016919d330",
"md5": "0dceb9b9853d4be84dd9c00588e9b30b",
"sha256": "f1c7dd123bff09681c14013a74c18030d5dbe96ff135d0f3a2d394f5badc4dd3"
},
"downloads": -1,
"filename": "gitlab-watchman-3.0.0.tar.gz",
"has_sig": false,
"md5_digest": "0dceb9b9853d4be84dd9c00588e9b30b",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.10",
"size": 41070,
"upload_time": "2023-05-20T13:52:20",
"upload_time_iso_8601": "2023-05-20T13:52:20.737548Z",
"url": "https://files.pythonhosted.org/packages/bb/81/1d42c14ef2887c435556812ae6c3cdf34c0d10ce56c08e4604016919d330/gitlab-watchman-3.0.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-05-20 13:52:20",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "PaperMtn",
"github_project": "gitlab-watchman",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"requirements": [],
"lcname": "gitlab-watchman"
}
JSON
