<img src="https://i.imgur.com/6uh3Gh4.png" width="550">
# GitLab Watchman
## About GitLab Watchman
GitLab Watchman is an application that uses the GitLab API to detect exposed secrets and personal data. It also enumerates the GitLab instance for any useful information.
### Features
#### Secrets Detection
It searches GitLab for internally shared projects and looks at:
- Code
- Commits
- Wiki pages
- Issues
- Merge requests
- Milestones
- Notes
- Snippets
For the following data:
- GCP keys and service account files
- AWS keys
- Azure keys and service account files
- Google API keys
- Slack API tokens & webhooks
- Private keys (SSH, PGP, any other misc private key)
- Exposed tokens (Bearer tokens, access tokens, client_secret etc.)
- S3 config files
- Tokens for services such as Heroku, PayPal and more
- Passwords in plaintext
- and more
##### Time based searching
You can run GitLab Watchman to look for results going back as far as:
- 24 hours
- 7 days
- 30 days
- All time
This means after one deep scan, you can schedule GitLab Watchman to run regularly and only return results from your chosen timeframe.
#### Enumeration
GitLab Watchman can enumerate potentially useful information from a GitLab instance:
- Instance metadata
- Information on the calling user/token being used
- Output all users to CSV file
- Output all projects to CSV file
- Output all groups to CSV file
### Signatures
GitLab Watchman uses custom YAML signatures to detect matches in GitLab. These signatures are pulled from the central [Watchman Signatures repository](https://github.com/PaperMtn/watchman-signatures). Slack Watchman automatically updates its signature base at runtime to ensure its using the latest signatures to detect secrets.
#### Suppressing Signatures
You can define signatures that you want to disable when running GitLab Watchman by adding their IDs to the `disabled_signatures` section of the `watchman.conf` file. For example:
```yaml
gitlab_watchman:
disabled_signatures:
- tokens_generic_bearer_tokens
- tokens_generic_access_tokens
```
You can find the ID of a signature in the individual YAML files in [Watchman Signatures repository](https://github.com/PaperMtn/watchman-signatures).
### Logging
GitLab Watchman gives the following logging options:
- Terminal-friendly Stdout
- JSON to Stdout
GitLab Watchman defaults to terminal-friendly stdout logging if no option is given. This is designed to be easier for humans to read.
JSON logging is also available, which is perfect for ingesting into a SIEM or other log analysis platforms.
JSON formatted logging can be easily redirected to a file as below:
```commandline
gitlab-watchman --timeframe a --all --output json >> gitlab_watchman_log.json
```
## Requirements
### GitLab versions
GitLab Watchman uses the v4 API, and works with GitLab Enterprise Edition versions:
- 13.0 and above - Yes
- GitLab.com - Yes
- 12.0 - 12.10 - Maybe, untested but if using v4 of the API then it could work
### GitLab Licence & Elasticsearch
To search the scopes:
- blobs
- wiki_blobs
- commits
The GitLab instance must have [Elasticsearch](https://docs.gitlab.com/ee/integration/elasticsearch.html) configured, and be running Enterprise Edition with a minimum GitLab Starter or Bronze Licence.
### GitLab personal access token
To run GitLab Watchman, you will need a GitLab personal access token.
You can create a personal access token in the GitLab GUI via Settings -> Access Tokens -> Add a personal access token
The token needs permission for the following scopes:
```
api
```
**Note**: Personal access tokens act on behalf of the user who creates them, so I would suggest you create a token using a service account, otherwise the app will have access to your private repositories.
### GitLab URL
You also need to provide the URL of your GitLab instance.
#### Providing token & URL
GitLab Watchman will get the GitLab token and URL from the environment variables `GITLAB_WATCHMAN_TOKEN` and `GITLAB_WATCHMAN_URL`.
### watchman.conf file
Configuration options can be passed in a file named `watchman.conf` which must be stored in your home directory. The file should follow the YAML format, and should look like below:
```yaml
gitlab_watchman:
disabled_signatures:
- tokens_generic_bearer_tokens
- tokens_generic_access_tokens
```
GitLab Watchman will look for this file at runtime, and use the configuration options from here.
## Installation
You can install the latest stable version via pip:
`python3 -m pip install gitlab-watchman`
Or build from source yourself. Download the release source files, then from the top level repository run:
```shell
python3 -m build
python3 -m pip install --force-reinstall dist/*.whl
```
## Docker Image
GitLab Watchman is also available from the Docker hub as a Docker image:
`docker pull papermountain/gitlab-watchman:latest`
You can then run GitLab Watchman in a container, making sure you pass the required environment variables:
```
// help
docker run --rm papermountain/gitlab-watchman -h
// scan all
docker run --rm -e GITLAB_WATCHMAN_TOKEN=abc123 -e GITLAB_WATCHMAN_URL=https://example.gitlab.com papermountain/gitlab-watchman --timeframe a --all
docker run --rm --env-file .env papermountain/gitlab-watchman --timeframe a --all
```
## Usage
GitLab Watchman will be installed as a global command, use as follows:
```
usage: gitlab-watchman [-h] --timeframe {d,w,m,a} [--output {json,stdout}] [--version] [--all] [--blobs] [--commits] [--wiki-blobs] [--issues]
[--merge-requests] [--milestones] [--notes] [--snippets] [--enumerate] [--debug] [--verbose]
Finding exposed secrets and personal data in GitLab
options:
-h, --help show this help message and exit
--output {json,stdout}, -o {json,stdout}
Where to send results
--version, -v show program's version number and exit
--all, -a Find everything
--blobs, -b Search code blobs
--commits, -c Search commits
--wiki-blobs, -w Search wiki blobs
--issues, -i Search issues
--merge-requests, -mr
Search merge requests
--milestones, -m Search milestones
--notes, -n Search notes
--snippets, -s Search snippets
--enumerate, -e Enumerate this GitLab instance for users, groups, projects.Output will be saved to CSV files
--debug, -d Turn on debug level logging
--verbose, -V Turn on more verbose output for JSON logging. This includes more fields, but is larger
required arguments:
--timeframe {d,w,m,a}
How far back to search: d = 24 hours w = 7 days, m = 30 days, a = all time
```
## Other Watchman apps
You may be interested in the other apps in the Watchman family:
- [Slack Watchman](https://github.com/PaperMtn/slack-watchman)
- [Slack Watchman for Enterprise Grid](https://github.com/PaperMtn/slack-watchman-enterprise-grid)
- [GitHub Watchman](https://github.com/PaperMtn/github-watchman)
## License
The source code for this project is released under the [GNU General Public Licence](https://www.gnu.org/licenses/licenses.html#GPL). This project is not associated with GitLab.
Raw data
{
"_id": null,
"home_page": "https://github.com/PaperMtn/gitlab-watchman",
"name": "gitlab-watchman",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.10",
"maintainer_email": null,
"keywords": "audit, dlp, gitlab, gitlab-watchman, watchman, blue-team, red-team, threat-hunting",
"author": "PaperMtn",
"author_email": "papermtn@protonmail.com",
"download_url": "https://files.pythonhosted.org/packages/5b/98/cc6aac0ee8f0eeab9b7cb004fc8f5835fca87201f294cc2f9e604ccd7a32/gitlab_watchman-3.1.0.tar.gz",
"platform": null,
"description": "<img src=\"https://i.imgur.com/6uh3Gh4.png\" width=\"550\">\n\n# GitLab Watchman\n\n\n\n\n## About GitLab Watchman\n\nGitLab Watchman is an application that uses the GitLab API to detect exposed secrets and personal data. It also enumerates the GitLab instance for any useful information.\n\n### Features\n\n#### Secrets Detection\nIt searches GitLab for internally shared projects and looks at:\n- Code\n- Commits\n- Wiki pages\n- Issues\n- Merge requests\n- Milestones\n- Notes\n- Snippets\n\nFor the following data:\n- GCP keys and service account files\n- AWS keys\n- Azure keys and service account files\n- Google API keys\n- Slack API tokens & webhooks\n- Private keys (SSH, PGP, any other misc private key)\n- Exposed tokens (Bearer tokens, access tokens, client_secret etc.)\n- S3 config files\n- Tokens for services such as Heroku, PayPal and more\n- Passwords in plaintext\n- and more\n\n##### Time based searching\nYou can run GitLab Watchman to look for results going back as far as:\n- 24 hours\n- 7 days\n- 30 days\n- All time\n\nThis means after one deep scan, you can schedule GitLab Watchman to run regularly and only return results from your chosen timeframe.\n\n#### Enumeration\nGitLab Watchman can enumerate potentially useful information from a GitLab instance:\n- Instance metadata\n- Information on the calling user/token being used\n- Output all users to CSV file\n- Output all projects to CSV file\n- Output all groups to CSV file\n\n### Signatures\nGitLab Watchman uses custom YAML signatures to detect matches in GitLab. These signatures are pulled from the central [Watchman Signatures repository](https://github.com/PaperMtn/watchman-signatures). Slack Watchman automatically updates its signature base at runtime to ensure its using the latest signatures to detect secrets.\n\n#### Suppressing Signatures\nYou can define signatures that you want to disable when running GitLab Watchman by adding their IDs to the `disabled_signatures` section of the `watchman.conf` file. For example:\n\n```yaml\ngitlab_watchman:\n disabled_signatures:\n - tokens_generic_bearer_tokens\n - tokens_generic_access_tokens\n```\n\nYou can find the ID of a signature in the individual YAML files in [Watchman Signatures repository](https://github.com/PaperMtn/watchman-signatures).\n\n### Logging\n\nGitLab Watchman gives the following logging options:\n- Terminal-friendly Stdout\n- JSON to Stdout\n\nGitLab Watchman defaults to terminal-friendly stdout logging if no option is given. This is designed to be easier for humans to read.\n\nJSON logging is also available, which is perfect for ingesting into a SIEM or other log analysis platforms.\n\nJSON formatted logging can be easily redirected to a file as below:\n```commandline\ngitlab-watchman --timeframe a --all --output json >> gitlab_watchman_log.json \n```\n\n## Requirements\n\n### GitLab versions\nGitLab Watchman uses the v4 API, and works with GitLab Enterprise Edition versions:\n- 13.0 and above - Yes\n\n- GitLab.com - Yes\n- 12.0 - 12.10 - Maybe, untested but if using v4 of the API then it could work\n\n### GitLab Licence & Elasticsearch\nTo search the scopes:\n- blobs\n- wiki_blobs\n- commits\n\nThe GitLab instance must have [Elasticsearch](https://docs.gitlab.com/ee/integration/elasticsearch.html) configured, and be running Enterprise Edition with a minimum GitLab Starter or Bronze Licence.\n\n### GitLab personal access token\nTo run GitLab Watchman, you will need a GitLab personal access token.\n\nYou can create a personal access token in the GitLab GUI via Settings -> Access Tokens -> Add a personal access token\n\nThe token needs permission for the following scopes:\n```\napi\n```\n\n**Note**: Personal access tokens act on behalf of the user who creates them, so I would suggest you create a token using a service account, otherwise the app will have access to your private repositories.\n\n### GitLab URL\n\nYou also need to provide the URL of your GitLab instance.\n\n#### Providing token & URL\nGitLab Watchman will get the GitLab token and URL from the environment variables `GITLAB_WATCHMAN_TOKEN` and `GITLAB_WATCHMAN_URL`.\n\n### watchman.conf file\nConfiguration options can be passed in a file named `watchman.conf` which must be stored in your home directory. The file should follow the YAML format, and should look like below:\n```yaml\ngitlab_watchman:\n disabled_signatures:\n - tokens_generic_bearer_tokens\n - tokens_generic_access_tokens\n```\nGitLab Watchman will look for this file at runtime, and use the configuration options from here.\n\n## Installation\nYou can install the latest stable version via pip:\n\n`python3 -m pip install gitlab-watchman`\n\nOr build from source yourself. Download the release source files, then from the top level repository run:\n```shell\npython3 -m build\npython3 -m pip install --force-reinstall dist/*.whl\n```\n\n## Docker Image\n\nGitLab Watchman is also available from the Docker hub as a Docker image:\n\n`docker pull papermountain/gitlab-watchman:latest`\n\nYou can then run GitLab Watchman in a container, making sure you pass the required environment variables:\n\n```\n// help\ndocker run --rm papermountain/gitlab-watchman -h\n\n// scan all\ndocker run --rm -e GITLAB_WATCHMAN_TOKEN=abc123 -e GITLAB_WATCHMAN_URL=https://example.gitlab.com papermountain/gitlab-watchman --timeframe a --all\ndocker run --rm --env-file .env papermountain/gitlab-watchman --timeframe a --all\n```\n\n## Usage\nGitLab Watchman will be installed as a global command, use as follows:\n```\nusage: gitlab-watchman [-h] --timeframe {d,w,m,a} [--output {json,stdout}] [--version] [--all] [--blobs] [--commits] [--wiki-blobs] [--issues]\n [--merge-requests] [--milestones] [--notes] [--snippets] [--enumerate] [--debug] [--verbose]\n\nFinding exposed secrets and personal data in GitLab\n\noptions:\n -h, --help show this help message and exit\n --output {json,stdout}, -o {json,stdout}\n Where to send results\n --version, -v show program's version number and exit\n --all, -a Find everything\n --blobs, -b Search code blobs\n --commits, -c Search commits\n --wiki-blobs, -w Search wiki blobs\n --issues, -i Search issues\n --merge-requests, -mr\n Search merge requests\n --milestones, -m Search milestones\n --notes, -n Search notes\n --snippets, -s Search snippets\n --enumerate, -e Enumerate this GitLab instance for users, groups, projects.Output will be saved to CSV files\n --debug, -d Turn on debug level logging\n --verbose, -V Turn on more verbose output for JSON logging. This includes more fields, but is larger\n\nrequired arguments:\n --timeframe {d,w,m,a}\n How far back to search: d = 24 hours w = 7 days, m = 30 days, a = all time\n\n ```\n\n## Other Watchman apps\nYou may be interested in the other apps in the Watchman family:\n- [Slack Watchman](https://github.com/PaperMtn/slack-watchman)\n- [Slack Watchman for Enterprise Grid](https://github.com/PaperMtn/slack-watchman-enterprise-grid)\n- [GitHub Watchman](https://github.com/PaperMtn/github-watchman)\n\n## License\nThe source code for this project is released under the [GNU General Public Licence](https://www.gnu.org/licenses/licenses.html#GPL). This project is not associated with GitLab.\n",
"bugtrack_url": null,
"license": "GPL-3.0",
"summary": "Finding exposed secrets and personal data in GitLab",
"version": "3.1.0",
"project_urls": {
"Blog": "https://papermtn.co.uk/category/tools/gitlab-watchman/",
"Homepage": "https://github.com/PaperMtn/gitlab-watchman",
"Repository": "https://github.com/PaperMtn/gitlab-watchman"
},
"split_keywords": [
"audit",
" dlp",
" gitlab",
" gitlab-watchman",
" watchman",
" blue-team",
" red-team",
" threat-hunting"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "820aa543f3170afc0a5954d79bcc85f71369e26f5c8e6c5ae97f9d7a5e0dddd7",
"md5": "4710e278c5b1b5d532731318e6a76484",
"sha256": "77109e2441a1740e4501e72481530ea5751e2af3aff95bba93415f788e337d2a"
},
"downloads": -1,
"filename": "gitlab_watchman-3.1.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "4710e278c5b1b5d532731318e6a76484",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.10",
"size": 43477,
"upload_time": "2024-11-18T14:32:25",
"upload_time_iso_8601": "2024-11-18T14:32:25.921961Z",
"url": "https://files.pythonhosted.org/packages/82/0a/a543f3170afc0a5954d79bcc85f71369e26f5c8e6c5ae97f9d7a5e0dddd7/gitlab_watchman-3.1.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "5b98cc6aac0ee8f0eeab9b7cb004fc8f5835fca87201f294cc2f9e604ccd7a32",
"md5": "cb6bf44d5d6bc1dc2b75c158c565f6d4",
"sha256": "7160ab1668b65cd45c6168eef4affa6329699ebf5f5572a1c097b8ad370604e6"
},
"downloads": -1,
"filename": "gitlab_watchman-3.1.0.tar.gz",
"has_sig": false,
"md5_digest": "cb6bf44d5d6bc1dc2b75c158c565f6d4",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.10",
"size": 37500,
"upload_time": "2024-11-18T14:32:27",
"upload_time_iso_8601": "2024-11-18T14:32:27.603142Z",
"url": "https://files.pythonhosted.org/packages/5b/98/cc6aac0ee8f0eeab9b7cb004fc8f5835fca87201f294cc2f9e604ccd7a32/gitlab_watchman-3.1.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-11-18 14:32:27",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "PaperMtn",
"github_project": "gitlab-watchman",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "gitlab-watchman"
}
JSON
