# Cloudflare Web Application Firewall downloader
[](https://badge.fury.io/py/waf-downloader)
[](https://pypi.org/project/waf-downloader/)
[](LICENSE)
A library, CLI, and docker image that downloads Cloudflare WAF logs for a specified zone and time range.
## Schema
See [src/waf_logs/resources/db/](./src/waf_logs/resources/db) for a list of schemas that are auto-applied at start. This can be disabled by passing `--ensure_schema False`.
## Quickstart
The project is published at <https://pypi.org/project/waf-downloader/>.
Install it via:
```shell
pip install waf-downloader
# or alternatively, directly from git
pip install "git+https://github.com/MihaiBojin/waf-downloader@main"
```
Or with Docker:
```shell
docker pull docker.io/mihaibojin/waf-downloader:latest
```
The list of published images can be found at:
<https://github.com/MihaiBojin/waf-downloader/pkgs/container/waf-downloader>
and
<https://hub.docker.com/repository/docker/mihaibojin/waf-downloader/tags>
## Or with Helm
See [charts/waf-downloader/README.md](./charts/waf-downloader/README.md) for more details.
## Development
### Build and run with Docker
Define secrets in an `.env` file (do not quote values):
```properties
CLOUDFLARE_API_TOKEN=...
DB_CONN_STR=...
```
The Cloudflare token is required (see [required permissions](https://developers.cloudflare.com/analytics/graphql-api/getting-started/authentication/api-token-auth/)),
but the connection string is optional. If skipped, it will result in logs being printed to stdout.
IMPORTANT: This project uses [taskfile.dev](https://taskfile.dev/installation/),
which you will need to install for running the following commands:
```shell
# Build
task docker-build
# Load all logs in zone, starting 5 minutes prior
task docker-run -- --zone_id zoneid1 --start_minutes_ago 5
# And alternatively, only output the logs
task docker-run -- --zone_id zoneid1 --start_minutes_ago 5 2>/dev/null
# Do not specify a start time, relying on a starting timestamp stored in the database
# If a timestamp is not found in the database, or specified with --start_minutes_ago, the downloader will start 5 minutes prior
# This functionality makes it easy to run waf-downloader as a cron job
# NOTE: specifying --start_minutes_ago will always override the timestamp stored in the database, causing potential gaps in the data
task docker-run -- --zone_id zoneid1
# Do not exit and keep downloading new logs forever
# These will be recent up to the last minute
task docker-run -- --zone_id zoneid1 --follow
# Multiple zones can be specified via a comma-separated string
task docker-run -- --zone_id zoneid1,zoneid2,zoneid3,etc
# Or by repeating the flag
task docker-run -- --zone_id zoneid1 --zone_id zoneid2 --zone_id zoneid3 ...
```
## Publishing to PyPI
### GitHub-based version publishing
The simplest way to publish a new version (if you have committer rights) is to tag a commit and push it to the repo:
```shell
# At a certain commit, ideally after merging a PR to main
git tag v0.1.x
git push origin v0.1.x
```
A [GitHub Action](https://github.com/MihaiBojin/waf-downloader/actions) will run, build the library and publish it to the PyPI repositories.
### Manual publish
These steps can also be performed locally. For these commands to work, you will need to export two environment variables (or define them in `.env`):
```shell
export TESTPYPI_PASSWORD=... # token for https://test.pypi.org/legacy/
export PYPI_PASSWORD=... # token for https://upload.pypi.org/legacy/
```
First, publish to the test repo and inspect the package:
```shell
task publish-test
```
If correct, distribute the wheel to the PyPI index:
```shell
task publish
```
Verify the distributed code
```shell
task publish-verify
```
## Cloudflare WAF documentation
- <https://developers.cloudflare.com/analytics/graphql-api/tutorials/querying-firewall-events>
- <https://developers.cloudflare.com/analytics/graphql-api/tutorials/export-graphql-to-csv/>
Raw data
{
"_id": null,
"home_page": null,
"name": "waf-downloader",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.9",
"maintainer_email": null,
"keywords": "application, cloudflare, download, firewall, logs, waf, web",
"author": null,
"author_email": "Mihai Bojin <557584+MihaiBojin@users.noreply.github.com>",
"download_url": "https://files.pythonhosted.org/packages/c0/b7/a96db3f77fbe361001263595a4dcab05fb347674fe16c6c05b30e0442820/waf_downloader-0.2.24.tar.gz",
"platform": null,
"description": "# Cloudflare Web Application Firewall downloader\n\n\n[](https://badge.fury.io/py/waf-downloader)\n[](https://pypi.org/project/waf-downloader/)\n[](LICENSE)\n\nA library, CLI, and docker image that downloads Cloudflare WAF logs for a specified zone and time range.\n\n\n\n## Schema\n\nSee [src/waf_logs/resources/db/](./src/waf_logs/resources/db) for a list of schemas that are auto-applied at start. This can be disabled by passing `--ensure_schema False`.\n\n## Quickstart\n\nThe project is published at <https://pypi.org/project/waf-downloader/>.\n\nInstall it via:\n\n```shell\npip install waf-downloader\n\n# or alternatively, directly from git\npip install \"git+https://github.com/MihaiBojin/waf-downloader@main\"\n```\n\nOr with Docker:\n\n```shell\ndocker pull docker.io/mihaibojin/waf-downloader:latest\n```\n\nThe list of published images can be found at:\n<https://github.com/MihaiBojin/waf-downloader/pkgs/container/waf-downloader>\n\nand\n\n<https://hub.docker.com/repository/docker/mihaibojin/waf-downloader/tags>\n\n## Or with Helm\n\nSee [charts/waf-downloader/README.md](./charts/waf-downloader/README.md) for more details.\n\n## Development\n\n### Build and run with Docker\n\nDefine secrets in an `.env` file (do not quote values):\n\n```properties\nCLOUDFLARE_API_TOKEN=...\nDB_CONN_STR=...\n```\n\nThe Cloudflare token is required (see [required permissions](https://developers.cloudflare.com/analytics/graphql-api/getting-started/authentication/api-token-auth/)),\nbut the connection string is optional. If skipped, it will result in logs being printed to stdout.\n\nIMPORTANT: This project uses [taskfile.dev](https://taskfile.dev/installation/),\nwhich you will need to install for running the following commands:\n\n```shell\n# Build\ntask docker-build\n\n# Load all logs in zone, starting 5 minutes prior\ntask docker-run -- --zone_id zoneid1 --start_minutes_ago 5\n\n# And alternatively, only output the logs\ntask docker-run -- --zone_id zoneid1 --start_minutes_ago 5 2>/dev/null\n\n# Do not specify a start time, relying on a starting timestamp stored in the database\n# If a timestamp is not found in the database, or specified with --start_minutes_ago, the downloader will start 5 minutes prior\n# This functionality makes it easy to run waf-downloader as a cron job\n# NOTE: specifying --start_minutes_ago will always override the timestamp stored in the database, causing potential gaps in the data\ntask docker-run -- --zone_id zoneid1\n\n# Do not exit and keep downloading new logs forever\n# These will be recent up to the last minute\ntask docker-run -- --zone_id zoneid1 --follow\n\n# Multiple zones can be specified via a comma-separated string\ntask docker-run -- --zone_id zoneid1,zoneid2,zoneid3,etc\n\n# Or by repeating the flag\ntask docker-run -- --zone_id zoneid1 --zone_id zoneid2 --zone_id zoneid3 ...\n```\n\n## Publishing to PyPI\n\n### GitHub-based version publishing\n\nThe simplest way to publish a new version (if you have committer rights) is to tag a commit and push it to the repo:\n\n```shell\n# At a certain commit, ideally after merging a PR to main\ngit tag v0.1.x\ngit push origin v0.1.x\n```\n\nA [GitHub Action](https://github.com/MihaiBojin/waf-downloader/actions) will run, build the library and publish it to the PyPI repositories.\n\n### Manual publish\n\nThese steps can also be performed locally. For these commands to work, you will need to export two environment variables (or define them in `.env`):\n\n```shell\nexport TESTPYPI_PASSWORD=... # token for https://test.pypi.org/legacy/\nexport PYPI_PASSWORD=... # token for https://upload.pypi.org/legacy/\n```\n\nFirst, publish to the test repo and inspect the package:\n\n```shell\ntask publish-test\n```\n\nIf correct, distribute the wheel to the PyPI index:\n\n```shell\ntask publish\n```\n\nVerify the distributed code\n\n```shell\ntask publish-verify\n```\n\n## Cloudflare WAF documentation\n\n- <https://developers.cloudflare.com/analytics/graphql-api/tutorials/querying-firewall-events>\n- <https://developers.cloudflare.com/analytics/graphql-api/tutorials/export-graphql-to-csv/>\n",
"bugtrack_url": null,
"license": "Apache-2.0",
"summary": "Web Application Firewall logs downloader.",
"version": "0.2.24",
"project_urls": {
"Bug reports": "https://github.com/MihaiBojin/waf-downloader/issues/new",
"Documentation": "https://github.com/MihaiBojin/waf-downloader/blob/main/python/README.md",
"Source": "https://github.com/MihaiBojin/waf-downloader"
},
"split_keywords": [
"application",
" cloudflare",
" download",
" firewall",
" logs",
" waf",
" web"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "f11a78ce615883552221485eae9fadd0f53cdfd574797409976ec1ad7cfad64c",
"md5": "f9ae8a45b05f1740125b9da42a162287",
"sha256": "362e5bbda4954213969f33e07a4d73401752d299120f5f02bb60e4b111845758"
},
"downloads": -1,
"filename": "waf_downloader-0.2.24-py3-none-any.whl",
"has_sig": false,
"md5_digest": "f9ae8a45b05f1740125b9da42a162287",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.9",
"size": 21424,
"upload_time": "2024-12-03T02:17:24",
"upload_time_iso_8601": "2024-12-03T02:17:24.561981Z",
"url": "https://files.pythonhosted.org/packages/f1/1a/78ce615883552221485eae9fadd0f53cdfd574797409976ec1ad7cfad64c/waf_downloader-0.2.24-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "c0b7a96db3f77fbe361001263595a4dcab05fb347674fe16c6c05b30e0442820",
"md5": "84f1a8d6c3109d7651324c79e4ba4b0c",
"sha256": "b7a6d1d1d132e2c71441a60183b4f81fa1fb54fc141798ccfb17674d6d2a9159"
},
"downloads": -1,
"filename": "waf_downloader-0.2.24.tar.gz",
"has_sig": false,
"md5_digest": "84f1a8d6c3109d7651324c79e4ba4b0c",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.9",
"size": 226296,
"upload_time": "2024-12-03T02:17:26",
"upload_time_iso_8601": "2024-12-03T02:17:26.342106Z",
"url": "https://files.pythonhosted.org/packages/c0/b7/a96db3f77fbe361001263595a4dcab05fb347674fe16c6c05b30e0442820/waf_downloader-0.2.24.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-12-03 02:17:26",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "MihaiBojin",
"github_project": "waf-downloader",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"requirements": [
{
"name": "more_itertools",
"specs": [
[
"==",
"10.5.0"
]
]
},
{
"name": "psycopg-binary",
"specs": [
[
"==",
"3.2.3"
]
]
},
{
"name": "psycopg",
"specs": [
[
"==",
"3.2.3"
]
]
},
{
"name": "psycopg_pool",
"specs": [
[
"==",
"3.2.4"
]
]
},
{
"name": "requests",
"specs": [
[
"==",
"2.32.3"
]
]
}
],
"lcname": "waf-downloader"
}
JSON
