| Name | beametrics JSON |
| Version |
0.2.5
JSON |
| download |
| home_page | None |
| Summary | A streaming pipeline that transforms PubSub messages into metrics using Apache Beam |
| upload_time | 2024-11-10 09:30:33 |
| maintainer | None |
| docs_url | None |
| author | kesompochy |
| requires_python | <4.0,>=3.12 |
| license | MIT |
| keywords |
apache beam
streaming
|
| VCS |
|
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
# Beametrics
Let your logs be metrics in real-time with Apache Beam.
Beametrics transfers structured messages from a queue into metrics in real-time. Primarily designed to work with Cloud Pub/Sub to export metrics to Cloud Monitoring.
## Usage
### Direct Runner
```bash
$ python -m beametrics.main \
--project=YOUR_PROJECT_ID \
--subscription=projects/YOUR_PROJECT_ID/subscriptions/YOUR_SUBSCRIPTION \
--metric-name=YOUR_METRIC_NAME \
--metric-labels='{"LABEL": "HOGE"}' \
--filter-conditions='[{"field": "user_agent", "value": "dummy_data", "operator": "equals"}]' \
--runner=DirectRunner \
--metric-type=count \
--export-type=monitoring
```
### Dataflow Runner
#### 1. Build Docker image
```bash
$ docker build -t LOCATION-docker.pkg.dev/PROJECT_ID/REPOSITORY/beametrics:latest .
```
#### 2. Push Docker image to Artifact Registry
```bash
$ docker push LOCATION-docker.pkg.dev/PROJECT_ID/REPOSITORY/beametrics:latest
```
#### 3. Build Dataflow Flex Template
```bash
$ gcloud dataflow flex-template build gs://BUCKET/beametrics.json \
--image "LOCATION-docker.pkg.dev/PROJECT_ID/REPOSITORY/beametrics:latest" \
--sdk-language "PYTHON" \
--metadata-file "metadata.json"
```
#### 4. Run Dataflow job
```bash
$ cat flags.yaml
--parameters:
project-id: YOUR_PROJECT_ID
subscription: projects/YOUR_PROJECT_ID/subscriptions/YOUR_SUBSCRIPTION
metric-name: YOUR_METRIC_NAME
metric-labels: '{"LABEL": "HOGE"}'
filter-conditions: '[{"field":"user_agent","value":"dummy_data","operator":"equals"}]'
metric-type: count
window-size: "120"
$ gcloud dataflow flex-template run "beametrics-job-$(date +%Y%m%d-%H%M%S)" \
--template-file-gcs-location gs://BUCKET/beametrics.json \
--region REGION \
--flags-file=flags.yaml
```
Raw data
{
"_id": null,
"home_page": null,
"name": "beametrics",
"maintainer": null,
"docs_url": null,
"requires_python": "<4.0,>=3.12",
"maintainer_email": null,
"keywords": "apache beam, streaming",
"author": "kesompochy",
"author_email": null,
"download_url": "https://files.pythonhosted.org/packages/15/ed/7a95d373a99c20d22224305de1be34f66be09edb217e0ac585418eba30bd/beametrics-0.2.5.tar.gz",
"platform": null,
"description": "# Beametrics\n\nLet your logs be metrics in real-time with Apache Beam.\n\nBeametrics transfers structured messages from a queue into metrics in real-time. Primarily designed to work with Cloud Pub/Sub to export metrics to Cloud Monitoring.\n\n## Usage\n\n### Direct Runner\n\n```bash\n$ python -m beametrics.main \\\n --project=YOUR_PROJECT_ID \\\n --subscription=projects/YOUR_PROJECT_ID/subscriptions/YOUR_SUBSCRIPTION \\\n --metric-name=YOUR_METRIC_NAME \\\n --metric-labels='{\"LABEL\": \"HOGE\"}' \\\n --filter-conditions='[{\"field\": \"user_agent\", \"value\": \"dummy_data\", \"operator\": \"equals\"}]' \\\n --runner=DirectRunner \\\n --metric-type=count \\\n --export-type=monitoring\n```\n\n### Dataflow Runner\n\n#### 1. Build Docker image\n\n```bash\n$ docker build -t LOCATION-docker.pkg.dev/PROJECT_ID/REPOSITORY/beametrics:latest .\n```\n\n#### 2. Push Docker image to Artifact Registry\n\n```bash\n$ docker push LOCATION-docker.pkg.dev/PROJECT_ID/REPOSITORY/beametrics:latest\n```\n\n#### 3. Build Dataflow Flex Template\n\n```bash\n$ gcloud dataflow flex-template build gs://BUCKET/beametrics.json \\\n--image \"LOCATION-docker.pkg.dev/PROJECT_ID/REPOSITORY/beametrics:latest\" \\\n--sdk-language \"PYTHON\" \\\n--metadata-file \"metadata.json\"\n```\n\n#### 4. Run Dataflow job\n\n```bash\n$ cat flags.yaml\n--parameters:\n project-id: YOUR_PROJECT_ID\n subscription: projects/YOUR_PROJECT_ID/subscriptions/YOUR_SUBSCRIPTION\n metric-name: YOUR_METRIC_NAME\n metric-labels: '{\"LABEL\": \"HOGE\"}'\n filter-conditions: '[{\"field\":\"user_agent\",\"value\":\"dummy_data\",\"operator\":\"equals\"}]'\n metric-type: count\n window-size: \"120\"\n$ gcloud dataflow flex-template run \"beametrics-job-$(date +%Y%m%d-%H%M%S)\" \\\n--template-file-gcs-location gs://BUCKET/beametrics.json \\\n--region REGION \\\n--flags-file=flags.yaml\n```\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "A streaming pipeline that transforms PubSub messages into metrics using Apache Beam",
"version": "0.2.5",
"project_urls": null,
"split_keywords": [
"apache beam",
" streaming"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "4ee09d06c2c877c2141b891742b4b2d450770eeffad62ece9693fdbd40a7444b",
"md5": "1f1f4d1b3450c71862bd4709fe80c53d",
"sha256": "c0da94565bacf366099e8941606f1bfc3191ddd9fb06ff01b6c2e1506bbbe974"
},
"downloads": -1,
"filename": "beametrics-0.2.5-py3-none-any.whl",
"has_sig": false,
"md5_digest": "1f1f4d1b3450c71862bd4709fe80c53d",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": "<4.0,>=3.12",
"size": 9467,
"upload_time": "2024-11-10T09:30:31",
"upload_time_iso_8601": "2024-11-10T09:30:31.628595Z",
"url": "https://files.pythonhosted.org/packages/4e/e0/9d06c2c877c2141b891742b4b2d450770eeffad62ece9693fdbd40a7444b/beametrics-0.2.5-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "15ed7a95d373a99c20d22224305de1be34f66be09edb217e0ac585418eba30bd",
"md5": "854bef4e0b33603df4b41d5d188e68e9",
"sha256": "23c6e459e24a7f66d8f52a1073166d14c202de0bc23ae01c5385deaea61f536f"
},
"downloads": -1,
"filename": "beametrics-0.2.5.tar.gz",
"has_sig": false,
"md5_digest": "854bef4e0b33603df4b41d5d188e68e9",
"packagetype": "sdist",
"python_version": "source",
"requires_python": "<4.0,>=3.12",
"size": 7757,
"upload_time": "2024-11-10T09:30:33",
"upload_time_iso_8601": "2024-11-10T09:30:33.059719Z",
"url": "https://files.pythonhosted.org/packages/15/ed/7a95d373a99c20d22224305de1be34f66be09edb217e0ac585418eba30bd/beametrics-0.2.5.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-11-10 09:30:33",
"github": false,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"lcname": "beametrics"
}
