| Name | beametrics JSON |
| Version |
0.3.0
JSON |
| download |
| home_page | None |
| Summary | A streaming pipeline that transforms PubSub messages into metrics using Apache Beam |
| upload_time | 2024-12-31 10:31:35 |
| maintainer | None |
| docs_url | None |
| author | kesompochy |
| requires_python | <4.0,>=3.12 |
| license | MIT |
| keywords |
apache beam
streaming
|
| VCS |
|
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
# Beametrics
Let your logs be metrics in real-time with Apache Beam.
Beametrics transfers structured messages from a queue into metrics in real-time. Primarily designed to work with Cloud Pub/Sub to export metrics to Cloud Monitoring.
## Usage
### Direct Runner
```bash
$ python -m beametrics.main \
--project=YOUR_PROJECT_ID \
--subscription=projects/YOUR_PROJECT_ID/subscriptions/YOUR_SUBSCRIPTION \
--metric-name=YOUR_METRIC_NAME \
--metric-labels='{"LABEL": "HOGE"}' \
--filter-conditions='[{"field": "user_agent", "value": "dummy_data", "operator": "equals"}]' \
--runner=DirectRunner \
--metric-type=count \
--export-type=monitoring
```
### Dataflow Runner
#### 1. Build Docker image
```bash
$ docker build -t LOCATION-docker.pkg.dev/PROJECT_ID/REPOSITORY/beametrics:latest .
```
#### 2. Push Docker image to Artifact Registry
```bash
$ docker push LOCATION-docker.pkg.dev/PROJECT_ID/REPOSITORY/beametrics:latest
```
#### 3. Build Dataflow Flex Template
```bash
$ gcloud dataflow flex-template build gs://BUCKET/beametrics.json \
--image "LOCATION-docker.pkg.dev/PROJECT_ID/REPOSITORY/beametrics:latest" \
--sdk-language "PYTHON" \
--metadata-file "metadata.json"
```
#### 4. Run Dataflow job
```bash
$ cat flags.yaml
--parameters:
project-id: YOUR_PROJECT_ID
subscription: projects/YOUR_PROJECT_ID/subscriptions/YOUR_SUBSCRIPTION
metric-name: YOUR_METRIC_NAME
metric-labels: '{"LABEL": "HOGE"}'
filter-conditions: '[{"field":"user_agent","value":"dummy_data","operator":"equals"}]'
metric-type: count
window-size: "120"
$ gcloud dataflow flex-template run "beametrics-job-$(date +%Y%m%d-%H%M%S)" \
--template-file-gcs-location gs://BUCKET/beametrics.json \
--region REGION \
--flags-file=flags.yaml
```
##### As a Dataflow Flex Template with external config file
```bash
$ cat flags_external_config.yaml
--parameters:
project-id: YOUR_PROJECT_ID
subscription: projects/YOUR_PROJECT_ID/subscriptions/YOUR_SUBSCRIPTION
window-size: "60"
config: gs://YOUR_BUCKET/YOUR_CONFIG_FILE.yaml
$ cat YOUR_CONFIG_FILE.yaml
metrics:
- name: beametrics-test-1
labels:
LABEL: HOGE1
dynamic_labels:
label_key: label_value
filter-conditions:
- field: user_agent
value: dummy_data
operator: equals
type: count
export_type: google-cloud-monitoring
- name: beametrics-test-2
labels:
LABEL: HOGE2
dynamic_labels:
label_key: label_value
filter-conditions:
- field: user_agent
value: dummy_data
operator: equals
type: count
export_type: local
$ gcloud dataflow flex-template run "beametrics-job-$(date +%Y%m%d-%H%M%S)" \
--template-file-gcs-location gs://BUCKET/beametrics.json \
--region REGION \
--flags-file=flags_external_config.yaml
```
Raw data
{
"_id": null,
"home_page": null,
"name": "beametrics",
"maintainer": null,
"docs_url": null,
"requires_python": "<4.0,>=3.12",
"maintainer_email": null,
"keywords": "apache beam, streaming",
"author": "kesompochy",
"author_email": null,
"download_url": "https://files.pythonhosted.org/packages/cc/fd/772a73c27711a6b822befd1f20738002971d11a17903f1117192efa1ab2b/beametrics-0.3.0.tar.gz",
"platform": null,
"description": "# Beametrics\n\nLet your logs be metrics in real-time with Apache Beam.\n\nBeametrics transfers structured messages from a queue into metrics in real-time. Primarily designed to work with Cloud Pub/Sub to export metrics to Cloud Monitoring.\n\n## Usage\n\n### Direct Runner\n\n```bash\n$ python -m beametrics.main \\\n --project=YOUR_PROJECT_ID \\\n --subscription=projects/YOUR_PROJECT_ID/subscriptions/YOUR_SUBSCRIPTION \\\n --metric-name=YOUR_METRIC_NAME \\\n --metric-labels='{\"LABEL\": \"HOGE\"}' \\\n --filter-conditions='[{\"field\": \"user_agent\", \"value\": \"dummy_data\", \"operator\": \"equals\"}]' \\\n --runner=DirectRunner \\\n --metric-type=count \\\n --export-type=monitoring\n```\n\n### Dataflow Runner\n\n#### 1. Build Docker image\n\n```bash\n$ docker build -t LOCATION-docker.pkg.dev/PROJECT_ID/REPOSITORY/beametrics:latest .\n```\n\n#### 2. Push Docker image to Artifact Registry\n\n```bash\n$ docker push LOCATION-docker.pkg.dev/PROJECT_ID/REPOSITORY/beametrics:latest\n```\n\n#### 3. Build Dataflow Flex Template\n\n```bash\n$ gcloud dataflow flex-template build gs://BUCKET/beametrics.json \\\n--image \"LOCATION-docker.pkg.dev/PROJECT_ID/REPOSITORY/beametrics:latest\" \\\n--sdk-language \"PYTHON\" \\\n--metadata-file \"metadata.json\"\n```\n\n#### 4. Run Dataflow job\n\n```bash\n$ cat flags.yaml\n--parameters:\n project-id: YOUR_PROJECT_ID\n subscription: projects/YOUR_PROJECT_ID/subscriptions/YOUR_SUBSCRIPTION\n metric-name: YOUR_METRIC_NAME\n metric-labels: '{\"LABEL\": \"HOGE\"}'\n filter-conditions: '[{\"field\":\"user_agent\",\"value\":\"dummy_data\",\"operator\":\"equals\"}]'\n metric-type: count\n window-size: \"120\"\n$ gcloud dataflow flex-template run \"beametrics-job-$(date +%Y%m%d-%H%M%S)\" \\\n--template-file-gcs-location gs://BUCKET/beametrics.json \\\n--region REGION \\\n--flags-file=flags.yaml\n```\n\n##### As a Dataflow Flex Template with external config file\n\n```bash\n $ cat flags_external_config.yaml\n--parameters:\n project-id: YOUR_PROJECT_ID\n subscription: projects/YOUR_PROJECT_ID/subscriptions/YOUR_SUBSCRIPTION\n window-size: \"60\"\n config: gs://YOUR_BUCKET/YOUR_CONFIG_FILE.yaml\n$ cat YOUR_CONFIG_FILE.yaml\nmetrics:\n - name: beametrics-test-1\n labels:\n LABEL: HOGE1\n dynamic_labels:\n label_key: label_value\n filter-conditions:\n - field: user_agent\n value: dummy_data\n operator: equals\n type: count\n export_type: google-cloud-monitoring\n - name: beametrics-test-2\n labels:\n LABEL: HOGE2\n dynamic_labels:\n label_key: label_value\n filter-conditions:\n - field: user_agent\n value: dummy_data\n operator: equals\n type: count\n export_type: local\n$ gcloud dataflow flex-template run \"beametrics-job-$(date +%Y%m%d-%H%M%S)\" \\\n--template-file-gcs-location gs://BUCKET/beametrics.json \\\n--region REGION \\\n--flags-file=flags_external_config.yaml\n```\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "A streaming pipeline that transforms PubSub messages into metrics using Apache Beam",
"version": "0.3.0",
"project_urls": null,
"split_keywords": [
"apache beam",
" streaming"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "515604fc774adc69fd5cbae8170b6fecc80bf3fa07b8aef2dfd2ce8159ecce00",
"md5": "2ff855728d4d0af53d680b029ccfc2a3",
"sha256": "8248b1a860093a33615a350c1e7bf472b41aad22c5692b3cfe69c8a5ee22b3be"
},
"downloads": -1,
"filename": "beametrics-0.3.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "2ff855728d4d0af53d680b029ccfc2a3",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": "<4.0,>=3.12",
"size": 11425,
"upload_time": "2024-12-31T10:31:32",
"upload_time_iso_8601": "2024-12-31T10:31:32.917046Z",
"url": "https://files.pythonhosted.org/packages/51/56/04fc774adc69fd5cbae8170b6fecc80bf3fa07b8aef2dfd2ce8159ecce00/beametrics-0.3.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "ccfd772a73c27711a6b822befd1f20738002971d11a17903f1117192efa1ab2b",
"md5": "0f5f40929b33b7acd6e6b8c883fc2758",
"sha256": "c22e577eb35cb663f0f4d609fbcfb65d6df07cbff26b4931b7f92be738ad1936"
},
"downloads": -1,
"filename": "beametrics-0.3.0.tar.gz",
"has_sig": false,
"md5_digest": "0f5f40929b33b7acd6e6b8c883fc2758",
"packagetype": "sdist",
"python_version": "source",
"requires_python": "<4.0,>=3.12",
"size": 9529,
"upload_time": "2024-12-31T10:31:35",
"upload_time_iso_8601": "2024-12-31T10:31:35.039864Z",
"url": "https://files.pythonhosted.org/packages/cc/fd/772a73c27711a6b822befd1f20738002971d11a17903f1117192efa1ab2b/beametrics-0.3.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-12-31 10:31:35",
"github": false,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"lcname": "beametrics"
}
