# GCP Log Forwarder
GCP Log Forwarder is a Python package that consumes events from a GCP Pub/Sub subscription, processes log files stored in Google Cloud Storage, and forwards the logs either to an HTTP endpoint or writes them to files.
## Features
- Consume object creation events from a GCP Pub/Sub subscription.
- Download and decompress gzipped JSON log files from Google Cloud Storage.
- Forward logs to an HTTP endpoint or write them to files.
- Support for TLS protocol.
- Support for different HTTP authentication methods (Token, API Key).
- Configurable through environment variables.
- Configurable timeout to exit the program if no messages are found for a specified duration.
## Prerequisites
- Python 3.8 or higher
- A GCP project with Pub/Sub and Cloud Storage enabled.
- GCP Service Account with appropriate permissions.
## Installation
### With internet access:
```sh
pip install sap-ecs-gcp-log-forwarder
pip install sap-ecs-gcp-log-forwarder==<version>
```
### Without internet access:
- Navigate to the **Download files** section on **pypi.org** for **sap-ecs-gcp-log-forwarder** and download the latest .whl (wheel) file. The file will typically have a name that includes details about compatibility (e.g., example_package-1.0-py3-none-any.whl).
- Navigate to the directory where your **.whl** file is located using the cd command, or provide the full path to the **.whl** file when running the pip command.
Run the following command:
```sh
pip install /path/to/your_file.whl
```
**Example**:
```sh
pip install /User/Downloads/sap_ecs_gcp_log_forwarder-1.0.2-py3-none-any.whl
```
## Configuration
Set up the following environment variables to configure the GCP Log Forwarder:
- `GOOGLE_APPLICATION_CREDENTIALS`: Path to your service account key JSON file.
- `GCP_PROJECT_ID`: Your GCP project ID.
- `PUBSUB_SUBSCRIPTION`: The name of the Pub/Sub subscription to consume events from.
- `TIMEOUT_DURATION`: The time duration in seconds to wait for messages before exiting.
- `OUTPUT_METHOD`: The method to forward logs (`http` or `files`).
- `HTTP_ENDPOINT`: The HTTP endpoint to forward logs to (required if `OUTPUT_METHOD` is `http`).
- `TLS_CERT_PATH`: Path to the TLS certificate for mutual TLS connections.
- `TLS_KEY_PATH`: Path to the TLS key for mutual TLS connections.
- `AUTH_METHOD`: Authentication method (`token`, `api_key`). Default is `token`.
- **Auth Token**: Set `AUTH_METHOD` to `token` and configure `AUTH_TOKEN`.
- `AUTH_TOKEN`: Bearer/OAuth token for HTTP endpoint authentication.
- **API Key**: Set `AUTH_METHOD` to `api_key` and configure `API_KEY`.
- `API_KEY`: API key for HTTP endpoint API key authentication.
- `OUTPUT_DIR`: The output directory to write log files to (required if `OUTPUT_METHOD` is `files`).
You can set these environment variables in your shell or in a `.env` file.
### Example of setting environment variables in a shell:
```sh
export GOOGLE_APPLICATION_CREDENTIALS="path/to/your/service-account-key.json"
export GCP_PROJECT_ID="your-gcp-project-id"
export PUBSUB_SUBSCRIPTION="your-pubsub-subscription-name"
export TIMEOUT_DURATION=120 # Timeout after 120 seconds of inactivity. DO NOT set for indefinite runs.
# For http
export OUTPUT_METHOD="http"
export HTTP_ENDPOINT="https://your-http-endpoint.com"
# For TLS certs
export TLS_CERT_PATH="/path/to/your/tls_cert.pem"
export TLS_KEY_PATH="/path/to/your/tls_key.pem"
# For Bearer/OAuth Token authentication
export AUTH_METHOD="token"
export AUTH_TOKEN="your_token"
# For API Key authentication
export AUTH_METHOD="api_key"
export API_KEY="your_api_key"
# For file output
export OUTPUT_METHOD="files"
export OUTPUT_DIR="/path/to/your/output/directory/"
```
## Usage
To run the GCP Log Forwarder, use the following command:
```sh
sap-ecs-gcp-log-forwarder
```
This will start the process of consuming events from the Pub/Sub subscription, downloading and processing log files, and forwarding them according to the specified method. The program will exit if no messages are found within the specified timeout duration. If no timeout duration is specified, the program will run indefinitely.
## Things to remember
- If you do not set a **TIMEOUT_DURATION**, the program will run indefinitely.
- The service account in your project needs to have the '**Storage Object Viewer**' permission on the GCS bucket and the '**Pub/Sub Subscriber**' permission on the Pub/Sub subscription for this app to work.
## References
1. [Pub/Sub notifications for Cloud Storage](https://cloud.google.com/storage/docs/pubsub-notifications)
2. [Create service accounts](https://cloud.google.com/iam/docs/service-accounts-create)
3. [Create and delete service account keys](https://cloud.google.com/iam/docs/keys-create-delete)
## License
This application and its source code are licensed under the terms of the SAP Developer License Agreement. See the LICENSE file for more information.
## Release Notes
### 1.0.1
- First proper release!
### 1.0.2
- Updated README with diagrams and instructions on installing the package without access to the internet.
Raw data
{
"_id": null,
"home_page": "https://www.sap.com/",
"name": "sap-ecs-gcp-log-forwarder",
"maintainer": null,
"docs_url": null,
"requires_python": "<4.0,>=3.8",
"maintainer_email": null,
"keywords": "SAP GCP Log Forwarder, SAP Log Forwarder",
"author": "SAP SE",
"author_email": null,
"download_url": null,
"platform": null,
"description": "# GCP Log Forwarder\n\nGCP Log Forwarder is a Python package that consumes events from a GCP Pub/Sub subscription, processes log files stored in Google Cloud Storage, and forwards the logs either to an HTTP endpoint or writes them to files.\n\n\n## Features\n\n- Consume object creation events from a GCP Pub/Sub subscription.\n- Download and decompress gzipped JSON log files from Google Cloud Storage.\n- Forward logs to an HTTP endpoint or write them to files.\n- Support for TLS protocol.\n- Support for different HTTP authentication methods (Token, API Key).\n- Configurable through environment variables.\n- Configurable timeout to exit the program if no messages are found for a specified duration.\n\n## Prerequisites\n\n- Python 3.8 or higher\n- A GCP project with Pub/Sub and Cloud Storage enabled.\n- GCP Service Account with appropriate permissions.\n\n## Installation\n### With internet access:\n```sh\npip install sap-ecs-gcp-log-forwarder\npip install sap-ecs-gcp-log-forwarder==<version>\n```\n\n### Without internet access:\n- Navigate to the **Download files** section on **pypi.org** for **sap-ecs-gcp-log-forwarder** and download the latest .whl (wheel) file. The file will typically have a name that includes details about compatibility (e.g., example_package-1.0-py3-none-any.whl).\n- Navigate to the directory where your **.whl** file is located using the cd command, or provide the full path to the **.whl** file when running the pip command.\n\nRun the following command:\n```sh\npip install /path/to/your_file.whl\n```\n**Example**:\n```sh\npip install /User/Downloads/sap_ecs_gcp_log_forwarder-1.0.2-py3-none-any.whl\n```\n\n## Configuration\n\nSet up the following environment variables to configure the GCP Log Forwarder:\n\n- `GOOGLE_APPLICATION_CREDENTIALS`: Path to your service account key JSON file.\n- `GCP_PROJECT_ID`: Your GCP project ID.\n- `PUBSUB_SUBSCRIPTION`: The name of the Pub/Sub subscription to consume events from.\n- `TIMEOUT_DURATION`: The time duration in seconds to wait for messages before exiting.\n- `OUTPUT_METHOD`: The method to forward logs (`http` or `files`).\n- `HTTP_ENDPOINT`: The HTTP endpoint to forward logs to (required if `OUTPUT_METHOD` is `http`).\n- `TLS_CERT_PATH`: Path to the TLS certificate for mutual TLS connections.\n- `TLS_KEY_PATH`: Path to the TLS key for mutual TLS connections.\n- `AUTH_METHOD`: Authentication method (`token`, `api_key`). Default is `token`.\n - **Auth Token**: Set `AUTH_METHOD` to `token` and configure `AUTH_TOKEN`.\n - `AUTH_TOKEN`: Bearer/OAuth token for HTTP endpoint authentication.\n - **API Key**: Set `AUTH_METHOD` to `api_key` and configure `API_KEY`.\n - `API_KEY`: API key for HTTP endpoint API key authentication.\n- `OUTPUT_DIR`: The output directory to write log files to (required if `OUTPUT_METHOD` is `files`).\n\nYou can set these environment variables in your shell or in a `.env` file.\n\n### Example of setting environment variables in a shell:\n\n```sh\nexport GOOGLE_APPLICATION_CREDENTIALS=\"path/to/your/service-account-key.json\"\nexport GCP_PROJECT_ID=\"your-gcp-project-id\"\nexport PUBSUB_SUBSCRIPTION=\"your-pubsub-subscription-name\"\nexport TIMEOUT_DURATION=120 # Timeout after 120 seconds of inactivity. DO NOT set for indefinite runs.\n\n# For http\nexport OUTPUT_METHOD=\"http\"\nexport HTTP_ENDPOINT=\"https://your-http-endpoint.com\"\n# For TLS certs\nexport TLS_CERT_PATH=\"/path/to/your/tls_cert.pem\"\nexport TLS_KEY_PATH=\"/path/to/your/tls_key.pem\"\n# For Bearer/OAuth Token authentication\nexport AUTH_METHOD=\"token\"\nexport AUTH_TOKEN=\"your_token\"\n# For API Key authentication\nexport AUTH_METHOD=\"api_key\"\nexport API_KEY=\"your_api_key\"\n\n# For file output\nexport OUTPUT_METHOD=\"files\"\nexport OUTPUT_DIR=\"/path/to/your/output/directory/\"\n```\n\n## Usage\n\nTo run the GCP Log Forwarder, use the following command:\n```sh\nsap-ecs-gcp-log-forwarder\n```\nThis will start the process of consuming events from the Pub/Sub subscription, downloading and processing log files, and forwarding them according to the specified method. The program will exit if no messages are found within the specified timeout duration. If no timeout duration is specified, the program will run indefinitely.\n\n## Things to remember\n- If you do not set a **TIMEOUT_DURATION**, the program will run indefinitely.\n- The service account in your project needs to have the '**Storage Object Viewer**' permission on the GCS bucket and the '**Pub/Sub Subscriber**' permission on the Pub/Sub subscription for this app to work.\n\n\n## References\n1. [Pub/Sub notifications for Cloud Storage](https://cloud.google.com/storage/docs/pubsub-notifications)\n2. [Create service accounts](https://cloud.google.com/iam/docs/service-accounts-create)\n3. [Create and delete service account keys](https://cloud.google.com/iam/docs/keys-create-delete)\n\n## License\nThis application and its source code are licensed under the terms of the SAP Developer License Agreement. See the LICENSE file for more information.\n\n## Release Notes\n### 1.0.1\n- First proper release!\n\n### 1.0.2\n- Updated README with diagrams and instructions on installing the package without access to the internet.\n",
"bugtrack_url": null,
"license": "SAP DEVELOPER LICENSE AGREEMENT",
"summary": "A package to consume events from an Google Pub/Sub, process log files, and forward them to a HTTP endpoint or file.",
"version": "1.0.2",
"project_urls": {
"Homepage": "https://www.sap.com/"
},
"split_keywords": [
"sap gcp log forwarder",
" sap log forwarder"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "011410545145fa62edd2e1e977f7728dd1bc621add04c63788db8662bc61c17e",
"md5": "948d23d49404747bd654681316295dd2",
"sha256": "b930a1a0df6699a6dbe1fba92e9260b2d3b9a489d239e84efe43ee1b7e8b1757"
},
"downloads": -1,
"filename": "sap_ecs_gcp_log_forwarder-1.0.2-py3-none-any.whl",
"has_sig": false,
"md5_digest": "948d23d49404747bd654681316295dd2",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": "<4.0,>=3.8",
"size": 12971,
"upload_time": "2024-09-16T23:27:21",
"upload_time_iso_8601": "2024-09-16T23:27:21.723173Z",
"url": "https://files.pythonhosted.org/packages/01/14/10545145fa62edd2e1e977f7728dd1bc621add04c63788db8662bc61c17e/sap_ecs_gcp_log_forwarder-1.0.2-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-09-16 23:27:21",
"github": false,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"lcname": "sap-ecs-gcp-log-forwarder"
}
JSON