# loki_logger_handler
[]()
[]()
[](https://pepy.tech/project/loki_logger_handler)
A logging handler that sends log messages to **(Grafana) Loki** in JSON format.
## Features
* Logs pushed in JSON format by default
* Custom labels definition
* Allows defining *loguru* and *logger* extra keys as labels
* Logger extra keys added automatically as keys into pushed JSON
* Publish in batch of Streams
* Publish logs compressed
## Args
* url (str): The URL of the Loki server.
* labels (dict): A dictionary of labels to attach to each log message.
* label_keys (dict, optional): A dictionary of keys to extract from each log message and use as labels. Defaults to None.
* additional_headers (dict, optional): Additional headers for the Loki request. Defaults to None.
* timeout (int, optional): Timeout interval in seconds to wait before flushing the buffer. Defaults to 10 seconds.
* compressed (bool, optional): Whether to compress the log messages before sending them to Loki. Defaults to True.
* loguru (bool, optional): Whether to use `LoguruFormatter`. Defaults to False.
* default_formatter (logging.Formatter, optional): Formatter for the log records. If not provided,`LoggerFormatter` or `LoguruFormatter` will be used.
* enable_self_errors (bool, optional): Set to True to show Hanlder errors on console. Defaults to False
### Loki 3.0
* enable_structured_loki_metadata (bool, optional): Whether to include structured loki_metadata in the logs. Defaults to False. Only supported for Loki 3.0 and above
* loki_metadata (dict, optional): Default loki_metadata values. Defaults to None. Only supported for Loki 3.0 and above
* loki_metadata_keys (arrray, optional): Specific log record keys to extract as loki_metadata. Only supported for Loki 3.0 and above
## Formatters
* **LoggerFormatter**: Formatter for default python logging implementation
* **LoguruFormatter**: Formatter for Loguru python library
## How to use
### Logger
```python
from loki_logger_handler.loki_logger_handler import LokiLoggerHandler,
import logging
import os
# Set up logging
logger = logging.getLogger("custom_logger")
logger.setLevel(logging.DEBUG)
# Create an instance of the custom handler
custom_handler = LokiLoggerHandler(
url=os.environ["LOKI_URL"],
labels={"application": "Test", "environment": "Develop"},
label_keys={},
timeout=10,
)
# Create an instance of the custom handler
logger.addHandler(custom_handler)
logger.debug("Debug message", extra={'custom_field': 'custom_value'})
```
### Loguru
```python
from loki_logger_handler.loki_logger_handler import LokiLoggerHandler
from loki_logger_handler.formatters.loguru_formatter import LoguruFormatter
from loguru import logger
import os
os.environ["LOKI_URL"]="https://USER:PASSWORD@logs-prod-eu-west-0.grafana.net/loki/api/v1/push"
custom_handler = LokiLoggerHandler(
url=os.environ["LOKI_URL"],
labels={"application": "Test", "environment": "Develop"},
label_keys={},
timeout=10,
default_formatter=LoguruFormatter(),
)
logger.configure(handlers=[{"sink": custom_handler, "serialize": True}])
logger.info(
"Response code {code} HTTP/1.1 GET {url}", code=200, url="https://loki_handler.io"
)
```
## Loki messages samples
### Without extra
```json
{
"message": "Starting service",
"timestamp": 1681638266.542849,
"process": 48906,
"thread": 140704422327936,
"function": "run",
"module": "test",
"name": "__main__"
}
```
### With extra
```json
{
"message": "Response code 200 HTTP/1.1 GET https://loki_handler.io",
"timestamp": 1681638225.877143,
"process": 48870,
"thread": 140704422327936,
"function": "run",
"module": "test",
"name": "__main__",
"code": 200,
"url": "https://loki_handler.io"
}
```
### Exceptions
```json
{
"message": "name 'plan' is not defined",
"timestamp": 1681638284.358464,
"process": 48906,
"thread": 140704422327936,
"function": "run",
"module": "test",
"name": "__main__",
"file": "test.py",
"path": "/test.py",
"line": 39
}
```
## Loki Query Sample
Loki query sample :
```
{environment="Develop"} |= `` | json
```
Filter by level:
```
{environment="Develop", level="INFO"} |= `` | json
```
Filter by extra:
```
{environment="Develop", level="INFO"} |= `` | json | code=`200`
```
## Loki Structured Metadata
Loki structured metadata to include additional context in your logs. This can be useful for filtering and querying logs in Loki.
We can add metadata in 3 ways:
1. Defile static loki_metadata that will be injected into all logs lines
2. Use logger extra options adding metadata inside `loki_metadata` key
3. Use logger `loki_metadata_keys` to move logs keys to loki metadata.
### Example global metadata
```python
from loki_logger_handler.loki_logger_handler import LokiLoggerHandler
import logging
import os
# Set up logging
logger = logging.getLogger("custom_logger")
logger.setLevel(logging.DEBUG)
# Create an instance of the custom handler with structured metadata
custom_handler = LokiLoggerHandler(
url=os.environ["LOKI_URL"],
labels={"application": "Test", "environment": "Develop"},
label_keys={},
timeout=10,
enable_structured_loki_metadata=True,
loki_metadata={"service": "user-service", "version": "1.0.0"}
)
logger.addHandler(custom_handler)
```
In this example, the `loki_metadata` dictionary includes metadata that will be attached to every log message. The `enable_structured_loki_metadata` flag ensures that this metadata is included in the logs.
### Example log metadata
```python
from loki_logger_handler.loki_logger_handler import LokiLoggerHandler
import logging
import os
# Set up logging
logger = logging.getLogger("custom_logger")
logger.setLevel(logging.DEBUG)
# Create an instance of the custom handler with structured metadata
custom_handler = LokiLoggerHandler(
url=os.environ["LOKI_URL"],
labels={"application": "Test", "environment": "Develop"},
label_keys={},
timeout=10,
enable_structured_loki_metadata=True,
loki_metadata={"service": "user-service", "version": "1.0.0"}
)
logger.addHandler(custom_handler)
logger.info("User acction", extra={"loki_metadata": {"user_id": 12345, "operation": "update", "status": "success"}})
```
```python
from loki_logger_handler.loki_logger_handler import LokiLoggerHandler
import logging
import os
# Set up logging
logger = logging.getLogger("custom_logger")
logger.setLevel(logging.DEBUG)
# Create an instance of the custom handler with structured metadata
custom_handler = LokiLoggerHandler(
url=os.environ["LOKI_URL"],
labels={"application": "Test", "environment": "Develop"},
label_keys={},
timeout=10,
enable_structured_loki_metadata=True,
loki_metadata={"service": "user-service", "version": "1.0.0"},
loki_metadata_keys=["trace_id"]
)
logger.addHandler(custom_handler)
logger.info("User acction", extra={"loki_metadata": {"user_id": 12345, "operation": "update", "status": "success"}, "trace_id": "000-000000-0000"})
```
### Querying Logs with Structured Metadata
You can query logs in Loki using the structured metadata.
This query will return all logs where the `service` metadata is set to `user-service`.
```
{application="Test"} |= `` | service="user-service"
```
This query will return all logs where the `user_id` metadata is set to `12345`.
```
{application="Test"} |= `` | user_id="12345"
```
This query will return all logs where the `trace_id` metadata is set to `000-000000-0000`.
```
{application="Test"} |= `` | trace_id="000-000000-0000"
```
## Development Environment: Dev Container
This project uses a **Dev Container** to provide a consistent and reproducible development environment. A Dev Container ensures all team members have the same tools, dependencies, and configurations, avoiding "works on my machine" issues.
---
### **Why Use a Dev Container?**
- **Consistency**: Ensures everyone works in the same environment, regardless of the host OS.
- **Isolation**: Keeps project dependencies separate from your system.
- **Portability**: Easily onboard new developers by setting up the environment with a single command.
- **Pre-configured Tools**: Includes all required tools and dependencies for the project.
---
### **Getting Started with the Dev Container**
To start working with the Dev Container, follow these steps:
#### **Prerequisites**
1. Install [Docker Desktop](https://www.docker.com/products/docker-desktop) (required for running containers).
2. Install [Visual Studio Code (VS Code)](https://code.visualstudio.com/).
3. Install the **Dev Containers** extension in VS Code:
- Go to Extensions (`Ctrl+Shift+X` / `Cmd+Shift+X`) and search for `Dev Containers`.
- Install the extension by Microsoft.
#### **Setup Instructions**
1. Clone the repository
2. Open in VS Code
3. Open the Command Palette (Ctrl+Shift+P / Cmd+Shift+P) and select: **Dev Containers: Reopen in Container**
- VS Code will:
- Pull the Dev Container image.
- Install all dependencies and tools specified.
#### Resources
The loki_logger_handler Dev Container provides the following resources:
- Grafana: Accessible externally at http://localhost:3000.
- Loki: Accessible internally at http://loki:3100/loki/api/v1/push.
You can use this URL in your code as the publish endpoint for logs.
Logs can be viewed and queried via the Grafana interface.
```
os.environ["LOKI_URL"]=http://loki:3100/loki/api/v1/push
````
## License
The MIT License
Raw data
{
"_id": null,
"home_page": "https://github.com/xente/loki-logger-handler",
"name": "loki-logger-handler",
"maintainer": null,
"docs_url": null,
"requires_python": ">=2.7",
"maintainer_email": null,
"keywords": "loki, loguru, logging, logger, handler",
"author": "Xente",
"author_email": null,
"download_url": "https://files.pythonhosted.org/packages/a5/d5/8ba38c0be6a86419c62091e066cc16c164479d47529c69be47827980b4bd/loki_logger_handler-1.1.0.tar.gz",
"platform": null,
"description": "# loki_logger_handler\n\n[]()\n[]()\n[](https://pepy.tech/project/loki_logger_handler)\n\nA logging handler that sends log messages to **(Grafana) Loki** in JSON format.\n\n## Features\n\n* Logs pushed in JSON format by default\n* Custom labels definition\n* Allows defining *loguru* and *logger* extra keys as labels\n* Logger extra keys added automatically as keys into pushed JSON\n* Publish in batch of Streams\n* Publish logs compressed\n\n## Args\n\n* url (str): The URL of the Loki server.\n* labels (dict): A dictionary of labels to attach to each log message.\n* label_keys (dict, optional): A dictionary of keys to extract from each log message and use as labels. Defaults to None.\n* additional_headers (dict, optional): Additional headers for the Loki request. Defaults to None.\n* timeout (int, optional): Timeout interval in seconds to wait before flushing the buffer. Defaults to 10 seconds.\n* compressed (bool, optional): Whether to compress the log messages before sending them to Loki. Defaults to True.\n* loguru (bool, optional): Whether to use `LoguruFormatter`. Defaults to False.\n* default_formatter (logging.Formatter, optional): Formatter for the log records. If not provided,`LoggerFormatter` or `LoguruFormatter` will be used.\n* enable_self_errors (bool, optional): Set to True to show Hanlder errors on console. Defaults to False\n### Loki 3.0 \n* enable_structured_loki_metadata (bool, optional): Whether to include structured loki_metadata in the logs. Defaults to False. Only supported for Loki 3.0 and above\n* loki_metadata (dict, optional): Default loki_metadata values. Defaults to None. Only supported for Loki 3.0 and above\n* loki_metadata_keys (arrray, optional): Specific log record keys to extract as loki_metadata. Only supported for Loki 3.0 and above\n\n## Formatters\n* **LoggerFormatter**: Formatter for default python logging implementation\n* **LoguruFormatter**: Formatter for Loguru python library\n\n## How to use \n\n### Logger\n```python\nfrom loki_logger_handler.loki_logger_handler import LokiLoggerHandler,\nimport logging\nimport os \n\n# Set up logging\nlogger = logging.getLogger(\"custom_logger\")\nlogger.setLevel(logging.DEBUG)\n\n# Create an instance of the custom handler\ncustom_handler = LokiLoggerHandler(\n url=os.environ[\"LOKI_URL\"],\n labels={\"application\": \"Test\", \"environment\": \"Develop\"},\n label_keys={},\n timeout=10,\n)\n# Create an instance of the custom handler\n\nlogger.addHandler(custom_handler)\nlogger.debug(\"Debug message\", extra={'custom_field': 'custom_value'})\n```\n\n\n### Loguru\n\n```python\nfrom loki_logger_handler.loki_logger_handler import LokiLoggerHandler\nfrom loki_logger_handler.formatters.loguru_formatter import LoguruFormatter\nfrom loguru import logger\nimport os \n\nos.environ[\"LOKI_URL\"]=\"https://USER:PASSWORD@logs-prod-eu-west-0.grafana.net/loki/api/v1/push\"\n\ncustom_handler = LokiLoggerHandler(\n url=os.environ[\"LOKI_URL\"],\n labels={\"application\": \"Test\", \"environment\": \"Develop\"},\n label_keys={},\n timeout=10,\n default_formatter=LoguruFormatter(),\n)\nlogger.configure(handlers=[{\"sink\": custom_handler, \"serialize\": True}])\n\nlogger.info(\n \"Response code {code} HTTP/1.1 GET {url}\", code=200, url=\"https://loki_handler.io\"\n)\n```\n\n## Loki messages samples\n\n### Without extra\n\n```json\n{\n \"message\": \"Starting service\",\n \"timestamp\": 1681638266.542849,\n \"process\": 48906,\n \"thread\": 140704422327936,\n \"function\": \"run\",\n \"module\": \"test\",\n \"name\": \"__main__\"\n}\n\n```\n\n### With extra\n\n```json\n{\n \"message\": \"Response code 200 HTTP/1.1 GET https://loki_handler.io\",\n \"timestamp\": 1681638225.877143,\n \"process\": 48870,\n \"thread\": 140704422327936,\n \"function\": \"run\",\n \"module\": \"test\",\n \"name\": \"__main__\",\n \"code\": 200,\n \"url\": \"https://loki_handler.io\"\n}\n```\n\n### Exceptions\n\n```json\n{\n \"message\": \"name 'plan' is not defined\",\n \"timestamp\": 1681638284.358464,\n \"process\": 48906,\n \"thread\": 140704422327936,\n \"function\": \"run\",\n \"module\": \"test\",\n \"name\": \"__main__\",\n \"file\": \"test.py\",\n \"path\": \"/test.py\",\n \"line\": 39\n}\n```\n\n## Loki Query Sample\n\nLoki query sample :\n\n ```\n {environment=\"Develop\"} |= `` | json\n ```\n\nFilter by level:\n\n```\n{environment=\"Develop\", level=\"INFO\"} |= `` | json\n```\nFilter by extra:\n\n```\n{environment=\"Develop\", level=\"INFO\"} |= `` | json | code=`200`\n```\n\n\n## Loki Structured Metadata\n\nLoki structured metadata to include additional context in your logs. This can be useful for filtering and querying logs in Loki.\n\nWe can add metadata in 3 ways:\n\n1. Defile static loki_metadata that will be injected into all logs lines\n2. Use logger extra options adding metadata inside `loki_metadata` key\n3. Use logger `loki_metadata_keys` to move logs keys to loki metadata. \n\n### Example global metadata\n\n```python\nfrom loki_logger_handler.loki_logger_handler import LokiLoggerHandler\nimport logging\nimport os\n\n# Set up logging\nlogger = logging.getLogger(\"custom_logger\")\nlogger.setLevel(logging.DEBUG)\n\n# Create an instance of the custom handler with structured metadata\ncustom_handler = LokiLoggerHandler(\n url=os.environ[\"LOKI_URL\"],\n labels={\"application\": \"Test\", \"environment\": \"Develop\"},\n label_keys={},\n timeout=10,\n enable_structured_loki_metadata=True,\n loki_metadata={\"service\": \"user-service\", \"version\": \"1.0.0\"}\n)\n\nlogger.addHandler(custom_handler)\n\n```\n\nIn this example, the `loki_metadata` dictionary includes metadata that will be attached to every log message. The `enable_structured_loki_metadata` flag ensures that this metadata is included in the logs.\n\n### Example log metadata\n\n\n```python\nfrom loki_logger_handler.loki_logger_handler import LokiLoggerHandler\nimport logging\nimport os\n\n# Set up logging\nlogger = logging.getLogger(\"custom_logger\")\nlogger.setLevel(logging.DEBUG)\n\n# Create an instance of the custom handler with structured metadata\ncustom_handler = LokiLoggerHandler(\n url=os.environ[\"LOKI_URL\"],\n labels={\"application\": \"Test\", \"environment\": \"Develop\"},\n label_keys={},\n timeout=10,\n enable_structured_loki_metadata=True,\n loki_metadata={\"service\": \"user-service\", \"version\": \"1.0.0\"}\n)\n\nlogger.addHandler(custom_handler)\n\nlogger.info(\"User acction\", extra={\"loki_metadata\": {\"user_id\": 12345, \"operation\": \"update\", \"status\": \"success\"}})\n\n```\n\n```python\nfrom loki_logger_handler.loki_logger_handler import LokiLoggerHandler\nimport logging\nimport os\n\n# Set up logging\nlogger = logging.getLogger(\"custom_logger\")\nlogger.setLevel(logging.DEBUG)\n\n# Create an instance of the custom handler with structured metadata\ncustom_handler = LokiLoggerHandler(\n url=os.environ[\"LOKI_URL\"],\n labels={\"application\": \"Test\", \"environment\": \"Develop\"},\n label_keys={},\n timeout=10,\n enable_structured_loki_metadata=True,\n loki_metadata={\"service\": \"user-service\", \"version\": \"1.0.0\"},\n loki_metadata_keys=[\"trace_id\"]\n)\n\nlogger.addHandler(custom_handler)\n\nlogger.info(\"User acction\", extra={\"loki_metadata\": {\"user_id\": 12345, \"operation\": \"update\", \"status\": \"success\"}, \"trace_id\": \"000-000000-0000\"})\n\n```\n\n\n### Querying Logs with Structured Metadata\n\nYou can query logs in Loki using the structured metadata.\n\nThis query will return all logs where the `service` metadata is set to `user-service`.\n\n```\n{application=\"Test\"} |= `` | service=\"user-service\"\n```\n\nThis query will return all logs where the `user_id` metadata is set to `12345`.\n\n```\n{application=\"Test\"} |= `` | user_id=\"12345\"\n```\n\nThis query will return all logs where the `trace_id` metadata is set to `000-000000-0000`.\n\n```\n{application=\"Test\"} |= `` | trace_id=\"000-000000-0000\"\n```\n\n\n## Development Environment: Dev Container\n\nThis project uses a **Dev Container** to provide a consistent and reproducible development environment. A Dev Container ensures all team members have the same tools, dependencies, and configurations, avoiding \"works on my machine\" issues.\n\n---\n\n### **Why Use a Dev Container?**\n\n- **Consistency**: Ensures everyone works in the same environment, regardless of the host OS.\n- **Isolation**: Keeps project dependencies separate from your system.\n- **Portability**: Easily onboard new developers by setting up the environment with a single command.\n- **Pre-configured Tools**: Includes all required tools and dependencies for the project.\n\n---\n\n### **Getting Started with the Dev Container**\n\nTo start working with the Dev Container, follow these steps:\n\n#### **Prerequisites**\n1. Install [Docker Desktop](https://www.docker.com/products/docker-desktop) (required for running containers).\n2. Install [Visual Studio Code (VS Code)](https://code.visualstudio.com/).\n3. Install the **Dev Containers** extension in VS Code:\n - Go to Extensions (`Ctrl+Shift+X` / `Cmd+Shift+X`) and search for `Dev Containers`.\n - Install the extension by Microsoft.\n\n#### **Setup Instructions**\n1. Clone the repository\n2. Open in VS Code\n3. Open the Command Palette (Ctrl+Shift+P / Cmd+Shift+P) and select: **Dev Containers: Reopen in Container**\n\n- VS Code will:\n - Pull the Dev Container image. \n - Install all dependencies and tools specified.\n\n\n#### Resources\n\nThe loki_logger_handler Dev Container provides the following resources:\n\n- Grafana: Accessible externally at http://localhost:3000.\n- Loki: Accessible internally at http://loki:3100/loki/api/v1/push.\nYou can use this URL in your code as the publish endpoint for logs.\nLogs can be viewed and queried via the Grafana interface.\n\n```\nos.environ[\"LOKI_URL\"]=http://loki:3100/loki/api/v1/push\n````\n\n## License\nThe MIT License\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "Handler designed for transmitting logs to Grafana Loki in JSON format.",
"version": "1.1.0",
"project_urls": {
"Homepage": "https://github.com/xente/loki-logger-handler"
},
"split_keywords": [
"loki",
" loguru",
" logging",
" logger",
" handler"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "c7d3852f40a3461e6c5049c08a1a257365df59c55bb4c69946f0d374997721ee",
"md5": "907f6248563649e14b09e45eea9e5502",
"sha256": "0198c6ec0cda01e90a569b2ed2e1bb92d8bbfc19c0f9d47014238d9a0fa5df86"
},
"downloads": -1,
"filename": "loki_logger_handler-1.1.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "907f6248563649e14b09e45eea9e5502",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=2.7",
"size": 16584,
"upload_time": "2024-12-30T18:13:30",
"upload_time_iso_8601": "2024-12-30T18:13:30.128826Z",
"url": "https://files.pythonhosted.org/packages/c7/d3/852f40a3461e6c5049c08a1a257365df59c55bb4c69946f0d374997721ee/loki_logger_handler-1.1.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "a5d58ba38c0be6a86419c62091e066cc16c164479d47529c69be47827980b4bd",
"md5": "1f6a5abaa3c3b0a8da8075f68c80547e",
"sha256": "4ec0cfecaa8ba724f3d7d429cf5a505d5e53ff6ca2cf4e980e4262b12cb980fb"
},
"downloads": -1,
"filename": "loki_logger_handler-1.1.0.tar.gz",
"has_sig": false,
"md5_digest": "1f6a5abaa3c3b0a8da8075f68c80547e",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=2.7",
"size": 15635,
"upload_time": "2024-12-30T18:13:32",
"upload_time_iso_8601": "2024-12-30T18:13:32.617677Z",
"url": "https://files.pythonhosted.org/packages/a5/d5/8ba38c0be6a86419c62091e066cc16c164479d47529c69be47827980b4bd/loki_logger_handler-1.1.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-12-30 18:13:32",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "xente",
"github_project": "loki-logger-handler",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"requirements": [
{
"name": "requests",
"specs": [
[
">=",
"2.27.1"
]
]
},
{
"name": "requests",
"specs": [
[
">=",
"2.31.0"
]
]
},
{
"name": "requests",
"specs": [
[
">=",
"2.32.3"
]
]
}
],
"lcname": "loki-logger-handler"
}
JSON
