<h1 align="center" style="border-bottom: 0">
🪵<br>
Troncos <br/>
</h1>
<p align="center">
<em>
Collection of Python logging, tracing and profiling tools
</em>
<br>
<a href="https://github.com/kolonialno/troncos/actions?workflow=CI">
<img src="https://github.com/kolonialno/troncos/actions/workflows/ci.yml/badge.svg" alt="CI status">
</a>
<a href="https://pypi.python.org/pypi/troncos">
<img src="https://img.shields.io/pypi/v/troncos.svg" alt="Troncos version">
</a>
<img src="https://img.shields.io/pypi/pyversions/troncos" alt="Supported Python version">
<a href="https://github.com/kolonialno/troncos/blob/master/LICENSE">
<img src="https://img.shields.io/github/license/kolonialno/troncos.svg" alt="licenece">
</a>
</p>
<!-- TOC -->
- [Etymology](#etymology)
- [Installation](#installation)
- [Tracing](#tracing)
- [Profiling](#profiling)
- [Logging](#logging)
<!-- TOC -->
## Etymology
"Troncos" is the plural of the spanish word "Tronco", which translates to "trunk" or "log".
## Installation
```console
# With pip
pip install troncos
```
## Tracing
Troncos is designed to take advantage of `ddtrace` made by DataDog.
The ddtrace docs can be found [here](https://ddtrace.readthedocs.io/en/stable/).
[Best practices for traces](https://grafana.com/docs/tempo/latest/operations/best-practices/#naming-conventions-for-span-and-resource-attributes) is a good guide to get started.
### Span vs resource attributes
- A `span attribute` is a key/value pair that provides context for its span.
- A `resource attribute` is a key/value pair that describes the context of how the span was collected.
For more information, read the [Attribute and Resource](https://opentelemetry.io/docs/specs/otel/overview/) sections in the OpenTelemetry specification.
### Enabling the tracer
Configure ddtrace as usual and run `configure_tracer` to send spans to Tempo.
This is typically done in `settings.py` of you want to profile a Django application,
or in `__init__.py` in the root project package.
`TRACE_HOST` is usually the host IP of the K8s pod, `TRACE_PORT` is usually 4318
when the Grafana agent is used to collect spans using HTTP.
```python
import ddtrace
from troncos.tracing import configure_tracer
# Configure tracer as described in the ddtrace docs.
ddtrace.config.django["service_name"] = 'SERVICE_NAME'
# These are added as span attributes
ddtrace.tracer.set_tags(
tags={
"key": "value",
}
)
# Patch third-party modules
ddtrace.patch_all()
# Configure the ddtrace tracer to send traces to Tempo.
TRACE_HOST = "127.0.0.1" # Usually obtained from env variables.
TRACE_PORT = "4318"
configure_tracer(
enabled=False, # Set to True when TRACE_HOST is configured.
service_name='SERVICE_NAME',
endpoint=f"http://{TRACE_HOST}:{TRACE_PORT}/v1/traces",
resource_attributes={
"app": "app",
"component": "component",
"role": "role",
"tenant": "tenant",
"owner": "owner",
"version": "version",
}
)
```
ddtrace also uses env variables to configure the service name, environment and version etc.
Add the following environment variables to your application.
```console
DD_ENV="{{ environment }}"
DD_SERVICE="{{ app }}"
DD_VERSION="{{ version }}"
# tracecontext/w3c is usually used to propagate distributed traces across services.
DD_TRACE_PROPAGATION_STYLE_EXTRACT="tracecontext"
DD_TRACE_PROPAGATION_STYLE_INJECT="tracecontext"
```
#### Debugging during development
By setting the environment variable `OTEL_TRACE_DEBUG=True` you will enable traces
to be printed to `stdout` via the ConsoleSpanExporter as well as through http/grpc.
Also specifying `OTEL_TRACE_DEBUG_FILE=/some/file/path` will output traces to the
specified file path instead of the console/stdout.
### Using the GRPC span exporter
Using the GRPC span exporter gives you significant performance gains.
If you are running a critical service with high load in production,
we recommend using GRPC.
`TRACE_PORT` is usually 4317 when the Grafana agent is used to collect
spans using GRPC.
```console
poetry add troncos -E grpc
```
or
```toml
[tool.poetry.dependencies]
troncos = {version="?", extras = ["grpc"]}
```
```python
from troncos.tracing import configure_tracer, Exporter
TRACE_HOST = "127.0.0.1" # Usually obtained from env variables.
TRACE_PORT = "4317"
configure_tracer(
enabled=False, # Set to True when TRACE_HOST is configured.
service_name='SERVICE_NAME',
endpoint=f"http://{TRACE_HOST}:{TRACE_PORT}",
exporter=Exporter.GRPC
)
```
### Setting headers for the exporter
```python
from troncos.tracing import configure_tracer, Exporter, ExporterType
TRACE_HOST = "127.0.0.1" # Usually obtained from env variables.
TRACE_PORT = "4317"
configure_tracer(
enabled=False, # Set to True when TRACE_HOST is configured.
service_name='SERVICE_NAME',
endpoint=f"http://{TRACE_HOST}:{TRACE_PORT}",
exporter=Exporter(ExporterType.GRPC, headers={"my", "header"}),
)
```
### Instrument your code
Manual instrumentation of your code is described in the [ddtrace docs](https://ddtrace.readthedocs.io/en/stable/basic_usage.html#manual-instrumentation).
### Add tracing context to your log
Adding the tracing context to your log makes it easier to find relevant traces in Grafana.
Troncos include a Structlog processor designed to do this.
```python
import structlog
from troncos.contrib.structlog.processors import trace_injection_processor
structlog.configure(
processors=[
trace_injection_processor,
],
)
```
### Logging of major actions in your application
Finding relevant traces in Grafana can be difficult. One way to make finding the relevant traces
easier it to log every major action in your application. This typically means logging every
incoming HTTP request to your server or every task executed by your Celery worker.
The structlog processor above needs to be enabled before logging your major actions is relevant.
#### ASGI middleware
Log ASGI requests.
```python
from starlette.applications import Starlette
from troncos.contrib.asgi.logging.middleware import AsgiLoggingMiddleware
application = AsgiLoggingMiddleware(Starlette())
```
#### Django middleware
Log Django requests. This is not needed if you run Django with ASGI and use the
ASGI middleware.
```python
MIDDLEWARE = [
"troncos.contrib.django.logging.middleware.DjangoLoggingMiddleware",
...
]
```
#### Celery signals
`
Log Celery tasks. Run the code bellow when you configure Celery.
```python
from troncos.contrib.celery.logging.signals import (
connect_troncos_logging_celery_signals,
)
connect_troncos_logging_celery_signals()
```
## Profiling
### Enabling the continuous py-spy profiler
Start the profiler by running the `start_py_spy_profiler` method early in your application. This is
typically done in `settings.py` of you want to profile a Django application, or in `__init__.py`
in the root project package.
```python
from troncos.profiling import start_py_spy_profiler
start_py_spy_profiler(server_address="http://127.0.0.1:4100")
```
### Enabling the ddtrace profiler
Start the profiler by importing the profiler module early in your application. This is
typically done in `settings.py` of you want to profile a Django application, or in `__init__.py`
in the root project package.
<!--pytest.mark.skip-->
```python
import troncos.profiling.auto
```
#### Setup profile endpoint
Use one of the methods bellow based on your selected framework.
##### Django
Add the profile view to the url config.
```python
from django.urls import path
from troncos.contrib.django.profiling.views import profiling_view
urlpatterns = [
path("/debug/pprof", profiling_view, name="profiling"),
]
```
##### Starlette
Add the profile view to your router.
```python
from starlette.routing import Route
from troncos.contrib.starlette.profiling.views import profiling_view
routes = [
Route("/debug/pprof", profiling_view),
]
```
##### ASGI
Mount the generic ASGI profiling application. There is no generic way to do this,
please check the relevant ASGI framework documentation.
```python
from troncos.contrib.asgi.profiling.app import profiling_asgi_app
# FastAPI example
from fastapi import FastAPI
app = FastAPI()
app.mount("/debug/pprof", profiling_asgi_app)
```
#### Verify setup
You can verify that your setup works with the [pprof](https://github.com/google/pprof) cli:
```console
pprof -http :6060 "http://localhost:8080/debug/pprof"
```
#### Enable scraping
When you deploy your application, be sure to use the custom oda annotation for scraping:
```yaml
annotations:
phlare.oda.com/port: "8080"
phlare.oda.com/scrape: "true"
```
## Logging
Troncos is not designed to take control over your logger. But, we do include logging
related tools to make instrumenting your code easier.
### Configure Structlog
Troncos contains a helper method that lets you configure Structlog.
First, run `poetry add structlog` to install structlog in your project.
You can now replace your existing logger config with
```python
from troncos.contrib.structlog import configure_structlog
configure_structlog(format="json", level="INFO")
```
### Adding tracing context to your log
Troncos has a Structlog processor that can be used to add the `span_id` and `trace_id`
properties to your log. More information can be found in the [Tracing](#tracing)
section in this document. This is used by the `configure_structlog` helper method
by default.
### Request logging middleware
Finding the relevant traces in Tempo and Grafana can be difficult. The request logging
middleware exist to make it easier to connect HTTP requests to traces. More information
can be found in the [Tracing](#tracing) section in this document.
Raw data
{
"_id": null,
"home_page": "https://github.com/kolonialno/troncos",
"name": "troncos",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.10,<4.0",
"maintainer_email": "",
"keywords": "logs,traces,opentelemetry",
"author": "Gu\u00f0mundur Bj\u00f6rn Birkisson",
"author_email": "gudmundur.birkisson@oda.com",
"download_url": "https://files.pythonhosted.org/packages/51/2b/cc449ccedca23be5be24311bdb538819b7e5213c813f910cf540d65bdf2d/troncos-4.4.2.tar.gz",
"platform": null,
"description": "<h1 align=\"center\" style=\"border-bottom: 0\">\n \ud83e\udeb5<br>\n Troncos <br/>\n</h1>\n\n<p align=\"center\">\n <em>\n Collection of Python logging, tracing and profiling tools\n </em>\n <br>\n <a href=\"https://github.com/kolonialno/troncos/actions?workflow=CI\">\n <img src=\"https://github.com/kolonialno/troncos/actions/workflows/ci.yml/badge.svg\" alt=\"CI status\">\n </a>\n <a href=\"https://pypi.python.org/pypi/troncos\">\n <img src=\"https://img.shields.io/pypi/v/troncos.svg\" alt=\"Troncos version\">\n </a>\n <img src=\"https://img.shields.io/pypi/pyversions/troncos\" alt=\"Supported Python version\">\n <a href=\"https://github.com/kolonialno/troncos/blob/master/LICENSE\">\n <img src=\"https://img.shields.io/github/license/kolonialno/troncos.svg\" alt=\"licenece\">\n </a>\n</p>\n\n<!-- TOC -->\n- [Etymology](#etymology)\n- [Installation](#installation)\n- [Tracing](#tracing)\n- [Profiling](#profiling)\n- [Logging](#logging)\n<!-- TOC -->\n\n## Etymology\n\n\"Troncos\" is the plural of the spanish word \"Tronco\", which translates to \"trunk\" or \"log\".\n\n## Installation\n\n```console\n# With pip\npip install troncos\n```\n\n## Tracing\n\nTroncos is designed to take advantage of `ddtrace` made by DataDog.\n\nThe ddtrace docs can be found [here](https://ddtrace.readthedocs.io/en/stable/).\n\n[Best practices for traces](https://grafana.com/docs/tempo/latest/operations/best-practices/#naming-conventions-for-span-and-resource-attributes) is a good guide to get started.\n\n### Span vs resource attributes\n\n- A `span attribute` is a key/value pair that provides context for its span.\n- A `resource attribute` is a key/value pair that describes the context of how the span was collected.\n\nFor more information, read the [Attribute and Resource](https://opentelemetry.io/docs/specs/otel/overview/) sections in the OpenTelemetry specification.\n\n### Enabling the tracer\n\nConfigure ddtrace as usual and run `configure_tracer` to send spans to Tempo.\n\nThis is typically done in `settings.py` of you want to profile a Django application,\nor in `__init__.py` in the root project package.\n\n`TRACE_HOST` is usually the host IP of the K8s pod, `TRACE_PORT` is usually 4318\nwhen the Grafana agent is used to collect spans using HTTP.\n\n```python\nimport ddtrace\n\nfrom troncos.tracing import configure_tracer\n\n# Configure tracer as described in the ddtrace docs.\nddtrace.config.django[\"service_name\"] = 'SERVICE_NAME'\n# These are added as span attributes\nddtrace.tracer.set_tags(\n tags={\n \"key\": \"value\",\n }\n)\n\n# Patch third-party modules\nddtrace.patch_all()\n\n# Configure the ddtrace tracer to send traces to Tempo.\nTRACE_HOST = \"127.0.0.1\" # Usually obtained from env variables.\nTRACE_PORT = \"4318\"\nconfigure_tracer(\n enabled=False, # Set to True when TRACE_HOST is configured.\n service_name='SERVICE_NAME',\n endpoint=f\"http://{TRACE_HOST}:{TRACE_PORT}/v1/traces\",\n resource_attributes={\n \"app\": \"app\",\n \"component\": \"component\",\n \"role\": \"role\",\n \"tenant\": \"tenant\",\n \"owner\": \"owner\",\n \"version\": \"version\",\n }\n)\n```\n\nddtrace also uses env variables to configure the service name, environment and version etc.\n\nAdd the following environment variables to your application.\n\n```console\nDD_ENV=\"{{ environment }}\"\nDD_SERVICE=\"{{ app }}\"\nDD_VERSION=\"{{ version }}\"\n# tracecontext/w3c is usually used to propagate distributed traces across services.\nDD_TRACE_PROPAGATION_STYLE_EXTRACT=\"tracecontext\"\nDD_TRACE_PROPAGATION_STYLE_INJECT=\"tracecontext\"\n```\n\n#### Debugging during development\n\nBy setting the environment variable `OTEL_TRACE_DEBUG=True` you will enable traces\nto be printed to `stdout` via the ConsoleSpanExporter as well as through http/grpc.\nAlso specifying `OTEL_TRACE_DEBUG_FILE=/some/file/path` will output traces to the\nspecified file path instead of the console/stdout.\n\n### Using the GRPC span exporter\n\nUsing the GRPC span exporter gives you significant performance gains.\nIf you are running a critical service with high load in production,\nwe recommend using GRPC.\n\n`TRACE_PORT` is usually 4317 when the Grafana agent is used to collect\nspans using GRPC.\n\n```console\npoetry add troncos -E grpc\n```\n\nor\n\n```toml\n[tool.poetry.dependencies]\ntroncos = {version=\"?\", extras = [\"grpc\"]}\n```\n\n```python\nfrom troncos.tracing import configure_tracer, Exporter\n\nTRACE_HOST = \"127.0.0.1\" # Usually obtained from env variables.\nTRACE_PORT = \"4317\"\n\nconfigure_tracer(\n enabled=False, # Set to True when TRACE_HOST is configured.\n service_name='SERVICE_NAME',\n endpoint=f\"http://{TRACE_HOST}:{TRACE_PORT}\",\n exporter=Exporter.GRPC\n)\n```\n\n### Setting headers for the exporter\n\n```python\nfrom troncos.tracing import configure_tracer, Exporter, ExporterType\n\nTRACE_HOST = \"127.0.0.1\" # Usually obtained from env variables.\nTRACE_PORT = \"4317\"\n\nconfigure_tracer(\n enabled=False, # Set to True when TRACE_HOST is configured.\n service_name='SERVICE_NAME',\n endpoint=f\"http://{TRACE_HOST}:{TRACE_PORT}\",\n exporter=Exporter(ExporterType.GRPC, headers={\"my\", \"header\"}),\n)\n```\n\n### Instrument your code\n\nManual instrumentation of your code is described in the [ddtrace docs](https://ddtrace.readthedocs.io/en/stable/basic_usage.html#manual-instrumentation).\n\n### Add tracing context to your log\n\nAdding the tracing context to your log makes it easier to find relevant traces in Grafana.\nTroncos include a Structlog processor designed to do this.\n\n```python\nimport structlog\n\nfrom troncos.contrib.structlog.processors import trace_injection_processor\n\nstructlog.configure(\n processors=[\n trace_injection_processor,\n ],\n)\n```\n\n### Logging of major actions in your application\n\nFinding relevant traces in Grafana can be difficult. One way to make finding the relevant traces\neasier it to log every major action in your application. This typically means logging every\nincoming HTTP request to your server or every task executed by your Celery worker.\n\nThe structlog processor above needs to be enabled before logging your major actions is relevant.\n\n#### ASGI middleware\n\nLog ASGI requests.\n\n```python\nfrom starlette.applications import Starlette\n\nfrom troncos.contrib.asgi.logging.middleware import AsgiLoggingMiddleware\n\napplication = AsgiLoggingMiddleware(Starlette())\n```\n\n#### Django middleware\n\nLog Django requests. This is not needed if you run Django with ASGI and use the\nASGI middleware.\n\n```python\nMIDDLEWARE = [\n \"troncos.contrib.django.logging.middleware.DjangoLoggingMiddleware\",\n ...\n]\n```\n\n#### Celery signals\n\n`\nLog Celery tasks. Run the code bellow when you configure Celery.\n\n```python\nfrom troncos.contrib.celery.logging.signals import (\n connect_troncos_logging_celery_signals,\n)\n\nconnect_troncos_logging_celery_signals()\n```\n\n## Profiling\n\n### Enabling the continuous py-spy profiler\n\nStart the profiler by running the `start_py_spy_profiler` method early in your application. This is\ntypically done in `settings.py` of you want to profile a Django application, or in `__init__.py`\nin the root project package.\n\n```python\nfrom troncos.profiling import start_py_spy_profiler\n\nstart_py_spy_profiler(server_address=\"http://127.0.0.1:4100\")\n```\n\n### Enabling the ddtrace profiler\n\nStart the profiler by importing the profiler module early in your application. This is\ntypically done in `settings.py` of you want to profile a Django application, or in `__init__.py`\nin the root project package.\n\n<!--pytest.mark.skip-->\n```python\nimport troncos.profiling.auto\n```\n\n#### Setup profile endpoint\n\nUse one of the methods bellow based on your selected framework.\n\n##### Django\n\nAdd the profile view to the url config.\n\n```python\nfrom django.urls import path\n\nfrom troncos.contrib.django.profiling.views import profiling_view\n\nurlpatterns = [\n path(\"/debug/pprof\", profiling_view, name=\"profiling\"),\n]\n```\n\n##### Starlette\n\nAdd the profile view to your router.\n\n```python\nfrom starlette.routing import Route\n\nfrom troncos.contrib.starlette.profiling.views import profiling_view\n\nroutes = [\n Route(\"/debug/pprof\", profiling_view),\n]\n```\n\n##### ASGI\n\nMount the generic ASGI profiling application. There is no generic way to do this,\nplease check the relevant ASGI framework documentation.\n\n```python\nfrom troncos.contrib.asgi.profiling.app import profiling_asgi_app\n\n# FastAPI example\nfrom fastapi import FastAPI\n\napp = FastAPI()\n\napp.mount(\"/debug/pprof\", profiling_asgi_app)\n```\n\n#### Verify setup\n\nYou can verify that your setup works with the [pprof](https://github.com/google/pprof) cli:\n\n```console\npprof -http :6060 \"http://localhost:8080/debug/pprof\"\n```\n\n#### Enable scraping\n\nWhen you deploy your application, be sure to use the custom oda annotation for scraping:\n\n```yaml\nannotations:\n phlare.oda.com/port: \"8080\"\n phlare.oda.com/scrape: \"true\"\n```\n\n## Logging\n\nTroncos is not designed to take control over your logger. But, we do include logging\nrelated tools to make instrumenting your code easier.\n\n### Configure Structlog\n\nTroncos contains a helper method that lets you configure Structlog.\n\nFirst, run `poetry add structlog` to install structlog in your project.\n\nYou can now replace your existing logger config with\n\n```python\nfrom troncos.contrib.structlog import configure_structlog\n\nconfigure_structlog(format=\"json\", level=\"INFO\")\n```\n\n### Adding tracing context to your log\n\nTroncos has a Structlog processor that can be used to add the `span_id` and `trace_id`\nproperties to your log. More information can be found in the [Tracing](#tracing)\nsection in this document. This is used by the `configure_structlog` helper method\nby default.\n\n### Request logging middleware\n\nFinding the relevant traces in Tempo and Grafana can be difficult. The request logging\nmiddleware exist to make it easier to connect HTTP requests to traces. More information\ncan be found in the [Tracing](#tracing) section in this document.\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "Collection of Python logging, tracing and profiling tools",
"version": "4.4.2",
"project_urls": {
"Documentation": "https://github.com/kolonialno/troncos",
"Homepage": "https://github.com/kolonialno/troncos",
"Repository": "https://github.com/kolonialno/troncos"
},
"split_keywords": [
"logs",
"traces",
"opentelemetry"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "6b468e2973a5d2373e04560efaddbf3bb6a14d750da147a048b84d9308ca5d31",
"md5": "12a82758754253d75d94e5fb29722b24",
"sha256": "5413040d8e193810caaeaddbdb25ba184170854bb0b82647a6f108b6db78355f"
},
"downloads": -1,
"filename": "troncos-4.4.2-py3-none-any.whl",
"has_sig": false,
"md5_digest": "12a82758754253d75d94e5fb29722b24",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.10,<4.0",
"size": 25696,
"upload_time": "2024-02-05T08:45:47",
"upload_time_iso_8601": "2024-02-05T08:45:47.328902Z",
"url": "https://files.pythonhosted.org/packages/6b/46/8e2973a5d2373e04560efaddbf3bb6a14d750da147a048b84d9308ca5d31/troncos-4.4.2-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "512bcc449ccedca23be5be24311bdb538819b7e5213c813f910cf540d65bdf2d",
"md5": "4d38ea5fb9c47bd14d5d0f1dc343e6b6",
"sha256": "4ef8ae3bde343dc790a59e5a7ec07591c1b4ce296e7311677fcf6bd711341049"
},
"downloads": -1,
"filename": "troncos-4.4.2.tar.gz",
"has_sig": false,
"md5_digest": "4d38ea5fb9c47bd14d5d0f1dc343e6b6",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.10,<4.0",
"size": 21559,
"upload_time": "2024-02-05T08:45:49",
"upload_time_iso_8601": "2024-02-05T08:45:49.328537Z",
"url": "https://files.pythonhosted.org/packages/51/2b/cc449ccedca23be5be24311bdb538819b7e5213c813f910cf540d65bdf2d/troncos-4.4.2.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-02-05 08:45:49",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "kolonialno",
"github_project": "troncos",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "troncos"
}
JSON
