# Track-o-tron
Improved decorator for Langfuse. It aims to address some issues of the official Langfuse decorator :
- No tracking of default values
- Messy typing
- Inconsistant behavior between traces and observations
- Poor error handling
> [!WARNING]
> This is still very experimental and definitely not production ready.
## Installation
First, install the package with your favorie package manager (`pip`, `poetry`, etc.).
```python
pip install trackotron
```
## Usage
The observers offer some flexibility. They can be used both as context managers or decorators. When they are used as decorators, the lib is able to observe more data (inputs and outputs, module name, function name, etc.).
In both cases, it offers an access to the observation client through a proxy. This is useful to inject a score, or any other custom thing. The proxy can store updates until the function is terminated (or the context manager exits) to avoid useless API calls.
First, an observer needs to be instantiated. It will create required contexts.
```python
from langfuse import Langfuse
from trackotron import Observer
client = Langfuse()
observer = Observer(client)
```
Then, any function can be decorated. The proxy will be automatically injected. By default, it will capture both the input and the output, but this can be tuned with `capture_input=False` or `capture_output=False`.
```python
from trackotron import GenerationProxyAlias, GenerationUpdate
@observer.observe(type_="generation")
def run(proxy: GenerationProxyAlias, model: str = "gpt-4o-mini") -> str:
# ...
proxy.update(GenerationUpdate(model=model))
```
It can also be used as a context manager. Some parameters (such as `capture_input` or `capture_output`) won't have any effect in such situation.
```python
with observer.observe(name="context") as proxy:
# ...
proxy.score("perplexity", 1.0, comment="Lorem ipsum")
```
### Limitations
The decorator currently only supports functions. Instance and class methods are not supported regarding typing (but it should work).
Coroutines are not supported yet but it should be pretty straight forward to implement. The only issue is to find a way to avoid code duplication.
## Performances
The overhead appears to be quite small. The scripts ([`benchmark`](./scripts/benchmark.py) and [`baseline`](./scripts/baseline.py)) are used to quickly measure performances of Langfuse (both when enabled or disabled) compared to a raw script. It contains some sleeps (1.50s) to fake some execution or I/O.
| Baseline | Langfuse Disabled | Langfuse Enabled |
| -------- | ----------------- | ----------------------- |
| 1.5017 | 1.5062 (+0.30%) | 1.5080 (+0.42%, +0.12%) |
The overhead is between 0.30% and 0.42% (depending if Langfuse is enabled). This should be pretty okay for most of the LLM applications which often only do HTTP calls.
## Development
The whole project relies on Poetry for setup. It has been developed with Python 3.9 for backward compatibility. Python 3.8 is not supported as it will reach end-of-life in October 2024.
### Environment
```bash
poetry env use python3.9
poetry lock
poetry install --with dev
poetry run pre-commit install
```
### Quality
```bash
poetry run black . # Format code
poetry run ruff check --fix --force-exclude . # Lint code
poetry run mypy --ignore-missing-imports --scripts-are-modules . # Type check code
```
```bash
poetry run python -m pytest --cov=src/trackotron tests # Run all tests
```
### Build
```bash
poetry build -f wheel
```
### TODOs
Here are some ideas to improve the current decorator :
- [ ] Support coroutines
- [ ] Fix typing for instance and class methods
- [ ] Make the proxy injection optional (_i.e._ do not inject the proxy if not present in the function/method signature)
- [ ] Add more tests to ensure behaviors and consistency
- [ ] Support context reuse if that makes sense
Raw data
{
"_id": null,
"home_page": "https://github.com/bendabir/trackotron",
"name": "trackotron",
"maintainer": null,
"docs_url": null,
"requires_python": "<4,>=3.9",
"maintainer_email": null,
"keywords": "langfuse",
"author": "Bendabir",
"author_email": null,
"download_url": null,
"platform": null,
"description": "# Track-o-tron\n\n\n\n\n\n\n\n\nImproved decorator for Langfuse. It aims to address some issues of the official Langfuse decorator :\n\n- No tracking of default values\n- Messy typing\n- Inconsistant behavior between traces and observations\n- Poor error handling\n\n> [!WARNING]\n> This is still very experimental and definitely not production ready.\n\n## Installation\n\nFirst, install the package with your favorie package manager (`pip`, `poetry`, etc.).\n\n```python\npip install trackotron\n```\n\n## Usage\n\nThe observers offer some flexibility. They can be used both as context managers or decorators. When they are used as decorators, the lib is able to observe more data (inputs and outputs, module name, function name, etc.).\n\nIn both cases, it offers an access to the observation client through a proxy. This is useful to inject a score, or any other custom thing. The proxy can store updates until the function is terminated (or the context manager exits) to avoid useless API calls.\n\nFirst, an observer needs to be instantiated. It will create required contexts.\n\n```python\nfrom langfuse import Langfuse\nfrom trackotron import Observer\n\nclient = Langfuse()\nobserver = Observer(client)\n```\n\nThen, any function can be decorated. The proxy will be automatically injected. By default, it will capture both the input and the output, but this can be tuned with `capture_input=False` or `capture_output=False`.\n\n```python\nfrom trackotron import GenerationProxyAlias, GenerationUpdate\n\n@observer.observe(type_=\"generation\")\ndef run(proxy: GenerationProxyAlias, model: str = \"gpt-4o-mini\") -> str:\n # ...\n proxy.update(GenerationUpdate(model=model))\n```\n\nIt can also be used as a context manager. Some parameters (such as `capture_input` or `capture_output`) won't have any effect in such situation.\n\n```python\nwith observer.observe(name=\"context\") as proxy:\n # ...\n proxy.score(\"perplexity\", 1.0, comment=\"Lorem ipsum\")\n```\n\n### Limitations\n\nThe decorator currently only supports functions. Instance and class methods are not supported regarding typing (but it should work).\n\nCoroutines are not supported yet but it should be pretty straight forward to implement. The only issue is to find a way to avoid code duplication.\n\n## Performances\n\nThe overhead appears to be quite small. The scripts ([`benchmark`](./scripts/benchmark.py) and [`baseline`](./scripts/baseline.py)) are used to quickly measure performances of Langfuse (both when enabled or disabled) compared to a raw script. It contains some sleeps (1.50s) to fake some execution or I/O.\n\n| Baseline | Langfuse Disabled | Langfuse Enabled |\n| -------- | ----------------- | ----------------------- |\n| 1.5017 | 1.5062 (+0.30%) | 1.5080 (+0.42%, +0.12%) |\n\nThe overhead is between 0.30% and 0.42% (depending if Langfuse is enabled). This should be pretty okay for most of the LLM applications which often only do HTTP calls.\n\n## Development\n\nThe whole project relies on Poetry for setup. It has been developed with Python 3.9 for backward compatibility. Python 3.8 is not supported as it will reach end-of-life in October 2024.\n\n### Environment\n\n```bash\npoetry env use python3.9\npoetry lock\npoetry install --with dev\npoetry run pre-commit install\n```\n\n### Quality\n\n```bash\npoetry run black . # Format code\npoetry run ruff check --fix --force-exclude . # Lint code\npoetry run mypy --ignore-missing-imports --scripts-are-modules . # Type check code\n```\n\n```bash\npoetry run python -m pytest --cov=src/trackotron tests # Run all tests\n```\n\n### Build\n\n```bash\npoetry build -f wheel\n```\n\n### TODOs\n\nHere are some ideas to improve the current decorator :\n\n- [ ] Support coroutines\n- [ ] Fix typing for instance and class methods\n- [ ] Make the proxy injection optional (_i.e._ do not inject the proxy if not present in the function/method signature)\n- [ ] Add more tests to ensure behaviors and consistency\n- [ ] Support context reuse if that makes sense\n\n",
"bugtrack_url": null,
"license": "GPL-3.0-or-later",
"summary": "Improved decorator for Langfuse.",
"version": "0.1.0",
"project_urls": {
"Documentation": "https://github.com/Bendabir/trackotron/README.md",
"Homepage": "https://github.com/bendabir/trackotron",
"Repository": "https://github.com/bendabir/trackotron"
},
"split_keywords": [
"langfuse"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "2566b56240e7190c156d552b893a71760126248a856592ee25d00d65c712e39c",
"md5": "9cb9b1078e37e650509922a077cbad32",
"sha256": "71ca735286f309029397ee4e0af74ef1c205a76b16b554d9df3a67edd3cf4143"
},
"downloads": -1,
"filename": "trackotron-0.1.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "9cb9b1078e37e650509922a077cbad32",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": "<4,>=3.9",
"size": 27512,
"upload_time": "2024-09-08T12:00:05",
"upload_time_iso_8601": "2024-09-08T12:00:05.391217Z",
"url": "https://files.pythonhosted.org/packages/25/66/b56240e7190c156d552b893a71760126248a856592ee25d00d65c712e39c/trackotron-0.1.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-09-08 12:00:05",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "bendabir",
"github_project": "trackotron",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "trackotron"
}
JSON
