# es2loki
[](https://github.com/ktsstudio/es2loki/actions)
[](https://github.com/ktsstudio/es2loki/actions)
[](https://pypi.python.org/pypi/es2loki)
[](https://hub.docker.com/repository/docker/ktshub/es2loki)
`es2loki` is a migration library that helps to transfer logs from
Elasticsearch to Grafana Loki.
To use es2loki currently you need to define your own mapping of elasticsearch documents
to labels for Grafana Loki.
## Demo
You may find helpful a [demo](demo) folder which contains a fully-sufficient demo stand
that demonstrates transferring logs using `es2loki`.
## Usage
In the simplest form you don't need to write any Python code at all,
Loki will receive no meaningful labels, but nevertheless - let's see how it works.
```bash
$ pip install -U es2loki
$ ELASTIC_HOSTS=http://localhost:9200 \
ELASTIC_INDEX="filebeat-*" \
LOKI_URL=http://localhost:3100 \
python -m es2loki
```
In order to override default `es2loki` behaviour you need to subclass
a `es2loki.BaseTransfer` class.
To declare how documents map to Loki labels you have to override a
`extract_doc_labels` method (see [demo/example.py](demo/example.py)):
```python
from es2loki import BaseTransfer
class TransferLogs(BaseTransfer):
def extract_doc_labels(self, source: dict) -> Optional[MutableMapping[str, str]]:
return dict(
app=source.get("fields", {}).get("service_name"),
job="logs",
level=source.get("level"),
node_name=source.get("host", {}).get("name"),
logger_name=source.get("logger_name"),
)
```
You can run this using the following code:
```python
import sys
from es2loki import run_transfer
if __name__ == "__main__":
sys.exit(run_transfer(TransferLogs()))
```
You can find more examples in the [demo](demo) folder.
### Sorting
By default `es2loki` assumes that in the documents returned from Elasticsearch
there are fields `@timestamp` (you can change the name - see below) and `log.offset`.
Using these 2 fields we can be sure that we will not reread the same lines multiple times.
But if you have your fields that could guarantee such a behaviour - please
override a `make_es_sort` and `make_es_search_after` methods.
* `make_es_sort` defines by which fields the sorting will happen.
* `make_es_search_after` defines an initial "offset". It is needed to resume es2loki after a shutdown. By default it
extracts information from the internal state, which can be saved persistently.
### Persistence
`es2loki` has a mechanism to store the Elasticsearch scrolling state
in the database (highly recommended). In this mode `es2loki` saves
the scrolling state inside an SQL database (PostgreSQL, MySQL, SQLite, ...).
You can opt out of enabling persistence completely using `STATE_MODE=none` env variable, which is the default.
But we highly recommend to enable persistence with some SQL storage.
### Deployment
You can deploy `es2loki` via our helm chart.
Add `kts` repo:
```bash
helm repo add kts https://charts.kts.studio
helm repo update
```
Install the chart:
```bash
helm upgrade --install RELEASE_NAME kts/es2loki
```
More information about helm chart deployment can be found [here](https://github.com/ktsstudio/helm-charts/tree/main/charts/es2loki).
## Configuration
You can configure `es2loki` using the following environment variables:
| name | default | description |
|-------------------------|------------------------------------|----------------------------------------------------------------------------------------------------|
| ELASTIC_HOSTS | http://localhost:9200 | Elasticsearch hosts. Separate multiple hosts using `,` |
| ELASTIC_USER | "" | Elasticsearch username |
| ELASTIC_PASSWORD | "" | Elasticsearch password |
| ELASTIC_INDEX | "" | Elasticsearch index pattern to search documents in |
| ELASTIC_BATCH_SIZE | 3000 | How much documents to extract from ES in one batch |
| ELASTIC_TIMEOUT | 120 | Elasticsearch `search` query timeout |
| ELASTIC_MAX_DATE | | Upper date limit (format is the same as @timestamp field) |
| ELASTIC_TIMESTAMP_FIELD | @timestamp | Name of timesteamp field in Elasticsearch |
| LOKI_URL | http://localhost:3100 | Loki instance URL |
| LOKI_USERNAME | "" | Loki username |
| LOKI_PASSWORD | "" | Loki password |
| LOKI_TENANT_ID | "" | Loki Tenant ID (Org ID) |
| LOKI_BATCH_SIZE | 1048576 | Maximum batch size (in bytes) |
| LOKI_POOL_LOAD_FACTOR | 10 | Maximum number of push non-waiting requests |
| LOKI_PUSH_MODE | pb | `pb` - protobuf + snappy, `gzip` - json + gzip, `json` - just json |
| LOKI_WAIT_TIMEOUT | 0 | How much time (in seconds) to wait after a Loki push request |
| STATE_MODE | none | Configures es2loki persistence (`db` is recommended). Use `none` to disable persistence completely |
| STATE_START_OVER | | Clean up persisted data and start over |
| STATE_DB_URL | postgres://127.0.0.1:5432/postgres | Database URL for `db` persistence |
Raw data
{
"_id": null,
"home_page": "https://github.com/ktsstudio/es2loki",
"name": "es2loki",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.9,<3.11",
"maintainer_email": "",
"keywords": "elasticsearch,loki",
"author": "igorcoding",
"author_email": "igorcoding@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/bc/02/d002d2e599cf0805fc5b916bd1f91ad72ad9c4f8818a852099bc4390598c/es2loki-0.1.6.tar.gz",
"platform": null,
"description": "# es2loki\n\n[](https://github.com/ktsstudio/es2loki/actions)\n[](https://github.com/ktsstudio/es2loki/actions)\n[](https://pypi.python.org/pypi/es2loki)\n[](https://hub.docker.com/repository/docker/ktshub/es2loki)\n\n`es2loki` is a migration library that helps to transfer logs from\nElasticsearch to Grafana Loki.\n\nTo use es2loki currently you need to define your own mapping of elasticsearch documents\nto labels for Grafana Loki.\n\n## Demo\nYou may find helpful a [demo](demo) folder which contains a fully-sufficient demo stand\nthat demonstrates transferring logs using `es2loki`.\n\n## Usage\nIn the simplest form you don't need to write any Python code at all,\nLoki will receive no meaningful labels, but nevertheless - let's see how it works.\n\n```bash\n$ pip install -U es2loki\n$ ELASTIC_HOSTS=http://localhost:9200 \\\n ELASTIC_INDEX=\"filebeat-*\" \\\n LOKI_URL=http://localhost:3100 \\\n python -m es2loki\n```\n\nIn order to override default `es2loki` behaviour you need to subclass\na `es2loki.BaseTransfer` class.\n\nTo declare how documents map to Loki labels you have to override a\n`extract_doc_labels` method (see [demo/example.py](demo/example.py)):\n\n```python\n\nfrom es2loki import BaseTransfer\n\n\nclass TransferLogs(BaseTransfer):\n def extract_doc_labels(self, source: dict) -> Optional[MutableMapping[str, str]]:\n return dict(\n app=source.get(\"fields\", {}).get(\"service_name\"),\n job=\"logs\",\n level=source.get(\"level\"),\n node_name=source.get(\"host\", {}).get(\"name\"),\n logger_name=source.get(\"logger_name\"),\n )\n```\n\nYou can run this using the following code:\n```python\nimport sys\nfrom es2loki import run_transfer\n\nif __name__ == \"__main__\":\n sys.exit(run_transfer(TransferLogs()))\n```\n\nYou can find more examples in the [demo](demo) folder.\n\n### Sorting\n\nBy default `es2loki` assumes that in the documents returned from Elasticsearch\nthere are fields `@timestamp` (you can change the name - see below) and `log.offset`.\nUsing these 2 fields we can be sure that we will not reread the same lines multiple times.\nBut if you have your fields that could guarantee such a behaviour - please\noverride a `make_es_sort` and `make_es_search_after` methods.\n\n* `make_es_sort` defines by which fields the sorting will happen.\n* `make_es_search_after` defines an initial \"offset\". It is needed to resume es2loki after a shutdown. By default it\n extracts information from the internal state, which can be saved persistently.\n\n### Persistence\n\n`es2loki` has a mechanism to store the Elasticsearch scrolling state\nin the database (highly recommended). In this mode `es2loki` saves\nthe scrolling state inside an SQL database (PostgreSQL, MySQL, SQLite, ...).\n\nYou can opt out of enabling persistence completely using `STATE_MODE=none` env variable, which is the default.\nBut we highly recommend to enable persistence with some SQL storage.\n\n### Deployment\n\nYou can deploy `es2loki` via our helm chart.\n\nAdd `kts` repo:\n```bash\nhelm repo add kts https://charts.kts.studio\nhelm repo update\n```\n\nInstall the chart:\n```bash\nhelm upgrade --install RELEASE_NAME kts/es2loki\n```\n\nMore information about helm chart deployment can be found [here](https://github.com/ktsstudio/helm-charts/tree/main/charts/es2loki).\n\n## Configuration\n\nYou can configure `es2loki` using the following environment variables:\n\n| name | default | description |\n|-------------------------|------------------------------------|----------------------------------------------------------------------------------------------------|\n| ELASTIC_HOSTS | http://localhost:9200 | Elasticsearch hosts. Separate multiple hosts using `,` |\n| ELASTIC_USER | \"\" | Elasticsearch username |\n| ELASTIC_PASSWORD | \"\" | Elasticsearch password |\n| ELASTIC_INDEX | \"\" | Elasticsearch index pattern to search documents in |\n| ELASTIC_BATCH_SIZE | 3000 | How much documents to extract from ES in one batch |\n| ELASTIC_TIMEOUT | 120 | Elasticsearch `search` query timeout |\n| ELASTIC_MAX_DATE | | Upper date limit (format is the same as @timestamp field) |\n| ELASTIC_TIMESTAMP_FIELD | @timestamp | Name of timesteamp field in Elasticsearch |\n| LOKI_URL | http://localhost:3100 | Loki instance URL |\n| LOKI_USERNAME | \"\" | Loki username |\n| LOKI_PASSWORD | \"\" | Loki password |\n| LOKI_TENANT_ID | \"\" | Loki Tenant ID (Org ID) |\n| LOKI_BATCH_SIZE | 1048576 | Maximum batch size (in bytes) |\n| LOKI_POOL_LOAD_FACTOR | 10 | Maximum number of push non-waiting requests |\n| LOKI_PUSH_MODE | pb | `pb` - protobuf + snappy, `gzip` - json + gzip, `json` - just json |\n| LOKI_WAIT_TIMEOUT | 0 | How much time (in seconds) to wait after a Loki push request |\n| STATE_MODE | none | Configures es2loki persistence (`db` is recommended). Use `none` to disable persistence completely |\n| STATE_START_OVER | | Clean up persisted data and start over |\n| STATE_DB_URL | postgres://127.0.0.1:5432/postgres | Database URL for `db` persistence |\n\n\n\n\n",
"bugtrack_url": null,
"license": "Apache 2.0",
"summary": "es2loki is a migration library that helps to transfer logs from Elasticsearch to Grafana Loki",
"version": "0.1.6",
"split_keywords": [
"elasticsearch",
"loki"
],
"urls": [
{
"comment_text": "",
"digests": {
"md5": "b35dac2dd09e0505b34af7e3ee41b466",
"sha256": "3ba8d392f7d8843cfbc9403085d61e9766594f80777ecd02f7fff077dd742d11"
},
"downloads": -1,
"filename": "es2loki-0.1.6-py3-none-any.whl",
"has_sig": false,
"md5_digest": "b35dac2dd09e0505b34af7e3ee41b466",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.9,<3.11",
"size": 32035,
"upload_time": "2022-12-12T11:51:41",
"upload_time_iso_8601": "2022-12-12T11:51:41.917659Z",
"url": "https://files.pythonhosted.org/packages/ba/61/854b66eff958efde3536f122e9eaf8fa6de17275d5f48982901c07584944/es2loki-0.1.6-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"md5": "8b2300d53cfaaec5751ed250dc79fc06",
"sha256": "e61835f3bda08373b069fa8de41b2ed107ccb018f1639b5ff24bf3e9f30eeda5"
},
"downloads": -1,
"filename": "es2loki-0.1.6.tar.gz",
"has_sig": false,
"md5_digest": "8b2300d53cfaaec5751ed250dc79fc06",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.9,<3.11",
"size": 25364,
"upload_time": "2022-12-12T11:51:43",
"upload_time_iso_8601": "2022-12-12T11:51:43.484423Z",
"url": "https://files.pythonhosted.org/packages/bc/02/d002d2e599cf0805fc5b916bd1f91ad72ad9c4f8818a852099bc4390598c/es2loki-0.1.6.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2022-12-12 11:51:43",
"github": true,
"gitlab": false,
"bitbucket": false,
"github_user": "ktsstudio",
"github_project": "es2loki",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "es2loki"
}
JSON
