# OpenSearch Logger for Python
<p>
<a href="https://github.com/vduseev/opensearch-logger/actions/workflows/test.yml"><img alt="Tests (main branch)" src="https://img.shields.io/github/actions/workflow/status/vduseev/opensearch-logger/test.yml?logo=github"></a>
<a href="https://codecov.io/gh/vduseev/opensearch-logger"><img alt="Code coverage" src="https://img.shields.io/codecov/c/github/vduseev/opensearch-logger?logo=codecov&logoColor=white"></a>
<a href="https://pypi.org/pypi/opensearch-logger"><img alt="Package version" src="https://img.shields.io/pypi/v/opensearch-logger?logo=python&logoColor=white&color=blue"></a>
<a href="https://pypi.org/pypi/opensearch-logger"><img alt="Supported python versions" src="https://img.shields.io/pypi/pyversions/opensearch-logger?logo=python&logoColor=white"></a>
<a href="https://pypistats.org/packages/opensearch-logger"><img alt="PyPI Downloads" src="https://img.shields.io/pypi/dm/opensearch-logger?logo=python&logoColor=white&color=blue"></a>
</p>
This library provides a standard Python [logging][logging] handler compatible with [OpenSearch][opensearch] suite.
The **goals** of this project are
* to provide a **simple** and direct logging from Python to OpenSearch without *fluentd*, *logstash* or other middleware;
* keep it up to date with the growing difference between OpenSearch and Elasticsearch projects;
* keep the library easy to use, robust, and simple.
The library has been open-sourced from an internal project where it has been successfully used in production
since the release of OpenSearch 1.0.
Generated log records follow the [Elastic Common Schema (ECS)][ecs] field naming convention.
For better performance, it is recommended to set up a proper mapping for your logging indices.
However, everything will work fine without it.
A ready to use [compatible JSON mapping][ecs-mapping] can be found [in the repository][ecs-mapping].
## Installation
```shell
pip install opensearch-logger
```
## Usage
Just add the OpenSearch handler to your Python logger
```python
import logging
from opensearch_logger import OpenSearchHandler
handler = OpenSearchHandler(
index_name="my-logs",
hosts=["https://localhost:9200"],
http_auth=("admin", "admin"),
http_compress=True,
use_ssl=True,
verify_certs=False,
ssl_assert_hostname=False,
ssl_show_warn=False,
)
logger = logging.getLogger(__name__)
logger.setLevel(logging.INFO)
logger.addHandler(handler)
```
To send logs to OpenSearch simply use the same regular logging commands
```python
# Logging a simple text message
logger.info("This message will be indexed in OpenSearch")
# Now, as an example, let's measure how long some operation takes
start_time = time.perf_counter()
heavy_database_operation()
elapsed_time = time.perf_counter() - start_time
# Let's send elapsed_time as an exatra parameter to the log record below.
# This will make the `elapsed_time` field searchable and aggregatable.
logger.info(
f"Database operation took {elapsed_time:.3f} seconds",
extra={"elapsed_time": elapsed_time},
)
```
## Configuration
The `OpenSearchHandler` constructor takes several arguments described in the table below.
These parameters specify the name of the index, buffering settings, and some general behavior.
None of this parameters are mandatory.
All other keyword arguments are passed directly "as is" to the underlying `OpenSearch` python client.
Full list of connection parameters can be found in [`opensearch-py`][opensearch-py] docs.
At least one connection parameter **must** be provided, otherwise a `TypeError` will be thrown.
## Logging parameters
| Parameter | Default | Description |
| - | - | - |
| `index_name` | `"python-logs"` | Base name of the OpenSearch index name that will be created. Or name of the data stream if `is_data_stream` is set to `True`. |
| `index_rotate` | `DAILY` | Frequency that controls what date is appended to index name during its creation. `OpenSearchHandler.DAILY`. |
| `index_date_format` | `"%Y.%m.%d"` | Format of the date that gets appended to the base index name. |
| `index_name_sep` | `"-"` | Separator string between `index_name` and the date, appended to the index name. |
| `is_data_stream` | `False` | A flag to indicate that the documents will get indexed into a data stream. If `True`, index rotation settings are ignored. |
| `buffer_size` | `1000` | Number of log records which when reached on the internal buffer results in a flush to OpenSearch. |
| `flush_frequency` | `1` | Float representing how often the buffer will be flushed (in seconds). |
| `extra_fields` | `{}` | Nested dictionary with extra fields that will be added to every log record. |
| `raise_on_index_exc` | `False` | Raise exception if indexing the log record in OpenSearch fails. |
## Connection parameters
Here are a few examples of the connection parameters supported by the OpenSearch client.
For more details please check the [`opensearch-py`][opensearch-py] documentation.
| Parameter | Example | Description |
| - | - | - |
| `hosts` | `["https://localhost:9200"]` | The list of hosts to connect to. Multiple hosts are allowed. |
| `http_auth` | `("admin", "admin")` | Username and password to authenticate against the OpenSearch servers. |
| `http_compress` | `True` | Enables gzip compression for request bodies. |
| `use_ssl` | `True` | Whether communications should be SSL encrypted. |
| `verify_certs` | `False` | Whether the SSL certificates are validated or not. |
| `ssl_assert_hostname` | `False` | Verify authenticity of host for encrypted connections. |
| `ssl_show_warn` | `False` | Enable warning for SSL connections. |
| `ca_certs` | `"/var/lib/root-ca.pem"` | CA bundle path for using intermediate CAs with your root CA. |
## Configuration with logging.config or in Django
Similarly to other log handlers, `opensearch-logger` supports configuration via `logging.config` facility.
Just specify the `opensearch_logger.OpenSearchHandler` as one of the handlers and provide it with parameters.
Full guide on tweaking `logging.config` can be found in the [official python documentation][logging-config].
```python
import logging.config
LOGGING = {
"version": 1,
"disable_existing_loggers": False,
"handlers": {
"file": {
"level": "DEBUG",
"class": "logging.handlers.RotatingFileHandler",
"filename": "./debug.log",
"maxBytes": 102400,
"backupCount": 5,
},
"opensearch": {
"level": "INFO",
"class": "opensearch_logger.OpenSearchHandler",
"index_name": "my-logs",
"extra_fields": {"App": "test", "Environment": "dev"},
"hosts": [{"host": "localhost", "port": 9200}],
"http_auth": ("admin", "admin"),
"http_compress": True,
"use_ssl": True,
"verify_certs": False,
"ssl_assert_hostname": False,
"ssl_show_warn": False,
},
},
"loggers": {
"root": {
"handlers": ["file", "opensearch"],
"level": "INFO",
"propogate": False,
},
"django": {
"handlers": ["file","opensearch"],
"level": "DEBUG",
"propagate": True,
},
},
}
logging.config.dictConfig(LOGGING)
```
## Using AWS OpenSearch
Package `requests_aws4auth` is required to connect to the AWS OpenSearch service.
```python
import boto3
from opensearch_logger import OpenSearchHandler
from requests_aws4auth import AWS4Auth
from opensearchpy import RequestsHttpConnection
host = "" # The OpenSearch domain endpoint starting with https://
region = "us-east-1" # AWS Region
service = "es"
creds = boto3.Session().get_credentials()
handler = OpenSearchHandler(
index_name="my-logs",
hosts=[host],
http_auth=AWS4Auth(creds.access_key, creds.secret_key, region, service, session_token=creds.token),
use_ssl=True,
verify_certs=True,
ssl_assert_hostname=True,
ssl_show_warn=True,
connection_class=RequestsHttpConnection,
)
```
## Using Kerberos Authentication
Package `requests_kerberos` is required to authenticate using Kerberos.
```python
from opensearch_logger import OpenSearchHandler
from requests_kerberos import HTTPKerberosAuth, DISABLED
handler = OpenSearchHandler(
index_name="my-logs",
hosts=["https://localhost:9200"],
http_auth=HTTPKerberosAuth(mutual_authentication=DISABLED),
use_ssl=True,
verify_certs=False,
ssl_assert_hostname=False,
ssl_show_warn=False,
)
```
## Using Data Streams
Indexing documents into data streams is supported by just setting the `is_data_stream` parameter to `True`.
In this configuration however, It is OpenSearch that solely manages the rollover of the data stream's write index.
The logging handler's rollover functionality and index rotation settings (e.g. `index_rotate`) are disabled.
The following is an example configuration to send documents to a data stream.
```python
handler = OpenSearchHandler(
index_name="logs-myapp-development",
is_data_stream=True,
hosts=["https://localhost:9200"],
http_auth=("admin", "admin")
)
```
## Dependencies
This library depends on the following packages
* [`opensearch-py`][opensearch-py]
## Building from source & Developing
This package uses [`pyenv`][pyenv] (optional) for development purposes.
It also uses Docker to run OpenSearch container for integration testing during development.
1. Clone the repo.
1. Create a virtual environment using any of the supported Python version.
```shell
# We are using Python 3.11 installed using pyenv for this example
pyenv local 3.11.0
# Create virtual env
python -m venv .venv
# Activate it
source .venv/bin/activate
```
1. Install [`pip-tools`][pip-tools] and [`flit`][flit]
```shell
# Update pip to the latest version, just in case
pip install --upgrade pip
# Install pip-compile and pip-sync, as well as flit
pip install pip-tools flit
```
1. Compile resolved dependency list
```shell
# Generates requirements.txt file.
# This might yield different results for different platforms.
pip-compile pyproject.toml
# Resolve dev requirements
pip-compile --extra dev -o dev-requirements.txt pyproject.toml
# If you want to upgrade dependencies, then call
pip-compile pyproject.toml --upgrade
```
1. Install resolved dependencies into virtual environment
```shell
# Sync current venv with both core and dev dependencies
pip-sync requirements.txt dev-requirements.txt
```
1. Install package itself locally.
Build, publishing, and local installation are done using [`flit`][flit].
```shell
flit install
```
1. Run tests
**WARNING**: You need opensearch running on `https://localhost:9200` to run the tests.
Part of the tests verifies that correct number of logs actually gets into OpenSearch.
Alternatively, you can specify the `TEST_OPENSEARCH_HOST` variable and set it to a different value pointing
to the running OpenSearch server.
There are not many tests, but they run with **5 seconds cooldown each** to allow OpenSearch to process the
newly sent log records properly and verify their count.
Small helper scripts are available in the `tests/` directory to start and stop OpenSearch using Docker.
```shell
# Give it 5-10 seconds to initialize before running tests
tests/start-opensearch-docker.sh
# Run tests
pytest
# Run coverage tests
pytest --cov --cov-report=html --cov-config=pyproject.toml
# Run mypy typing verification
pytest --mypy opensearch_logger --strict-markers
# Run flake8 to make sure code style is correct
flake8
# Turn off OpenSearch
tests/stop-opensearch-docker.sh
```
1. Bump package version
```shell
bump2version patch
```
1. Publish package (make sure you have correct credentials or `.pypirc` file)
```shell
flit publish
```
### Cheat Sheet for working with OpenSearch
1. List all created indices, including count of documents
```shell
$ curl -k -XGET "https://admin:admin@localhost:9200/_cat/indices/test*?v&s=index"
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
yellow open test-opensearch-logger-2021.11.08 N0BEEnG2RIuPP0l8RZE0Dg 1 1 7 0 29.7kb 29.7kb
```
1. Count documents in all indexes that start with `test`
```shell
$ curl -k -XGET "https://admin:admin@localhost:9200/test*/_count"
{"count":109,"_shards":{"total":1,"successful":1,"skipped":0,"failed":0}}
```
1. Retrieve all documents from indexes that start with `test`
```shell
$ curl -k -XGET "https://admin:admin@localhost:9200/test*/_search" -H 'Content-Type: application/json' -d '{"query":{"match_all":{}}}'
{
"took": 1,
"timed_out": false,
"hits": {
"total": {
"value": 109,
"relation": "eq"
}
...
```
1. Same, but limit the number of returned documents to 10
```shell
$ curl -k -XGET "https://admin:admin@localhost:9200/test*/_search?size=10" -H 'Content-Type: application/json' -d '{"query":{"match_all":{}}}'
{
"took": 1,
"timed_out": false,
"hits": {
"total": {
"value": 109,
"relation": "eq"
}
...
```
## Contributions
Contributions are welcome! 👏 🎉
Please create a GitHub issue and a Pull Request that references that issue as well as your proposed changes.
Your Pull Request will be automatically tested using GitHub actions.
After your pull request will be accepted, it will be merged and the version of the library will be bumped
and released to PyPI by project maintainers.
## History
This is a fork of [Python Elasticsearch ECS Log handler][python-elasticsearch-ecs-logger] project
which was in turn forked from [Python Elasticsearch Logger][python-elasticsearch-logger] project.
While original is perfectly suitable for logging to Elasticsearch, due to the split between
OpenSearch and Elasticsearch it makes sense to make a fork entirely tailored to work with OpenSearch
and based on the official [`opensearch-py`][opensearch-py] Python library.
The API between `python-elasticsearch-ecs-logger` and this project has slightly changed for better
compatibility with OpenSearch and for the purposes of simplification.
## Featured on
The `opensearch-logger` project is featured on the official
[OpenSearch Community Projects page][community-projects] 🚀.
## License
Distributed under the terms of [Apache 2.0][apache-2.0] license, opensearch-logger is free and open source software.
[opensearch]: https://opensearch.org/
[opensearch-py]: https://pypi.org/project/opensearch-py/
[logging]: https://docs.python.org/3/library/logging.html
[ecs]: https://www.elastic.co/guide/en/ecs/current/index.html
[logging-config]: https://docs.python.org/3/library/logging.config.html
[pyenv]: https://github.com/pyenv/pyenv
[ecs-mapping]: https://github.com/vduseev/opensearch-logger/blob/main/mappings/ecs1.4.0_compatible_minimal.json
[apache-2.0]: https://github.com/vduseev/opensearch-logger/blob/main/LICENSE.md
[python-elasticsearch-ecs-logger]: https://github.com/IMInterne/python-elasticsearch-ecs-logger
[python-elasticsearch-logger]: https://github.com/cmanaha/python-elasticsearch-logger
[community-projects]: https://opensearch.org/community_projects
[pip-tools]: https://pypi.org/project/pip-tools/
[flit]: https://flit.pypa.io/en/stable/
Raw data
{
"_id": null,
"home_page": null,
"name": "opensearch-logger",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.6",
"maintainer_email": "vduseev <vagiz@duseev.com>",
"keywords": "opensearch,logging,handler,logger",
"author": null,
"author_email": "vduseev <vagiz@duseev.com>, IMInterne <equipe_interne@innovmetric.com>",
"download_url": "https://files.pythonhosted.org/packages/e8/f8/475a4bb186c032fda2e1dc21c1ae461c41e64ba6a95b7688702639f4bbc4/opensearch_logger-1.3.0.tar.gz",
"platform": null,
"description": "# OpenSearch Logger for Python\n\n<p>\n <a href=\"https://github.com/vduseev/opensearch-logger/actions/workflows/test.yml\"><img alt=\"Tests (main branch)\" src=\"https://img.shields.io/github/actions/workflow/status/vduseev/opensearch-logger/test.yml?logo=github\"></a>\n <a href=\"https://codecov.io/gh/vduseev/opensearch-logger\"><img alt=\"Code coverage\" src=\"https://img.shields.io/codecov/c/github/vduseev/opensearch-logger?logo=codecov&logoColor=white\"></a>\n <a href=\"https://pypi.org/pypi/opensearch-logger\"><img alt=\"Package version\" src=\"https://img.shields.io/pypi/v/opensearch-logger?logo=python&logoColor=white&color=blue\"></a>\n <a href=\"https://pypi.org/pypi/opensearch-logger\"><img alt=\"Supported python versions\" src=\"https://img.shields.io/pypi/pyversions/opensearch-logger?logo=python&logoColor=white\"></a>\n <a href=\"https://pypistats.org/packages/opensearch-logger\"><img alt=\"PyPI Downloads\" src=\"https://img.shields.io/pypi/dm/opensearch-logger?logo=python&logoColor=white&color=blue\"></a>\n</p>\n\nThis library provides a standard Python [logging][logging] handler compatible with [OpenSearch][opensearch] suite.\n\nThe **goals** of this project are\n\n* to provide a **simple** and direct logging from Python to OpenSearch without *fluentd*, *logstash* or other middleware;\n* keep it up to date with the growing difference between OpenSearch and Elasticsearch projects;\n* keep the library easy to use, robust, and simple.\n\nThe library has been open-sourced from an internal project where it has been successfully used in production\nsince the release of OpenSearch 1.0.\n\nGenerated log records follow the [Elastic Common Schema (ECS)][ecs] field naming convention.\nFor better performance, it is recommended to set up a proper mapping for your logging indices.\nHowever, everything will work fine without it.\nA ready to use [compatible JSON mapping][ecs-mapping] can be found [in the repository][ecs-mapping].\n\n## Installation\n\n```shell\npip install opensearch-logger\n```\n\n## Usage\n\nJust add the OpenSearch handler to your Python logger\n\n```python\nimport logging\nfrom opensearch_logger import OpenSearchHandler\n\nhandler = OpenSearchHandler(\n index_name=\"my-logs\",\n hosts=[\"https://localhost:9200\"],\n http_auth=(\"admin\", \"admin\"),\n http_compress=True,\n use_ssl=True,\n verify_certs=False,\n ssl_assert_hostname=False,\n ssl_show_warn=False,\n)\n\nlogger = logging.getLogger(__name__)\nlogger.setLevel(logging.INFO)\nlogger.addHandler(handler)\n```\n\nTo send logs to OpenSearch simply use the same regular logging commands\n\n```python\n# Logging a simple text message\nlogger.info(\"This message will be indexed in OpenSearch\")\n\n# Now, as an example, let's measure how long some operation takes\nstart_time = time.perf_counter()\n\nheavy_database_operation()\n\nelapsed_time = time.perf_counter() - start_time\n\n# Let's send elapsed_time as an exatra parameter to the log record below.\n# This will make the `elapsed_time` field searchable and aggregatable.\nlogger.info(\n f\"Database operation took {elapsed_time:.3f} seconds\",\n extra={\"elapsed_time\": elapsed_time},\n)\n```\n\n## Configuration\n\nThe `OpenSearchHandler` constructor takes several arguments described in the table below.\nThese parameters specify the name of the index, buffering settings, and some general behavior.\nNone of this parameters are mandatory.\n\nAll other keyword arguments are passed directly \"as is\" to the underlying `OpenSearch` python client.\nFull list of connection parameters can be found in [`opensearch-py`][opensearch-py] docs.\nAt least one connection parameter **must** be provided, otherwise a `TypeError` will be thrown.\n\n## Logging parameters\n\n| Parameter | Default | Description |\n| - | - | - |\n| `index_name` | `\"python-logs\"` | Base name of the OpenSearch index name that will be created. Or name of the data stream if `is_data_stream` is set to `True`. |\n| `index_rotate` | `DAILY` | Frequency that controls what date is appended to index name during its creation. `OpenSearchHandler.DAILY`. |\n| `index_date_format` | `\"%Y.%m.%d\"` | Format of the date that gets appended to the base index name. |\n| `index_name_sep` | `\"-\"` | Separator string between `index_name` and the date, appended to the index name. |\n| `is_data_stream` | `False` | A flag to indicate that the documents will get indexed into a data stream. If `True`, index rotation settings are ignored. |\n| `buffer_size` | `1000` | Number of log records which when reached on the internal buffer results in a flush to OpenSearch. |\n| `flush_frequency` | `1` | Float representing how often the buffer will be flushed (in seconds). |\n| `extra_fields` | `{}` | Nested dictionary with extra fields that will be added to every log record. |\n| `raise_on_index_exc` | `False` | Raise exception if indexing the log record in OpenSearch fails. |\n\n## Connection parameters\n\nHere are a few examples of the connection parameters supported by the OpenSearch client.\nFor more details please check the [`opensearch-py`][opensearch-py] documentation.\n\n| Parameter | Example | Description |\n| - | - | - |\n| `hosts` | `[\"https://localhost:9200\"]` | The list of hosts to connect to. Multiple hosts are allowed. |\n| `http_auth` | `(\"admin\", \"admin\")` | Username and password to authenticate against the OpenSearch servers. |\n| `http_compress` | `True` | Enables gzip compression for request bodies. |\n| `use_ssl` | `True` | Whether communications should be SSL encrypted. |\n| `verify_certs` | `False` | Whether the SSL certificates are validated or not. |\n| `ssl_assert_hostname` | `False` | Verify authenticity of host for encrypted connections. |\n| `ssl_show_warn` | `False` | Enable warning for SSL connections. |\n| `ca_certs` | `\"/var/lib/root-ca.pem\"` | CA bundle path for using intermediate CAs with your root CA. |\n\n## Configuration with logging.config or in Django\n\nSimilarly to other log handlers, `opensearch-logger` supports configuration via `logging.config` facility.\nJust specify the `opensearch_logger.OpenSearchHandler` as one of the handlers and provide it with parameters.\n\nFull guide on tweaking `logging.config` can be found in the [official python documentation][logging-config].\n\n```python\nimport logging.config\n\nLOGGING = {\n \"version\": 1,\n \"disable_existing_loggers\": False,\n \"handlers\": {\n \"file\": {\n \"level\": \"DEBUG\",\n \"class\": \"logging.handlers.RotatingFileHandler\",\n \"filename\": \"./debug.log\",\n \"maxBytes\": 102400,\n \"backupCount\": 5,\n },\n \"opensearch\": {\n \"level\": \"INFO\",\n \"class\": \"opensearch_logger.OpenSearchHandler\",\n \"index_name\": \"my-logs\",\n \"extra_fields\": {\"App\": \"test\", \"Environment\": \"dev\"},\n \"hosts\": [{\"host\": \"localhost\", \"port\": 9200}],\n \"http_auth\": (\"admin\", \"admin\"),\n \"http_compress\": True,\n \"use_ssl\": True,\n \"verify_certs\": False,\n \"ssl_assert_hostname\": False,\n \"ssl_show_warn\": False,\n },\n },\n \"loggers\": {\n \"root\": {\n \"handlers\": [\"file\", \"opensearch\"],\n \"level\": \"INFO\",\n \"propogate\": False,\n },\n \"django\": {\n \"handlers\": [\"file\",\"opensearch\"],\n \"level\": \"DEBUG\",\n \"propagate\": True,\n },\n },\n}\n\nlogging.config.dictConfig(LOGGING)\n```\n\n## Using AWS OpenSearch\n\nPackage `requests_aws4auth` is required to connect to the AWS OpenSearch service.\n\n```python\nimport boto3\nfrom opensearch_logger import OpenSearchHandler\nfrom requests_aws4auth import AWS4Auth\nfrom opensearchpy import RequestsHttpConnection\n\nhost = \"\" # The OpenSearch domain endpoint starting with https://\nregion = \"us-east-1\" # AWS Region\nservice = \"es\"\ncreds = boto3.Session().get_credentials()\n\nhandler = OpenSearchHandler(\n index_name=\"my-logs\",\n hosts=[host],\n http_auth=AWS4Auth(creds.access_key, creds.secret_key, region, service, session_token=creds.token),\n use_ssl=True,\n verify_certs=True,\n ssl_assert_hostname=True,\n ssl_show_warn=True,\n connection_class=RequestsHttpConnection,\n)\n```\n\n## Using Kerberos Authentication\n\nPackage `requests_kerberos` is required to authenticate using Kerberos.\n\n```python\nfrom opensearch_logger import OpenSearchHandler\nfrom requests_kerberos import HTTPKerberosAuth, DISABLED\n\nhandler = OpenSearchHandler(\n index_name=\"my-logs\",\n hosts=[\"https://localhost:9200\"],\n http_auth=HTTPKerberosAuth(mutual_authentication=DISABLED),\n use_ssl=True,\n verify_certs=False,\n ssl_assert_hostname=False,\n ssl_show_warn=False,\n)\n```\n\n## Using Data Streams\n\nIndexing documents into data streams is supported by just setting the `is_data_stream` parameter to `True`.\n\nIn this configuration however, It is OpenSearch that solely manages the rollover of the data stream's write index.\nThe logging handler's rollover functionality and index rotation settings (e.g. `index_rotate`) are disabled.\n\nThe following is an example configuration to send documents to a data stream.\n\n```python\nhandler = OpenSearchHandler(\n index_name=\"logs-myapp-development\",\n is_data_stream=True,\n hosts=[\"https://localhost:9200\"],\n http_auth=(\"admin\", \"admin\")\n)\n```\n\n## Dependencies\n\nThis library depends on the following packages\n\n* [`opensearch-py`][opensearch-py]\n\n## Building from source & Developing\n\nThis package uses [`pyenv`][pyenv] (optional) for development purposes.\nIt also uses Docker to run OpenSearch container for integration testing during development.\n\n1. Clone the repo.\n1. Create a virtual environment using any of the supported Python version.\n\n ```shell\n # We are using Python 3.11 installed using pyenv for this example\n pyenv local 3.11.0\n\n # Create virtual env\n python -m venv .venv\n\n # Activate it\n source .venv/bin/activate\n ```\n\n1. Install [`pip-tools`][pip-tools] and [`flit`][flit]\n\n ```shell\n # Update pip to the latest version, just in case\n pip install --upgrade pip\n # Install pip-compile and pip-sync, as well as flit\n pip install pip-tools flit\n ```\n\n1. Compile resolved dependency list\n\n ```shell\n # Generates requirements.txt file.\n # This might yield different results for different platforms.\n pip-compile pyproject.toml\n \n # Resolve dev requirements\n pip-compile --extra dev -o dev-requirements.txt pyproject.toml\n \n # If you want to upgrade dependencies, then call\n pip-compile pyproject.toml --upgrade\n ```\n\n1. Install resolved dependencies into virtual environment\n\n ```shell\n # Sync current venv with both core and dev dependencies\n pip-sync requirements.txt dev-requirements.txt\n ```\n\n1. Install package itself locally.\n\n Build, publishing, and local installation are done using [`flit`][flit].\n\n ```shell\n flit install\n ```\n\n1. Run tests\n\n **WARNING**: You need opensearch running on `https://localhost:9200` to run the tests.\n Part of the tests verifies that correct number of logs actually gets into OpenSearch.\n Alternatively, you can specify the `TEST_OPENSEARCH_HOST` variable and set it to a different value pointing\n to the running OpenSearch server.\n\n There are not many tests, but they run with **5 seconds cooldown each** to allow OpenSearch to process the\n newly sent log records properly and verify their count.\n\n Small helper scripts are available in the `tests/` directory to start and stop OpenSearch using Docker.\n\n ```shell\n # Give it 5-10 seconds to initialize before running tests\n tests/start-opensearch-docker.sh\n\n # Run tests\n pytest\n\n # Run coverage tests\n pytest --cov --cov-report=html --cov-config=pyproject.toml\n\n # Run mypy typing verification\n pytest --mypy opensearch_logger --strict-markers\n\n # Run flake8 to make sure code style is correct\n flake8\n\n # Turn off OpenSearch\n tests/stop-opensearch-docker.sh\n ```\n\n1. Bump package version\n\n ```shell\n bump2version patch\n ```\n\n1. Publish package (make sure you have correct credentials or `.pypirc` file)\n\n ```shell\n flit publish\n ```\n\n### Cheat Sheet for working with OpenSearch\n\n1. List all created indices, including count of documents\n\n ```shell\n $ curl -k -XGET \"https://admin:admin@localhost:9200/_cat/indices/test*?v&s=index\"\n health status index uuid pri rep docs.count docs.deleted store.size pri.store.size\n yellow open test-opensearch-logger-2021.11.08 N0BEEnG2RIuPP0l8RZE0Dg 1 1 7 0 29.7kb 29.7kb\n ```\n\n1. Count documents in all indexes that start with `test`\n\n ```shell\n $ curl -k -XGET \"https://admin:admin@localhost:9200/test*/_count\"\n {\"count\":109,\"_shards\":{\"total\":1,\"successful\":1,\"skipped\":0,\"failed\":0}}\n ```\n\n1. Retrieve all documents from indexes that start with `test`\n\n ```shell\n $ curl -k -XGET \"https://admin:admin@localhost:9200/test*/_search\" -H 'Content-Type: application/json' -d '{\"query\":{\"match_all\":{}}}'\n {\n \"took\": 1,\n \"timed_out\": false,\n \"hits\": {\n \"total\": {\n \"value\": 109,\n \"relation\": \"eq\"\n }\n ...\n ```\n\n1. Same, but limit the number of returned documents to 10\n\n ```shell\n $ curl -k -XGET \"https://admin:admin@localhost:9200/test*/_search?size=10\" -H 'Content-Type: application/json' -d '{\"query\":{\"match_all\":{}}}'\n {\n \"took\": 1,\n \"timed_out\": false,\n \"hits\": {\n \"total\": {\n \"value\": 109,\n \"relation\": \"eq\"\n }\n ...\n ```\n\n## Contributions\n\nContributions are welcome! \ud83d\udc4f \ud83c\udf89\n\nPlease create a GitHub issue and a Pull Request that references that issue as well as your proposed changes.\nYour Pull Request will be automatically tested using GitHub actions.\n\nAfter your pull request will be accepted, it will be merged and the version of the library will be bumped\nand released to PyPI by project maintainers.\n\n## History\n\nThis is a fork of [Python Elasticsearch ECS Log handler][python-elasticsearch-ecs-logger] project\nwhich was in turn forked from [Python Elasticsearch Logger][python-elasticsearch-logger] project.\nWhile original is perfectly suitable for logging to Elasticsearch, due to the split between\nOpenSearch and Elasticsearch it makes sense to make a fork entirely tailored to work with OpenSearch\nand based on the official [`opensearch-py`][opensearch-py] Python library.\n\nThe API between `python-elasticsearch-ecs-logger` and this project has slightly changed for better\ncompatibility with OpenSearch and for the purposes of simplification.\n\n## Featured on\n\nThe `opensearch-logger` project is featured on the official\n[OpenSearch Community Projects page][community-projects] \ud83d\ude80.\n\n\n\n## License\n\nDistributed under the terms of [Apache 2.0][apache-2.0] license, opensearch-logger is free and open source software.\n\n[opensearch]: https://opensearch.org/\n[opensearch-py]: https://pypi.org/project/opensearch-py/\n[logging]: https://docs.python.org/3/library/logging.html\n[ecs]: https://www.elastic.co/guide/en/ecs/current/index.html\n[logging-config]: https://docs.python.org/3/library/logging.config.html\n[pyenv]: https://github.com/pyenv/pyenv\n[ecs-mapping]: https://github.com/vduseev/opensearch-logger/blob/main/mappings/ecs1.4.0_compatible_minimal.json\n[apache-2.0]: https://github.com/vduseev/opensearch-logger/blob/main/LICENSE.md\n[python-elasticsearch-ecs-logger]: https://github.com/IMInterne/python-elasticsearch-ecs-logger\n[python-elasticsearch-logger]: https://github.com/cmanaha/python-elasticsearch-logger\n[community-projects]: https://opensearch.org/community_projects\n[pip-tools]: https://pypi.org/project/pip-tools/\n[flit]: https://flit.pypa.io/en/stable/\n",
"bugtrack_url": null,
"license": null,
"summary": "OpenSearch logging handler",
"version": "1.3.0",
"project_urls": {
"documentation": "https://github.com/vduseev/opensearch-logger",
"homepage": "https://github.com/vduseev/opensearch-logger",
"repository": "https://github.com/vduseev/opensearch-logger"
},
"split_keywords": [
"opensearch",
"logging",
"handler",
"logger"
],
"urls": [
{
"comment_text": null,
"digests": {
"blake2b_256": "4a70e4417ba2e75f34e690430ce82b2b9727a898721f6fc2a507216ca69f2639",
"md5": "9c03a5ec9a491ea02a1ba16ea3049011",
"sha256": "c396b174d06b619c58fa06497100ccd5b337e21a8319b03e0682e45a24c4e053"
},
"downloads": -1,
"filename": "opensearch_logger-1.3.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "9c03a5ec9a491ea02a1ba16ea3049011",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.6",
"size": 14130,
"upload_time": "2023-11-26T23:57:45",
"upload_time_iso_8601": "2023-11-26T23:57:45.698183Z",
"url": "https://files.pythonhosted.org/packages/4a/70/e4417ba2e75f34e690430ce82b2b9727a898721f6fc2a507216ca69f2639/opensearch_logger-1.3.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "e8f8475a4bb186c032fda2e1dc21c1ae461c41e64ba6a95b7688702639f4bbc4",
"md5": "8053f6ea1ba52728240779b3ceda0ff9",
"sha256": "b0b031cf9c19cffebcae052dd8cf3e1a96bba57d59b75fc19b55efbccd5a8959"
},
"downloads": -1,
"filename": "opensearch_logger-1.3.0.tar.gz",
"has_sig": false,
"md5_digest": "8053f6ea1ba52728240779b3ceda0ff9",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.6",
"size": 97038,
"upload_time": "2023-11-26T23:57:48",
"upload_time_iso_8601": "2023-11-26T23:57:48.516558Z",
"url": "https://files.pythonhosted.org/packages/e8/f8/475a4bb186c032fda2e1dc21c1ae461c41e64ba6a95b7688702639f4bbc4/opensearch_logger-1.3.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-11-26 23:57:48",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "vduseev",
"github_project": "opensearch-logger",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "opensearch-logger"
}
JSON
