# PromQL HTTP API
This python package provides a [Prometheus](https://prometheus.io/) HTTP API client library.
It encapsulates and simplifies the collection of data from a Prometheus server.
One major feature of this library is that responses to queries are returned as [Pandas](https://pandas.pydata.org/) DataFrames.
Prometheus is an open-source system monitoring and alerting toolkit. It collects metrics from configured targets at given intervals, evaluates rule expressions, displays the results, and can trigger alerts if some condition is observed to be true. The Prometheus server exposes an HTTP API for querying the collected data, and a query language called PromQL.
This library is intended to help data scientists who would like to harvest data from a Prometheus server for analysis and visualization. The library is design to be simple to use, and to provide a convenient interface to the Prometheus HTTP API. It is also designed to be performant and scalable, by using the [requests](https://requests.readthedocs.io/en/master/) library and caching HTTP connections to the Prometheus server between API accesses.
For unstable connections, the library supports retrying failed requests. The user may specify the number of retries, the time-out between retries, and the back-off factor for the retry interval.
Version 0.2.0 improves timezone awareness. Starting in this version, the HTTP API query is issued with a timestamp value in the 'time', 'start', and 'end' fields. In addition, the query schema now supports a 'timezone' element (provided as a timezone object), which controls the timestamp column formatting in the returned dataframe.
## Installation
To install as a root user:
```commandline
python3 -m pip install promql-http-api
```
To install as a non-root user:
```commandline
python3 -m pip install --user promql-http-api
```
To uninstall:
```commandline
python3 -m pip uninstall promql-http-api
```
## Usage Examples
Here is a basic usage example:
```python
from promql_http_api import PromqlHttpApi
api = PromqlHttpApi('http://localhost:9090')
tz = pytz.timezone('UTC')
q_time = datetime.now()
q = api.query('up', tz.localize(q_time))
df = q.to_dataframe()
print(df)
```
On the first line we create a PromqlHttpApi object named `api`. This example assumes that a Prometheus server is running on the local host, and it is listening to port 9090.
Replace this URL as needed with the appropriate URL for your server.
Next, we use the `api` object to create a Query object named `q`. The `query()` function takes two parameters: a query string and a date-time string.
To execute the query explicitly, without converting the result to a DataFrame, you can use:
```python
# Execute the query explicitly
promql_response_data = q()
# Convert the cached result to a DataFrame
df = q.to_dataframe()
```
Alternately, by calling the to_dataframe() method alone, we will implicitly execute the query.
```python
# Execute the query implicitly
df = q.to_dataframe()
```
Adding retries and time-out to the query work only with explicit execution:
```python
# Execute the query explicitly
# with 5 retries and retry intervals of 5, 10, 20, and 40 seconds
promql_response_data = q(retries=5, timeout=5, backoff=2)
# Convert the cached result to a DataFrame
df = q.to_dataframe()
```
### HTTP Authentication (and other headers)
The `PromqlHttpApi` object takes an optional `headers` parameter. This parameter is a dictionary of HTTP headers to be included in the request. The `headers` parameter is useful for including authentication information in the request. Here is an example of how to use the `headers` parameter:
```python
api = PromqlHttpApi('http://localhost:9090', headers={'Authorization': 'token 0123456789ABCDEF'})
```
### Working with schemas
The `to_dataframe()` method takes an optional `schema` parameter. The schema is a dictionary that controls several elements of the query. A schema may include the following element keys: `columns`, `dtype`, and `timezone`.
The `columns` element controls the PromQL response elements to be included as columns in the returned DataFrame. The returned DataFrame will always include a timestamp column (in seconds since the epoch), and a `value` column. If `columns` is not provided, all the fields returned in a PromQL response will be included in the returned DataFrame. If `columns` is provided, only the fields listed in `columns` will be included in the returned DataFrame.
The `dtype` element controls the data type of the `value` column in the returned DataFrame. The default is `str`. The `dtype` element may be set to any valid Pandas data type.
The `timezone` element allows the user to request an additional `datetime` column which is formatted in the specified timezone. The `timezone` element must be a timezone object from the [pytz](https://pypi.org/project/pytz/) library. If the `timezone` element is not provided, the returned DataFrame will not include a `datetime` column.
Here is an example of how to use a schema:
```python
schema = {
'columns': ['node', 'sensor'],
'dtype': float,
'timezone': pytz.timezone('US/Eastern')
}
df = q.to_dataframe(schema)
```
## Debugging
If something goes wrong, you can look at the HTTP response and the PromQL response information. Here are some examples:
```python
from promql_http_api import PromqlHttpApi
api = PromqlHttpApi('http://localhost:9090')
tz = pytz.timezone('UTC')
q_time = datetime.now()
q = api.query('up', tz.localize(q_time))
q()
promql_response = q.response
http_response = promql_response.response
print(f'HTTP response status code = {http_response.status_code}')
print(f'HTTP response encoding = {http_response.encoding}')
print(f'PromQL response status = {promql_response.status()}')
print(f'PromQL response data = {promql_response.data()}')
print(f'PromQL response error type = {promql_response.error_type()}')
print(f'PromQL response error = {promql_response.error()}')
```
---
# List of Supported APIs
| API | Method |
|--------------------- |---------------------------------------|
| /api/v1/query | query(query, time) |
| /api/v1/query_range | query_range(query, start, end, step) |
| /api/v1/format_query | format_query(query) |
| /api/v1/series | series(match) |
| /api/v1/labels | labels() |
| /api/v1/label/<label_name>/values | label_values(label) |
| /api/v1/targets | targets(state) |
| /api/v1/rules | rules(type) |
| /api/v1/alerts | alerts() |
| /api/v1/alertmanagers | alertmanagers() |
| /api/v1/status/config | config() |
| /api/v1/status/flags | flags() |
| /api/v1/status/runtimeinfo | runtimeinfo() |
| /api/v1/status/buildinfo | buildinfo() |
---
# Testing
The package contains limited unit testing.
Run the tests from the package top folder using:
```commandline
pytest
```
---
# Future work
Implement a CI/CD pipeline with a Prometheus instance in a Docker container to test API accesses.
If you use this library and would like to help - please contact the author.
---
# References
[Prometheus / HTTP API](https://prometheus.io/docs/prometheus/latest/querying/api/)
Raw data
{
"_id": null,
"home_page": "https://github.com/nir-arad/promql-http-api",
"name": "promql-http-api",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.8,<4.0",
"maintainer_email": "",
"keywords": "prometheus,promql,pandas,dataframe",
"author": "Nir Arad",
"author_email": "",
"download_url": "https://files.pythonhosted.org/packages/41/03/3c57b86b8fb7444cbdcb2572160615cbefa524807fc2a4b4f36693d50398/promql_http_api-0.3.3.tar.gz",
"platform": null,
"description": "# PromQL HTTP API\n\nThis python package provides a [Prometheus](https://prometheus.io/) HTTP API client library.\nIt encapsulates and simplifies the collection of data from a Prometheus server.\nOne major feature of this library is that responses to queries are returned as [Pandas](https://pandas.pydata.org/) DataFrames.\n\nPrometheus is an open-source system monitoring and alerting toolkit. It collects metrics from configured targets at given intervals, evaluates rule expressions, displays the results, and can trigger alerts if some condition is observed to be true. The Prometheus server exposes an HTTP API for querying the collected data, and a query language called PromQL.\n\nThis library is intended to help data scientists who would like to harvest data from a Prometheus server for analysis and visualization. The library is design to be simple to use, and to provide a convenient interface to the Prometheus HTTP API. It is also designed to be performant and scalable, by using the [requests](https://requests.readthedocs.io/en/master/) library and caching HTTP connections to the Prometheus server between API accesses.\n\nFor unstable connections, the library supports retrying failed requests. The user may specify the number of retries, the time-out between retries, and the back-off factor for the retry interval.\n\nVersion 0.2.0 improves timezone awareness. Starting in this version, the HTTP API query is issued with a timestamp value in the 'time', 'start', and 'end' fields. In addition, the query schema now supports a 'timezone' element (provided as a timezone object), which controls the timestamp column formatting in the returned dataframe.\n\n\n## Installation\n\nTo install as a root user:\n\n```commandline\npython3 -m pip install promql-http-api\n```\n\nTo install as a non-root user:\n\n```commandline\npython3 -m pip install --user promql-http-api\n```\n\nTo uninstall:\n```commandline\npython3 -m pip uninstall promql-http-api\n```\n\n## Usage Examples\n\nHere is a basic usage example:\n\n```python\nfrom promql_http_api import PromqlHttpApi\n\napi = PromqlHttpApi('http://localhost:9090')\ntz = pytz.timezone('UTC')\nq_time = datetime.now()\nq = api.query('up', tz.localize(q_time))\ndf = q.to_dataframe()\nprint(df)\n```\n\nOn the first line we create a PromqlHttpApi object named `api`. This example assumes that a Prometheus server is running on the local host, and it is listening to port 9090.\nReplace this URL as needed with the appropriate URL for your server.\n\nNext, we use the `api` object to create a Query object named `q`. The `query()` function takes two parameters: a query string and a date-time string.\n\nTo execute the query explicitly, without converting the result to a DataFrame, you can use:\n```python\n# Execute the query explicitly\npromql_response_data = q()\n\n# Convert the cached result to a DataFrame\ndf = q.to_dataframe()\n```\n\nAlternately, by calling the to_dataframe() method alone, we will implicitly execute the query.\n\n```python\n# Execute the query implicitly\ndf = q.to_dataframe()\n```\n\nAdding retries and time-out to the query work only with explicit execution:\n\n```python\n# Execute the query explicitly\n# with 5 retries and retry intervals of 5, 10, 20, and 40 seconds\npromql_response_data = q(retries=5, timeout=5, backoff=2)\n\n# Convert the cached result to a DataFrame\ndf = q.to_dataframe()\n```\n\n### HTTP Authentication (and other headers)\n\nThe `PromqlHttpApi` object takes an optional `headers` parameter. This parameter is a dictionary of HTTP headers to be included in the request. The `headers` parameter is useful for including authentication information in the request. Here is an example of how to use the `headers` parameter:\n\n```python\napi = PromqlHttpApi('http://localhost:9090', headers={'Authorization': 'token 0123456789ABCDEF'})\n```\n\n### Working with schemas\n\nThe `to_dataframe()` method takes an optional `schema` parameter. The schema is a dictionary that controls several elements of the query. A schema may include the following element keys: `columns`, `dtype`, and `timezone`.\n\nThe `columns` element controls the PromQL response elements to be included as columns in the returned DataFrame. The returned DataFrame will always include a timestamp column (in seconds since the epoch), and a `value` column. If `columns` is not provided, all the fields returned in a PromQL response will be included in the returned DataFrame. If `columns` is provided, only the fields listed in `columns` will be included in the returned DataFrame.\n\nThe `dtype` element controls the data type of the `value` column in the returned DataFrame. The default is `str`. The `dtype` element may be set to any valid Pandas data type.\n\nThe `timezone` element allows the user to request an additional `datetime` column which is formatted in the specified timezone. The `timezone` element must be a timezone object from the [pytz](https://pypi.org/project/pytz/) library. If the `timezone` element is not provided, the returned DataFrame will not include a `datetime` column.\n\nHere is an example of how to use a schema:\n```python\nschema = {\n 'columns': ['node', 'sensor'],\n 'dtype': float,\n 'timezone': pytz.timezone('US/Eastern')\n}\n\ndf = q.to_dataframe(schema)\n```\n\n\n## Debugging\n\nIf something goes wrong, you can look at the HTTP response and the PromQL response information. Here are some examples:\n```python\nfrom promql_http_api import PromqlHttpApi\napi = PromqlHttpApi('http://localhost:9090')\ntz = pytz.timezone('UTC')\nq_time = datetime.now()\nq = api.query('up', tz.localize(q_time))\nq()\npromql_response = q.response\nhttp_response = promql_response.response\nprint(f'HTTP response status code = {http_response.status_code}')\nprint(f'HTTP response encoding = {http_response.encoding}')\nprint(f'PromQL response status = {promql_response.status()}')\nprint(f'PromQL response data = {promql_response.data()}')\nprint(f'PromQL response error type = {promql_response.error_type()}')\nprint(f'PromQL response error = {promql_response.error()}')\n```\n\n---\n# List of Supported APIs\n\n| API | Method |\n|--------------------- |---------------------------------------|\n| /api/v1/query | query(query, time) |\n| /api/v1/query_range | query_range(query, start, end, step) |\n| /api/v1/format_query | format_query(query) |\n| /api/v1/series | series(match) |\n| /api/v1/labels | labels() |\n| /api/v1/label/<label_name>/values | label_values(label) |\n| /api/v1/targets | targets(state) |\n| /api/v1/rules | rules(type) |\n| /api/v1/alerts | alerts() |\n| /api/v1/alertmanagers | alertmanagers() |\n| /api/v1/status/config | config() |\n| /api/v1/status/flags | flags() |\n| /api/v1/status/runtimeinfo | runtimeinfo() |\n| /api/v1/status/buildinfo | buildinfo() |\n\n\n---\n# Testing\n\nThe package contains limited unit testing.\nRun the tests from the package top folder using:\n\n```commandline\npytest\n```\n\n---\n# Future work\n\nImplement a CI/CD pipeline with a Prometheus instance in a Docker container to test API accesses.\n\nIf you use this library and would like to help - please contact the author.\n\n---\n# References\n\n[Prometheus / HTTP API](https://prometheus.io/docs/prometheus/latest/querying/api/)\n",
"bugtrack_url": null,
"license": "Apache-2.0",
"summary": "Query a Prometheus server and get a Pandas DataFrame",
"version": "0.3.3",
"project_urls": {
"Bug Tracker": "https://github.com/nir-arad/promql-http-api/issues",
"Homepage": "https://github.com/nir-arad/promql-http-api",
"Repository": "https://github.com/nir-arad/promql-http-api"
},
"split_keywords": [
"prometheus",
"promql",
"pandas",
"dataframe"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "901affe2621cb89a8b0472afa7f2d9184f09a4415859fe566f109845798c5023",
"md5": "c1b659606d0c17b17776864b64ea724a",
"sha256": "9b09ec762d253b22b3e40953bd1655b58adf4723024c96369b9dd54165c8739e"
},
"downloads": -1,
"filename": "promql_http_api-0.3.3-py3-none-any.whl",
"has_sig": false,
"md5_digest": "c1b659606d0c17b17776864b64ea724a",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8,<4.0",
"size": 24630,
"upload_time": "2024-03-11T21:31:17",
"upload_time_iso_8601": "2024-03-11T21:31:17.411392Z",
"url": "https://files.pythonhosted.org/packages/90/1a/ffe2621cb89a8b0472afa7f2d9184f09a4415859fe566f109845798c5023/promql_http_api-0.3.3-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "41033c57b86b8fb7444cbdcb2572160615cbefa524807fc2a4b4f36693d50398",
"md5": "fa18f58d3937d268bf433a8562398358",
"sha256": "ce71d38d185bf93ddc2bce44e8f344d5af8a2145a329f437592a2d3cd12a5866"
},
"downloads": -1,
"filename": "promql_http_api-0.3.3.tar.gz",
"has_sig": false,
"md5_digest": "fa18f58d3937d268bf433a8562398358",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8,<4.0",
"size": 15068,
"upload_time": "2024-03-11T21:31:18",
"upload_time_iso_8601": "2024-03-11T21:31:18.710391Z",
"url": "https://files.pythonhosted.org/packages/41/03/3c57b86b8fb7444cbdcb2572160615cbefa524807fc2a4b4f36693d50398/promql_http_api-0.3.3.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-03-11 21:31:18",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "nir-arad",
"github_project": "promql-http-api",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"lcname": "promql-http-api"
}
JSON
