# ProgressivePostgres
**ProgressivePostgres** is a Python client library for interacting with a TimescaleDB (or PostgreSQL) instance, configured via environment variables. It integrates seamlessly with the [Zeitgleich](https://git.ift.tuwien.ac.at/lab/ift/sis/researchlin-x/pandastso) library for advanced time-series handling and includes optional features for bridging MQTT data ingestion through [SwampClient](https://your-swampclient-docs).
Developed at [TU Wien](https://www.tuwien.ac.at/), ProgressivePostgres provides:
- **Time-Series Data Models** — Integration with `TimeSeriesData` and `MultiOriginTimeSeries`.
- **Environment-Based Configuration** — Minimizes boilerplate; simply use a `.env` file.
- **Automatic Table Creation** — Optionally create hypertables for your data if they do not already exist.
- **Extra Columns Handling** — Decide how to manage columns beyond the expected set: `ignore`, `error`, or `append`.
- **Asynchronous MQTT Integration** — If combined with an MQTT client, seamlessly push sensor data into TimescaleDB.
---
## Table of Contents
- [Features](#features)
- [Installation](#installation)
- [Configuration](#configuration)
- [Environment Variables](#environment-variables)
- [Usage & Examples](#usage--examples)
- [TODOs](#todos)
- [License](#license)
---
## Features
- **Simple DB Client:** Quickly execute queries, insert data, and retrieve time-series rows.
- **Timestamp Normalization:** `TimeSeriesData` can parse and convert timestamps among multiple formats (`ISO`, `RFC3339`, `UNIX`, etc.).
- **Automatic Hypertable Creation:** Create TimescaleDB hypertables on-the-fly if desired.
- **Multi-Origin Data Model:** Use `MultiOriginTimeSeries` for simultaneously managing data from multiple sensors or devices.
- **Optional MQTT Bridge:** Combine with an MQTT client for real-time sensor data ingestion.
---
## Installation
Install **ProgressivePostgres** (and any needed dependencies) via:
```bash
pip install ProgressivePostgres
```
or clone the repository locally with the provided Makefile:
```bash
make install
```
---
## Configuration
ProgressivePostgres uses environment variables for configuration, read from a prefix that you pass to the `Client(name="TS")` constructor.
### Environment Variables
| Variable | Description | Default | Options |
|---------------------------------------------------|-----------------------------------------------------------------------------|-------------|-----------------------------------|
| `{PREFIX}_DB_HOST` | Hostname for TimescaleDB/PostgreSQL. | `localhost` | |
| `{PREFIX}_DB_PORT` | Port for TimescaleDB. | `5432` | |
| `{PREFIX}_DB_NAME` | Database name. | `timescale` | |
| `{PREFIX}_DB_USER` | Username for DB authentication. | `postgres` | |
| `{PREFIX}_DB_PASS` | Password for DB authentication. | *None* | |
| `{PREFIX}_DB_AUTOCOMMIT` | Whether to auto-commit each statement. | `true` | `true`, `false` |
| `{PREFIX}_LOG_LEVEL` | Log level. | `DEBUG` | `DEBUG`, `INFO`, `WARNING`, `ERROR` |
| `{PREFIX}_ORIGIN_SPLIT_CHAR` | Char used to split origins (e.g. `machine1/sensorA`). | `/` | |
| `{PREFIX}_ORIGIN_JOIN_CHAR` | Char used to join origin parts for table naming. | `/` | |
| `{PREFIX}_TIMESTAMP_COLUMN` | Name of the time column in DB. | `timestamp` | |
| `{PREFIX}_VALUE_COLUMN` | Name of the primary value column. | `value` | |
| `{PREFIX}_CREATE_TABLES_IF_NOT_EXIST` | Automatically create hypertables if missing. | `true` | `true`, `false` |
| `{PREFIX}_EXTRA_COLUMNS_HANDLING` | Handling of extra columns. | `append` | `ignore`, `error`, `append` |
---
## Usage & Examples
To see **working code samples**, please refer to the [examples directory](examples/) in this repository. Highlights include:
- **Basic Example:** Demonstrates how to connect to a local TimescaleDB instance, run simple queries, and handle `.env` environment variables.
- **MQTT Logger Example:** Combines ProgressivePostgres with an MQTT client, pushing messages from topics into TimescaleDB.
- **Zeitgleich Example:** Showcases using `TimeSeriesData` and `MultiOriginTimeSeries` for multi-sensor data insertion and retrieval.
### Example `.env`
A typical `.env` file might look like:
```
TS_DB_HOST="localhost"
TS_DB_PORT="5444"
TS_DB_NAME="timeseries"
TS_DB_USER="postgres"
TS_DB_PASS="pwd"
TS_LOG_LEVEL="DEBUG"
TS_ORIGIN_SPLIT_CHAR="/"
TS_ORIGIN_JOIN_CHAR="_"
TS_TIMESTAMP_COLUMN="timestamp"
TS_VALUE_COLUMN="value"
TS_EXTRA_COLUMNS_HANDLING="append"
TS_CREATE_TABLES_IF_NOT_EXIST="true"
```
Load these environment variables in your Python script using [`python-dotenv`](https://pypi.org/project/python-dotenv):
```python
from dotenv import load_dotenv
load_dotenv()
```
Then instantiate a client:
```python
from ProgressivePostgres import Client
client = Client(name="TS") # "TS" will be the prefix for env variables
# ...
```
---
## TODOs
- **Automatic Migrations:** Provide tools to manage schema migrations automatically.
- **Advanced Query Builder:** Add an optional query builder for more complex queries (joins, filters, etc.).
- **Transaction Handling:** More robust transaction management (automatic rollback on certain errors).
- **Comprehensive Testing:** Add unit and integration tests across various DB versions.
- **Enhanced MQTT Integration:** Provide additional examples.
---
## License
Licensed under the [MIT License](LICENSE).
Raw data
{
"_id": null,
"home_page": null,
"name": "ProgressivePostgres",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.6",
"maintainer_email": null,
"keywords": "mqtt, postgresql, time-series, timescale, timestamp",
"author": null,
"author_email": "Jakob Friedl <jakob.friedl@tuwien.ac.at>",
"download_url": "https://files.pythonhosted.org/packages/bf/a8/fb2de05d5e20c336e22e2c5b4d5323c778175cd5ecc01d6ef411fae92054/progressivepostgres-0.0.3.tar.gz",
"platform": null,
"description": "# ProgressivePostgres\n\n**ProgressivePostgres** is a Python client library for interacting with a TimescaleDB (or PostgreSQL) instance, configured via environment variables. It integrates seamlessly with the [Zeitgleich](https://git.ift.tuwien.ac.at/lab/ift/sis/researchlin-x/pandastso) library for advanced time-series handling and includes optional features for bridging MQTT data ingestion through [SwampClient](https://your-swampclient-docs).\n\nDeveloped at [TU Wien](https://www.tuwien.ac.at/), ProgressivePostgres provides:\n\n- **Time-Series Data Models** \u2014 Integration with `TimeSeriesData` and `MultiOriginTimeSeries`.\n- **Environment-Based Configuration** \u2014 Minimizes boilerplate; simply use a `.env` file.\n- **Automatic Table Creation** \u2014 Optionally create hypertables for your data if they do not already exist.\n- **Extra Columns Handling** \u2014 Decide how to manage columns beyond the expected set: `ignore`, `error`, or `append`.\n- **Asynchronous MQTT Integration** \u2014 If combined with an MQTT client, seamlessly push sensor data into TimescaleDB.\n\n---\n\n## Table of Contents\n\n- [Features](#features)\n- [Installation](#installation)\n- [Configuration](#configuration)\n - [Environment Variables](#environment-variables)\n- [Usage & Examples](#usage--examples)\n- [TODOs](#todos)\n- [License](#license)\n\n---\n\n## Features\n\n- **Simple DB Client:** Quickly execute queries, insert data, and retrieve time-series rows.\n- **Timestamp Normalization:** `TimeSeriesData` can parse and convert timestamps among multiple formats (`ISO`, `RFC3339`, `UNIX`, etc.).\n- **Automatic Hypertable Creation:** Create TimescaleDB hypertables on-the-fly if desired.\n- **Multi-Origin Data Model:** Use `MultiOriginTimeSeries` for simultaneously managing data from multiple sensors or devices.\n- **Optional MQTT Bridge:** Combine with an MQTT client for real-time sensor data ingestion.\n\n---\n\n## Installation\n\nInstall **ProgressivePostgres** (and any needed dependencies) via:\n\n```bash\npip install ProgressivePostgres\n```\n\nor clone the repository locally with the provided Makefile:\n\n```bash\nmake install\n```\n\n---\n\n## Configuration\n\nProgressivePostgres uses environment variables for configuration, read from a prefix that you pass to the `Client(name=\"TS\")` constructor.\n\n### Environment Variables\n\n| Variable | Description | Default | Options |\n|---------------------------------------------------|-----------------------------------------------------------------------------|-------------|-----------------------------------|\n| `{PREFIX}_DB_HOST` | Hostname for TimescaleDB/PostgreSQL. | `localhost` | |\n| `{PREFIX}_DB_PORT` | Port for TimescaleDB. | `5432` | |\n| `{PREFIX}_DB_NAME` | Database name. | `timescale` | |\n| `{PREFIX}_DB_USER` | Username for DB authentication. | `postgres` | |\n| `{PREFIX}_DB_PASS` | Password for DB authentication. | *None* | |\n| `{PREFIX}_DB_AUTOCOMMIT` | Whether to auto-commit each statement. | `true` | `true`, `false` |\n| `{PREFIX}_LOG_LEVEL` | Log level. | `DEBUG` | `DEBUG`, `INFO`, `WARNING`, `ERROR` |\n| `{PREFIX}_ORIGIN_SPLIT_CHAR` | Char used to split origins (e.g. `machine1/sensorA`). | `/` | |\n| `{PREFIX}_ORIGIN_JOIN_CHAR` | Char used to join origin parts for table naming. | `/` | |\n| `{PREFIX}_TIMESTAMP_COLUMN` | Name of the time column in DB. | `timestamp` | |\n| `{PREFIX}_VALUE_COLUMN` | Name of the primary value column. | `value` | |\n| `{PREFIX}_CREATE_TABLES_IF_NOT_EXIST` | Automatically create hypertables if missing. | `true` | `true`, `false` |\n| `{PREFIX}_EXTRA_COLUMNS_HANDLING` | Handling of extra columns. | `append` | `ignore`, `error`, `append` |\n\n---\n\n## Usage & Examples\n\nTo see **working code samples**, please refer to the [examples directory](examples/) in this repository. Highlights include:\n\n- **Basic Example:** Demonstrates how to connect to a local TimescaleDB instance, run simple queries, and handle `.env` environment variables.\n- **MQTT Logger Example:** Combines ProgressivePostgres with an MQTT client, pushing messages from topics into TimescaleDB.\n- **Zeitgleich Example:** Showcases using `TimeSeriesData` and `MultiOriginTimeSeries` for multi-sensor data insertion and retrieval.\n\n### Example `.env`\n\nA typical `.env` file might look like:\n\n```\nTS_DB_HOST=\"localhost\"\nTS_DB_PORT=\"5444\"\nTS_DB_NAME=\"timeseries\"\nTS_DB_USER=\"postgres\"\nTS_DB_PASS=\"pwd\"\nTS_LOG_LEVEL=\"DEBUG\"\nTS_ORIGIN_SPLIT_CHAR=\"/\"\nTS_ORIGIN_JOIN_CHAR=\"_\"\nTS_TIMESTAMP_COLUMN=\"timestamp\"\nTS_VALUE_COLUMN=\"value\"\nTS_EXTRA_COLUMNS_HANDLING=\"append\"\nTS_CREATE_TABLES_IF_NOT_EXIST=\"true\"\n```\n\nLoad these environment variables in your Python script using [`python-dotenv`](https://pypi.org/project/python-dotenv):\n\n```python\nfrom dotenv import load_dotenv\n\nload_dotenv()\n```\n\nThen instantiate a client:\n\n```python\nfrom ProgressivePostgres import Client\n\nclient = Client(name=\"TS\") # \"TS\" will be the prefix for env variables\n# ...\n```\n\n---\n\n## TODOs\n\n- **Automatic Migrations:** Provide tools to manage schema migrations automatically.\n- **Advanced Query Builder:** Add an optional query builder for more complex queries (joins, filters, etc.).\n- **Transaction Handling:** More robust transaction management (automatic rollback on certain errors).\n- **Comprehensive Testing:** Add unit and integration tests across various DB versions.\n- **Enhanced MQTT Integration:** Provide additional examples.\n\n---\n\n## License\n\nLicensed under the [MIT License](LICENSE).",
"bugtrack_url": null,
"license": null,
"summary": "A TimescaleDB client library with environment-based config and time-series data handling.",
"version": "0.0.3",
"project_urls": null,
"split_keywords": [
"mqtt",
" postgresql",
" time-series",
" timescale",
" timestamp"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "8ae55f84d0914a4fecb273aa095ae24c22455c5962cbfea1553b3d0810b358eb",
"md5": "eb12cc5de9a13bcc64984edb5872a1b8",
"sha256": "a3059b4d977658ab34564dafe05a59cdbc715507992cdf825e8c6f63f3c622fd"
},
"downloads": -1,
"filename": "progressivepostgres-0.0.3-py3-none-any.whl",
"has_sig": false,
"md5_digest": "eb12cc5de9a13bcc64984edb5872a1b8",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.6",
"size": 11088,
"upload_time": "2025-01-09T12:07:36",
"upload_time_iso_8601": "2025-01-09T12:07:36.723233Z",
"url": "https://files.pythonhosted.org/packages/8a/e5/5f84d0914a4fecb273aa095ae24c22455c5962cbfea1553b3d0810b358eb/progressivepostgres-0.0.3-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "bfa8fb2de05d5e20c336e22e2c5b4d5323c778175cd5ecc01d6ef411fae92054",
"md5": "431d22d59c4e36adf6b19c3602759015",
"sha256": "d050fa953ed1036b20dccc4b1cf19a48c8f5ca188a902c168ec1dbee073b608b"
},
"downloads": -1,
"filename": "progressivepostgres-0.0.3.tar.gz",
"has_sig": false,
"md5_digest": "431d22d59c4e36adf6b19c3602759015",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.6",
"size": 161993,
"upload_time": "2025-01-09T12:07:41",
"upload_time_iso_8601": "2025-01-09T12:07:41.825960Z",
"url": "https://files.pythonhosted.org/packages/bf/a8/fb2de05d5e20c336e22e2c5b4d5323c778175cd5ecc01d6ef411fae92054/progressivepostgres-0.0.3.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-01-09 12:07:41",
"github": false,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"lcname": "progressivepostgres"
}
JSON