.. image:: https://github.com/earthobservations/luftdatenpumpe/workflows/Tests/badge.svg
:target: https://github.com/earthobservations/luftdatenpumpe/actions?workflow=Tests
:alt: CI outcome
.. image:: https://codecov.io/gh/earthobservations/luftdatenpumpe/branch/main/graph/badge.svg
:target: https://codecov.io/gh/earthobservations/luftdatenpumpe
:alt: Test suite code coverage
.. image:: https://pepy.tech/badge/luftdatenpumpe/month
:target: https://pypi.org/project/luftdatenpumpe/
:alt: PyPI downloads per month
.. image:: https://img.shields.io/pypi/v/luftdatenpumpe.svg
:target: https://pypi.org/project/luftdatenpumpe/
:alt: Package version on PyPI
.. image:: https://img.shields.io/pypi/status/luftdatenpumpe.svg
:target: https://pypi.org/project/luftdatenpumpe/
:alt: Project status (alpha, beta, stable)
.. image:: https://img.shields.io/pypi/pyversions/luftdatenpumpe.svg
:target: https://pypi.org/project/luftdatenpumpe/
:alt: Supported Python versions
.. image:: https://img.shields.io/pypi/l/luftdatenpumpe.svg
:target: https://github.com/earthobservations/luftdatenpumpe/blob/main/LICENSE
:alt: Project license
|
##############
Luftdatenpumpe
##############
.. image:: https://assets.okfn.org/images/ok_buttons/od_80x15_red_green.png
:target: https://okfn.org/opendata/
.. image:: https://assets.okfn.org/images/ok_buttons/oc_80x15_blue.png
:target: https://okfn.org/opendata/
.. image:: https://assets.okfn.org/images/ok_buttons/os_80x15_orange_grey.png
:target: https://okfn.org/opendata/
*****
About
*****
Process live and historical data from `luftdaten.info`_, irceline_ and OpenAQ_.
Filter by station-id, sensor-id and sensor-type, apply reverse geocoding,
store into TSDB_ and RDBMS_ databases (InfluxDB_ and PostGIS_),
publish to MQTT_ or just output as JSON.
.. figure:: https://cdn.jsdelivr.net/gh/earthobservations/luftdatenpumpe@main/doc/logo.svg
:target: https://github.com/earthobservations/luftdatenpumpe
:height: 200px
:width: 200px
********
Features
********
1. Luftdatenpumpe_ acquires the measurement readings either from the livedata API
of `luftdaten.info`_ or from its archived CSV files published to `archive.luftdaten.info`.
To minimize impact on the upstream servers, all data gets reasonably cached.
2. While iterating the readings, it optionally filters on station-id, sensor-id or sensor-type
and restrains information processing to the corresponding stations and sensors.
3. Then, each station's location information gets enhanced by
- attaching its geospatial position as a Geohash_.
- attaching a synthetic real-world address resolved using the reverse geocoding service Nominatim_ by OpenStreetMap_.
4. Information about stations can be
- displayed on STDOUT or STDERR in JSON format.
- filtered and transformed interactively through jq_, the swiss army knife of JSON manipulation.
- stored into RDBMS_ databases like PostgreSQL_ using the fine dataset_ package.
Being built on top of SQLAlchemy_, this supports all major databases.
- queried using advanced geospatial features when running PostGIS_, please
follow up reading the `Luftdatenpumpe PostGIS tutorial <doc-postgis_>`_.
5. Measurement readings can be
- displayed on STDOUT or STDERR in JSON format, which allows for piping into jq_ again.
- forwarded to MQTT_.
- stored to InfluxDB_ and then
- displayed in Grafana_.
********
Synopsis
********
::
# List networks
luftdatenpumpe networks
# List LDI stations
luftdatenpumpe stations --network=ldi --station=49,1033 --reverse-geocode
# Store list of LDI stations and metadata into RDBMS database (PostgreSQL), also display on STDERR
luftdatenpumpe stations --network=ldi --station=49,1033 --reverse-geocode --target=postgresql://luftdatenpumpe@localhost/weatherbase
# Store LDI readings into InfluxDB
luftdatenpumpe readings --network=ldi --station=49,1033 --target=influxdb://luftdatenpumpe@localhost/luftdaten_info
# Forward LDI readings to MQTT
luftdatenpumpe readings --network=ldi --station=49,1033 --target=mqtt://mqtt.example.org/luftdaten.info
For a full overview about all program options including meaningful examples,
you might just want to run ``luftdatenpumpe --help`` on your command line
or visit `luftdatenpumpe --help`_.
***********
Screenshots
***********
Luftdaten-Viewer displays stations and measurements from luftdaten.info (LDI) in Grafana.
Map display and filtering
=========================
- Filter by different synthesized address components and sensor type.
- Display measurements from filtered stations on Grafana Worldmap Panel.
- Display filtered list of stations with corresponding information in tabular form.
- Measurement values are held against configured thresholds so points are colored appropriately.
.. image:: https://community.hiveeyes.org/uploads/default/original/2X/f/f455d3afcd20bfa316fefbe69e43ca2fe159e62d.png
:target: https://weather.hiveeyes.org/grafana/d/9d9rnePmk/amo-ldi-stations-5-map-by-sensor-type
Map popup labels
================
- Humanized label computed from synthesized OpenStreetMap address.
- Numeric station identifier.
- Measurement value, unit and field name.
.. image:: https://community.hiveeyes.org/uploads/default/original/2X/4/48eeda1a1d418eaf698b241a65080666abcf2497.png
:target: https://weather.hiveeyes.org/grafana/d/9d9rnePmk/amo-ldi-stations-5-map-by-sensor-type
************
Installation
************
If you are running Python 3 already, installing the program should be as easy as::
pip install luftdatenpumpe
At this point, you should be able to conduct simple tests like
``luftdatenpumpe stations`` as seen in the synopsis section above.
At least, you should verify the installation succeeded by running::
luftdatenpumpe --version
However, you might have to resolve some prerequisites so you want to follow
the detailed installation instructions at `install Luftdatenpumpe`_.
****************
Luftdaten-Viewer
****************
About
=====
Using Luftdatenpumpe, you can build user-friendly interactive GIS systems
on top of PostGIS, InfluxDB and Grafana. We are calling this "Luftdaten-Viewer".
Without further ado, you might enjoy reading about existing "Luftdaten-Viewer"
installations at `Testimonials for Luftdatenpumpe`_.
Instructions
============
These installation instructions outline how to setup the whole system to build
similar interactive data visualization compositions of map-, graph- and other
panel-widgets like outlined in the "Testimonials" section.
- `Luftdaten-Viewer Applications`_
- `Luftdaten-Viewer Databases`_
- `Luftdaten-Viewer Grafana`_
*******
License
*******
This project is licensed under the terms of the GNU AGPL license.
********************
Content attributions
********************
The copyright of particular images and pictograms are held by their respective owners, unless otherwise noted.
Icons and pictograms
====================
- `Water Pump Free Icon <https://www.onlinewebfonts.com/icon/97990>`_ from
`Icon Fonts <http://www.onlinewebfonts.com/icon>`_ is licensed by CC BY 3.0.
.. _doc-virtualenv: https://github.com/earthobservations/luftdatenpumpe/blob/main/doc/virtualenv.rst
.. _doc-postgis: https://github.com/earthobservations/luftdatenpumpe/blob/main/doc/postgis.rst
.. _luftdaten.info: https://luftdaten.info/
.. _irceline: http://www.irceline.be/en/documentation/open-data
.. _OpenAQ: https://openaq.org/
.. _Luftdatenpumpe: https://github.com/earthobservations/luftdatenpumpe
.. _Testimonials for Luftdatenpumpe: https://github.com/earthobservations/luftdatenpumpe/blob/main/doc/testimonials.rst
.. _luftdatenpumpe --help: https://github.com/earthobservations/luftdatenpumpe/blob/main/doc/usage.rst
.. _install Luftdatenpumpe: https://github.com/earthobservations/luftdatenpumpe/blob/main/doc/setup/luftdatenpumpe.rst
.. _Luftdaten-Viewer Applications: https://github.com/earthobservations/luftdatenpumpe/blob/main/doc/setup/ldview-applications.rst
.. _Luftdaten-Viewer Databases: https://github.com/earthobservations/luftdatenpumpe/blob/main/doc/setup/ldview-databases.rst
.. _Luftdaten-Viewer Grafana: https://github.com/earthobservations/luftdatenpumpe/blob/main/doc/setup/ldview-grafana.rst
.. _Luftdaten-Viewer Cron Job: https://github.com/earthobservations/luftdatenpumpe/blob/main/doc/setup/ldview-cronjob.rst
.. _Erneuerung der Luftdatenpumpe: https://community.hiveeyes.org/t/erneuerung-der-luftdatenpumpe/1199
.. _The Hiveeyes Project: https://hiveeyes.org/
.. _OpenStreetMap: https://en.wikipedia.org/wiki/OpenStreetMap
.. _Nominatim: https://wiki.openstreetmap.org/wiki/Nominatim
.. _Geohash: https://en.wikipedia.org/wiki/Geohash
.. _dataset: https://dataset.readthedocs.io/
.. _SQLAlchemy: https://www.sqlalchemy.org/
.. _TSDB: https://en.wikipedia.org/wiki/Time_series_database
.. _RDBMS: https://en.wikipedia.org/wiki/Relational_database_management_system
.. _MQTT: http://mqtt.org/
.. _PostgreSQL: https://www.postgresql.org/
.. _PostGIS: https://postgis.net/
.. _InfluxDB: https://github.com/influxdata/influxdb
.. _Grafana: https://github.com/grafana/grafana
.. _jq: https://stedolan.github.io/jq/
Raw data
{
"_id": null,
"home_page": "https://github.com/earthobservations/luftdatenpumpe",
"name": "luftdatenpumpe",
"maintainer": "",
"docs_url": null,
"requires_python": "",
"maintainer_email": "",
"keywords": "luftdaten.info irceline openaq air quality particulate matter pollution feinstaub luftdaten ogc sos sensor network observation opendata data acquisition transformation export geospatial temporal timeseries http rest json api rdbms sql mysql mosquitto mqtt openstreetmap nominatim postgis postgresql influxdb grafana 52north",
"author": "Andreas Motl",
"author_email": "andreas.motl@panodata.org",
"download_url": "https://files.pythonhosted.org/packages/09/e5/a5f0886c37fc0d8b89cb3dbb9da7766210389318e9dbb633546df4bf35ec/luftdatenpumpe-0.21.1.tar.gz",
"platform": null,
"description": ".. image:: https://github.com/earthobservations/luftdatenpumpe/workflows/Tests/badge.svg\n :target: https://github.com/earthobservations/luftdatenpumpe/actions?workflow=Tests\n :alt: CI outcome\n\n.. image:: https://codecov.io/gh/earthobservations/luftdatenpumpe/branch/main/graph/badge.svg\n :target: https://codecov.io/gh/earthobservations/luftdatenpumpe\n :alt: Test suite code coverage\n\n.. image:: https://pepy.tech/badge/luftdatenpumpe/month\n :target: https://pypi.org/project/luftdatenpumpe/\n :alt: PyPI downloads per month\n\n.. image:: https://img.shields.io/pypi/v/luftdatenpumpe.svg\n :target: https://pypi.org/project/luftdatenpumpe/\n :alt: Package version on PyPI\n\n.. image:: https://img.shields.io/pypi/status/luftdatenpumpe.svg\n :target: https://pypi.org/project/luftdatenpumpe/\n :alt: Project status (alpha, beta, stable)\n\n.. image:: https://img.shields.io/pypi/pyversions/luftdatenpumpe.svg\n :target: https://pypi.org/project/luftdatenpumpe/\n :alt: Supported Python versions\n\n.. image:: https://img.shields.io/pypi/l/luftdatenpumpe.svg\n :target: https://github.com/earthobservations/luftdatenpumpe/blob/main/LICENSE\n :alt: Project license\n\n|\n\n##############\nLuftdatenpumpe\n##############\n\n.. image:: https://assets.okfn.org/images/ok_buttons/od_80x15_red_green.png\n :target: https://okfn.org/opendata/\n\n.. image:: https://assets.okfn.org/images/ok_buttons/oc_80x15_blue.png\n :target: https://okfn.org/opendata/\n\n.. image:: https://assets.okfn.org/images/ok_buttons/os_80x15_orange_grey.png\n :target: https://okfn.org/opendata/\n\n\n*****\nAbout\n*****\n\nProcess live and historical data from `luftdaten.info`_, irceline_ and OpenAQ_.\nFilter by station-id, sensor-id and sensor-type, apply reverse geocoding,\nstore into TSDB_ and RDBMS_ databases (InfluxDB_ and PostGIS_),\npublish to MQTT_ or just output as JSON.\n\n.. figure:: https://cdn.jsdelivr.net/gh/earthobservations/luftdatenpumpe@main/doc/logo.svg\n :target: https://github.com/earthobservations/luftdatenpumpe\n :height: 200px\n :width: 200px\n\n\n********\nFeatures\n********\n\n1. Luftdatenpumpe_ acquires the measurement readings either from the livedata API\n of `luftdaten.info`_ or from its archived CSV files published to `archive.luftdaten.info`.\n To minimize impact on the upstream servers, all data gets reasonably cached.\n\n2. While iterating the readings, it optionally filters on station-id, sensor-id or sensor-type\n and restrains information processing to the corresponding stations and sensors.\n\n3. Then, each station's location information gets enhanced by\n\n - attaching its geospatial position as a Geohash_.\n - attaching a synthetic real-world address resolved using the reverse geocoding service Nominatim_ by OpenStreetMap_.\n\n4. Information about stations can be\n\n - displayed on STDOUT or STDERR in JSON format.\n - filtered and transformed interactively through jq_, the swiss army knife of JSON manipulation.\n - stored into RDBMS_ databases like PostgreSQL_ using the fine dataset_ package.\n Being built on top of SQLAlchemy_, this supports all major databases.\n - queried using advanced geospatial features when running PostGIS_, please\n follow up reading the `Luftdatenpumpe PostGIS tutorial <doc-postgis_>`_.\n\n5. Measurement readings can be\n\n - displayed on STDOUT or STDERR in JSON format, which allows for piping into jq_ again.\n - forwarded to MQTT_.\n - stored to InfluxDB_ and then\n - displayed in Grafana_.\n\n\n********\nSynopsis\n********\n::\n\n # List networks\n luftdatenpumpe networks\n\n # List LDI stations\n luftdatenpumpe stations --network=ldi --station=49,1033 --reverse-geocode\n\n # Store list of LDI stations and metadata into RDBMS database (PostgreSQL), also display on STDERR\n luftdatenpumpe stations --network=ldi --station=49,1033 --reverse-geocode --target=postgresql://luftdatenpumpe@localhost/weatherbase\n\n # Store LDI readings into InfluxDB\n luftdatenpumpe readings --network=ldi --station=49,1033 --target=influxdb://luftdatenpumpe@localhost/luftdaten_info\n\n # Forward LDI readings to MQTT\n luftdatenpumpe readings --network=ldi --station=49,1033 --target=mqtt://mqtt.example.org/luftdaten.info\n\n\nFor a full overview about all program options including meaningful examples,\nyou might just want to run ``luftdatenpumpe --help`` on your command line\nor visit `luftdatenpumpe --help`_.\n\n\n\n***********\nScreenshots\n***********\n\nLuftdaten-Viewer displays stations and measurements from luftdaten.info (LDI) in Grafana.\n\n\nMap display and filtering\n=========================\n- Filter by different synthesized address components and sensor type.\n- Display measurements from filtered stations on Grafana Worldmap Panel.\n- Display filtered list of stations with corresponding information in tabular form.\n- Measurement values are held against configured thresholds so points are colored appropriately.\n\n.. image:: https://community.hiveeyes.org/uploads/default/original/2X/f/f455d3afcd20bfa316fefbe69e43ca2fe159e62d.png\n :target: https://weather.hiveeyes.org/grafana/d/9d9rnePmk/amo-ldi-stations-5-map-by-sensor-type\n\n\nMap popup labels\n================\n- Humanized label computed from synthesized OpenStreetMap address.\n- Numeric station identifier.\n- Measurement value, unit and field name.\n\n.. image:: https://community.hiveeyes.org/uploads/default/original/2X/4/48eeda1a1d418eaf698b241a65080666abcf2497.png\n :target: https://weather.hiveeyes.org/grafana/d/9d9rnePmk/amo-ldi-stations-5-map-by-sensor-type\n\n\n************\nInstallation\n************\n\nIf you are running Python 3 already, installing the program should be as easy as::\n\n pip install luftdatenpumpe\n\nAt this point, you should be able to conduct simple tests like\n``luftdatenpumpe stations`` as seen in the synopsis section above.\nAt least, you should verify the installation succeeded by running::\n\n luftdatenpumpe --version\n\nHowever, you might have to resolve some prerequisites so you want to follow\nthe detailed installation instructions at `install Luftdatenpumpe`_.\n\n\n****************\nLuftdaten-Viewer\n****************\n\nAbout\n=====\nUsing Luftdatenpumpe, you can build user-friendly interactive GIS systems\non top of PostGIS, InfluxDB and Grafana. We are calling this \"Luftdaten-Viewer\".\n\nWithout further ado, you might enjoy reading about existing \"Luftdaten-Viewer\"\ninstallations at `Testimonials for Luftdatenpumpe`_.\n\nInstructions\n============\nThese installation instructions outline how to setup the whole system to build\nsimilar interactive data visualization compositions of map-, graph- and other\npanel-widgets like outlined in the \"Testimonials\" section.\n\n- `Luftdaten-Viewer Applications`_\n- `Luftdaten-Viewer Databases`_\n- `Luftdaten-Viewer Grafana`_\n\n\n*******\nLicense\n*******\n\nThis project is licensed under the terms of the GNU AGPL license.\n\n\n********************\nContent attributions\n********************\n\nThe copyright of particular images and pictograms are held by their respective owners, unless otherwise noted.\n\nIcons and pictograms\n====================\n- `Water Pump Free Icon <https://www.onlinewebfonts.com/icon/97990>`_ from\n `Icon Fonts <http://www.onlinewebfonts.com/icon>`_ is licensed by CC BY 3.0.\n\n\n\n.. _doc-virtualenv: https://github.com/earthobservations/luftdatenpumpe/blob/main/doc/virtualenv.rst\n.. _doc-postgis: https://github.com/earthobservations/luftdatenpumpe/blob/main/doc/postgis.rst\n\n\n.. _luftdaten.info: https://luftdaten.info/\n.. _irceline: http://www.irceline.be/en/documentation/open-data\n.. _OpenAQ: https://openaq.org/\n\n.. _Luftdatenpumpe: https://github.com/earthobservations/luftdatenpumpe\n\n.. _Testimonials for Luftdatenpumpe: https://github.com/earthobservations/luftdatenpumpe/blob/main/doc/testimonials.rst\n.. _luftdatenpumpe --help: https://github.com/earthobservations/luftdatenpumpe/blob/main/doc/usage.rst\n.. _install Luftdatenpumpe: https://github.com/earthobservations/luftdatenpumpe/blob/main/doc/setup/luftdatenpumpe.rst\n.. _Luftdaten-Viewer Applications: https://github.com/earthobservations/luftdatenpumpe/blob/main/doc/setup/ldview-applications.rst\n.. _Luftdaten-Viewer Databases: https://github.com/earthobservations/luftdatenpumpe/blob/main/doc/setup/ldview-databases.rst\n.. _Luftdaten-Viewer Grafana: https://github.com/earthobservations/luftdatenpumpe/blob/main/doc/setup/ldview-grafana.rst\n.. _Luftdaten-Viewer Cron Job: https://github.com/earthobservations/luftdatenpumpe/blob/main/doc/setup/ldview-cronjob.rst\n\n.. _Erneuerung der Luftdatenpumpe: https://community.hiveeyes.org/t/erneuerung-der-luftdatenpumpe/1199\n\n.. _The Hiveeyes Project: https://hiveeyes.org/\n\n.. _OpenStreetMap: https://en.wikipedia.org/wiki/OpenStreetMap\n.. _Nominatim: https://wiki.openstreetmap.org/wiki/Nominatim\n.. _Geohash: https://en.wikipedia.org/wiki/Geohash\n.. _dataset: https://dataset.readthedocs.io/\n.. _SQLAlchemy: https://www.sqlalchemy.org/\n.. _TSDB: https://en.wikipedia.org/wiki/Time_series_database\n.. _RDBMS: https://en.wikipedia.org/wiki/Relational_database_management_system\n.. _MQTT: http://mqtt.org/\n\n.. _PostgreSQL: https://www.postgresql.org/\n.. _PostGIS: https://postgis.net/\n.. _InfluxDB: https://github.com/influxdata/influxdb\n.. _Grafana: https://github.com/grafana/grafana\n\n.. _jq: https://stedolan.github.io/jq/\n",
"bugtrack_url": null,
"license": "AGPL 3, EUPL 1.2",
"summary": "Process live and historical data from luftdaten.info, IRCELINE and OpenAQ. Filter by station-id, sensor-id and sensor-type, apply reverse geocoding, store into time-series and RDBMS databases, publish to MQTT, output as JSON, or visualize in Grafana.",
"version": "0.21.1",
"split_keywords": [
"luftdaten.info",
"irceline",
"openaq",
"air",
"quality",
"particulate",
"matter",
"pollution",
"feinstaub",
"luftdaten",
"ogc",
"sos",
"sensor",
"network",
"observation",
"opendata",
"data",
"acquisition",
"transformation",
"export",
"geospatial",
"temporal",
"timeseries",
"http",
"rest",
"json",
"api",
"rdbms",
"sql",
"mysql",
"mosquitto",
"mqtt",
"openstreetmap",
"nominatim",
"postgis",
"postgresql",
"influxdb",
"grafana",
"52north"
],
"urls": [
{
"comment_text": "",
"digests": {
"md5": "39bddf6ea6cabab9e69e4ddad29d9239",
"sha256": "b029ad99299efa8027dbcc1d645bfdbc70825d433c75d04bf1cb77feea4239db"
},
"downloads": -1,
"filename": "luftdatenpumpe-0.21.1-py3-none-any.whl",
"has_sig": false,
"md5_digest": "39bddf6ea6cabab9e69e4ddad29d9239",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": null,
"size": 71165,
"upload_time": "2022-12-05T20:20:47",
"upload_time_iso_8601": "2022-12-05T20:20:47.052935Z",
"url": "https://files.pythonhosted.org/packages/4a/df/d9042023f3c4cdc19abd956b586ca1344ca037ce3a1c7c527384419f4bce/luftdatenpumpe-0.21.1-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"md5": "5f424a0bc7b1d5f2faebc9b56a5e2a0a",
"sha256": "a2a1b839fc1fc9d7d228184ba457d251a6b081687e5e8d88bbe73480a6699954"
},
"downloads": -1,
"filename": "luftdatenpumpe-0.21.1.tar.gz",
"has_sig": false,
"md5_digest": "5f424a0bc7b1d5f2faebc9b56a5e2a0a",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 67017,
"upload_time": "2022-12-05T20:20:50",
"upload_time_iso_8601": "2022-12-05T20:20:50.125369Z",
"url": "https://files.pythonhosted.org/packages/09/e5/a5f0886c37fc0d8b89cb3dbb9da7766210389318e9dbb633546df4bf35ec/luftdatenpumpe-0.21.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2022-12-05 20:20:50",
"github": true,
"gitlab": false,
"bitbucket": false,
"github_user": "earthobservations",
"github_project": "luftdatenpumpe",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "luftdatenpumpe"
}
JSON
