[](https://github.com/wmo-im/pywis-pubsub/actions)
[](https://github.com/wmo-im/pywis-pubsub/actions)
# pywis-pubsub
## Overview
pywis-pubsub provides subscription and download capability of WMO data from WIS2
infrastructure services.
## Installation
The easiest way to install pywis-pubsub is via the Python [pip](https://pip.pypa.io)
utility:
```bash
pip3 install pywis-pubsub
```
### Requirements
- Python 3
- [virtualenv](https://virtualenv.pypa.io)
### Dependencies
Dependencies are listed in [requirements.txt](requirements.txt). Dependencies
are automatically installed during pywis-pubsub installation.
#### Windows installations
Note that you will need Cython and [Shapely Windows wheels](https://pypi.org/project/shapely/#files) for windows for your architecture
prior to installing pywis-pubsub.
### Installing pywis-pubsub
```bash
# setup virtualenv
python3 -m venv --system-site-packages pywis-pubsub
cd pywis-pubsub
source bin/activate
# clone codebase and install
git clone https://github.com/wmo-im/pywis-pubsub.git
cd pywis-pubsub
python3 setup.py install
```
## Running
First check pywis-pubsub was correctly installed
```bash
pywis-pubsub --version
```
Create configuration
```bash
cp pywis-pubsub-config.yml local.yml
vim local.yml # update accordingly to configure subscribe options
```
### Subscribing
```bash
# sync WIS2 notification schema
pywis-pubsub schema sync
# connect, and simply echo messages
pywis-pubsub subscribe --config local.yml
# subscribe, and download data from message
pywis-pubsub subscribe --config local.yml --download
# subscribe, and filter messages by geometry
pywis-pubsub subscribe --config local.yml --bbox=-142,42,-52,84
# subscribe, and filter messages by geometry, increase debugging verbosity
pywis-pubsub subscribe --config local.yml --bbox=-142,42,-52,84 --verbosity=DEBUG
```
### Validating a message and verifying data
```bash
# validate a message
pywis-pubsub message validate /path/to/message1.json
# verify data from a message
pywis-pubsub message verify /path/to/message1.json
# validate WNM against abstract test suite (file on disk)
pywis-pubsub ets validate /path/to/file.json
# validate WNM against abstract test suite (URL)
pywis-pubsub ets validate https://example.org/path/to/file.json
# validate WNM against abstract test suite (URL), but turn JSON Schema validation off
pywis-pubsub ets validate https://example.org/path/to/file.json --no-fail-on-schema-validation
# key performance indicators
# set environment variable for GDC URL
export PYWIS_PUBSUB_GDC_URL=https://api.weather.gc.ca/collections/wis2-discovery-metadata
# all key performance indicators at once
pywis-pubsub kpi validate https://example.org/path/to/file.json --verbosity DEBUG
# all key performance indicators at once, but turn ETS validation off
pywis-pubsub kpi validate https://example.org/path/to/file.json --no-fail-on-ets --verbosity DEBUG
# all key performance indicators at once, in summary
pywis-pubsub kpi validate https://example.org/path/to/file.json --verbosity DEBUG --summary
# selected key performance indicator
pywis-pubsub kpi validate --kpi metadata_id /path/to/file.json -v INFO
```
### Publishing
```bash
cp pub-config-example.yml pub-local.yml
vim pub-local.yml # update accordingly to configure publishing options
# example publishing a WIS2 notification message with attributes:
# unique-id=stationXYZ-20221111085500
# data-url=http://www.meteo.xx/stationXYZ-20221111085500.bufr4
# lon,lat,elevation=33.8,11.8,112
# wigos_station_identifier=0-20000-12345
pywis-pubsub publish --topic origin/a/wis2/centre-id/data/core/weather --config pub-local.yml -i stationXYZ-20221111085500 -u https://example.org/stationXYZ-20221111085500.bufr4 -g 33.8,-11.8,8.112 -w 0-20000-12345
# publish a message with a WCMP2 metadata id
pywis-pubsub publish --topic origin/a/wis2/centre-id/data/core/weather --config pub-local.yml -i stationXYZ-20221111085500 -u https://example.org/stationXYZ-20221111085500.bufr4 -g 33.8,-11.8,8.112 -w 0-20000-12345 --metadata-id "x-urn:wmo:md:test-foo:htebmal2001"
# publish a message with a datetime (instant)
pywis-pubsub publish --topic origin/a/wis2/centre-id/data/core/weather --config pub-local.yml -i stationXYZ-20221111085500 -u https://example.org/stationXYZ-20221111085500.bufr4 -g 33.8,-11.8,8.112 -w 0-20000-12345 --metadata-id "x-urn:wmo:md:test-foo:htebmal2001" --datetime 2024-01-08T22:56:23Z
# publish a message with a start and end datetime (extent)
pywis-pubsub publish --topic origin/a/wis2/centre-id/data/core/weather --config pub-local.yml -i stationXYZ-20221111085500 -u https://example.org/stationXYZ-20221111085500.bufr4 -g 33.8,-11.8,8.112 -w 0-20000-12345 --metadata-id "x-urn:wmo:md:test-foo:htebmal2001" --datetime 2024-01-08T20:56:23Z/2024-01-08T22:56:43Z
# publish a message as a data update
pywis-pubsub publish --topic origin/a/wis2/centre-id/data/core/weather --config pub-local.yml -i stationXYZ-20221111085500 -u https://example.org/stationXYZ-20221111085500.bufr4 -g 33.8,-11.8,8.112 -w 0-20000-12345 --metadata-id "x-urn:wmo:md:test-foo:htebmal2001" --operation update
# publish a message as a data deletion
pywis-pubsub publish --topic origin/a/wis2/centre-id/data/core/weather --config pub-local.yml -i stationXYZ-20221111085500 -u https://example.org/stationXYZ-20221111085500.bufr4 -g 33.8,-11.8,8.112 -w 0-20000-12345 --metadata-id "x-urn:wmo:md:test-foo:htebmal2001" --operation delete
# publish a message from file on disk
pywis-pubsub publish --topic origin/a/wis2/centre-id/data/core/weather --config pub-local.yml --file my_message.json
```
### Using the API
Python examples:
Subscribing to a WIS2 Global Broker
```python
from pywis_pubsub.mqtt import MQTTPubSubClient
options = {
'storage': {
'type': 'fs',
'basedir': '/tmp'
},
'bbox': [-90, -180, 90, 180]
}
topics = [
'topic1',
'topic2'
]
m = MQTTPubSubClient('mqtt://localhost:1883', options)
m.sub(topics)
```
Publishing a WIS2 Notification Message
```python
from datetime import datetime, timezone
from pywis_pubsub.mqtt import MQTTPubSubClient
from pywis_pubsub.publish import create_message
message = create_message(
topic='foo/bar',
content_type='application/x-bufr',
url='http://www.meteo.xx/stationXYZ-20221111085500.bufr4',
identifier='stationXYZ-20221111085500',
datetime=datetime.now(timezone.utc),
geometry=[33.8, -11.8, 123],
metadata_id='x-urn:wmo:md:test-foo:htebmal2001',
wigos_station_identifier='0-20000-12345',
operation='update'
)
m = MQTTPubSubClient('mqtt://localhost:1883')
m.pub(topic, json.dumps(message))
```
Running KPIs
```pycon
>>> # test KPI
>>> import json
>>> from pywis_pubsub.kpi import WNMKeyPerformanceIndicators
>>> with open('/path/to/file.json') as fh:
... data = json.load(fh)
>>> kpis = WNMKeyPerformanceIndicators(data)
>>> results = kpis.evaluate()
>>> results['summary']
```
## Development
### Running Tests
```bash
# install dev requirements
pip3 install -r requirements-dev.txt
# run tests like this:
python3 tests/run_tests.py
# or this:
python3 setup.py test
```
## Releasing
```bash
# create release (x.y.z is the release version)
vi pywis_pubsub/__init__.py # update __version__
vi debian/changelog # add changelog entry
git commit -am 'update release version x.y.z'
git push origin main
git tag -a x.y.z -m 'tagging release version x.y.z'
git push --tags
# upload to PyPI
rm -fr build dist *.egg-info
python3 setup.py sdist bdist_wheel --universal
twine upload dist/*
# publish release on GitHub (https://github.com/wmo-im/pywis-pubsub/releases/new)
# bump version back to dev
vi pywis_pubsub/__init__.py # update __version__
git commit -am 'back to dev'
git push origin main
```
### Code Conventions
* [PEP8](https://www.python.org/dev/peps/pep-0008)
### Bugs and Issues
All bugs, enhancements and issues are managed on [GitHub](https://github.com/wmo-im/pywis-pubsub/issues).
## Contact
* [Antje Schremmer](https://github.com/antje-s)
* [Tom Kralidis](https://github.com/tomkralidis)
* [Maaike Limper](https://github.com/maaikelimper)
Raw data
{
"_id": null,
"home_page": "https://github.com/wmo-im/pywis-pubsub",
"name": "pywis-pubsub",
"maintainer": "Tom Kralidis",
"docs_url": null,
"requires_python": null,
"maintainer_email": "tomkraldis@gmail.com",
"keywords": "WIS2 PubSub broker topic",
"author": "Antje Schremmer",
"author_email": "antje.schremmer@dwd.de",
"download_url": "https://files.pythonhosted.org/packages/8c/ec/823fa150e3b6bee6c31cc398174819042ef89c4625ba1d6801ab4686045f/pywis-pubsub-0.7.1.tar.gz",
"platform": "all",
"description": "[](https://github.com/wmo-im/pywis-pubsub/actions)\n[](https://github.com/wmo-im/pywis-pubsub/actions)\n\n# pywis-pubsub\n\n## Overview\n\npywis-pubsub provides subscription and download capability of WMO data from WIS2\ninfrastructure services.\n\n## Installation\n\nThe easiest way to install pywis-pubsub is via the Python [pip](https://pip.pypa.io)\nutility:\n\n```bash\npip3 install pywis-pubsub\n```\n\n### Requirements\n- Python 3\n- [virtualenv](https://virtualenv.pypa.io)\n\n### Dependencies\nDependencies are listed in [requirements.txt](requirements.txt). Dependencies\nare automatically installed during pywis-pubsub installation.\n\n#### Windows installations\nNote that you will need Cython and [Shapely Windows wheels](https://pypi.org/project/shapely/#files) for windows for your architecture\nprior to installing pywis-pubsub.\n\n\n### Installing pywis-pubsub\n\n```bash\n# setup virtualenv\npython3 -m venv --system-site-packages pywis-pubsub\ncd pywis-pubsub\nsource bin/activate\n\n# clone codebase and install\ngit clone https://github.com/wmo-im/pywis-pubsub.git\ncd pywis-pubsub\npython3 setup.py install\n```\n\n## Running\n\nFirst check pywis-pubsub was correctly installed\n\n```bash\npywis-pubsub --version\n```\n\nCreate configuration\n\n```bash\ncp pywis-pubsub-config.yml local.yml\nvim local.yml # update accordingly to configure subscribe options\n```\n\n### Subscribing\n\n```bash\n# sync WIS2 notification schema\npywis-pubsub schema sync\n\n# connect, and simply echo messages\npywis-pubsub subscribe --config local.yml\n\n# subscribe, and download data from message\npywis-pubsub subscribe --config local.yml --download\n\n# subscribe, and filter messages by geometry\npywis-pubsub subscribe --config local.yml --bbox=-142,42,-52,84\n\n# subscribe, and filter messages by geometry, increase debugging verbosity\npywis-pubsub subscribe --config local.yml --bbox=-142,42,-52,84 --verbosity=DEBUG\n```\n\n### Validating a message and verifying data\n\n```bash\n# validate a message\npywis-pubsub message validate /path/to/message1.json\n\n# verify data from a message\npywis-pubsub message verify /path/to/message1.json\n\n# validate WNM against abstract test suite (file on disk)\npywis-pubsub ets validate /path/to/file.json\n\n# validate WNM against abstract test suite (URL)\npywis-pubsub ets validate https://example.org/path/to/file.json\n\n# validate WNM against abstract test suite (URL), but turn JSON Schema validation off\npywis-pubsub ets validate https://example.org/path/to/file.json --no-fail-on-schema-validation\n\n# key performance indicators\n\n# set environment variable for GDC URL\nexport PYWIS_PUBSUB_GDC_URL=https://api.weather.gc.ca/collections/wis2-discovery-metadata\n\n# all key performance indicators at once\npywis-pubsub kpi validate https://example.org/path/to/file.json --verbosity DEBUG\n\n# all key performance indicators at once, but turn ETS validation off\npywis-pubsub kpi validate https://example.org/path/to/file.json --no-fail-on-ets --verbosity DEBUG\n\n# all key performance indicators at once, in summary\npywis-pubsub kpi validate https://example.org/path/to/file.json --verbosity DEBUG --summary\n\n# selected key performance indicator\npywis-pubsub kpi validate --kpi metadata_id /path/to/file.json -v INFO\n```\n\n### Publishing\n\n```bash\ncp pub-config-example.yml pub-local.yml\nvim pub-local.yml # update accordingly to configure publishing options\n\n# example publishing a WIS2 notification message with attributes: \n# unique-id=stationXYZ-20221111085500 \n# data-url=http://www.meteo.xx/stationXYZ-20221111085500.bufr4 \n# lon,lat,elevation=33.8,11.8,112\n# wigos_station_identifier=0-20000-12345\npywis-pubsub publish --topic origin/a/wis2/centre-id/data/core/weather --config pub-local.yml -i stationXYZ-20221111085500 -u https://example.org/stationXYZ-20221111085500.bufr4 -g 33.8,-11.8,8.112 -w 0-20000-12345\n\n# publish a message with a WCMP2 metadata id\npywis-pubsub publish --topic origin/a/wis2/centre-id/data/core/weather --config pub-local.yml -i stationXYZ-20221111085500 -u https://example.org/stationXYZ-20221111085500.bufr4 -g 33.8,-11.8,8.112 -w 0-20000-12345 --metadata-id \"x-urn:wmo:md:test-foo:htebmal2001\"\n\n# publish a message with a datetime (instant)\npywis-pubsub publish --topic origin/a/wis2/centre-id/data/core/weather --config pub-local.yml -i stationXYZ-20221111085500 -u https://example.org/stationXYZ-20221111085500.bufr4 -g 33.8,-11.8,8.112 -w 0-20000-12345 --metadata-id \"x-urn:wmo:md:test-foo:htebmal2001\" --datetime 2024-01-08T22:56:23Z\n\n# publish a message with a start and end datetime (extent)\npywis-pubsub publish --topic origin/a/wis2/centre-id/data/core/weather --config pub-local.yml -i stationXYZ-20221111085500 -u https://example.org/stationXYZ-20221111085500.bufr4 -g 33.8,-11.8,8.112 -w 0-20000-12345 --metadata-id \"x-urn:wmo:md:test-foo:htebmal2001\" --datetime 2024-01-08T20:56:23Z/2024-01-08T22:56:43Z\n\n# publish a message as a data update\npywis-pubsub publish --topic origin/a/wis2/centre-id/data/core/weather --config pub-local.yml -i stationXYZ-20221111085500 -u https://example.org/stationXYZ-20221111085500.bufr4 -g 33.8,-11.8,8.112 -w 0-20000-12345 --metadata-id \"x-urn:wmo:md:test-foo:htebmal2001\" --operation update\n\n# publish a message as a data deletion\npywis-pubsub publish --topic origin/a/wis2/centre-id/data/core/weather --config pub-local.yml -i stationXYZ-20221111085500 -u https://example.org/stationXYZ-20221111085500.bufr4 -g 33.8,-11.8,8.112 -w 0-20000-12345 --metadata-id \"x-urn:wmo:md:test-foo:htebmal2001\" --operation delete\n\n# publish a message from file on disk\npywis-pubsub publish --topic origin/a/wis2/centre-id/data/core/weather --config pub-local.yml --file my_message.json\n```\n\n### Using the API\n\nPython examples:\n\nSubscribing to a WIS2 Global Broker\n```python\nfrom pywis_pubsub.mqtt import MQTTPubSubClient\n\noptions = {\n 'storage': {\n 'type': 'fs',\n 'basedir': '/tmp'\n },\n 'bbox': [-90, -180, 90, 180]\n}\ntopics = [\n 'topic1',\n 'topic2'\n]\n\nm = MQTTPubSubClient('mqtt://localhost:1883', options)\nm.sub(topics)\n```\n\nPublishing a WIS2 Notification Message\n```python\nfrom datetime import datetime, timezone\n\nfrom pywis_pubsub.mqtt import MQTTPubSubClient\nfrom pywis_pubsub.publish import create_message\n\nmessage = create_message(\n topic='foo/bar',\n content_type='application/x-bufr',\n url='http://www.meteo.xx/stationXYZ-20221111085500.bufr4', \n identifier='stationXYZ-20221111085500', \n datetime=datetime.now(timezone.utc),\n geometry=[33.8, -11.8, 123],\n metadata_id='x-urn:wmo:md:test-foo:htebmal2001',\n wigos_station_identifier='0-20000-12345',\n operation='update'\n)\n\nm = MQTTPubSubClient('mqtt://localhost:1883')\nm.pub(topic, json.dumps(message))\n```\n\nRunning KPIs\n```pycon\n>>> # test KPI\n>>> import json\n>>> from pywis_pubsub.kpi import WNMKeyPerformanceIndicators\n>>> with open('/path/to/file.json') as fh:\n... data = json.load(fh)\n>>> kpis = WNMKeyPerformanceIndicators(data)\n>>> results = kpis.evaluate()\n>>> results['summary']\n```\n\n## Development\n\n### Running Tests\n\n```bash\n# install dev requirements\npip3 install -r requirements-dev.txt\n\n# run tests like this:\npython3 tests/run_tests.py\n\n# or this:\npython3 setup.py test\n```\n\n## Releasing\n\n```bash\n# create release (x.y.z is the release version)\nvi pywis_pubsub/__init__.py # update __version__\nvi debian/changelog # add changelog entry\ngit commit -am 'update release version x.y.z'\ngit push origin main\ngit tag -a x.y.z -m 'tagging release version x.y.z'\ngit push --tags\n\n# upload to PyPI\nrm -fr build dist *.egg-info\npython3 setup.py sdist bdist_wheel --universal\ntwine upload dist/*\n\n# publish release on GitHub (https://github.com/wmo-im/pywis-pubsub/releases/new)\n\n# bump version back to dev\nvi pywis_pubsub/__init__.py # update __version__\ngit commit -am 'back to dev'\ngit push origin main\n```\n\n### Code Conventions\n\n* [PEP8](https://www.python.org/dev/peps/pep-0008)\n\n### Bugs and Issues\n\nAll bugs, enhancements and issues are managed on [GitHub](https://github.com/wmo-im/pywis-pubsub/issues).\n\n## Contact\n\n* [Antje Schremmer](https://github.com/antje-s)\n* [Tom Kralidis](https://github.com/tomkralidis)\n* [Maaike Limper](https://github.com/maaikelimper)\n",
"bugtrack_url": null,
"license": "Apache Software License",
"summary": "pywis-pubsub provides subscription and download capability of WMO data from WIS2 infrastructure services",
"version": "0.7.1",
"project_urls": {
"Homepage": "https://github.com/wmo-im/pywis-pubsub"
},
"split_keywords": [
"wis2",
"pubsub",
"broker",
"topic"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "df8701116c59577753a44e51102cc088bc70b99c0280d67280c844184d83d4db",
"md5": "157a6634378bea04a9e6748148a01616",
"sha256": "efeeba642cba7b521df176ff9564616442061fd5af36bb63b522925f6a037cc1"
},
"downloads": -1,
"filename": "pywis_pubsub-0.7.1-py2.py3-none-any.whl",
"has_sig": false,
"md5_digest": "157a6634378bea04a9e6748148a01616",
"packagetype": "bdist_wheel",
"python_version": "py2.py3",
"requires_python": null,
"size": 36235,
"upload_time": "2024-03-26T02:56:46",
"upload_time_iso_8601": "2024-03-26T02:56:46.096467Z",
"url": "https://files.pythonhosted.org/packages/df/87/01116c59577753a44e51102cc088bc70b99c0280d67280c844184d83d4db/pywis_pubsub-0.7.1-py2.py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "8cec823fa150e3b6bee6c31cc398174819042ef89c4625ba1d6801ab4686045f",
"md5": "cd0f863601223e5e75f617521f85f593",
"sha256": "bdb9353bc233c2b6f07da66d663fca3c18214766b6d98021c8936d6070a7a366"
},
"downloads": -1,
"filename": "pywis-pubsub-0.7.1.tar.gz",
"has_sig": false,
"md5_digest": "cd0f863601223e5e75f617521f85f593",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 27535,
"upload_time": "2024-03-26T02:56:48",
"upload_time_iso_8601": "2024-03-26T02:56:48.014537Z",
"url": "https://files.pythonhosted.org/packages/8c/ec/823fa150e3b6bee6c31cc398174819042ef89c4625ba1d6801ab4686045f/pywis-pubsub-0.7.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-03-26 02:56:48",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "wmo-im",
"github_project": "pywis-pubsub",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"requirements": [
{
"name": "boto3",
"specs": []
},
{
"name": "click",
"specs": []
},
{
"name": "jsonschema",
"specs": []
},
{
"name": "paho-mqtt",
"specs": [
[
"<",
"2"
]
]
},
{
"name": "pyyaml",
"specs": []
},
{
"name": "requests",
"specs": []
},
{
"name": "shapely",
"specs": [
[
"<",
"2"
]
]
},
{
"name": "urllib3",
"specs": [
[
"<",
"2"
]
]
}
],
"lcname": "pywis-pubsub"
}
JSON
