# EM27 Metadata
## The purpose of this library
This repository is the single source of truth for our EM27 measurement logistics: "Where has each station been on each day of measurements?" We selected this format over putting it in a database due to various reasons:
- Easy to read, modify and extend by selective group members using GitHub permissions
- Changes to this are more evident here than in database logs
- Versioning (easy to revert mistakes)
- Automatic testing of the files integrities
- Easy import as a statically typed Python library
<br/>
## How it works
This repository only contains a Python library to interact with the metadata. The metadata itself is stored in local files or a GitHub repository. The library can load the metadata from both sources and provides a unified interface with static types to access it.
<br/>
## Library Usage
Install as a library:
```bash
poetry add em27-metadata
# or
pip install em27-metadata
```
```python
import pendulum
import em27_metadata
em27_metadata_store = em27_metadata.load_from_github(
github_repository="org-name/repo-name",
access_token="your-github-access-token",
)
# or load it from local files
em27_metadata_store = em27_metadata.load_from_local_files(
locations_path="location-data/locations.json",
sensors_path="location-data/sensors.json",
campaigns_path="location-data/campaigns.json",
)
metadata = em27_metadata_store.get(
sensor_id = "ma",
from_datetime = pendulum.DateTime(
year=2022, month=6, day=1, hour=0, minute=0, second=0
),
to_datetime = pendulum.DateTime(
year=2022, month=6, day=1, hour=23, minute=59, second=59
),
)
print(metadata)
```
Prints out:
```json
[
{
"sensor_id": "ma",
"serial_number": 61,
"from_datetime": "2022-06-01T00:00:00+00:00",
"to_datetime": "2022-06-01T23:59:59+00:00",
"location": {
"location_id": "TUM_I",
"details": "TUM Dach Innenstadt",
"lon": 11.569,
"lat": 48.151,
"alt": 539
},
"utc_offset": 0,
"pressure_data_source": "ma",
"pressure_calibration_factor": 1,
"output_calibration_factors_xco2": 1,
"output_calibration_factors_xch4": 1,
"output_calibration_factors_xco": 1,
"output_calibration_scheme": null
}
]
```
The object returned by `em27_metadata_store.get()` is of type `list[em27_metadata.types.SensorDataContext]`. It is a Pydantic model (https://docs.pydantic.dev/) but can be converted to a dictionary using `metadata.model_dump()`.
The list will contain one item per time period where the metadata properties are continuous (same location, etc.). You can find dummy data in the `data/` folder.
<br/>
## Set up an EM27 Metadata Storage Directory
You can use the repository https://github.com/tum-esm/em27-metadata-storage-template to create your own repository for storing the metadata. It contains a GitHub Actions workflow that automatically validates the metadata on every commit in any branch.
<br/>
## For Developers
Run tests:
```bash
# used inside the GitHub CI for this repo
pytest -m "ci"
# used inside the GitHub Actions workflow for storage repos
pytest -m "action"
# can be used for local development (skips pulling from GitHub)
pytest -m "local"
```
Publish the Package to PyPI:
```bash
poetry build
poetry publish
```
In order to test the "get metadata for a time period" function, the following example is used:
The test `tests/test_data_integrity.py` requests the time period `00:00` to `23:59`. The UTC offsets are specified (to be non-zero) from `02:00` to `15:59`, where it has one non-zero value, and from `16:00` to `21:59`, where it has another non-zero value. Each property has two (non-default) values over the day. There should be eight resulting chunks of metadata. All properties of each chunk are validated in the test.
Raw data
{
"_id": null,
"home_page": "https://github.com/tum-esm/em27-metadata",
"name": "em27-metadata",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.9,<4.0",
"maintainer_email": "",
"keywords": "",
"author": "Moritz Makowski",
"author_email": "moritz.makowski@tum.de",
"download_url": "https://files.pythonhosted.org/packages/00/a7/8f17ad893238a0cea015fb257a6aa5286f7632eb2b605adf70163ec17a9b/em27_metadata-1.0.0rc1.tar.gz",
"platform": null,
"description": "# EM27 Metadata\n\n## The purpose of this library\n\nThis repository is the single source of truth for our EM27 measurement logistics: \"Where has each station been on each day of measurements?\" We selected this format over putting it in a database due to various reasons:\n\n- Easy to read, modify and extend by selective group members using GitHub permissions\n- Changes to this are more evident here than in database logs\n- Versioning (easy to revert mistakes)\n- Automatic testing of the files integrities\n- Easy import as a statically typed Python library\n\n<br/>\n\n## How it works\n\nThis repository only contains a Python library to interact with the metadata. The metadata itself is stored in local files or a GitHub repository. The library can load the metadata from both sources and provides a unified interface with static types to access it.\n\n<br/>\n\n## Library Usage\n\nInstall as a library:\n\n```bash\npoetry add em27-metadata\n# or\npip install em27-metadata\n```\n\n```python\nimport pendulum\nimport em27_metadata\n\nem27_metadata_store = em27_metadata.load_from_github(\n github_repository=\"org-name/repo-name\",\n access_token=\"your-github-access-token\",\n)\n\n# or load it from local files\nem27_metadata_store = em27_metadata.load_from_local_files(\n locations_path=\"location-data/locations.json\",\n sensors_path=\"location-data/sensors.json\",\n campaigns_path=\"location-data/campaigns.json\",\n)\n\nmetadata = em27_metadata_store.get(\n sensor_id = \"ma\",\n from_datetime = pendulum.DateTime(\n year=2022, month=6, day=1, hour=0, minute=0, second=0\n ),\n to_datetime = pendulum.DateTime(\n year=2022, month=6, day=1, hour=23, minute=59, second=59\n ),\n)\n\nprint(metadata)\n\n```\n\nPrints out:\n\n```json\n[\n {\n \"sensor_id\": \"ma\",\n \"serial_number\": 61,\n \"from_datetime\": \"2022-06-01T00:00:00+00:00\",\n \"to_datetime\": \"2022-06-01T23:59:59+00:00\",\n \"location\": {\n \"location_id\": \"TUM_I\",\n \"details\": \"TUM Dach Innenstadt\",\n \"lon\": 11.569,\n \"lat\": 48.151,\n \"alt\": 539\n },\n \"utc_offset\": 0,\n \"pressure_data_source\": \"ma\",\n \"pressure_calibration_factor\": 1,\n \"output_calibration_factors_xco2\": 1,\n \"output_calibration_factors_xch4\": 1,\n \"output_calibration_factors_xco\": 1,\n \"output_calibration_scheme\": null\n }\n]\n```\n\nThe object returned by `em27_metadata_store.get()` is of type `list[em27_metadata.types.SensorDataContext]`. It is a Pydantic model (https://docs.pydantic.dev/) but can be converted to a dictionary using `metadata.model_dump()`.\n\nThe list will contain one item per time period where the metadata properties are continuous (same location, etc.). You can find dummy data in the `data/` folder.\n\n<br/>\n\n## Set up an EM27 Metadata Storage Directory\n\nYou can use the repository https://github.com/tum-esm/em27-metadata-storage-template to create your own repository for storing the metadata. It contains a GitHub Actions workflow that automatically validates the metadata on every commit in any branch.\n\n<br/>\n\n## For Developers\n\nRun tests:\n\n```bash\n# used inside the GitHub CI for this repo\npytest -m \"ci\"\n\n# used inside the GitHub Actions workflow for storage repos\npytest -m \"action\"\n\n# can be used for local development (skips pulling from GitHub)\npytest -m \"local\"\n```\n\nPublish the Package to PyPI:\n\n```bash\npoetry build\npoetry publish\n```\n\nIn order to test the \"get metadata for a time period\" function, the following example is used:\n\n\n\nThe test `tests/test_data_integrity.py` requests the time period `00:00` to `23:59`. The UTC offsets are specified (to be non-zero) from `02:00` to `15:59`, where it has one non-zero value, and from `16:00` to `21:59`, where it has another non-zero value. Each property has two (non-default) values over the day. There should be eight resulting chunks of metadata. All properties of each chunk are validated in the test.\n",
"bugtrack_url": null,
"license": "",
"summary": "Single source of truth for ESM's EM27 measurement logistics",
"version": "1.0.0rc1",
"project_urls": {
"Homepage": "https://github.com/tum-esm/em27-metadata",
"Repository": "https://github.com/tum-esm/em27-metadata"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "f48572c915a6fad18aaa80a442dff2f44fcb0f7afe0c8abbdab6b6224a0f277e",
"md5": "0b0a5bc41cb1df332debd99938cf9770",
"sha256": "1355cc0f4dbb743e2afcb5c3825338ef00f8b0513c40dc87d33d950db894df90"
},
"downloads": -1,
"filename": "em27_metadata-1.0.0rc1-py3-none-any.whl",
"has_sig": false,
"md5_digest": "0b0a5bc41cb1df332debd99938cf9770",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.9,<4.0",
"size": 9340,
"upload_time": "2023-09-28T20:58:06",
"upload_time_iso_8601": "2023-09-28T20:58:06.242833Z",
"url": "https://files.pythonhosted.org/packages/f4/85/72c915a6fad18aaa80a442dff2f44fcb0f7afe0c8abbdab6b6224a0f277e/em27_metadata-1.0.0rc1-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "00a78f17ad893238a0cea015fb257a6aa5286f7632eb2b605adf70163ec17a9b",
"md5": "b20c5049fdcd1a20024087920687ff44",
"sha256": "120dbc94c29510d163d860660bb0a78a919881d219d0b72f766b62ee01b53a15"
},
"downloads": -1,
"filename": "em27_metadata-1.0.0rc1.tar.gz",
"has_sig": false,
"md5_digest": "b20c5049fdcd1a20024087920687ff44",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.9,<4.0",
"size": 9576,
"upload_time": "2023-09-28T20:58:07",
"upload_time_iso_8601": "2023-09-28T20:58:07.735478Z",
"url": "https://files.pythonhosted.org/packages/00/a7/8f17ad893238a0cea015fb257a6aa5286f7632eb2b605adf70163ec17a9b/em27_metadata-1.0.0rc1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-09-28 20:58:07",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "tum-esm",
"github_project": "em27-metadata",
"github_not_found": true,
"lcname": "em27-metadata"
}
JSON
