| Name | parkapi-sources JSON |
| Version |
0.2.0
JSON |
| download |
| home_page | None |
| Summary | ParkAPI Sources is a collection of converters from several different data sources to normalized ParkAPI data. |
| upload_time | 2024-04-18 18:45:43 |
| maintainer | None |
| docs_url | None |
| author | None |
| requires_python | >=3.10 |
| license | None |
| keywords |
data
parking
converter
mobility
car
bike
|
| VCS |
|
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
# ParkAPI Sources
ParkAPI Sources is a collection of converters from several different data sources to normalized ParkAPI data. ParkAPI does support parking
for cars, for bikes and lockers. The data model is based on the original ParkAPI project and tries to stay compatible to DATEX II Parking
Publication Light at any extension of the model.
We support following data sources:
| name | purpose | type | uid | realtime |
|-----------------------------------------------------------------------------------|---------|-------------|----------------------|----------|
| Deutsche Bahn | car | pull | `bahn_v2` | no |
| Barrierefreie Reisekette Baden-Württemberg: PKW-Parkplätze an Bahnhöfen | car | push (csv) | `bfrk_bw_oepnv_car` | no |
| Barrierefreie Reisekette Baden-Württemberg: PKW-Parkplätze an Bushaltestellen | car | push (csv) | `bfrk_bw_spnv_car` | no |
| Barrierefreie Reisekette Baden-Württemberg: Fahrrad-Parkplätze an Bahnhöfen | bike | push (csv) | `bfrk_bw_oepnv_bike` | no |
| Barrierefreie Reisekette Baden-Württemberg: Fahrrad-Parkplätze an Bushaltestellen | bike | push (csv) | `bfrk_bw_spnv_bike` | no |
| Stadt Freiburg | car | pull | `freiburg` | yes |
| Stadt Heidelberg | car | pull | `heidelberg` | yes |
| Stadt Karlsruhe | car | pull | `karlsruhe` | yes |
| Stadt Mannheim | car | pull | `mannheim` | yes |
| Stadt Karlsruhe | car | push (csv) | `neckarsulm` | no |
| Parkraumgesellschaft Baden-Württemberg | car | pull | `pbw` | yes |
| Stadt Pforzheim | car | push (csv) | `pforzheim` | no |
| Stadt Reutlingen | car | push (csv) | `reutlingen` | no |
| Stadt Stuttgart | car | push (json) | `stuttgart` | yes |
| Stadt Ulm | car | pull | `ulm` | yes |
| Verband Region Stuttgart: Park and Ride | car | pull | `vrs_p_r` | yes |
New converters for new sources are always welcome, please have a look at "Contribute" below.
## Install
ParkAPI Sources is a python module published at [PyPI](https://pypi.org/project/parkapi-sources/). Therefore, you can install it by
```shell
pip install parkapi-sources
```
If you use parkapi-sources in a project, we recommend to fix the version. As long as parkapi-sources is beta, breaking changes might be
introduced on minor version level (like: change from 0.1.1 to 0.2.0). As soon as 1.0 is released, we will follow
[Semantic Versioning](https://semver.org), which means that breaking changes will just appear on major version changes (like: change from 1.1.2 to 2.0.0).
You can expect a lot of changes in the minor version level, as any new converter is a new feature.
## Usage
Your starting point is always the `ParkAPISources` where all Sources are registered.
```python
from parkapi_sources import ParkAPISources
my_sources = ParkAPISources()
```
`ParkAPISources` accepts following parameters:
- `config: Optional[dict] = None` is a dictionary for config values, especially secrets.
- `converter_uids: Optional[list[str]] = None` is used for loading just the converter uids you want to load
- `no_pull_converter: bool = False` is used for limiting converters to pull converters
- `no_push_converter: bool = False` is used for limiting converters to push converters
### Configuration
Config values are mostly individual for specific converters: if there are required config values, they are defined at the converter
definition right at the top:
```
required_config_keys = ['MY_SECRET']
```
`ParkAPISources` offers a method to check if all config values are set:
```python
from parkapi_sources import ParkAPISources
my_sources = ParkAPISources()
my_sources.check_credentials()
```
If not all config values are set, a `MissingConfigException` is thrown. It's recommended to run this check after initializing the module
to prevent exceptions during runtime.
Besides converter-individual config values, there are two global values which can be used to configure the source of GeoJSON files. Per
default, static GeoJSON files are fetched from this repository. This behaviour can be changed:
- `STATIC_GEOJSON_BASE_URL` defines another base URL for GeoJSON files
- `STATIC_GEOJSON_BASE_PATH` defines a lokal path instead, so the application will load files locally without network requests
### Use converters
After initializing, you will find all initialized converters at `ParkAPISources.converter_by_uid`. As the input is very different, so are
the methods you have to use. In general, you can differ between two major strategies:
### Pull converters
Pull converters are responsible for getting data from an external data source. This can be an REST endpoints as well as HTML which is
scraped. Pull converters always split up in static and realtime data, because at most sources, this is not the same. Each pull converter
has two methods:
1) `get_static_parking_sites(self) -> tuple[list[StaticParkingSiteInput], list[ImportParkingSiteException]]:`
2) `def get_realtime_parking_sites(self) -> tuple[list[RealtimeParkingSiteInput], list[ImportParkingSiteException]]:`
### Push converters
Push converters are responsible to handle data which is pushed to the service using defined endpoints. Usually, these converters are used
as a handler behind HTTP endpoints, but of course you can use them in other ways, too, for example command line scripts.
Push converters always handle specific formats, therefore, there are multiple types of push converters. All push converters return a
`tuple[list[StaticParkingSiteInput | RealtimeParkingSiteInput], list[ImportParkingSiteException]]`, therefore they decided based on the
given data if it's static or realtime data they got - or even both, then each dataset ends up in two entries in the first list.
1) A `CsvConverter` handles CSV files: `handle_csv_string(self, data: StringIO)`
2) A `JsonConverter` handles JSON based data: `handle_json(self, data: dict | list)`
3) A `XlsxConverter` handles XMLX data: `def handle_xlsx(self, workbook: Workbook)` parsed by `openpyxl`
4) A `XmlConverter` handles XML data: `def handle_xml(self, root: Element)` parsed by `lxml`
### Results
At `webapp/models/parking_site_inputs.py`, you can find the definition of `StaticParkingSiteInput` and `RealtimeParkingSiteInput`. These
`dataclasses` are also [`validataclasses`](https://pypi.org/project/validataclass/), so you can be sure that the data you get is validated.
## Contribute
We are happy about any contributions. We do not expect that yoy can develop Python, but of course things might speed up if
### Report bugs
As ParkAPI-Sources integrates a lot of external data sources, sometimes without a proper definition, converters might run into issues
because of changes on the data source side. If you see that happening, please add a bug report at the
[issues](https://github.com/ParkenDD/parkapi-sources-v3/issues). If possible, please add the data which fails.
### Contribute new source data
If you see a nice data source which is not covered by ParkAPI-sources, you can always create a feature request at our
[issues](https://github.com/ParkenDD/parkapi-sources-v3/issues). If you do so, please add the data you found, so we can actually build
the converter. If possible, please try to find out a licence, too.
### Write a new Converters
We always welcome merge requests with new converters. A merge request should contain the following:
* MIT licenced code
* A converter which validates the input in a way that the output follows the data model
* A test with example data to ensure that the converter works with current data
## Write a new converter
First you have to determine which type of converter you need. If you get the data from an endpoint, you will need a `PushConverter`, if
you have a file you want to push via HTTP or CLI, you need a `PullConverter`.
### Write the converter
In order to write a converter, you need a directory at `converters`. Please name your directory in a way that it points to the actual
converter you will write. If it's just one converter, the `uid` is usually the best approach.
At `converters/your-converter`, you will at least need a `converter.py` and an `__init__.py`. In most cases, you will also need some
`validataclasses` you can put in `models.py`. Validation is crucial in this library, because the users of this library should not think
invalid data. Additionally, if you have very specific new data types, you can write new `validataclass` validators you can usually put in
`validators.py`.
### Testing the converter
In order to proof that the validator works, we need to test the basic functionality. In order to do this, we need a snapshot of the data
which should be integrated. Common place for this data is `tests/converters/data/filename-starting-with-uid.ending`. This data should be
used in one or multiple tests (in several cases two tests, one for static, one for realtime data) stored at `tests/converters/uid_test.py`.
If you test a `PullConverter`, you will need no mock requests. This can be done using the fantastic
[`requests_mock`](https://pypi.org/project/requests-mock/) library.
If you created new validators, these should be tested with different inputs. Usually, `pytest.parametrize` is a nice approach to do this.
### Linting
As we try to keep a consistent code style, please lint your code before creating the merge request. We use `black` and `ruff` for linting.
There is Makefile target to do both: `make lint`. It runs the following commands:
```bash
black ./src ./tests
ruff check --fix ./src ./tests
```
If you don't have `black` and `ruff` installed globally, you can create a virtual environment for these tools:
```bash
virtualenv venv
source venv/bin/activate
pip install -r requirements.txt -r requirements-dev.txt
black ./src ./tests
ruff check --fix ./src ./tests
```
### Make your new converter available
All available converters should be registered at the `ParkAPISources` class in order to make them accessible for users of this library, so
please register your converter there. The new converter should also be added to the table in this README.md file.
### Release process
If you created a merge request, the maintainers will review your code. If everything is fine, it will be merged to `main`, and a new
release will be created soon. As written above, we follow SemVer, so any new converter will add plus one to the minor version. In order to
use this new release, please keep in mind to update your `requirements.txt` / update the dependency manager you use.
## Licence
This library is under MIT licence. Please look at `LICENCE.txt` for details.
Raw data
{
"_id": null,
"home_page": null,
"name": "parkapi-sources",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.10",
"maintainer_email": "\"Ernesto Ruge, binary butterfly GmbH\" <ernesto.ruge@binary-butterfly.de>",
"keywords": "data, parking, converter, mobility, car, bike",
"author": null,
"author_email": "\"Ernesto Ruge, binary butterfly GmbH\" <ernesto.ruge@binary-butterfly.de>",
"download_url": "https://files.pythonhosted.org/packages/cb/87/f119b32d2af8e24bd284e5c9a837fb61de4761fd2ddc8eb21119fb505cb9/parkapi_sources-0.2.0.tar.gz",
"platform": null,
"description": "# ParkAPI Sources\n\nParkAPI Sources is a collection of converters from several different data sources to normalized ParkAPI data. ParkAPI does support parking\nfor cars, for bikes and lockers. The data model is based on the original ParkAPI project and tries to stay compatible to DATEX II Parking\nPublication Light at any extension of the model.\n\nWe support following data sources:\n\n| name | purpose | type | uid | realtime |\n|-----------------------------------------------------------------------------------|---------|-------------|----------------------|----------|\n| Deutsche Bahn | car | pull | `bahn_v2` | no |\n| Barrierefreie Reisekette Baden-W\u00fcrttemberg: PKW-Parkpl\u00e4tze an Bahnh\u00f6fen | car | push (csv) | `bfrk_bw_oepnv_car` | no |\n| Barrierefreie Reisekette Baden-W\u00fcrttemberg: PKW-Parkpl\u00e4tze an Bushaltestellen | car | push (csv) | `bfrk_bw_spnv_car` | no |\n| Barrierefreie Reisekette Baden-W\u00fcrttemberg: Fahrrad-Parkpl\u00e4tze an Bahnh\u00f6fen | bike | push (csv) | `bfrk_bw_oepnv_bike` | no |\n| Barrierefreie Reisekette Baden-W\u00fcrttemberg: Fahrrad-Parkpl\u00e4tze an Bushaltestellen | bike | push (csv) | `bfrk_bw_spnv_bike` | no |\n| Stadt Freiburg | car | pull | `freiburg` | yes |\n| Stadt Heidelberg | car | pull | `heidelberg` | yes |\n| Stadt Karlsruhe | car | pull | `karlsruhe` | yes |\n| Stadt Mannheim | car | pull | `mannheim` | yes |\n| Stadt Karlsruhe | car | push (csv) | `neckarsulm` | no |\n| Parkraumgesellschaft Baden-W\u00fcrttemberg | car | pull | `pbw` | yes |\n| Stadt Pforzheim | car | push (csv) | `pforzheim` | no |\n| Stadt Reutlingen | car | push (csv) | `reutlingen` | no |\n| Stadt Stuttgart | car | push (json) | `stuttgart` | yes |\n| Stadt Ulm | car | pull | `ulm` | yes |\n| Verband Region Stuttgart: Park and Ride | car | pull | `vrs_p_r` | yes |\n\nNew converters for new sources are always welcome, please have a look at \"Contribute\" below.\n\n\n## Install\n\nParkAPI Sources is a python module published at [PyPI](https://pypi.org/project/parkapi-sources/). Therefore, you can install it by \n\n```shell\npip install parkapi-sources\n```\n\nIf you use parkapi-sources in a project, we recommend to fix the version. As long as parkapi-sources is beta, breaking changes might be\nintroduced on minor version level (like: change from 0.1.1 to 0.2.0). As soon as 1.0 is released, we will follow \n[Semantic Versioning](https://semver.org), which means that breaking changes will just appear on major version changes (like: change from 1.1.2 to 2.0.0). \nYou can expect a lot of changes in the minor version level, as any new converter is a new feature.\n\n\n## Usage\n\nYour starting point is always the `ParkAPISources` where all Sources are registered.\n\n```python\nfrom parkapi_sources import ParkAPISources\n\nmy_sources = ParkAPISources()\n```\n\n`ParkAPISources` accepts following parameters:\n\n- `config: Optional[dict] = None` is a dictionary for config values, especially secrets.\n- `converter_uids: Optional[list[str]] = None` is used for loading just the converter uids you want to load\n- `no_pull_converter: bool = False` is used for limiting converters to pull converters\n- `no_push_converter: bool = False` is used for limiting converters to push converters\n\n\n### Configuration\n\nConfig values are mostly individual for specific converters: if there are required config values, they are defined at the converter \ndefinition right at the top:\n\n```\nrequired_config_keys = ['MY_SECRET']\n```\n\n`ParkAPISources` offers a method to check if all config values are set:\n\n```python\nfrom parkapi_sources import ParkAPISources\n\nmy_sources = ParkAPISources()\nmy_sources.check_credentials()\n```\n\nIf not all config values are set, a `MissingConfigException` is thrown. It's recommended to run this check after initializing the module\nto prevent exceptions during runtime.\n\nBesides converter-individual config values, there are two global values which can be used to configure the source of GeoJSON files. Per\ndefault, static GeoJSON files are fetched from this repository. This behaviour can be changed:\n\n- `STATIC_GEOJSON_BASE_URL` defines another base URL for GeoJSON files\n- `STATIC_GEOJSON_BASE_PATH` defines a lokal path instead, so the application will load files locally without network requests\n\n\n### Use converters\n\nAfter initializing, you will find all initialized converters at `ParkAPISources.converter_by_uid`. As the input is very different, so are\nthe methods you have to use. In general, you can differ between two major strategies:\n\n\n### Pull converters\n\nPull converters are responsible for getting data from an external data source. This can be an REST endpoints as well as HTML which is\nscraped. Pull converters always split up in static and realtime data, because at most sources, this is not the same. Each pull converter\nhas two methods:\n\n1) `get_static_parking_sites(self) -> tuple[list[StaticParkingSiteInput], list[ImportParkingSiteException]]:`\n2) `def get_realtime_parking_sites(self) -> tuple[list[RealtimeParkingSiteInput], list[ImportParkingSiteException]]:`\n\n\n### Push converters\n\nPush converters are responsible to handle data which is pushed to the service using defined endpoints. Usually, these converters are used\nas a handler behind HTTP endpoints, but of course you can use them in other ways, too, for example command line scripts.\n\nPush converters always handle specific formats, therefore, there are multiple types of push converters. All push converters return a \n`tuple[list[StaticParkingSiteInput | RealtimeParkingSiteInput], list[ImportParkingSiteException]]`, therefore they decided based on the\ngiven data if it's static or realtime data they got - or even both, then each dataset ends up in two entries in the first list.\n\n1) A `CsvConverter` handles CSV files: `handle_csv_string(self, data: StringIO)`\n2) A `JsonConverter` handles JSON based data: `handle_json(self, data: dict | list)`\n3) A `XlsxConverter` handles XMLX data: `def handle_xlsx(self, workbook: Workbook)` parsed by `openpyxl`\n4) A `XmlConverter` handles XML data: `def handle_xml(self, root: Element)` parsed by `lxml`\n\n\n### Results\n\nAt `webapp/models/parking_site_inputs.py`, you can find the definition of `StaticParkingSiteInput` and `RealtimeParkingSiteInput`. These\n`dataclasses` are also [`validataclasses`](https://pypi.org/project/validataclass/), so you can be sure that the data you get is validated.\n\n\n## Contribute\n\nWe are happy about any contributions. We do not expect that yoy can develop Python, but of course things might speed up if \n\n\n### Report bugs\n\nAs ParkAPI-Sources integrates a lot of external data sources, sometimes without a proper definition, converters might run into issues\nbecause of changes on the data source side. If you see that happening, please add a bug report at the\n[issues](https://github.com/ParkenDD/parkapi-sources-v3/issues). If possible, please add the data which fails.\n\n\n### Contribute new source data\n\nIf you see a nice data source which is not covered by ParkAPI-sources, you can always create a feature request at our\n[issues](https://github.com/ParkenDD/parkapi-sources-v3/issues). If you do so, please add the data you found, so we can actually build\nthe converter. If possible, please try to find out a licence, too.\n\n\n### Write a new Converters\n\nWe always welcome merge requests with new converters. A merge request should contain the following:\n\n* MIT licenced code\n* A converter which validates the input in a way that the output follows the data model\n* A test with example data to ensure that the converter works with current data\n\n\n## Write a new converter\n\nFirst you have to determine which type of converter you need. If you get the data from an endpoint, you will need a `PushConverter`, if\nyou have a file you want to push via HTTP or CLI, you need a `PullConverter`.\n\n\n### Write the converter\n\nIn order to write a converter, you need a directory at `converters`. Please name your directory in a way that it points to the actual\nconverter you will write. If it's just one converter, the `uid` is usually the best approach.\n\nAt `converters/your-converter`, you will at least need a `converter.py` and an `__init__.py`. In most cases, you will also need some\n`validataclasses` you can put in `models.py`. Validation is crucial in this library, because the users of this library should not think\ninvalid data. Additionally, if you have very specific new data types, you can write new `validataclass` validators you can usually put in\n`validators.py`.\n\n\n### Testing the converter\n\nIn order to proof that the validator works, we need to test the basic functionality. In order to do this, we need a snapshot of the data\nwhich should be integrated. Common place for this data is `tests/converters/data/filename-starting-with-uid.ending`. This data should be \nused in one or multiple tests (in several cases two tests, one for static, one for realtime data) stored at `tests/converters/uid_test.py`.\n\nIf you test a `PullConverter`, you will need no mock requests. This can be done using the fantastic\n[`requests_mock`](https://pypi.org/project/requests-mock/) library.\n\nIf you created new validators, these should be tested with different inputs. Usually, `pytest.parametrize` is a nice approach to do this.\n\n\n### Linting\n\nAs we try to keep a consistent code style, please lint your code before creating the merge request. We use `black` and `ruff` for linting.\nThere is Makefile target to do both: `make lint`. It runs the following commands:\n\n```bash\nblack ./src ./tests\nruff check --fix ./src ./tests\n```\n\nIf you don't have `black` and `ruff` installed globally, you can create a virtual environment for these tools:\n\n```bash\nvirtualenv venv\nsource venv/bin/activate\npip install -r requirements.txt -r requirements-dev.txt\n\nblack ./src ./tests\nruff check --fix ./src ./tests\n```\n\n\n### Make your new converter available\n\nAll available converters should be registered at the `ParkAPISources` class in order to make them accessible for users of this library, so \nplease register your converter there. The new converter should also be added to the table in this README.md file.\n\n\n### Release process\n\nIf you created a merge request, the maintainers will review your code. If everything is fine, it will be merged to `main`, and a new\nrelease will be created soon. As written above, we follow SemVer, so any new converter will add plus one to the minor version. In order to\nuse this new release, please keep in mind to update your `requirements.txt` / update the dependency manager you use.\n\n\n## Licence\n\nThis library is under MIT licence. Please look at `LICENCE.txt` for details.\n",
"bugtrack_url": null,
"license": null,
"summary": "ParkAPI Sources is a collection of converters from several different data sources to normalized ParkAPI data.",
"version": "0.2.0",
"project_urls": null,
"split_keywords": [
"data",
" parking",
" converter",
" mobility",
" car",
" bike"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "7908f560c97bdb51c7800064547e3617be986eaed9e4779dc17d39c58c5a1b8f",
"md5": "0697b86cc4a84d80daef05897d559876",
"sha256": "48efb715adf066aed54ff6942ade1e0284da563d8cc77ab4f28af4f4bb849324"
},
"downloads": -1,
"filename": "parkapi_sources-0.2.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "0697b86cc4a84d80daef05897d559876",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.10",
"size": 78483,
"upload_time": "2024-04-18T18:53:04",
"upload_time_iso_8601": "2024-04-18T18:53:04.672321Z",
"url": "https://files.pythonhosted.org/packages/79/08/f560c97bdb51c7800064547e3617be986eaed9e4779dc17d39c58c5a1b8f/parkapi_sources-0.2.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "cb87f119b32d2af8e24bd284e5c9a837fb61de4761fd2ddc8eb21119fb505cb9",
"md5": "4124a57691c17ee54ddb529ef4ed9801",
"sha256": "c7a920334901f3cce7dfd65f999339ff3ecd302a9f424bf6ebd65e524f8ae527"
},
"downloads": -1,
"filename": "parkapi_sources-0.2.0.tar.gz",
"has_sig": false,
"md5_digest": "4124a57691c17ee54ddb529ef4ed9801",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.10",
"size": 554521,
"upload_time": "2024-04-18T18:45:43",
"upload_time_iso_8601": "2024-04-18T18:45:43.809421Z",
"url": "https://files.pythonhosted.org/packages/cb/87/f119b32d2af8e24bd284e5c9a837fb61de4761fd2ddc8eb21119fb505cb9/parkapi_sources-0.2.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-04-18 18:45:43",
"github": false,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"lcname": "parkapi-sources"
}
