# KohlrAHBi
<p align="center">
<img src="kohlrahbi-image.png" alt="kohlrahbi-logo" width="512" height="512">
</p>
Kohlrahbi generates machine-readable files from AHB documents.
Kohlrahbi's sister is [MIG_mose](https://github.com/Hochfrequenz/migmose).
## Rationale
German utilities exchange data using [EDIFACT](https://en.wikipedia.org/wiki/EDIFACT); This is called market communication (mako).
The _Forum Datenformate_ of the BDEW publishes the technical regulations of the EDIFACT based market communication on [`edi-energy.de`](https://www.edi-energy.de/).
These rules are not stable but change twice a year (in theory) or few times per year (in reality).
Specific rules, which are binding for every German utility are kind of formalised in so called "**A**nwendungs**h**and**b**üchern" (AHB).
Those AHBs are basically long tables that describe:
> As a utility, if I want to exchange data about business process XYZ with a market partner, then I have to provide the following information: [...]
In total the regulations from these Anwendungshandbücher span several thousand pages.
And by pages, we really _mean_ pages.
EDIFACT communication is basically the API between German utilities for most of their B2B processes.
However, the technical specifications of this API are
* prose
* on DIN A4 pages.
The Anwendungshandbücher are the epitome of digitization with some good intentions.
Although the AHBs are publicly available as PDF or Word files on `edi-energy.de`, they are hardly accessible in a technical sense:
* You cannot automatically extract information from the AHBs.
* You cannot run automatic comparisons between different versions.
* You cannot automatically test your own API against the set of rules, described in the AHBs (as prose).
* You cannot view or visualize the information from the AHBs in any more intuitive or practical way, than the raw tables from the AHB files.
* ...any many more...
The root cause for all these inaccessibility is a technical one:
Information that are theoretically structured are published in an unstructured format (PDF or Word), which is not suited for technical specifications in IT.
KohlrAHBi as a tool helps you to break those chains and access the AHBs as you'd expect it from technical specs: easy and automatically instead of with hours of mindless manual work.
**KohlrAHBi takes the `.docx` files published by `edi-energy.de` as an input and returns truly machine-readable data in a variety of formats (JSON, CSV...) as a result.**
Hence, KohlrAHBi is the key for unlocking any automation potential that is reliant on information hidden in the Anwendungshandbücher.
We're all hoping for the day of true digitization on which this repository will become obsolete.
## Installation
Kohlrahbi is a Python based tool.
Therefor you have to make sure, that Python is running on your machine.
We recommend to use virtual environments to keep your system clean.
Create a new virtual environment with
```bash
python -m venv .venv
```
The activation of the virtual environment depends on your used OS.
### Windows
```powershell
.venv\Scripts\activate
```
### MacOS/Linux
```zsh
source .venv/bin/activate
```
Finally, install the package with
```bash
pip install kohlrahbi
```
## Usage
Kohlrahbi is a command line tool.
You can use it in three different ways:
1. Extract AHB tables for all prüfidentifikatoren or a specific prüfidentifikator of a provided format version.
2. Extract all conditions for each format of a provided format version.
3. Extract the change history of a provided format version.
You can run the following command to get an overview of all available commands and options.
```bash
kohlrahbi --help
```
> [!NOTE]
> For the following steps we assume that you cloned our [edi_energy_mirror](https://github.com/Hochfrequenz/edi_energy_mirror/) to a neighbouring directory.
> The `edi_energy_mirror` contains the `.docx` files of the AHBs.
> The folder structure should look like this:
> ```plaintext
> .
> ├── edi_energy_mirror
> └── kohlrahbi
> ```
### Extract AHB table
To extract the all AHB tables for each pruefi of a specific format version, you can run the following command.
```bash
kohlrahbi ahb --edi-energy-mirror-path ../edi_energy_mirror/ --output-path ./output/ --file-type csv --format-version FV2310
```
To extract the AHB tables for a specific pruefi of a specific format version, you can run the following command.
```bash
kohlrahbi ahb -eemp ../edi_energy_mirror/ --output-path ./output/ --file-type csv --pruefis 13002 --format-version FV2310
```
You can also provide multiple pruefis.
```bash
kohlrahbi ahb -eemp ../edi_energy_mirror/ --output-path ./output/ --file-type csv --pruefis 13002 --pruefis 13003 --pruefis 13005 --format-version FV2310
```
And you can also provide multiple file types.
```bash
kohlrahbi ahb -eemp ../edi_energy_mirror/ --output-path ./output/ --file-type csv --file-type xlsx --file-type flatahb --pruefis 13002 --format-version FV2310
```
### Extract all conditions
To extract all conditions for each format of a specific format version, you can run the following command.
```bash
kohlrahbi conditions -eemp ../edi_energy_mirror/ --output-path ./output/ --format-version FV2310
```
This will provide you with:
* all conditions
* all packages
found in all AHBs (including the condition texts from package tables) within the specified folder with the .docx files.
The output will be saved for each Edifact format separately as `conditions.json` and `packages.json` in the specified output path.
Please note that the information regarding the conditions collected here may more comprehensive compared to the information collected for the AHBs above. This is because `conditions` uses a different routine than `ahb`.
### Extract change history
```bash
kohlrahbi changehistory -eemp ../edi_energy_mirror/ --output-path ./output/ --format-version FV2310
```
## `.docx` Data Sources
kohlrahbi internally relies on a [specific naming schema](https://github.com/Hochfrequenz/kohlrahbi/blob/22a78dc076c7d5f9248cb9e8707b0cc14a2981d3/src/kohlrahbi/read_functions.py#L57) of the `.docx` files in which the file name holds information about the edifact format and validity period of the AHBs contained within the file.
The easiest way to be compliant with this naming schema is to clone our [edi_energy_mirror](https://github.com/Hochfrequenz/edi_energy_mirror/) repository to your localhost.
## Results
There is a kohlrahbi based CI pipeline from the edi_energy_mirror mentioned above to the repository [machine-readable_anwendungshandbuecher](https://github.com/Hochfrequenz/machine-readable_anwendungshandbuecher) where you can find scraped AHBs as JSON, CSV or Excel files.
## Workflow
```mermaid
flowchart TB
S[Start] --> RD[Read docx]
RD --> RPT[Read all paragraphs <br> and tables]
RPT --> I[Start iterating]
I --> NI[Read next item]
%% check for text paragraph %%
NI --> CTP{Text Paragraph?}
CTP -- Yes --> NI
CTP -- No --> CCST{Is item just<br>Chapter or Section Title?}
CCST -- Yes --> CTAenderunghistorie{Is Chapter Title<br>'Änderungshistorie'?}
CTAenderunghistorie -- Yes --> EXPORT[Export Extract]
CCST -- No --> CT{Is item a table<br>with prüfis?}
CT -- Yes --> Extract[Create Extract]
```
## AHB page number per Format
The following table shows the page number of the AHBs for each format of the format version FV2310.
| Format | Page number | Hint | |
| ------------ | ----------- | --------------------------------------------------- | --- |
| UTILMD Strom | 1064 | | |
| UTILMD Gas | 345 | | |
| REQOTE | 264 | together with QUOTES, ORDERS, ORDRSP, ORDCHG | |
| QUOTES | 264 | together with REQOTE, ORDERS, ORDRSP, ORDCHG | |
| ORDRSP | 264 | together with REQOTE, QUOTES, ORDERS, ORDCHG | |
| ORDERS | 264 | together with REQOTE, QUOTES, ORDRSP, ORDCHG | |
| ORDCHG | 264 | together with REQOTE, QUOTES, ORDERS, ORDRSP | |
| MSCONS | 164 | | |
| UTILMD MaBis | 133 | | |
| REMADV | 91 | together with INVOIC | |
| INVOIC | 91 | together with REMADV | |
| IFTSTA | 82 | | |
| CONTRL | 72 | together with APERAK, contains no Prüfis | |
| APERAK | 72 | together with CONTRL, contains no Prüfis | |
| PARTIN | 69 | | |
| UTILTS | 34 | | |
| ORDRSP | 30 | together with ORDERS | |
| ORDERS | 30 | together with ORDRSP | |
| PRICAT | 25 | | |
| COMDIS | 10 | good test for tables which are above change history | |
## Development
### Setup
To set up the development environment, you have to install the dev dependencies.
```bash
tox -e dev
```
### Run all tests and linters
To run the tests, you can use tox.
```bash
tox
```
See our [Python Template Repository](https://github.com/Hochfrequenz/python_template_repository#how-to-use-this-repository-on-your-machine) for detailed explanations.
## Contribute
You are very welcome to contribute to this template repository by opening a pull request against the main branch.
## Related Tools and Context
This repository is part of the [Hochfrequenz Libraries and Tools for a truly digitized market communication](https://github.com/Hochfrequenz/digital_market_communication/).
Raw data
{
"_id": null,
"home_page": null,
"name": "kohlrahbi",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.11",
"maintainer_email": null,
"keywords": "ahb, automation, bdew, edi@energy",
"author": null,
"author_email": "Kevin Krechan <kevin.krechan@hochfrequenz.de>",
"download_url": "https://files.pythonhosted.org/packages/11/4f/400e1933f69ef18f5b0265cfdd22a38b9eef1a1927bbb468e16098359171/kohlrahbi-1.1.3.tar.gz",
"platform": null,
"description": "# KohlrAHBi\n\n<p align=\"center\">\n <img src=\"kohlrahbi-image.png\" alt=\"kohlrahbi-logo\" width=\"512\" height=\"512\">\n</p>\n\n\n\n\n\n\n\nKohlrahbi generates machine-readable files from AHB documents.\nKohlrahbi's sister is [MIG_mose](https://github.com/Hochfrequenz/migmose).\n\n## Rationale\n\nGerman utilities exchange data using [EDIFACT](https://en.wikipedia.org/wiki/EDIFACT); This is called market communication (mako).\nThe _Forum Datenformate_ of the BDEW publishes the technical regulations of the EDIFACT based market communication on [`edi-energy.de`](https://www.edi-energy.de/).\nThese rules are not stable but change twice a year (in theory) or few times per year (in reality).\n\nSpecific rules, which are binding for every German utility are kind of formalised in so called \"**A**nwendungs**h**and**b**\u00fcchern\" (AHB).\nThose AHBs are basically long tables that describe:\n> As a utility, if I want to exchange data about business process XYZ with a market partner, then I have to provide the following information: [...]\n\nIn total the regulations from these Anwendungshandb\u00fccher span several thousand pages.\nAnd by pages, we really _mean_ pages.\nEDIFACT communication is basically the API between German utilities for most of their B2B processes.\nHowever, the technical specifications of this API are\n\n* prose\n* on DIN A4 pages.\n\nThe Anwendungshandb\u00fccher are the epitome of digitization with some good intentions.\n\nAlthough the AHBs are publicly available as PDF or Word files on `edi-energy.de`, they are hardly accessible in a technical sense:\n\n* You cannot automatically extract information from the AHBs.\n* You cannot run automatic comparisons between different versions.\n* You cannot automatically test your own API against the set of rules, described in the AHBs (as prose).\n* You cannot view or visualize the information from the AHBs in any more intuitive or practical way, than the raw tables from the AHB files.\n* ...any many more...\n\nThe root cause for all these inaccessibility is a technical one:\nInformation that are theoretically structured are published in an unstructured format (PDF or Word), which is not suited for technical specifications in IT.\n\nKohlrAHBi as a tool helps you to break those chains and access the AHBs as you'd expect it from technical specs: easy and automatically instead of with hours of mindless manual work.\n\n**KohlrAHBi takes the `.docx` files published by `edi-energy.de` as an input and returns truly machine-readable data in a variety of formats (JSON, CSV...) as a result.**\n\nHence, KohlrAHBi is the key for unlocking any automation potential that is reliant on information hidden in the Anwendungshandb\u00fccher.\n\nWe're all hoping for the day of true digitization on which this repository will become obsolete.\n\n## Installation\n\nKohlrahbi is a Python based tool.\nTherefor you have to make sure, that Python is running on your machine.\n\nWe recommend to use virtual environments to keep your system clean.\n\nCreate a new virtual environment with\n\n```bash\npython -m venv .venv\n```\n\nThe activation of the virtual environment depends on your used OS.\n\n### Windows\n\n```powershell\n.venv\\Scripts\\activate\n```\n\n### MacOS/Linux\n\n```zsh\nsource .venv/bin/activate\n```\n\nFinally, install the package with\n\n```bash\npip install kohlrahbi\n```\n\n## Usage\n\nKohlrahbi is a command line tool.\nYou can use it in three different ways:\n\n1. Extract AHB tables for all pr\u00fcfidentifikatoren or a specific pr\u00fcfidentifikator of a provided format version.\n2. Extract all conditions for each format of a provided format version.\n3. Extract the change history of a provided format version.\n\nYou can run the following command to get an overview of all available commands and options.\n\n```bash\nkohlrahbi --help\n```\n\n> [!NOTE]\n> For the following steps we assume that you cloned our [edi_energy_mirror](https://github.com/Hochfrequenz/edi_energy_mirror/) to a neighbouring directory.\n> The `edi_energy_mirror` contains the `.docx` files of the AHBs.\n> The folder structure should look like this:\n> ```plaintext\n> .\n> \u251c\u2500\u2500 edi_energy_mirror\n> \u2514\u2500\u2500 kohlrahbi\n> ```\n\n### Extract AHB table\n\nTo extract the all AHB tables for each pruefi of a specific format version, you can run the following command.\n\n```bash\nkohlrahbi ahb --edi-energy-mirror-path ../edi_energy_mirror/ --output-path ./output/ --file-type csv --format-version FV2310\n```\n\nTo extract the AHB tables for a specific pruefi of a specific format version, you can run the following command.\n\n```bash\nkohlrahbi ahb -eemp ../edi_energy_mirror/ --output-path ./output/ --file-type csv --pruefis 13002 --format-version FV2310\n```\n\nYou can also provide multiple pruefis.\n\n```bash\nkohlrahbi ahb -eemp ../edi_energy_mirror/ --output-path ./output/ --file-type csv --pruefis 13002 --pruefis 13003 --pruefis 13005 --format-version FV2310\n```\n\nAnd you can also provide multiple file types.\n\n```bash\nkohlrahbi ahb -eemp ../edi_energy_mirror/ --output-path ./output/ --file-type csv --file-type xlsx --file-type flatahb --pruefis 13002 --format-version FV2310\n```\n\n### Extract all conditions\n\nTo extract all conditions for each format of a specific format version, you can run the following command.\n\n```bash\nkohlrahbi conditions -eemp ../edi_energy_mirror/ --output-path ./output/ --format-version FV2310\n```\nThis will provide you with:\n* all conditions\n* all packages\n\nfound in all AHBs (including the condition texts from package tables) within the specified folder with the .docx files.\nThe output will be saved for each Edifact format separately as `conditions.json` and `packages.json` in the specified output path.\nPlease note that the information regarding the conditions collected here may more comprehensive compared to the information collected for the AHBs above. This is because `conditions` uses a different routine than `ahb`.\n\n### Extract change history\n\n```bash\nkohlrahbi changehistory -eemp ../edi_energy_mirror/ --output-path ./output/ --format-version FV2310\n```\n\n## `.docx` Data Sources\n\nkohlrahbi internally relies on a [specific naming schema](https://github.com/Hochfrequenz/kohlrahbi/blob/22a78dc076c7d5f9248cb9e8707b0cc14a2981d3/src/kohlrahbi/read_functions.py#L57) of the `.docx` files in which the file name holds information about the edifact format and validity period of the AHBs contained within the file.\nThe easiest way to be compliant with this naming schema is to clone our [edi_energy_mirror](https://github.com/Hochfrequenz/edi_energy_mirror/) repository to your localhost.\n\n## Results\n\nThere is a kohlrahbi based CI pipeline from the edi_energy_mirror mentioned above to the repository [machine-readable_anwendungshandbuecher](https://github.com/Hochfrequenz/machine-readable_anwendungshandbuecher) where you can find scraped AHBs as JSON, CSV or Excel files.\n\n## Workflow\n\n```mermaid\nflowchart TB\n S[Start] --> RD[Read docx]\n RD --> RPT[Read all paragraphs <br> and tables]\n RPT --> I[Start iterating]\n I --> NI[Read next item]\n %% check for text paragraph %%\n NI --> CTP{Text Paragraph?}\n CTP -- Yes --> NI\n CTP -- No --> CCST{Is item just<br>Chapter or Section Title?}\n CCST -- Yes --> CTAenderunghistorie{Is Chapter Title<br>'\u00c4nderungshistorie'?}\n CTAenderunghistorie -- Yes --> EXPORT[Export Extract]\n CCST -- No --> CT{Is item a table<br>with pr\u00fcfis?}\n CT -- Yes --> Extract[Create Extract]\n```\n\n## AHB page number per Format\n\nThe following table shows the page number of the AHBs for each format of the format version FV2310.\n\n| Format | Page number | Hint | |\n| ------------ | ----------- | --------------------------------------------------- | --- |\n| UTILMD Strom | 1064 | | |\n| UTILMD Gas | 345 | | |\n| REQOTE | 264 | together with QUOTES, ORDERS, ORDRSP, ORDCHG | |\n| QUOTES | 264 | together with REQOTE, ORDERS, ORDRSP, ORDCHG | |\n| ORDRSP | 264 | together with REQOTE, QUOTES, ORDERS, ORDCHG | |\n| ORDERS | 264 | together with REQOTE, QUOTES, ORDRSP, ORDCHG | |\n| ORDCHG | 264 | together with REQOTE, QUOTES, ORDERS, ORDRSP | |\n| MSCONS | 164 | | |\n| UTILMD MaBis | 133 | | |\n| REMADV | 91 | together with INVOIC | |\n| INVOIC | 91 | together with REMADV | |\n| IFTSTA | 82 | | |\n| CONTRL | 72 | together with APERAK, contains no Pr\u00fcfis | |\n| APERAK | 72 | together with CONTRL, contains no Pr\u00fcfis | |\n| PARTIN | 69 | | |\n| UTILTS | 34 | | |\n| ORDRSP | 30 | together with ORDERS | |\n| ORDERS | 30 | together with ORDRSP | |\n| PRICAT | 25 | | |\n| COMDIS | 10 | good test for tables which are above change history | |\n\n## Development\n\n### Setup\n\nTo set up the development environment, you have to install the dev dependencies.\n\n```bash\ntox -e dev\n```\n\n### Run all tests and linters\n\nTo run the tests, you can use tox.\n\n```bash\ntox\n```\n\nSee our [Python Template Repository](https://github.com/Hochfrequenz/python_template_repository#how-to-use-this-repository-on-your-machine) for detailed explanations.\n\n## Contribute\n\nYou are very welcome to contribute to this template repository by opening a pull request against the main branch.\n\n## Related Tools and Context\n\nThis repository is part of the [Hochfrequenz Libraries and Tools for a truly digitized market communication](https://github.com/Hochfrequenz/digital_market_communication/).\n",
"bugtrack_url": null,
"license": "GPL",
"summary": "Tool to generate machine readable files from AHB documents",
"version": "1.1.3",
"project_urls": {
"Changelog": "https://github.com/Hochfrequenz/kohlrahbi/releases",
"Homepage": "https://github.com/Hochfrequenz/kohlrahbi"
},
"split_keywords": [
"ahb",
" automation",
" bdew",
" edi@energy"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "1eceef49475a5ee36d2484f7b98b11fa67a001773e0f3d88f8bbdd1be531aefa",
"md5": "fcdee6a6256161830f045b1b19097111",
"sha256": "73672c0ef47307198bbf2aa3ce0fffd1d1a0e2db79428bcf393dca9585a34c63"
},
"downloads": -1,
"filename": "kohlrahbi-1.1.3-py3-none-any.whl",
"has_sig": false,
"md5_digest": "fcdee6a6256161830f045b1b19097111",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.11",
"size": 72436,
"upload_time": "2024-05-21T16:20:09",
"upload_time_iso_8601": "2024-05-21T16:20:09.279685Z",
"url": "https://files.pythonhosted.org/packages/1e/ce/ef49475a5ee36d2484f7b98b11fa67a001773e0f3d88f8bbdd1be531aefa/kohlrahbi-1.1.3-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "114f400e1933f69ef18f5b0265cfdd22a38b9eef1a1927bbb468e16098359171",
"md5": "26cb03d74ade71581320b6ac61490bf3",
"sha256": "9423e3aadb7caa0fe3467e9c4df5f671039d6dd812dc8238e5514261eb94feaf"
},
"downloads": -1,
"filename": "kohlrahbi-1.1.3.tar.gz",
"has_sig": false,
"md5_digest": "26cb03d74ade71581320b6ac61490bf3",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.11",
"size": 269603,
"upload_time": "2024-05-21T16:20:10",
"upload_time_iso_8601": "2024-05-21T16:20:10.922891Z",
"url": "https://files.pythonhosted.org/packages/11/4f/400e1933f69ef18f5b0265cfdd22a38b9eef1a1927bbb468e16098359171/kohlrahbi-1.1.3.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-05-21 16:20:10",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "Hochfrequenz",
"github_project": "kohlrahbi",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"requirements": [
{
"name": "annotated-types",
"specs": [
[
"==",
"0.7.0"
]
]
},
{
"name": "attrs",
"specs": [
[
"==",
"23.2.0"
]
]
},
{
"name": "click",
"specs": [
[
"==",
"8.1.7"
]
]
},
{
"name": "colorama",
"specs": [
[
"==",
"0.4.6"
]
]
},
{
"name": "colorlog",
"specs": [
[
"==",
"6.8.2"
]
]
},
{
"name": "et-xmlfile",
"specs": [
[
"==",
"1.1.0"
]
]
},
{
"name": "lxml",
"specs": [
[
"==",
"5.2.2"
]
]
},
{
"name": "marshmallow",
"specs": [
[
"==",
"3.21.2"
]
]
},
{
"name": "maus",
"specs": [
[
"==",
"0.5.0"
]
]
},
{
"name": "more-itertools",
"specs": [
[
"==",
"10.2.0"
]
]
},
{
"name": "numpy",
"specs": [
[
"==",
"1.26.4"
]
]
},
{
"name": "openpyxl",
"specs": [
[
"==",
"3.1.2"
]
]
},
{
"name": "packaging",
"specs": [
[
"==",
"24.0"
]
]
},
{
"name": "pandas",
"specs": [
[
"==",
"2.2.2"
]
]
},
{
"name": "pydantic",
"specs": [
[
"==",
"2.7.1"
]
]
},
{
"name": "pydantic-core",
"specs": [
[
"==",
"2.18.2"
]
]
},
{
"name": "python-dateutil",
"specs": [
[
"==",
"2.9.0.post0"
]
]
},
{
"name": "python-docx",
"specs": [
[
"==",
"1.1.2"
]
]
},
{
"name": "pytz",
"specs": [
[
"==",
"2024.1"
]
]
},
{
"name": "six",
"specs": [
[
"==",
"1.16.0"
]
]
},
{
"name": "tomlkit",
"specs": [
[
"==",
"0.12.5"
]
]
},
{
"name": "typing-extensions",
"specs": [
[
"==",
"4.11.0"
]
]
},
{
"name": "tzdata",
"specs": [
[
"==",
"2024.1"
]
]
},
{
"name": "xlsxwriter",
"specs": [
[
"==",
"3.2.0"
]
]
}
],
"tox": true,
"lcname": "kohlrahbi"
}
JSON
