# edi-energy.de scraper
[](LICENSE)
The Python package `edi_energy_scraper` provides easy to use methods to mirror the free documents on bdew-mako.de.
### Rationale / Why?
If you'd like to be informed about new regulations or data formats being published on bdew-mako.de you can either
- visit the site every day and hope that you see the changes if this is your favourite hobby,
- or automate the task.
This repository helps you with the latter. It allows you to create an up-to-date copy of edi-energy.de on your local
computer. Other than if you mirrored the files using `wget` or `curl`, you'll get a clean and intuitive directory
structure.
From there you can e.g. commit the files into a VCS (like e.g. our [edi_energy_mirror](https://github.com/Hochfrequenz/edi_energy_mirror)), scrape the PDF/Word files for later use...
We're all hoping for the day of true digitization on which this repository will become obsolete.
## How to use the Package (as a user)
Install via pip:
```bash
pip install edi_energy_scraper
```
Create a directory in which you'd like to save the mirrored data:
```bash
mkdir edi_energy_de
```
Then import it and start the download:
```python
import asyncio
from edi_energy_scraper import EdiEnergyScraper
# add the following lines to enable debug logging to stdout (CLI)
# import logging
# import sys
# logging.basicConfig(stream=sys.stdout, level=logging.DEBUG)
async def mirror():
scraper = EdiEnergyScraper(path_to_mirror_directory="edi_energy_de")
await scraper.mirror()
if __name__ == "__main__":
loop = asyncio.new_event_loop()
asyncio.set_event_loop(loop)
asyncio.run(mirror())
```
This creates a directory structure:
```
-|-your_script_cwd.py
|-edi_energy_de
|- FV2310 (contains files valid since 2023-10-01)
|- ahb.pdf
|- ahb.docx
|- ...
|- FV2404 (contains files valid since 2024-04-03)
|- mig.pdf
|- mig.docx
|- ...
|- FV2504 (contains files valid since 2025-06-06)
|- allgemeine_festlegungen.pdf
|- schema.xsd
|- ...
```
> [!TIP]
> You can extract the information encoded into the filenames:
> ```python
> from edi_energy_scraper import DocumentMetadata
> structured_information = DocumentMetadata.from_filename("AHB_COMDIS_1.0f_99991231_20250605_20250605_8872.pdf")
> # DocumentMetadata(kind='MIG', edifact_format=<EdifactFormat.REQOTE: 'REQOTE'>, valid_from=datetime.date(2023, 9, 30), valid_unt...traordinary_publication=True, is_error_correction=False, is_informational_reading_version=True, additional_text=None, id=10071)
```
## How to use this Repository on Your Machine (for development)
Please follow the instructions in
our [Python Template Repository](https://github.com/Hochfrequenz/python_template_repository#how-to-use-this-repository-on-your-machine)
. And for further information, see the [Tox Repository](https://github.com/tox-dev/tox).
## Contribute
You are very welcome to contribute to this template repository by opening a pull request against the main branch.
Raw data
{
"_id": null,
"home_page": null,
"name": "edi-energy-scraper",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.10",
"maintainer_email": null,
"keywords": "ahb, automation, bdew, edi@energy",
"author": null,
"author_email": "Hochfrequenz Unternehmensberatung GmbH <info+github@hochfrequenz.de>",
"download_url": "https://files.pythonhosted.org/packages/73/9a/7ecaee3582c305950ead42ce16ce07642535fedae1aa2f370a4173592edf/edi_energy_scraper-2.0.4.tar.gz",
"platform": null,
"description": "# edi-energy.de scraper\n\n[](LICENSE)\n\n\n\n\n\n\n\nThe Python package `edi_energy_scraper` provides easy to use methods to mirror the free documents on bdew-mako.de.\n\n### Rationale / Why?\n\nIf you'd like to be informed about new regulations or data formats being published on bdew-mako.de you can either\n\n- visit the site every day and hope that you see the changes if this is your favourite hobby,\n- or automate the task.\n\nThis repository helps you with the latter. It allows you to create an up-to-date copy of edi-energy.de on your local\ncomputer. Other than if you mirrored the files using `wget` or `curl`, you'll get a clean and intuitive directory\nstructure.\n\nFrom there you can e.g. commit the files into a VCS (like e.g. our [edi_energy_mirror](https://github.com/Hochfrequenz/edi_energy_mirror)), scrape the PDF/Word files for later use...\n\nWe're all hoping for the day of true digitization on which this repository will become obsolete.\n\n## How to use the Package (as a user)\n\nInstall via pip:\n\n```bash\npip install edi_energy_scraper\n```\n\nCreate a directory in which you'd like to save the mirrored data:\n\n```bash\nmkdir edi_energy_de\n```\n\nThen import it and start the download:\n\n```python\nimport asyncio\nfrom edi_energy_scraper import EdiEnergyScraper\n\n\n# add the following lines to enable debug logging to stdout (CLI)\n# import logging\n# import sys\n# logging.basicConfig(stream=sys.stdout, level=logging.DEBUG)\n\nasync def mirror():\n scraper = EdiEnergyScraper(path_to_mirror_directory=\"edi_energy_de\")\n await scraper.mirror()\n\n\nif __name__ == \"__main__\":\n loop = asyncio.new_event_loop()\n asyncio.set_event_loop(loop)\n asyncio.run(mirror())\n\n```\n\nThis creates a directory structure:\n\n```\n-|-your_script_cwd.py\n |-edi_energy_de\n |- FV2310 (contains files valid since 2023-10-01)\n |- ahb.pdf\n |- ahb.docx\n |- ...\n |- FV2404 (contains files valid since 2024-04-03)\n |- mig.pdf\n |- mig.docx\n |- ...\n |- FV2504 (contains files valid since 2025-06-06)\n |- allgemeine_festlegungen.pdf\n |- schema.xsd\n |- ...\n```\n\n> [!TIP]\n> You can extract the information encoded into the filenames:\n> ```python\n> from edi_energy_scraper import DocumentMetadata\n> structured_information = DocumentMetadata.from_filename(\"AHB_COMDIS_1.0f_99991231_20250605_20250605_8872.pdf\")\n> # DocumentMetadata(kind='MIG', edifact_format=<EdifactFormat.REQOTE: 'REQOTE'>, valid_from=datetime.date(2023, 9, 30), valid_unt...traordinary_publication=True, is_error_correction=False, is_informational_reading_version=True, additional_text=None, id=10071)\n```\n\n## How to use this Repository on Your Machine (for development)\n\nPlease follow the instructions in\nour [Python Template Repository](https://github.com/Hochfrequenz/python_template_repository#how-to-use-this-repository-on-your-machine)\n. And for further information, see the [Tox Repository](https://github.com/tox-dev/tox).\n\n## Contribute\n\nYou are very welcome to contribute to this template repository by opening a pull request against the main branch.\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "a scraper to mirror edi-energy.de",
"version": "2.0.4",
"project_urls": {
"Changelog": "https://github.com/Hochfrequenz/edi_energy_scraper/releases",
"Homepage": "https://github.com/Hochfrequenz/edi_energy_scraper"
},
"split_keywords": [
"ahb",
" automation",
" bdew",
" edi@energy"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "f9bf3c55317544ec02659b79e888b5340839ee8fd27ea6208ddeb2b4646e05f4",
"md5": "5ae9f02b9f49cca6fc66a281efeb9feb",
"sha256": "4a0f7e49cd103e0eae45ed97abd26a420c6a14148210af35fe42383d1cde4d55"
},
"downloads": -1,
"filename": "edi_energy_scraper-2.0.4-py3-none-any.whl",
"has_sig": false,
"md5_digest": "5ae9f02b9f49cca6fc66a281efeb9feb",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.10",
"size": 12037,
"upload_time": "2025-01-12T11:08:40",
"upload_time_iso_8601": "2025-01-12T11:08:40.147505Z",
"url": "https://files.pythonhosted.org/packages/f9/bf/3c55317544ec02659b79e888b5340839ee8fd27ea6208ddeb2b4646e05f4/edi_energy_scraper-2.0.4-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "739a7ecaee3582c305950ead42ce16ce07642535fedae1aa2f370a4173592edf",
"md5": "c2420bb244193f3a6c1367efa1a4532c",
"sha256": "0c4fcaaf72d85881e84af98ccf9d262c94f9580d6ac6fe9d068655f94a5086e7"
},
"downloads": -1,
"filename": "edi_energy_scraper-2.0.4.tar.gz",
"has_sig": false,
"md5_digest": "c2420bb244193f3a6c1367efa1a4532c",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.10",
"size": 17219,
"upload_time": "2025-01-12T11:08:41",
"upload_time_iso_8601": "2025-01-12T11:08:41.940980Z",
"url": "https://files.pythonhosted.org/packages/73/9a/7ecaee3582c305950ead42ce16ce07642535fedae1aa2f370a4173592edf/edi_energy_scraper-2.0.4.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-01-12 11:08:41",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "Hochfrequenz",
"github_project": "edi_energy_scraper",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"requirements": [
{
"name": "aiohappyeyeballs",
"specs": [
[
"==",
"2.4.4"
]
]
},
{
"name": "aiohttp",
"specs": [
[
"==",
"3.11.11"
]
]
},
{
"name": "aiosignal",
"specs": [
[
"==",
"1.3.2"
]
]
},
{
"name": "annotated-types",
"specs": [
[
"==",
"0.7.0"
]
]
},
{
"name": "attrs",
"specs": [
[
"==",
"24.3.0"
]
]
},
{
"name": "efoli",
"specs": [
[
"==",
"1.4.0"
]
]
},
{
"name": "frozenlist",
"specs": [
[
"==",
"1.5.0"
]
]
},
{
"name": "idna",
"specs": [
[
"==",
"3.10"
]
]
},
{
"name": "more-itertools",
"specs": [
[
"==",
"10.5.0"
]
]
},
{
"name": "multidict",
"specs": [
[
"==",
"6.1.0"
]
]
},
{
"name": "propcache",
"specs": [
[
"==",
"0.2.1"
]
]
},
{
"name": "pydantic",
"specs": [
[
"==",
"2.10.5"
]
]
},
{
"name": "pydantic-core",
"specs": [
[
"==",
"2.27.2"
]
]
},
{
"name": "pypdf",
"specs": [
[
"==",
"5.1.0"
]
]
},
{
"name": "pytz",
"specs": [
[
"==",
"2024.2"
]
]
},
{
"name": "typing-extensions",
"specs": [
[
"==",
"4.12.2"
]
]
},
{
"name": "yarl",
"specs": [
[
"==",
"1.18.3"
]
]
}
],
"tox": true,
"lcname": "edi-energy-scraper"
}
JSON
