# edi-energy.de scraper
<!--- you need to replace the `organization/repo_name` in the status badge URLs --->
The Python package `edi_energy_scraper` provides easy to use methods to mirror the website edi-energy.de.
### Rationale / Why?
If you'd like to be informed about new regulations or data formats being published on edi-energy.de you can either
- visit the site every day and hope that you see the changes if this is your favourite hobby,
- or automate the task.
This repository helps you with the latter. It allows you to create an up-to-date copy of edi-energy.de on your local
computer. Other than if you mirrored the files using `wget` or `curl`, you'll get a clean and intuitive directory
structure.
From there you can e.g. commit the files into a VCS (like e.g. our [edi_energy_mirror](https://github.com/Hochfrequenz/edi_energy_mirror)), scrape the PDF/Word files for later use...
We're all hoping for the day of true digitization on which this repository will become obsolete.
## How to use the Package (as a user)
Install via pip:
```bash
pip install edi_energy_scraper
```
Create a directory in which you'd like to save the mirrored data:
```bash
mkdir edi_energy_de
```
Then import it and start the download:
```python
import asyncio
from edi_energy_scraper import EdiEnergyScraper
# add the following lines to enable debug logging to stdout (CLI)
# import logging
# import sys
# logging.basicConfig(stream=sys.stdout, level=logging.DEBUG)
async def mirror():
scraper = EdiEnergyScraper(path_to_mirror_directory="edi_energy_de")
await scraper.mirror()
if __name__ == "__main__":
loop = asyncio.new_event_loop()
asyncio.set_event_loop(loop)
asyncio.run(mirror())
```
This creates a directory structure:
```
-|-your_script_cwd.py
|-edi_energy_de
|- past (contains archived files)
|- ahb.pdf
|- ahb.docx
|- ...
|- current (contains files valid as of today)
|- mig.pdf
|- mig.docx
|- ...
|- future (contains files valid in the future)
|- allgemeine_festlegungen.pdf
|- schema.xsd
|- ...
```
## How to use this Repository on Your Machine (for development)
Please follow the instructions in
our [Python Template Repository](https://github.com/Hochfrequenz/python_template_repository#how-to-use-this-repository-on-your-machine)
. And for further information, see the [Tox Repository](https://github.com/tox-dev/tox).
## Contribute
You are very welcome to contribute to this template repository by opening a pull request against the main branch.
Raw data
{
"_id": null,
"home_page": null,
"name": "edi-energy-scraper",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.9",
"maintainer_email": null,
"keywords": "ahb, automation, bdew, edi@energy",
"author": null,
"author_email": "Hochfrequenz Unternehmensberatung GmbH <info+github@hochfrequenz.de>",
"download_url": "https://files.pythonhosted.org/packages/1b/27/3189f6c436537f9d2072776ad65feb91aaeacd13abd96e884d5bbc62b670/edi_energy_scraper-0.8.0.tar.gz",
"platform": null,
"description": "# edi-energy.de scraper\n\n<!--- you need to replace the `organization/repo_name` in the status badge URLs --->\n\n\n\n\n\n\n\nThe Python package `edi_energy_scraper` provides easy to use methods to mirror the website edi-energy.de.\n\n### Rationale / Why?\n\nIf you'd like to be informed about new regulations or data formats being published on edi-energy.de you can either\n\n- visit the site every day and hope that you see the changes if this is your favourite hobby,\n- or automate the task.\n\nThis repository helps you with the latter. It allows you to create an up-to-date copy of edi-energy.de on your local\ncomputer. Other than if you mirrored the files using `wget` or `curl`, you'll get a clean and intuitive directory\nstructure.\n\n\nFrom there you can e.g. commit the files into a VCS (like e.g. our [edi_energy_mirror](https://github.com/Hochfrequenz/edi_energy_mirror)), scrape the PDF/Word files for later use...\n\nWe're all hoping for the day of true digitization on which this repository will become obsolete.\n\n## How to use the Package (as a user)\n\nInstall via pip:\n\n```bash\npip install edi_energy_scraper\n```\n\nCreate a directory in which you'd like to save the mirrored data:\n\n```bash\nmkdir edi_energy_de\n```\n\nThen import it and start the download:\n\n```python\nimport asyncio\nfrom edi_energy_scraper import EdiEnergyScraper\n\n# add the following lines to enable debug logging to stdout (CLI)\n# import logging\n# import sys\n# logging.basicConfig(stream=sys.stdout, level=logging.DEBUG)\n\nasync def mirror():\n scraper = EdiEnergyScraper(path_to_mirror_directory=\"edi_energy_de\")\n await scraper.mirror()\n\n\nif __name__ == \"__main__\":\n loop = asyncio.new_event_loop()\n asyncio.set_event_loop(loop)\n asyncio.run(mirror())\n\n```\n\nThis creates a directory structure:\n\n```\n-|-your_script_cwd.py\n |-edi_energy_de\n |- past (contains archived files)\n |- ahb.pdf\n |- ahb.docx\n |- ...\n |- current (contains files valid as of today)\n |- mig.pdf\n |- mig.docx\n |- ...\n |- future (contains files valid in the future)\n |- allgemeine_festlegungen.pdf\n |- schema.xsd\n |- ...\n```\n\n## How to use this Repository on Your Machine (for development)\n\nPlease follow the instructions in\nour [Python Template Repository](https://github.com/Hochfrequenz/python_template_repository#how-to-use-this-repository-on-your-machine)\n. And for further information, see the [Tox Repository](https://github.com/tox-dev/tox).\n\n## Contribute\n\nYou are very welcome to contribute to this template repository by opening a pull request against the main branch.\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "a scraper to mirror edi-energy.de",
"version": "0.8.0",
"project_urls": {
"Changelog": "https://github.com/Hochfrequenz/edi_energy_scraper/releases",
"Homepage": "https://github.com/Hochfrequenz/edi_energy_scraper"
},
"split_keywords": [
"ahb",
" automation",
" bdew",
" edi@energy"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "0030f21dde05a8a4cd59089b31489625bb773f4f89d821a160a46f7d65fa002a",
"md5": "f7056cdfa629940d17e458c5014a016c",
"sha256": "030e5cbdfc4c9d798419cdf719329a9f37be41dc55d2d90e05b4fd82d7ce83af"
},
"downloads": -1,
"filename": "edi_energy_scraper-0.8.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "f7056cdfa629940d17e458c5014a016c",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.9",
"size": 11293,
"upload_time": "2024-12-13T13:05:45",
"upload_time_iso_8601": "2024-12-13T13:05:45.378955Z",
"url": "https://files.pythonhosted.org/packages/00/30/f21dde05a8a4cd59089b31489625bb773f4f89d821a160a46f7d65fa002a/edi_energy_scraper-0.8.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "1b273189f6c436537f9d2072776ad65feb91aaeacd13abd96e884d5bbc62b670",
"md5": "369bb6aa55a168aed252f3dc263876d7",
"sha256": "af67712ad6fd8ccf43816a83f6f2d875d96120b40f9d400a10645b8091e241f3"
},
"downloads": -1,
"filename": "edi_energy_scraper-0.8.0.tar.gz",
"has_sig": false,
"md5_digest": "369bb6aa55a168aed252f3dc263876d7",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.9",
"size": 17190,
"upload_time": "2024-12-13T13:05:47",
"upload_time_iso_8601": "2024-12-13T13:05:47.927160Z",
"url": "https://files.pythonhosted.org/packages/1b/27/3189f6c436537f9d2072776ad65feb91aaeacd13abd96e884d5bbc62b670/edi_energy_scraper-0.8.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-12-13 13:05:47",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "Hochfrequenz",
"github_project": "edi_energy_scraper",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"requirements": [
{
"name": "aiohttp",
"specs": [
[
"==",
"3.9.5"
]
]
},
{
"name": "aiohttp-requests",
"specs": [
[
"==",
"0.2.4"
]
]
},
{
"name": "aiosignal",
"specs": [
[
"==",
"1.3.1"
]
]
},
{
"name": "attrs",
"specs": [
[
"==",
"24.2.0"
]
]
},
{
"name": "beautifulsoup4",
"specs": [
[
"==",
"4.12.3"
]
]
},
{
"name": "brotli",
"specs": [
[
"==",
"1.1.0"
]
]
},
{
"name": "coworker",
"specs": [
[
"==",
"2.0.1"
]
]
},
{
"name": "efoli",
"specs": [
[
"==",
"1.4.0"
]
]
},
{
"name": "frozenlist",
"specs": [
[
"==",
"1.4.1"
]
]
},
{
"name": "idna",
"specs": [
[
"==",
"3.10"
]
]
},
{
"name": "multidict",
"specs": [
[
"==",
"6.1.0"
]
]
},
{
"name": "pypdf",
"specs": [
[
"==",
"5.1.0"
]
]
},
{
"name": "pytz",
"specs": [
[
"==",
"2024.1"
]
]
},
{
"name": "soupsieve",
"specs": [
[
"==",
"2.6"
]
]
},
{
"name": "yarl",
"specs": [
[
"==",
"1.18.3"
]
]
}
],
"tox": true,
"lcname": "edi-energy-scraper"
}
JSON
