Disclaimer: This library was written to experiment with various concepts.
The submodules `entsoe_client.ParameterTypes` and `entsoe_client.Queries` are straight forward.
To refactor `entsoe_client.Parsers`, the next steps would be to establish fixtures,
standardize flattening of the XML tree and decouple the records from the DataFrame library.
===============
ENTSO-E Client
===============
Formulate readable queries and handle data in Pandas,
including an exhaustive set of pre-defined queries.
.. code-block:: python
>>> import requests
>>> from lxml import objectify
>>> from lxml.etree import dump
>>> url = 'https://transparency.entsoe.eu/api?' \
... 'documentType=A81&businessType=A95&psrType=A04&type_MarketAgreement.Type=A01&controlArea_Domain=10YNL----------L' \
... f'&periodStart=202101010000&periodEnd=202104010000&securityToken={api_key}'
>>> response = requests.Session().get(url=url)
>>> element = objectify.fromstring(response.content)
>>> dump(element)
<Balancing_MarketDocument xmlns="urn:iec62325.351:tc57wg16:451-6:balancingdocument:3:0">
<mRID>051b91beed574b48b4548214e9001afc</mRID>
<revisionNumber>1</revisionNumber>
<type>A81</type>
<process.processType>A34</process.processType>
<sender_MarketParticipant.mRID codingScheme="A01">10X1001A1001A450</sender_MarketParticipant.mRID>
<sender_MarketParticipant.marketRole.type>A32</sender_MarketParticipant.marketRole.type>
<receiver_MarketParticipant.mRID codingScheme="A01">10X1001A1001A450</receiver_MarketParticipant.mRID>
<receiver_MarketParticipant.marketRole.type>A33</receiver_MarketParticipant.marketRole.type>
<createdDateTime>2021-10-04T18:12:43Z</createdDateTime>
<controlArea_Domain.mRID codingScheme="A01">10YNL----------L</controlArea_Domain.mRID>
<period.timeInterval>
<start>2020-12-31T23:00Z</start>
<end>2021-03-31T22:00Z</end>
</period.timeInterval>
<TimeSeries>
<mRID>1</mRID>
<businessType>A95</businessType>
<type_MarketAgreement.type>A01</type_MarketAgreement.type>
<mktPSRType.psrType>A04</mktPSRType.psrType>
<flowDirection.direction>A03</flowDirection.direction>
<quantity_Measure_Unit.name>MAW</quantity_Measure_Unit.name>
<curveType>A01</curveType>
<Period>
<timeInterval>
<start>2020-12-31T23:00Z</start>
<end>2021-01-01T23:00Z</end>
</timeInterval>
<resolution>PT60M</resolution>
<Point>
<position>1</position>
<quantity>44</quantity>
</Point>
<Point>
<position>2</position>
<quantity>44</quantity>
[...]
becomes
.. code-block:: python
>>> import entsoe_client as ec
>>> from entsoe_client.ParameterTypes import *
>>> client = ec.Client(api_key)
>>> parser = ec.Parser
>>> query = ec.Query(
... documentType=DocumentType("Contracted reserves"),
... psrType=PsrType("Generation"),
... businessType=BusinessType("Frequency containment reserve"),
... controlArea_Domain=Area("NL"),
... type_MarketAgreementType=MarketAgreementType("Daily"),
... periodStart="2021-01-01T00:00",
... periodEnd="2021-04-01T00:00"
... )
>>> response = client(query)
>>> df = parser.parse(response)
>>> df.iloc[:,:3].head()
position quantity Period.timeInterval.start...
2020-12-31 23:00:00+00:00 1 44 2020-12-31T23:00Z
2021-01-01 00:00:00+00:00 2 44 2020-12-31T23:00Z
2021-01-01 01:00:00+00:00 3 44 2020-12-31T23:00Z
2021-01-01 02:00:00+00:00 4 44 2020-12-31T23:00Z
2021-01-01 03:00:00+00:00 5 44 2020-12-31T23:00Z
...
predefined queries are subset of the generic Query class, covering all examples of the `ENTSO-E API guide <https://transparency.entsoe.eu/content/static_content/Static%20content/web%20api/Guide.html>`_.
.. code-block:: python
>>> predefined_query = ec.Queries.Balancing.AmountOfBalancingReservesUnderContract(
... controlArea_Domain=Area("NL"),
... type_MarketAgreementType=MarketAgreementType("Daily"),
... psrType=PsrType("Generation"),
... periodStart="2021-01-01T00:00",
... periodEnd="2021-04-01T00:00"
... )
...
>>> predefined_query() == query()
True
-----
| *ENTSO-E Client* enables straight-forward access to *all* of the data at `ENTSO-E Transparency Platform <https://transparency.entsoe.eu/>`_.
* Query templates abstract the API specifics through Enumerated types.
* Parse responses into Pandas DataFrames without loss of *any* information.
| The separation of Queries, Client and Parser with their hierarchical abstractions keep the package extensible and maintainable. A pipeline from Query to DataFrame is trivial, preserving the ability to customize steps in between.
| The implementation relies primarily on the
`Transparency Platform restful API - user guide <https://transparency.entsoe.eu/content/static_content/Static%20content/web%20api/Guide.html>`_.
The `Manual of Procedures (MoP) <https://www.entsoe.eu/data/transparency-platform/mop/>`_ documents provide
further insight on the *business requirements specification*.
Further information can be found in the
`Electronic Data Interchange (EDI) Library <https://www.entsoe.eu/publications/electronic-data-interchange-edi-library/>`_.
-----
Main contributions
* Exhaustive List of ParameterTypes.
These allow mapping between natural language and the codes required
for GET requests, e.g. :code:`DocumentType.A85 == DocumentType("Imbalance price")`.
This feature allows keeping track of queries without jumping between documents or adding comments.
* Exhaustive List of Pre-defined Queries from ENTSO-E API Guide.
`ENTSO-E API Guide <https://transparency.entsoe.eu/content/static_content/Static%20content/web%20api/Guide.html>`_
is a minial set for any API connector to implement and reflects all dashboards on
ENTSO-E Transparency Platform.
* Parsers
Response `Documents` come in XML schema which can be parsed into pandas DataFrames.
Implemented: GL_MarketDocuments, TransmissionNetwork_MarketDocuments,
Publication_MarketDocuments and Balancing_MarketDocuments.
Missing: Outages, Congestion Management and System Operations.
Nevertheless, ENTSO-E Client seeks to be minimal to go from Query to DataFrame and requires domain-
knowledge on how to formulate queries and interpret various columns of a parsed response.
-----
ENTSO-E relies on many codes (`Type`) to map to desired queries.
Types are encoded in Enum classes with a .help() function to list the all.
They can be addressed through Type[code] or Type(string), making interaction
easy. HTTP requests and responses usually require the `code`, whereas we
want to formulate the query as a human-readable `string`.
::
from entsoe_client import Queries
from entsoe_client.ParameterTypes import *
Queries.Transmission.CapacityAllocatedOutsideEU(
out_Domain=Area('SK'),
in_Domain=Area('UA_BEI'),
marketAgreementType=MarketAgreementType('Daily'), # Original code: A01
auctionType=AuctionType('Explicit'), # Original code: A02
auctionCategory=AuctionCategory('Hourly'), # Original code: A04
classificationSequence_AttributeInstanceComponent_Position=1,
periodStart=201601012300,
periodEnd=201601022300)
::
>>> ParameterTypes.DocumentType['A25'] == ParameterTypes.DocumentType('Allocation result document')
True
>>> ec.ParameterTypes.DocumentType.help()
--- DocumentType ---
API_PARAMETER: DESCRIPTION
[...]
A25: Allocation result document
A71: Generation forecast
A72: Reservoir filling information
A73: Actual generation
A85: Imbalance prices
A86: Imbalance volume
[...]
API_PARAMETER: DESCRIPTION
--- DocumentType ---
>>> ec.ParameterTypes.BusinessType.help()
--- BusinessType ---
API_PARAMETER: DESCRIPTION
[...]
A25: General Capacity Information
A29: Already allocated capacity(AAC)
A97: Manual frequency restoration reserve
B08: Total nominated capacity
C22: Shared Balancing Reserve Capacity
C24: Actual reserve capacity
[...]
API_PARAMETER: DESCRIPTION
--- BusinessType ---
::
#shortened from sample_plot.py
import entsoe_client as ec
from settings import api_key
# Instantiate Client, Parser and Query.
client = ec.Client(api_key)
parser = ec.Parser()
query = ec.Queries.Generation.AggregatedGenerationPerType(
in_Domain=ec.ParameterTypes.Area('DE_LU'),
periodStart=202109050200,
periodEnd=202109070200)
# Extract data.
response = client(query)
df = parser(response)
[...]
# Transform data.
production = df[~consumption_mask][['quantity', 'TimeSeries.MktPSRType.psrType']]
## PsrType, e.g. `B01` := `Biomass`.
production['GenerationType'] = production['TimeSeries.MktPSRType.psrType']. \
apply(lambda x: ParameterTypes.PsrType[x].value) # Map ENTSO-E PsrTypes into human-readable string.
production_by_type = pd.pivot_table(production,
index=production.index,
columns='GenerationType',
values='quantity')
[...]
# Plot.
production_by_type.plot.bar(title="Production by Generation Type in DE-LU",
xlabel="UTC",
ylabel='MWh',
ax=ax,
**plot_params)
[...]
.. image:: ./sample_plot.png
Raw data
{
"_id": null,
"home_page": "https://github.com/DarioHett/entsoe-client",
"name": "entsoe-client",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.9,<4.0",
"maintainer_email": "",
"keywords": "entsoe,entso-e,power,electricity,europe,tso,transparency platform",
"author": "Dario Hett",
"author_email": "dariohett@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/d9/86/f68a3d57fa273d8b4fef237ea10e67a4217c4e65727edf5b3e7198ef8d2f/entsoe_client-0.2.3.tar.gz",
"platform": null,
"description": "Disclaimer: This library was written to experiment with various concepts. \nThe submodules `entsoe_client.ParameterTypes` and `entsoe_client.Queries` are straight forward.\nTo refactor `entsoe_client.Parsers`, the next steps would be to establish fixtures, \nstandardize flattening of the XML tree and decouple the records from the DataFrame library.\n\n===============\nENTSO-E Client\n===============\n\nFormulate readable queries and handle data in Pandas,\nincluding an exhaustive set of pre-defined queries.\n\n.. code-block:: python\n\n >>> import requests\n >>> from lxml import objectify\n >>> from lxml.etree import dump\n >>> url = 'https://transparency.entsoe.eu/api?' \\\n ... 'documentType=A81&businessType=A95&psrType=A04&type_MarketAgreement.Type=A01&controlArea_Domain=10YNL----------L' \\\n ... f'&periodStart=202101010000&periodEnd=202104010000&securityToken={api_key}'\n >>> response = requests.Session().get(url=url)\n >>> element = objectify.fromstring(response.content)\n >>> dump(element)\n <Balancing_MarketDocument xmlns=\"urn:iec62325.351:tc57wg16:451-6:balancingdocument:3:0\">\n <mRID>051b91beed574b48b4548214e9001afc</mRID>\n <revisionNumber>1</revisionNumber>\n <type>A81</type>\n <process.processType>A34</process.processType>\n <sender_MarketParticipant.mRID codingScheme=\"A01\">10X1001A1001A450</sender_MarketParticipant.mRID>\n <sender_MarketParticipant.marketRole.type>A32</sender_MarketParticipant.marketRole.type>\n <receiver_MarketParticipant.mRID codingScheme=\"A01\">10X1001A1001A450</receiver_MarketParticipant.mRID>\n <receiver_MarketParticipant.marketRole.type>A33</receiver_MarketParticipant.marketRole.type>\n <createdDateTime>2021-10-04T18:12:43Z</createdDateTime>\n <controlArea_Domain.mRID codingScheme=\"A01\">10YNL----------L</controlArea_Domain.mRID>\n <period.timeInterval>\n <start>2020-12-31T23:00Z</start>\n <end>2021-03-31T22:00Z</end>\n </period.timeInterval>\n <TimeSeries>\n <mRID>1</mRID>\n <businessType>A95</businessType>\n <type_MarketAgreement.type>A01</type_MarketAgreement.type>\n <mktPSRType.psrType>A04</mktPSRType.psrType>\n <flowDirection.direction>A03</flowDirection.direction>\n <quantity_Measure_Unit.name>MAW</quantity_Measure_Unit.name>\n <curveType>A01</curveType>\n <Period>\n <timeInterval>\n <start>2020-12-31T23:00Z</start>\n <end>2021-01-01T23:00Z</end>\n </timeInterval>\n <resolution>PT60M</resolution>\n <Point>\n <position>1</position>\n <quantity>44</quantity>\n </Point>\n <Point>\n <position>2</position>\n <quantity>44</quantity>\n [...]\n\nbecomes\n\n.. code-block:: python\n\n >>> import entsoe_client as ec\n >>> from entsoe_client.ParameterTypes import *\n >>> client = ec.Client(api_key)\n >>> parser = ec.Parser\n >>> query = ec.Query(\n ... documentType=DocumentType(\"Contracted reserves\"),\n ... psrType=PsrType(\"Generation\"),\n ... businessType=BusinessType(\"Frequency containment reserve\"),\n ... controlArea_Domain=Area(\"NL\"),\n ... type_MarketAgreementType=MarketAgreementType(\"Daily\"),\n ... periodStart=\"2021-01-01T00:00\",\n ... periodEnd=\"2021-04-01T00:00\"\n ... )\n >>> response = client(query)\n >>> df = parser.parse(response)\n >>> df.iloc[:,:3].head()\n position quantity Period.timeInterval.start...\n 2020-12-31 23:00:00+00:00 1 44 2020-12-31T23:00Z\n 2021-01-01 00:00:00+00:00 2 44 2020-12-31T23:00Z\n 2021-01-01 01:00:00+00:00 3 44 2020-12-31T23:00Z\n 2021-01-01 02:00:00+00:00 4 44 2020-12-31T23:00Z\n 2021-01-01 03:00:00+00:00 5 44 2020-12-31T23:00Z\n ...\n\n\npredefined queries are subset of the generic Query class, covering all examples of the `ENTSO-E API guide <https://transparency.entsoe.eu/content/static_content/Static%20content/web%20api/Guide.html>`_.\n\n.. code-block:: python\n\n >>> predefined_query = ec.Queries.Balancing.AmountOfBalancingReservesUnderContract(\n ... controlArea_Domain=Area(\"NL\"),\n ... type_MarketAgreementType=MarketAgreementType(\"Daily\"),\n ... psrType=PsrType(\"Generation\"),\n ... periodStart=\"2021-01-01T00:00\",\n ... periodEnd=\"2021-04-01T00:00\"\n ... )\n ...\n >>> predefined_query() == query()\n True\n\n-----\n\n| *ENTSO-E Client* enables straight-forward access to *all* of the data at `ENTSO-E Transparency Platform <https://transparency.entsoe.eu/>`_.\n\n* Query templates abstract the API specifics through Enumerated types.\n\n* Parse responses into Pandas DataFrames without loss of *any* information.\n\n| The separation of Queries, Client and Parser with their hierarchical abstractions keep the package extensible and maintainable. A pipeline from Query to DataFrame is trivial, preserving the ability to customize steps in between.\n\n| The implementation relies primarily on the\n `Transparency Platform restful API - user guide <https://transparency.entsoe.eu/content/static_content/Static%20content/web%20api/Guide.html>`_.\n The `Manual of Procedures (MoP) <https://www.entsoe.eu/data/transparency-platform/mop/>`_ documents provide\n further insight on the *business requirements specification*.\n Further information can be found in the\n `Electronic Data Interchange (EDI) Library <https://www.entsoe.eu/publications/electronic-data-interchange-edi-library/>`_.\n\n-----\n\nMain contributions\n\n* Exhaustive List of ParameterTypes.\n These allow mapping between natural language and the codes required\n for GET requests, e.g. :code:`DocumentType.A85 == DocumentType(\"Imbalance price\")`.\n This feature allows keeping track of queries without jumping between documents or adding comments.\n\n* Exhaustive List of Pre-defined Queries from ENTSO-E API Guide.\n `ENTSO-E API Guide <https://transparency.entsoe.eu/content/static_content/Static%20content/web%20api/Guide.html>`_\n is a minial set for any API connector to implement and reflects all dashboards on\n ENTSO-E Transparency Platform.\n\n* Parsers\n Response `Documents` come in XML schema which can be parsed into pandas DataFrames.\n\n Implemented: GL_MarketDocuments, TransmissionNetwork_MarketDocuments,\n Publication_MarketDocuments and Balancing_MarketDocuments.\n\n Missing: Outages, Congestion Management and System Operations.\n\nNevertheless, ENTSO-E Client seeks to be minimal to go from Query to DataFrame and requires domain-\nknowledge on how to formulate queries and interpret various columns of a parsed response.\n\n-----\n\nENTSO-E relies on many codes (`Type`) to map to desired queries.\nTypes are encoded in Enum classes with a .help() function to list the all.\nThey can be addressed through Type[code] or Type(string), making interaction\neasy. HTTP requests and responses usually require the `code`, whereas we\nwant to formulate the query as a human-readable `string`.\n\n::\n\n from entsoe_client import Queries\n from entsoe_client.ParameterTypes import *\n\n Queries.Transmission.CapacityAllocatedOutsideEU(\n out_Domain=Area('SK'),\n in_Domain=Area('UA_BEI'),\n marketAgreementType=MarketAgreementType('Daily'), # Original code: A01\n auctionType=AuctionType('Explicit'), # Original code: A02\n auctionCategory=AuctionCategory('Hourly'), # Original code: A04\n classificationSequence_AttributeInstanceComponent_Position=1,\n periodStart=201601012300,\n periodEnd=201601022300)\n\n::\n\n >>> ParameterTypes.DocumentType['A25'] == ParameterTypes.DocumentType('Allocation result document')\n True\n >>> ec.ParameterTypes.DocumentType.help()\n --- DocumentType ---\n API_PARAMETER: DESCRIPTION\n [...]\n A25: Allocation result document\n A71: Generation forecast\n A72: Reservoir filling information\n A73: Actual generation\n A85: Imbalance prices\n A86: Imbalance volume\n [...]\n API_PARAMETER: DESCRIPTION\n --- DocumentType ---\n >>> ec.ParameterTypes.BusinessType.help()\n --- BusinessType ---\n API_PARAMETER: DESCRIPTION\n [...]\n A25: General Capacity Information\n A29: Already allocated capacity(AAC)\n A97: Manual frequency restoration reserve\n B08: Total nominated capacity\n C22: Shared Balancing Reserve Capacity\n C24: Actual reserve capacity\n [...]\n API_PARAMETER: DESCRIPTION\n --- BusinessType ---\n\n::\n\n #shortened from sample_plot.py\n import entsoe_client as ec\n from settings import api_key\n\n # Instantiate Client, Parser and Query.\n client = ec.Client(api_key)\n parser = ec.Parser()\n query = ec.Queries.Generation.AggregatedGenerationPerType(\n in_Domain=ec.ParameterTypes.Area('DE_LU'),\n periodStart=202109050200,\n periodEnd=202109070200)\n\n # Extract data.\n response = client(query)\n df = parser(response)\n [...]\n\n # Transform data.\n production = df[~consumption_mask][['quantity', 'TimeSeries.MktPSRType.psrType']]\n ## PsrType, e.g. `B01` := `Biomass`.\n production['GenerationType'] = production['TimeSeries.MktPSRType.psrType']. \\\n apply(lambda x: ParameterTypes.PsrType[x].value) # Map ENTSO-E PsrTypes into human-readable string.\n production_by_type = pd.pivot_table(production,\n index=production.index,\n columns='GenerationType',\n values='quantity')\n [...]\n # Plot.\n production_by_type.plot.bar(title=\"Production by Generation Type in DE-LU\",\n xlabel=\"UTC\",\n ylabel='MWh',\n ax=ax,\n **plot_params)\n [...]\n\n\n.. image:: ./sample_plot.png\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "Formulate human-readable queries and retrieve data from ENTSO-E into pandas.DataFrame format.",
"version": "0.2.3",
"project_urls": {
"Homepage": "https://github.com/DarioHett/entsoe-client",
"Repository": "https://github.com/DarioHett/entsoe-client"
},
"split_keywords": [
"entsoe",
"entso-e",
"power",
"electricity",
"europe",
"tso",
"transparency platform"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "f8cefd7e1872febbf62f0873df74db3a9add52a00f6103ec9c6a937e2c4e4f96",
"md5": "ea3db2e05490dee6365ad4dc8c126e4d",
"sha256": "0dfd8f9999927695adfcbfe1fdf5ae60f880879834d2a6f38f3764571d2a9519"
},
"downloads": -1,
"filename": "entsoe_client-0.2.3-py3-none-any.whl",
"has_sig": false,
"md5_digest": "ea3db2e05490dee6365ad4dc8c126e4d",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.9,<4.0",
"size": 37766,
"upload_time": "2023-12-10T17:35:34",
"upload_time_iso_8601": "2023-12-10T17:35:34.921263Z",
"url": "https://files.pythonhosted.org/packages/f8/ce/fd7e1872febbf62f0873df74db3a9add52a00f6103ec9c6a937e2c4e4f96/entsoe_client-0.2.3-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "d986f68a3d57fa273d8b4fef237ea10e67a4217c4e65727edf5b3e7198ef8d2f",
"md5": "dc64c2fad049edf0edd69d6f1bb84792",
"sha256": "008cdb666cec23af8e8f3b98d7b7a47918c74f0ac46aada05631a18d305fdef0"
},
"downloads": -1,
"filename": "entsoe_client-0.2.3.tar.gz",
"has_sig": false,
"md5_digest": "dc64c2fad049edf0edd69d6f1bb84792",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.9,<4.0",
"size": 26040,
"upload_time": "2023-12-10T17:35:37",
"upload_time_iso_8601": "2023-12-10T17:35:37.113793Z",
"url": "https://files.pythonhosted.org/packages/d9/86/f68a3d57fa273d8b4fef237ea10e67a4217c4e65727edf5b3e7198ef8d2f/entsoe_client-0.2.3.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-12-10 17:35:37",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "DarioHett",
"github_project": "entsoe-client",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "entsoe-client"
}
JSON
