<!--
Copyright (c) 2022-2023 Geosiris.
SPDX-License-Identifier: Apache-2.0
-->
energyml-utils
==============
[](https://badge.fury.io/py/energyml-utils)
[](https://github.com/geosiris-technologies/geosiris-technologies/blob/main/energyml-utils/LICENSE)
[](https://geosiris-technologies.readthedocs.io/en/latest/?badge=latest)
Installation
------------
energyml-utils can be installed with pip :
```console
pip install energyml-utils
```
or with poetry:
```console
poetry add energyml-utils
```
Features
--------
### Supported packages versions
This package supports read/write in xml/json the following packages :
- EML (common) : 2.0, 2.1, 2.2, 2.3
- RESQML : 2.0.1, 2.2dev3, 2.2
- WITSMl : 2.0, 2.1
- PRODML : 2.0, 2.2
/!\\ By default, these packages are not installed and are published independently.
You can install only the versions you need by adding the following lines in the .toml file :
```toml
energyml-common2-0 = "^1.12.0"
energyml-common2-1 = "^1.12.0"
energyml-common2-2 = "^1.12.0"
energyml-common2-3 = "^1.12.0"
energyml-resqml2-0-1 = "^1.12.0"
energyml-resqml2-2-dev3 = "^1.12.0"
energyml-resqml2-2 = "^1.12.0"
energyml-witsml2-0 = "^1.12.0"
energyml-witsml2-1 = "^1.12.0"
energyml-prodml2-0 = "^1.12.0"
energyml-prodml2-2 = "^1.12.0"
```
### Content of the package :
- Support EPC + h5 read and write
- *.rels* files are automatically generated, but it is possible to add custom Relations.
- You can add "raw files" such as PDF or anything else, in your EPC instance, and it will be package with other files in the ".epc" file when you call the "export" function.
- You can work with local files, but also with IO (BytesIO). This is usefull to work with cloud application to avoid local storage.
- Supports xml / json read and write (for energyml objects)
- *Work in progress* : Supports the read of 3D data inside the "AbstractMesh" class (and sub-classes "PointSetMesh", "PolylineSetMesh", "SurfaceMesh"). This gives you a instance containing a list of point and a list of indices to easily re-create a 3D representation of the data.
- These "mesh" classes provides *.obj*, *.off*, and *.geojson* export.
- Introspection : This package includes functions to ease the access of specific values inside energyml objects.
- Functions to access to UUID, object Version, and more generic functions for any other attributes with regex like ".Citation.Title" or "Cit\\.*.Title" (regular dots are used as in python object attribute access. To use dot in regex, you must escape them with a '\\')
- Functions to parse, or generate from an energyml object the "ContentType" or "QualifiedType"
- Generation of random data : you can generate random values for a specific energyml object. For example, you can generate a WITSML Tubular object with random values in it.
- Objects correctness validation :
- You can verify if your objects are valid following the energyml norm (a check is done on regex contraint attributes, maxCount, minCount, mandatory etc...)
- The DOR validation is tested : check if the DOR has correct information (title, ContentType/QualifiedType, object version), and also if the referenced object exists in the context of the EPC instance (or a list of object).
- Abstractions done to ease use with *ETP* (Energistics Transfer Protocol) :
- The "EnergymlWorkspace" class allows to abstract the access of numerical data like "ExternalArrays". This class can thus be extended to interact with ETP "GetDataArray" request etc...
- ETP URI support : the "Uri" class allows to parse/write an etp uri.
## EPC Stream Reader
The **EpcStreamReader** provides memory-efficient handling of large EPC files through lazy loading and smart caching. Unlike the standard `Epc` class which loads all objects into memory, the stream reader loads objects on-demand, making it ideal for handling very large EPC files with thousands of objects.
### Key Features
- **Lazy Loading**: Objects are loaded only when accessed, reducing memory footprint
- **Smart Caching**: LRU (Least Recently Used) cache with configurable size
- **Automatic EPC Version Detection**: Supports both CLASSIC and EXPANDED EPC formats
- **Add/Remove/Update Operations**: Full CRUD operations with automatic file structure maintenance
- **Context Management**: Automatic resource cleanup with `with` statements
- **Memory Monitoring**: Track cache efficiency and memory usage statistics
### Basic Usage
```python
from energyml.utils.epc_stream import EpcStreamReader
# Open EPC file with context manager (recommended)
with EpcStreamReader('large_file.epc', cache_size=50) as reader:
# List all objects without loading them
print(f"Total objects: {reader.stats.total_objects}")
# Get object by identifier
obj: Any = reader.get_object_by_identifier("uuid.version")
# Get objects by type
features: List[Any] = reader.get_objects_by_type("BoundaryFeature")
# Get all objects with same UUID
versions: List[Any] = reader.get_object_by_uuid("12345678-1234-1234-1234-123456789abc")
```
### Adding Objects
```python
from energyml.utils.epc_stream import EpcStreamReader
from energyml.utils.constants import gen_uuid
import energyml.resqml.v2_2.resqmlv2 as resqml
import energyml.eml.v2_3.commonv2 as eml
# Create a new EnergyML object
boundary_feature = resqml.BoundaryFeature()
boundary_feature.uuid = gen_uuid()
boundary_feature.citation = eml.Citation(title="My Feature")
with EpcStreamReader('my_file.epc') as reader:
# Add object - path is automatically generated based on EPC version
identifier = reader.add_object(boundary_feature)
print(f"Added object with identifier: {identifier}")
# Or specify custom path (optional)
identifier = reader.add_object(boundary_feature, "custom/path/MyFeature.xml")
```
### Removing Objects
```python
with EpcStreamReader('my_file.epc') as reader:
# Remove specific version by full identifier
success = reader.remove_object("uuid.version")
# Remove ALL versions by UUID only
success = reader.remove_object("12345678-1234-1234-1234-123456789abc")
if success:
print("Object(s) removed successfully")
```
### Updating Objects
```python
...
from energyml.utils.introspection import set_attribute_from_path
with EpcStreamReader('my_file.epc') as reader:
# Get existing object
obj = reader.get_object_by_identifier("uuid.version")
# Modify the object
set_attribute_from_path(obj, "citation.title", "Updated Title")
# Update in EPC file
new_identifier = reader.update_object(obj)
print(f"Updated object: {new_identifier}")
```
### Performance Monitoring
```python
with EpcStreamReader('large_file.epc', cache_size=100) as reader:
# Access some objects...
for i in range(10):
obj = reader.get_object_by_identifier(f"uuid-{i}.1")
# Check performance statistics
print(f"Cache hit rate: {reader.stats.cache_hit_rate:.1f}%")
print(f"Memory efficiency: {reader.stats.memory_efficiency:.1f}%")
print(f"Objects in cache: {reader.stats.loaded_objects}/{reader.stats.total_objects}")
```
### EPC Version Support
The EpcStreamReader automatically detects and handles both EPC packaging formats:
- **CLASSIC Format**: Flat file structure (e.g., `obj_BoundaryFeature_{uuid}.xml`)
- **EXPANDED Format**: Namespace structure (e.g., `namespace_resqml201/version_{id}/obj_BoundaryFeature_{uuid}.xml` or `namespace_resqml201/obj_BoundaryFeature_{uuid}.xml`)
```python
with EpcStreamReader('my_file.epc') as reader:
print(f"Detected EPC version: {reader.export_version}")
# Objects added will use the same format as the existing EPC file
```
### Advanced Usage
```python
# Initialize without preloading metadata for faster startup
reader = EpcStreamReader('huge_file.epc', preload_metadata=False, cache_size=200)
try:
# Manual metadata loading when needed
reader._load_metadata()
# Get object dependencies
deps = reader.get_object_dependencies("uuid.version")
# Batch processing with memory monitoring
for obj_type in ["BoundaryFeature", "PropertyKind"]:
objects = reader.get_objects_by_type(obj_type)
print(f"Processing {len(objects)} {obj_type} objects")
finally:
reader.close() # Manual cleanup if not using context manager
```
The EpcStreamReader is perfect for applications that need to work with large EPC files efficiently, such as data processing pipelines, web applications, or analysis tools where memory usage is a concern.
# Poetry scripts :
- extract_3d : extract a representation into an 3D file (obj/off)
- csv_to_dataset : translate csv data into h5 dataset
- generate_data : generate a random data from a qualified_type
- xml_to_json : translate an energyml xml file into json.
- json_to_xml : translate an energyml json file into an xml file
- describe_as_csv : create a csv description of an EPC content
- validate : validate an energyml object or an EPC instance (or a folder containing energyml objects)
## Installation to test poetry scripts :
```bash
poetry install
```
if you fail to run a script, you may have to add "src" to your PYTHONPATH environment variable. For example, in powershell :
```powershell
$env:PYTHONPATH="src"
```
## Validation examples :
An epc file:
```bash
poetry run validate --file "path/to/your/energyml/object.epc" *> output_logs.json
```
An xml file:
```bash
poetry run validate --file "path/to/your/energyml/object.xml" *> output_logs.json
```
A json file:
```bash
poetry run validate --file "path/to/your/energyml/object.json" *> output_logs.json
```
A folder containing Epc/xml/json files:
```bash
poetry run validate --file "path/to/your/folder" *> output_logs.json
```
Raw data
{
"_id": null,
"home_page": "http://www.geosiris.com",
"name": "energyml-utils",
"maintainer": "Valentin Gauthier",
"docs_url": null,
"requires_python": "<4.0,>=3.9",
"maintainer_email": "valentin.gauthier@geosiris.com",
"keywords": "energyml, resqml, witsml, prodml",
"author": "Valentin Gauthier",
"author_email": "valentin.gauthier@geosiris.com",
"download_url": "https://files.pythonhosted.org/packages/46/5e/41d9d624dd6358bc2f68bc65e1b7a6652e8f3fd2e582042e1f6b23a3b2bd/energyml_utils-1.9.1.tar.gz",
"platform": null,
"description": "<!--\nCopyright (c) 2022-2023 Geosiris.\nSPDX-License-Identifier: Apache-2.0\n-->\nenergyml-utils\n==============\n\n[](https://badge.fury.io/py/energyml-utils)\n[](https://github.com/geosiris-technologies/geosiris-technologies/blob/main/energyml-utils/LICENSE)\n[](https://geosiris-technologies.readthedocs.io/en/latest/?badge=latest)\n\n\n\n\n\n\nInstallation\n------------\n\nenergyml-utils can be installed with pip : \n\n```console\npip install energyml-utils\n```\n\nor with poetry: \n```console\npoetry add energyml-utils\n```\n\n\nFeatures\n--------\n\n### Supported packages versions\n\nThis package supports read/write in xml/json the following packages : \n- EML (common) : 2.0, 2.1, 2.2, 2.3\n- RESQML : 2.0.1, 2.2dev3, 2.2\n- WITSMl : 2.0, 2.1\n- PRODML : 2.0, 2.2\n\n/!\\\\ By default, these packages are not installed and are published independently.\nYou can install only the versions you need by adding the following lines in the .toml file : \n```toml\nenergyml-common2-0 = \"^1.12.0\"\nenergyml-common2-1 = \"^1.12.0\"\nenergyml-common2-2 = \"^1.12.0\"\nenergyml-common2-3 = \"^1.12.0\"\nenergyml-resqml2-0-1 = \"^1.12.0\"\nenergyml-resqml2-2-dev3 = \"^1.12.0\"\nenergyml-resqml2-2 = \"^1.12.0\"\nenergyml-witsml2-0 = \"^1.12.0\"\nenergyml-witsml2-1 = \"^1.12.0\"\nenergyml-prodml2-0 = \"^1.12.0\"\nenergyml-prodml2-2 = \"^1.12.0\"\n```\n\n### Content of the package :\n\n- Support EPC + h5 read and write\n - *.rels* files are automatically generated, but it is possible to add custom Relations.\n - You can add \"raw files\" such as PDF or anything else, in your EPC instance, and it will be package with other files in the \".epc\" file when you call the \"export\" function.\n - You can work with local files, but also with IO (BytesIO). This is usefull to work with cloud application to avoid local storage.\n- Supports xml / json read and write (for energyml objects)\n- *Work in progress* : Supports the read of 3D data inside the \"AbstractMesh\" class (and sub-classes \"PointSetMesh\", \"PolylineSetMesh\", \"SurfaceMesh\"). This gives you a instance containing a list of point and a list of indices to easily re-create a 3D representation of the data.\n - These \"mesh\" classes provides *.obj*, *.off*, and *.geojson* export.\n- Introspection : This package includes functions to ease the access of specific values inside energyml objects.\n - Functions to access to UUID, object Version, and more generic functions for any other attributes with regex like \".Citation.Title\" or \"Cit\\\\.*.Title\" (regular dots are used as in python object attribute access. To use dot in regex, you must escape them with a '\\\\')\n - Functions to parse, or generate from an energyml object the \"ContentType\" or \"QualifiedType\"\n - Generation of random data : you can generate random values for a specific energyml object. For example, you can generate a WITSML Tubular object with random values in it.\n- Objects correctness validation :\n - You can verify if your objects are valid following the energyml norm (a check is done on regex contraint attributes, maxCount, minCount, mandatory etc...)\n - The DOR validation is tested : check if the DOR has correct information (title, ContentType/QualifiedType, object version), and also if the referenced object exists in the context of the EPC instance (or a list of object).\n- Abstractions done to ease use with *ETP* (Energistics Transfer Protocol) :\n - The \"EnergymlWorkspace\" class allows to abstract the access of numerical data like \"ExternalArrays\". This class can thus be extended to interact with ETP \"GetDataArray\" request etc...\n- ETP URI support : the \"Uri\" class allows to parse/write an etp uri.\n\n## EPC Stream Reader\n\nThe **EpcStreamReader** provides memory-efficient handling of large EPC files through lazy loading and smart caching. Unlike the standard `Epc` class which loads all objects into memory, the stream reader loads objects on-demand, making it ideal for handling very large EPC files with thousands of objects.\n\n### Key Features\n\n- **Lazy Loading**: Objects are loaded only when accessed, reducing memory footprint\n- **Smart Caching**: LRU (Least Recently Used) cache with configurable size \n- **Automatic EPC Version Detection**: Supports both CLASSIC and EXPANDED EPC formats\n- **Add/Remove/Update Operations**: Full CRUD operations with automatic file structure maintenance\n- **Context Management**: Automatic resource cleanup with `with` statements\n- **Memory Monitoring**: Track cache efficiency and memory usage statistics\n\n### Basic Usage\n\n```python\nfrom energyml.utils.epc_stream import EpcStreamReader\n\n# Open EPC file with context manager (recommended)\nwith EpcStreamReader('large_file.epc', cache_size=50) as reader:\n # List all objects without loading them\n print(f\"Total objects: {reader.stats.total_objects}\")\n \n # Get object by identifier\n obj: Any = reader.get_object_by_identifier(\"uuid.version\")\n \n # Get objects by type\n features: List[Any] = reader.get_objects_by_type(\"BoundaryFeature\")\n \n # Get all objects with same UUID\n versions: List[Any] = reader.get_object_by_uuid(\"12345678-1234-1234-1234-123456789abc\")\n```\n\n### Adding Objects\n\n```python\nfrom energyml.utils.epc_stream import EpcStreamReader\nfrom energyml.utils.constants import gen_uuid\nimport energyml.resqml.v2_2.resqmlv2 as resqml\nimport energyml.eml.v2_3.commonv2 as eml\n\n# Create a new EnergyML object\nboundary_feature = resqml.BoundaryFeature()\nboundary_feature.uuid = gen_uuid()\nboundary_feature.citation = eml.Citation(title=\"My Feature\")\n\nwith EpcStreamReader('my_file.epc') as reader:\n # Add object - path is automatically generated based on EPC version\n identifier = reader.add_object(boundary_feature)\n print(f\"Added object with identifier: {identifier}\")\n \n # Or specify custom path (optional)\n identifier = reader.add_object(boundary_feature, \"custom/path/MyFeature.xml\")\n```\n\n### Removing Objects\n\n```python\nwith EpcStreamReader('my_file.epc') as reader:\n # Remove specific version by full identifier\n success = reader.remove_object(\"uuid.version\")\n \n # Remove ALL versions by UUID only\n success = reader.remove_object(\"12345678-1234-1234-1234-123456789abc\")\n \n if success:\n print(\"Object(s) removed successfully\")\n```\n\n### Updating Objects\n\n```python\n...\nfrom energyml.utils.introspection import set_attribute_from_path\n\nwith EpcStreamReader('my_file.epc') as reader:\n # Get existing object\n obj = reader.get_object_by_identifier(\"uuid.version\")\n \n # Modify the object\n set_attribute_from_path(obj, \"citation.title\", \"Updated Title\")\n \n # Update in EPC file\n new_identifier = reader.update_object(obj)\n print(f\"Updated object: {new_identifier}\")\n```\n\n### Performance Monitoring\n\n```python\nwith EpcStreamReader('large_file.epc', cache_size=100) as reader:\n # Access some objects...\n for i in range(10):\n obj = reader.get_object_by_identifier(f\"uuid-{i}.1\")\n \n # Check performance statistics\n print(f\"Cache hit rate: {reader.stats.cache_hit_rate:.1f}%\")\n print(f\"Memory efficiency: {reader.stats.memory_efficiency:.1f}%\") \n print(f\"Objects in cache: {reader.stats.loaded_objects}/{reader.stats.total_objects}\")\n```\n\n### EPC Version Support\n\nThe EpcStreamReader automatically detects and handles both EPC packaging formats:\n\n- **CLASSIC Format**: Flat file structure (e.g., `obj_BoundaryFeature_{uuid}.xml`)\n- **EXPANDED Format**: Namespace structure (e.g., `namespace_resqml201/version_{id}/obj_BoundaryFeature_{uuid}.xml` or `namespace_resqml201/obj_BoundaryFeature_{uuid}.xml`)\n\n```python\nwith EpcStreamReader('my_file.epc') as reader:\n print(f\"Detected EPC version: {reader.export_version}\")\n # Objects added will use the same format as the existing EPC file\n```\n\n### Advanced Usage\n\n```python\n# Initialize without preloading metadata for faster startup\nreader = EpcStreamReader('huge_file.epc', preload_metadata=False, cache_size=200)\n\ntry:\n # Manual metadata loading when needed\n reader._load_metadata()\n \n # Get object dependencies\n deps = reader.get_object_dependencies(\"uuid.version\")\n \n # Batch processing with memory monitoring\n for obj_type in [\"BoundaryFeature\", \"PropertyKind\"]:\n objects = reader.get_objects_by_type(obj_type)\n print(f\"Processing {len(objects)} {obj_type} objects\")\n \nfinally:\n reader.close() # Manual cleanup if not using context manager\n```\n\nThe EpcStreamReader is perfect for applications that need to work with large EPC files efficiently, such as data processing pipelines, web applications, or analysis tools where memory usage is a concern.\n\n\n# Poetry scripts : \n\n- extract_3d : extract a representation into an 3D file (obj/off)\n- csv_to_dataset : translate csv data into h5 dataset\n- generate_data : generate a random data from a qualified_type \n- xml_to_json : translate an energyml xml file into json.\n- json_to_xml : translate an energyml json file into an xml file\n- describe_as_csv : create a csv description of an EPC content\n- validate : validate an energyml object or an EPC instance (or a folder containing energyml objects)\n\n\n\n## Installation to test poetry scripts : \n\n```bash\npoetry install\n```\n\nif you fail to run a script, you may have to add \"src\" to your PYTHONPATH environment variable. For example, in powershell : \n\n```powershell\n$env:PYTHONPATH=\"src\"\n```\n\n\n## Validation examples : \n\nAn epc file:\n```bash\npoetry run validate --file \"path/to/your/energyml/object.epc\" *> output_logs.json\n```\n\nAn xml file:\n```bash\npoetry run validate --file \"path/to/your/energyml/object.xml\" *> output_logs.json\n```\n\nA json file:\n```bash\npoetry run validate --file \"path/to/your/energyml/object.json\" *> output_logs.json\n```\n\nA folder containing Epc/xml/json files:\n```bash\npoetry run validate --file \"path/to/your/folder\" *> output_logs.json\n```\n\n",
"bugtrack_url": null,
"license": "Apache-2.0",
"summary": "Energyml helper",
"version": "1.9.1",
"project_urls": {
"Homepage": "http://www.geosiris.com",
"Repository": "https://github.com/geosiris-technologies/energyml-python"
},
"split_keywords": [
"energyml",
" resqml",
" witsml",
" prodml"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "33a8c89a0bce11398cb11683fbff703752f1edf0fad56f3c4abf869b7efddd1a",
"md5": "117d998a9da1a13135b3f6a8c887a6d7",
"sha256": "665455ad629bceb6de7014b0fe4fc3251b63e4978fd54ace8fa9f9c06d62e435"
},
"downloads": -1,
"filename": "energyml_utils-1.9.1-py3-none-any.whl",
"has_sig": false,
"md5_digest": "117d998a9da1a13135b3f6a8c887a6d7",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": "<4.0,>=3.9",
"size": 611214,
"upload_time": "2025-10-07T15:37:36",
"upload_time_iso_8601": "2025-10-07T15:37:36.941549Z",
"url": "https://files.pythonhosted.org/packages/33/a8/c89a0bce11398cb11683fbff703752f1edf0fad56f3c4abf869b7efddd1a/energyml_utils-1.9.1-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "465e41d9d624dd6358bc2f68bc65e1b7a6652e8f3fd2e582042e1f6b23a3b2bd",
"md5": "3a4f8600a6e26c1cfe57c8f1725802eb",
"sha256": "491532ca3fdb3d5b1efe2781a5438a46206adecc260460f46bdd11faee99cd47"
},
"downloads": -1,
"filename": "energyml_utils-1.9.1.tar.gz",
"has_sig": false,
"md5_digest": "3a4f8600a6e26c1cfe57c8f1725802eb",
"packagetype": "sdist",
"python_version": "source",
"requires_python": "<4.0,>=3.9",
"size": 597201,
"upload_time": "2025-10-07T15:37:38",
"upload_time_iso_8601": "2025-10-07T15:37:38.687601Z",
"url": "https://files.pythonhosted.org/packages/46/5e/41d9d624dd6358bc2f68bc65e1b7a6652e8f3fd2e582042e1f6b23a3b2bd/energyml_utils-1.9.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-10-07 15:37:38",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "geosiris-technologies",
"github_project": "energyml-python",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "energyml-utils"
}
JSON
