# VTL Engine
| | |
|---------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| Testing | [](https://github.com/Meaningful-Data/vtlengine/actions/workflows/testing.yml) |
| Package | [](https://pypi.org/project/vtlengine/) |
| License | [](https://github.com/Meaningful-Data/vtlengine/blob/main/LICENSE.md) |
## Introduction
The VTL Engine is a Python library for validating and running VTL scripts.
It is a Python-based library around the [VTL Language](http://sdmx.org/?page_id=5096).
## Installation
### Requirements
The VTL Engine requires Python 3.10 or higher.
### Install with pip
To install the VTL Engine on any Operating System, you can use pip:
```bash
pip install vtlengine
```
*Note: it is recommended to install the VTL Engine in a virtual environment.*
## Usage
The VTL Engine API implements two basic methods:
* **Semantic Analysis**: aimed at validating the correctness of a script and computing the data
structures of the data sets created in the script.
* **Run**: aimed at executing the provided input on the provided input datasets.
Any action with VTL requires the following elements as input:
* **VTL Script**: Is the VTL to be executed, which includes the transformation scheme, as well as de
User Defined Operators, Hierarchical Rulesets and Datapoint Rulesets. It is provided as a string
or as a Path object to a vtl file.
* **Data structures** : Provides the structure of the input artifacts of the VTL script, according
to
the VTL Information model. Given that the current version doesn't prescribe a standard format for
providing the information, the VTL Engine is implementing a JSON format that can be found here.
Data Structures can be provided as Dictionaries or as Paths to JSON files. It is possible to have
* **External routines**: The VTL Engine allows using SQL (SQLite) with the eval operator. Can be
provided as a string with the SQL or as a path object to an SQL file. Its default value is `None`,
which shall be used if external routines are not applicable to the VTL script.
* **Value domains**: Provides the value domains that are used in the VTL script, normally with an in
operator. Can be provided as a dictionary or as a path to a JSON file. Its default value
is `None`, which shall be used if value domains are not applicable to the VTL script.
### Semantic Analysis
The `semantic_analysis` method serves to validate the correctness of a VTL script, as well as to
calculate the data structures of the datasets generated by the VTL script itself (that calculation
is a pre-requisite for the semantic analysis).
* If the VTL script is correct, the method returns a dictionary with the data structures of all the
datasets generated by the script.
* If the VTL script is incorrect, raises a VTL Engine custom error Explaining the error.
#### Example 1: Correct VTL
```python
from vtlengine import semantic_analysis
script = """
DS_A := DS_1 * 10;
"""
data_structures = {
'datasets': [
{'name': 'DS_1',
'DataStructure': [
{'name': 'Id_1',
'type':
'Integer',
'role': 'Identifier',
'nullable': False},
{'name': 'Me_1',
'type': 'Number',
'role': 'Measure',
'nullable': True}
]
}
]
}
sa_result = semantic_analysis(script=script, data_structures=data_structures)
print(sa_result)
```
Returns:
```
{'DS_A': Dataset(name='DS_A', components={'Id_1': Component(name='Id_1', data_type=<class 'vtlengine.DataTypes.Integer'>, role=<Role.IDENTIFIER: 'Identifier'>, nullable=False), 'Me_1': Component(name='Me_1', data_type=<class 'vtlengine.DataTypes.Number'>, role=<Role.MEASURE: 'Measure'>, nullable=True)}, data=None)}
```
#### Example 2: Incorrect VTL
Note that, as compared to Example 1, the only change is that Me_1 is of the String data type,
instead of Number.
```python
from vtlengine import semantic_analysis
script = """
DS_A := DS_1 * 10;
"""
data_structures = {
'datasets': [
{'name': 'DS_1',
'DataStructure': [
{'name': 'Id_1',
'type':
'Integer',
'role': 'Identifier',
'nullable': False},
{'name': 'Me_1',
'type': 'String',
'role': 'Measure',
'nullable': True}
]
}
]
}
sa_result = semantic_analysis(script=script, data_structures=data_structures)
print(sa_result)
```
Will raise the following Error:
``` python
raise SemanticError(code="1-1-1-2",
vtlengine.Exceptions.SemanticError: ('Invalid implicit cast from String and Integer to Number.', '1-1-1-2')
```
### Run VTL Scripts
The `run` method serves to execute a VTL script with input datapoints.
Returns a dictionary with all the generated Datasets.
When the output parameter is set, the engine will write the result of the computation to the output
folder, else it will include the data in the dictionary of the computed datasets.
Two validations are performed before running, which can raise errors:
* Semantic analysis: Equivalent to running the `semantic_analysis` method
* Data load analysis: Basic check of the data structure (names and types)
#### Example 3: Simple run
```python
from vtlengine import run
import pandas as pd
script = """
DS_A := DS_1 * 10;
"""
data_structures = {
'datasets': [
{'name': 'DS_1',
'DataStructure': [
{'name': 'Id_1',
'type':
'Integer',
'role': 'Identifier',
'nullable': False},
{'name': 'Me_1',
'type': 'Number',
'role': 'Measure',
'nullable': True}
]
}
]
}
data_df = pd.DataFrame(
{"Id_1": [1, 2, 3],
"Me_1": [10, 20, 30]})
datapoints = {"DS_1": data_df}
run_result = run(script=script, data_structures=data_structures,
datapoints=datapoints)
print(run_result)
```
returns:
``` python
{'DS_A': Dataset(name='DS_A', components={'Id_1': Component(name='Id_1', data_type=<class 'vtlengine.DataTypes.Integer'>, role=<Role.IDENTIFIER: 'Identifier'>, nullable=False), 'Me_1': Component(name='Me_1', data_type=<class 'vtlengine.DataTypes.Number'>, role=<Role.MEASURE: 'Measure'>, nullable=True)}, data= Id_1 Me_1
0 1 100.0
1 2 200.0
2 3 300.0)}
```
For more information on usage, please refer to
the [API documentation](https://docs.vtlengine.meaningfuldata.eu/api.html).
Raw data
{
"_id": null,
"home_page": null,
"name": "vtlengine",
"maintainer": null,
"docs_url": null,
"requires_python": "<4.0,>=3.10",
"maintainer_email": null,
"keywords": "vtl, sdmx, vtlengine, Validation and Transformation Language",
"author": "MeaningfulData",
"author_email": "info@meaningfuldata.eu",
"download_url": "https://files.pythonhosted.org/packages/99/8e/2ae81d5dff6b59e22ca582ea82a8e87adbb662c035f163ed6279a6f381f1/vtlengine-1.0.3.tar.gz",
"platform": null,
"description": "# VTL Engine\n\n| | |\n|---------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|\n| Testing | [](https://github.com/Meaningful-Data/vtlengine/actions/workflows/testing.yml) |\n| Package | [](https://pypi.org/project/vtlengine/) |\n| License | [](https://github.com/Meaningful-Data/vtlengine/blob/main/LICENSE.md) |\n\n## Introduction\n\nThe VTL Engine is a Python library for validating and running VTL scripts.\n\nIt is a Python-based library around the [VTL Language](http://sdmx.org/?page_id=5096).\n\n## Installation\n\n### Requirements\n\nThe VTL Engine requires Python 3.10 or higher.\n\n### Install with pip\n\nTo install the VTL Engine on any Operating System, you can use pip:\n\n```bash\n\npip install vtlengine\n\n```\n\n*Note: it is recommended to install the VTL Engine in a virtual environment.*\n\n## Usage\n\nThe VTL Engine API implements two basic methods:\n\n* **Semantic Analysis**: aimed at validating the correctness of a script and computing the data\n structures of the data sets created in the script.\n* **Run**: aimed at executing the provided input on the provided input datasets.\n\nAny action with VTL requires the following elements as input:\n\n* **VTL Script**: Is the VTL to be executed, which includes the transformation scheme, as well as de\n User Defined Operators, Hierarchical Rulesets and Datapoint Rulesets. It is provided as a string\n or as a Path object to a vtl file.\n* **Data structures** : Provides the structure of the input artifacts of the VTL script, according\n to\n the VTL Information model. Given that the current version doesn't prescribe a standard format for\n providing the information, the VTL Engine is implementing a JSON format that can be found here.\n Data Structures can be provided as Dictionaries or as Paths to JSON files. It is possible to have\n* **External routines**: The VTL Engine allows using SQL (SQLite) with the eval operator. Can be\n provided as a string with the SQL or as a path object to an SQL file. Its default value is `None`,\n which shall be used if external routines are not applicable to the VTL script.\n* **Value domains**: Provides the value domains that are used in the VTL script, normally with an in\n operator. Can be provided as a dictionary or as a path to a JSON file. Its default value\n is `None`, which shall be used if value domains are not applicable to the VTL script.\n\n### Semantic Analysis\n\nThe `semantic_analysis` method serves to validate the correctness of a VTL script, as well as to\ncalculate the data structures of the datasets generated by the VTL script itself (that calculation\nis a pre-requisite for the semantic analysis).\n\n* If the VTL script is correct, the method returns a dictionary with the data structures of all the\n datasets generated by the script.\n* If the VTL script is incorrect, raises a VTL Engine custom error Explaining the error.\n\n#### Example 1: Correct VTL\n\n```python\nfrom vtlengine import semantic_analysis\n\nscript = \"\"\"\n DS_A := DS_1 * 10;\n\"\"\"\n\ndata_structures = {\n 'datasets': [\n {'name': 'DS_1',\n 'DataStructure': [\n {'name': 'Id_1',\n 'type':\n 'Integer',\n 'role': 'Identifier',\n 'nullable': False},\n {'name': 'Me_1',\n 'type': 'Number',\n 'role': 'Measure',\n 'nullable': True}\n ]\n }\n ]\n}\n\nsa_result = semantic_analysis(script=script, data_structures=data_structures)\n\nprint(sa_result)\n\n```\n\nReturns:\n\n```\n{'DS_A': Dataset(name='DS_A', components={'Id_1': Component(name='Id_1', data_type=<class 'vtlengine.DataTypes.Integer'>, role=<Role.IDENTIFIER: 'Identifier'>, nullable=False), 'Me_1': Component(name='Me_1', data_type=<class 'vtlengine.DataTypes.Number'>, role=<Role.MEASURE: 'Measure'>, nullable=True)}, data=None)}\n```\n\n#### Example 2: Incorrect VTL\n\nNote that, as compared to Example 1, the only change is that Me_1 is of the String data type,\ninstead of Number.\n\n```python\nfrom vtlengine import semantic_analysis\n\nscript = \"\"\"\n DS_A := DS_1 * 10;\n\"\"\"\n\ndata_structures = {\n 'datasets': [\n {'name': 'DS_1',\n 'DataStructure': [\n {'name': 'Id_1',\n 'type':\n 'Integer',\n 'role': 'Identifier',\n 'nullable': False},\n {'name': 'Me_1',\n 'type': 'String',\n 'role': 'Measure',\n 'nullable': True}\n ]\n }\n ]\n}\n\nsa_result = semantic_analysis(script=script, data_structures=data_structures)\n\nprint(sa_result)\n\n``` \n\nWill raise the following Error:\n\n``` python\nraise SemanticError(code=\"1-1-1-2\",\nvtlengine.Exceptions.SemanticError: ('Invalid implicit cast from String and Integer to Number.', '1-1-1-2')\n```\n\n### Run VTL Scripts\n\nThe `run` method serves to execute a VTL script with input datapoints.\n\nReturns a dictionary with all the generated Datasets.\nWhen the output parameter is set, the engine will write the result of the computation to the output\nfolder, else it will include the data in the dictionary of the computed datasets.\n\nTwo validations are performed before running, which can raise errors:\n\n* Semantic analysis: Equivalent to running the `semantic_analysis` method\n* Data load analysis: Basic check of the data structure (names and types)\n\n#### Example 3: Simple run\n\n```python\nfrom vtlengine import run\nimport pandas as pd\n\nscript = \"\"\"\n DS_A := DS_1 * 10;\n\"\"\"\n\ndata_structures = {\n 'datasets': [\n {'name': 'DS_1',\n 'DataStructure': [\n {'name': 'Id_1',\n 'type':\n 'Integer',\n 'role': 'Identifier',\n 'nullable': False},\n {'name': 'Me_1',\n 'type': 'Number',\n 'role': 'Measure',\n 'nullable': True}\n ]\n }\n ]\n}\n\ndata_df = pd.DataFrame(\n {\"Id_1\": [1, 2, 3],\n \"Me_1\": [10, 20, 30]})\n\ndatapoints = {\"DS_1\": data_df}\n\nrun_result = run(script=script, data_structures=data_structures,\n datapoints=datapoints)\n\nprint(run_result)\n```\n\nreturns:\n\n``` python\n{'DS_A': Dataset(name='DS_A', components={'Id_1': Component(name='Id_1', data_type=<class 'vtlengine.DataTypes.Integer'>, role=<Role.IDENTIFIER: 'Identifier'>, nullable=False), 'Me_1': Component(name='Me_1', data_type=<class 'vtlengine.DataTypes.Number'>, role=<Role.MEASURE: 'Measure'>, nullable=True)}, data= Id_1 Me_1\n0 1 100.0\n1 2 200.0\n2 3 300.0)}\n```\n\nFor more information on usage, please refer to\nthe [API documentation](https://docs.vtlengine.meaningfuldata.eu/api.html).",
"bugtrack_url": null,
"license": "AGPL-3.0",
"summary": "Run and Validate VTL Scripts",
"version": "1.0.3",
"project_urls": {
"Authors": "https://github.com/Meaningful-Data/vtlengine/graphs/contributors",
"Documentation": "https://docs.vtlengine.meaningfuldata.eu",
"IssueTracker": "https://github.com/Meaningful-Data/vtlengine/issues",
"MeaningfulData": "https://www.meaningfuldata.eu/",
"Repository": "https://github.com/Meaningful-Data/vtlengine"
},
"split_keywords": [
"vtl",
" sdmx",
" vtlengine",
" validation and transformation language"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "cfb994f2af00a0c91a8b807cd841dcb4945323316cab3fb6c318f3567902c267",
"md5": "c028bf27b75e3d755339c9ccbdf1e212",
"sha256": "46b5421231d685c8a7c65838a2f075e847c5ef03cf35ed24db4229eb49c2bcf9"
},
"downloads": -1,
"filename": "vtlengine-1.0.3-py3-none-any.whl",
"has_sig": false,
"md5_digest": "c028bf27b75e3d755339c9ccbdf1e212",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": "<4.0,>=3.10",
"size": 247780,
"upload_time": "2024-11-29T10:07:21",
"upload_time_iso_8601": "2024-11-29T10:07:21.371262Z",
"url": "https://files.pythonhosted.org/packages/cf/b9/94f2af00a0c91a8b807cd841dcb4945323316cab3fb6c318f3567902c267/vtlengine-1.0.3-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "998e2ae81d5dff6b59e22ca582ea82a8e87adbb662c035f163ed6279a6f381f1",
"md5": "dcbeb46880b4dd390037a592bc5f8327",
"sha256": "97c9756436f7e8f309f7070cbcaeb47d624dca1a32361f0810eb509ceafda6c4"
},
"downloads": -1,
"filename": "vtlengine-1.0.3.tar.gz",
"has_sig": false,
"md5_digest": "dcbeb46880b4dd390037a592bc5f8327",
"packagetype": "sdist",
"python_version": "source",
"requires_python": "<4.0,>=3.10",
"size": 225233,
"upload_time": "2024-11-29T10:07:22",
"upload_time_iso_8601": "2024-11-29T10:07:22.573845Z",
"url": "https://files.pythonhosted.org/packages/99/8e/2ae81d5dff6b59e22ca582ea82a8e87adbb662c035f163ed6279a6f381f1/vtlengine-1.0.3.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-11-29 10:07:22",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "Meaningful-Data",
"github_project": "vtlengine",
"travis_ci": false,
"coveralls": true,
"github_actions": true,
"requirements": [],
"lcname": "vtlengine"
}
JSON
