# ODS Tools
## Overview
ODS Tools is a Python package designed to support users of the Oasis Loss Modelling Framework (Oasis LMF).
This package includes a range of tools for working with Oasis data files, including loading, conversion and validation.
The package is based on a release of:
in accordance with the [ODS_OpenExposureData](https://github.com/OasisLMF/ODS_OpenExposureData/).
## Installation
ODS_Tools can be installed via pip by running the following command:
```
pip install ods-tools
```
## command line interface
ODS tools provide command line interface to quickly convert oed files:
example :
```
ods_tools convert --location path_to_location_file --path output folder
```
see `ods_tools convert --help` for more option
## Usage
### loading exposure data
in order to load oed file we use the concept of source.
A source will define how to retrieve the oed data. For the moment we only support files but other type of
source such as DataBase could be envisaged.
The loading itself support several format such as parquet, csv and all pandas read_csv supported compression
The path to the file can be absolute relative or even an url
config example:
```python
config = {
'location': 'SourceLocOEDPiWind.csv', # csv file
'account': 'SourceAccOEDPiWind.parquet', # parquet file
'ri_info': {
'cur_version_name': 'orig', # passing args to the reader function
'sources': {
'orig': {
'source_type': 'filepath',
'filepath': 'SourceReinsInfoOEDPiWind.csv',
'read_param': {
'usecols':[
'ReinsNumber', 'ReinsLayerNumber', 'ReinsName', 'ReinsPeril',
'ReinsInceptionDate', 'ReinsExpiryDate', 'CededPercent', 'RiskLimit',
'RiskAttachment', 'OccLimit', 'OccAttachment', 'PlacedPercent',
'ReinsCurrency', 'InuringPriority', 'ReinsType', 'RiskLevel', 'OEDVersion'
]
}
}
}
},
'ri_scope': 'https://raw.githubusercontent.com/OasisLMF/OasisPiWind/main/tests/inputs/SourceReinsScopeOEDPiWind.csv', # url
}
```
### Access Oed File as DataFrame
Once the config is done you can create your OedExposure Object
and access the Dataframe representation of the different sources.
Data Type in the DataFrame will correspond to the type
```python
import ods_tools
oed_exposure = ods_tools.oed.OedExposure(**config)
location = oed_exposure.location.dataframe
account = oed_exposure.account.dataframe
```
### Saving Change to the oed DataFrame
You can modify the DataFrame and save it as a new version
```python
oed_exposure.location.save(version_name='modified version',
source='path_to_save_the_file')
```
you may also save the exposure itself this will save the current dataframe to the specified directory_path.
if you specify version_name, oed files will be saved as f'{version_name}_{OED_NAME}' + compression (ex: version_2_location.csv)
if the version_name is an empty string, oed files will be saved as just f'{OED_NAME}' + compression (ex: location.parquet)
if version_name is None, oed files will take the same name as the current source if it is a filepath or f'{OED_NAME}' + compression otherwise
(ex: SourceLocOEDPiWind.csv)
compression let you specify the file extension (csv, parquet, zip, gzip, bz2, zstd)
if save_config is True the exposure config will also be saved in the directory
```python
oed_exposure.save(directory_path, version_name, compression, save_config)
```
### OED Validation
Validity of oed files can be checked at loading time with the argument check_oed
```python
oed_exposure = ods_tools.oed.OedExposure(check_oed=True, validation_config=validation_config, **config)
```
validation_config is a list of all check that you want to perform, if one oed source fail a check depending on validation_config
4 different action can be performed 'raise', 'log', 'ignore', 'return'.
- 'raise' will raise an OdsException
- 'log' will log the issue with a info level
- 'ignore' will ignore the issue
- 'return' will return the check issue in a list in order for the developer to perform its own treatment.
In that case the check method need to be called instead of relying on the constructor
```python
oed_exposure = ods_tools.oed.OedExposure(check_oed=False**config)
invalid_data = oed_exposure.check(custom_validation_config)
```
the curent default validation under ods_tools.oed.common DEFAULT_VALIDATION_CONFIG contains
```python
VALIDATOR_ON_ERROR_ACTION = {'raise', 'log', 'ignore', 'return'}
DEFAULT_VALIDATION_CONFIG = [
{'name': 'required_fields', 'on_error': 'raise'},
{'name': 'unknown_column', 'on_error': 'log'},
{'name': 'valid_values', 'on_error': 'raise'},
{'name': 'perils', 'on_error': 'raise'},
{'name': 'occupancy_code', 'on_error': 'raise'},
{'name': 'construction_code', 'on_error': 'raise'},
{'name': 'country_and_area_code', 'on_error': 'raise'},
]
```
An OdsException is raised with a message indicating which file is invalid and why.
### Currency Conversion Support
Exposure Data handles the conversion of relevant columns of the oed files to another currency
to do so you will need to provide information on the currency conversion method in the config or after loading
#### DictBasedCurrencyRates
DictBasedCurrencyRates is a solution where all the rate are provided via files and stored internally as a dictionary.
We support csv file (compressed or not) or a parquet file where they will be read as DataFrame.
exemple of currency_conversion_json ("source_type": "parquet" if parquet file is used):
```json
{
"currency_conversion_type": "DictBasedCurrencyRates",
"source_type": "csv",
"file_path": "tests/inputs/roe.csv"
}
```
The expected format is (roe being a float in parquet format):
```
cur_from,cur_to,roe
USD,GBP,0.85
USD,EUR,0.95
GBP,EUR,1.12
```
Rate can also be passed directly in currency_conversion_json
ex:
```json
{
"currency_conversion_type": "DictBasedCurrencyRates",
"source_type": "dict",
"currency_rates": [["USD", "GBP", 0.85],
["USD", "EUR", 0.95],
["GBP", "EUR", 1.12]
]
}
```
When looking for a key pair, DictBasedCurrencyRates check 1st for the key pair (cur1, cur2) then for (cur2, cur1).
So if a Currency pairs is only specified one way (ex: GBP=>EUR) then it is automatically assume that
roe EUR=>GBP = 1/(roe GPB=>EUR)
if a currency pair is missing ValueError(f"currency pair {(cur_from, cur_to)} is missing") is thrown
#### FxCurrencyRates
OasisLMF let you use the external package [forex-python](https://forex-python.readthedocs.io/en/latest/usage.html)
to perform the conversion. A date may be specified in ISO 8601 format (YYYY-MM-DD)
currency_conversion_json:
```json
{
"currency_conversion_type": "FxCurrencyRates",
"datetime": "2018-10-10"
}
```
those config can be added as a json file path of directly into the oed_config dict
```python
config_with_currency_rate = {
'location': 'SourceLocOEDPiWind.csv', # csv file
'currency_conversion': {
"currency_conversion_type": "DictBasedCurrencyRates",
"source_type": "dict",
"currency_rates": {
('USD', 'GBP'): 0.85,
('USD', 'EUR'): 0.952,
('GBP', 'EUR'): 1.12}
},
'reporting_currency': 'USD',
}
```
if reporting_currency is specified in the config, the oed file will be converted on load
It can also be set once the OedExposure object has been created
```python
import ods_tools
oed_exposure = ods_tools.oed.OedExposure(**config)
oed_exposure.currency_conversion = ods_tools.oed.forex.create_currency_rates(
{
"currency_conversion_type": "DictBasedCurrencyRates",
"source_type": "dict",
"currency_rates": {
('USD', 'GBP'): 0.85,
('USD', 'EUR'): 0.952,
('GBP', 'EUR'): 1.12}
}
)
oed_exposure.reporting_currency = 'EUR' # this line will trigger currency conversion
```
Raw data
{
"_id": null,
"home_page": "https://github.com/OasisLMF/OpenDataStandards",
"name": "ods-tools",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.7",
"maintainer_email": "",
"keywords": "",
"author": "Oasis LMF",
"author_email": "support@oasislmf.org",
"download_url": "https://files.pythonhosted.org/packages/5f/21/56b78badcc8616ac2fc8c8ebbd0cf511124883b8dbc08c6ae7c5f53cd4d2/ods_tools-3.2.1.tar.gz",
"platform": null,
"description": "# ODS Tools\n\n\n\n## Overview\n\nODS Tools is a Python package designed to support users of the Oasis Loss Modelling Framework (Oasis LMF).\nThis package includes a range of tools for working with Oasis data files, including loading, conversion and validation.\n\nThe package is based on a release of:\nin accordance with the [ODS_OpenExposureData](https://github.com/OasisLMF/ODS_OpenExposureData/).\n\n\n## Installation\n\nODS_Tools can be installed via pip by running the following command:\n\n```\npip install ods-tools\n```\n\n\n## command line interface\n\nODS tools provide command line interface to quickly convert oed files:\n\nexample :\n\n```\nods_tools convert --location path_to_location_file --path output folder\n```\n\nsee `ods_tools convert --help` for more option\n\n## Usage\n\n### loading exposure data\n\nin order to load oed file we use the concept of source.\nA source will define how to retrieve the oed data. For the moment we only support files but other type of\nsource such as DataBase could be envisaged.\nThe loading itself support several format such as parquet, csv and all pandas read_csv supported compression\nThe path to the file can be absolute relative or even an url\n\nconfig example:\n\n```python\nconfig = {\n 'location': 'SourceLocOEDPiWind.csv', # csv file\n 'account': 'SourceAccOEDPiWind.parquet', # parquet file\n 'ri_info': {\n 'cur_version_name': 'orig', # passing args to the reader function\n 'sources': {\n 'orig': {\n 'source_type': 'filepath',\n 'filepath': 'SourceReinsInfoOEDPiWind.csv',\n 'read_param': {\n 'usecols':[\n 'ReinsNumber', 'ReinsLayerNumber', 'ReinsName', 'ReinsPeril',\n 'ReinsInceptionDate', 'ReinsExpiryDate', 'CededPercent', 'RiskLimit',\n 'RiskAttachment', 'OccLimit', 'OccAttachment', 'PlacedPercent',\n 'ReinsCurrency', 'InuringPriority', 'ReinsType', 'RiskLevel', 'OEDVersion'\n ]\n }\n }\n }\n },\n 'ri_scope': 'https://raw.githubusercontent.com/OasisLMF/OasisPiWind/main/tests/inputs/SourceReinsScopeOEDPiWind.csv', # url\n}\n```\n\n### Access Oed File as DataFrame\n\nOnce the config is done you can create your OedExposure Object\nand access the Dataframe representation of the different sources.\nData Type in the DataFrame will correspond to the type\n\n```python\nimport ods_tools\noed_exposure = ods_tools.oed.OedExposure(**config)\nlocation = oed_exposure.location.dataframe\naccount = oed_exposure.account.dataframe\n```\n\n### Saving Change to the oed DataFrame\n\nYou can modify the DataFrame and save it as a new version\n\n```python\noed_exposure.location.save(version_name='modified version',\n source='path_to_save_the_file')\n```\n\nyou may also save the exposure itself this will save the current dataframe to the specified directory_path.\nif you specify version_name, oed files will be saved as f'{version_name}_{OED_NAME}' + compression (ex: version_2_location.csv)\nif the version_name is an empty string, oed files will be saved as just f'{OED_NAME}' + compression (ex: location.parquet)\nif version_name is None, oed files will take the same name as the current source if it is a filepath or f'{OED_NAME}' + compression otherwise\n(ex: SourceLocOEDPiWind.csv)\n\ncompression let you specify the file extension (csv, parquet, zip, gzip, bz2, zstd)\n\nif save_config is True the exposure config will also be saved in the directory\n```python\noed_exposure.save(directory_path, version_name, compression, save_config)\n```\n\n### OED Validation\n\nValidity of oed files can be checked at loading time with the argument check_oed\n\n```python\noed_exposure = ods_tools.oed.OedExposure(check_oed=True, validation_config=validation_config, **config)\n```\n\nvalidation_config is a list of all check that you want to perform, if one oed source fail a check depending on validation_config\n4 different action can be performed 'raise', 'log', 'ignore', 'return'.\n - 'raise' will raise an OdsException\n - 'log' will log the issue with a info level\n - 'ignore' will ignore the issue\n - 'return' will return the check issue in a list in order for the developer to perform its own treatment.\nIn that case the check method need to be called instead of relying on the constructor\n```python\noed_exposure = ods_tools.oed.OedExposure(check_oed=False**config)\ninvalid_data = oed_exposure.check(custom_validation_config)\n```\n\nthe curent default validation under ods_tools.oed.common DEFAULT_VALIDATION_CONFIG contains\n```python\nVALIDATOR_ON_ERROR_ACTION = {'raise', 'log', 'ignore', 'return'}\nDEFAULT_VALIDATION_CONFIG = [\n {'name': 'required_fields', 'on_error': 'raise'},\n {'name': 'unknown_column', 'on_error': 'log'},\n {'name': 'valid_values', 'on_error': 'raise'},\n {'name': 'perils', 'on_error': 'raise'},\n {'name': 'occupancy_code', 'on_error': 'raise'},\n {'name': 'construction_code', 'on_error': 'raise'},\n {'name': 'country_and_area_code', 'on_error': 'raise'},\n]\n```\n\nAn OdsException is raised with a message indicating which file is invalid and why.\n\n### Currency Conversion Support\n\nExposure Data handles the conversion of relevant columns of the oed files to another currency\nto do so you will need to provide information on the currency conversion method in the config or after loading\n\n#### DictBasedCurrencyRates\n\nDictBasedCurrencyRates is a solution where all the rate are provided via files and stored internally as a dictionary.\n\nWe support csv file (compressed or not) or a parquet file where they will be read as DataFrame.\nexemple of currency_conversion_json (\"source_type\": \"parquet\" if parquet file is used):\n\n```json\n{\n \"currency_conversion_type\": \"DictBasedCurrencyRates\",\n \"source_type\": \"csv\",\n \"file_path\": \"tests/inputs/roe.csv\"\n}\n```\n\nThe expected format is (roe being a float in parquet format):\n\n```\ncur_from,cur_to,roe\nUSD,GBP,0.85\nUSD,EUR,0.95\nGBP,EUR,1.12\n```\n\nRate can also be passed directly in currency_conversion_json\nex:\n\n```json\n{\n \"currency_conversion_type\": \"DictBasedCurrencyRates\",\n \"source_type\": \"dict\",\n \"currency_rates\": [[\"USD\", \"GBP\", 0.85],\n [\"USD\", \"EUR\", 0.95],\n [\"GBP\", \"EUR\", 1.12]\n ]\n}\n```\n\nWhen looking for a key pair, DictBasedCurrencyRates check 1st for the key pair (cur1, cur2) then for (cur2, cur1).\nSo if a Currency pairs is only specified one way (ex: GBP=>EUR) then it is automatically assume that\nroe EUR=>GBP = 1/(roe GPB=>EUR)\n\nif a currency pair is missing ValueError(f\"currency pair {(cur_from, cur_to)} is missing\") is thrown\n\n#### FxCurrencyRates\n\nOasisLMF let you use the external package [forex-python](https://forex-python.readthedocs.io/en/latest/usage.html)\nto perform the conversion. A date may be specified in ISO 8601 format (YYYY-MM-DD)\ncurrency_conversion_json:\n\n```json\n{\n \"currency_conversion_type\": \"FxCurrencyRates\",\n \"datetime\": \"2018-10-10\"\n}\n```\n\nthose config can be added as a json file path of directly into the oed_config dict\n\n```python\nconfig_with_currency_rate = {\n 'location': 'SourceLocOEDPiWind.csv', # csv file\n 'currency_conversion': {\n \"currency_conversion_type\": \"DictBasedCurrencyRates\",\n \"source_type\": \"dict\",\n \"currency_rates\": {\n ('USD', 'GBP'): 0.85,\n ('USD', 'EUR'): 0.952,\n ('GBP', 'EUR'): 1.12}\n },\n 'reporting_currency': 'USD',\n }\n```\n\nif reporting_currency is specified in the config, the oed file will be converted on load\nIt can also be set once the OedExposure object has been created\n\n```python\nimport ods_tools\noed_exposure = ods_tools.oed.OedExposure(**config)\noed_exposure.currency_conversion = ods_tools.oed.forex.create_currency_rates(\n {\n \"currency_conversion_type\": \"DictBasedCurrencyRates\",\n \"source_type\": \"dict\",\n \"currency_rates\": {\n ('USD', 'GBP'): 0.85,\n ('USD', 'EUR'): 0.952,\n ('GBP', 'EUR'): 1.12}\n }\n)\noed_exposure.reporting_currency = 'EUR' # this line will trigger currency conversion\n```\n\n\n\n\n",
"bugtrack_url": null,
"license": "",
"summary": "Tools to manage ODS files",
"version": "3.2.1",
"project_urls": {
"Homepage": "https://github.com/OasisLMF/OpenDataStandards"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "637a550b420b0aeb80712d068f9da13f4cb00fc945722c949406bcf5b0e8061a",
"md5": "cadaaff50a5d298c4a82e61ff1c25a19",
"sha256": "ee150ce21dd71aded12c115843ce343137664c49987dd18846223c9bd5e8bfc7"
},
"downloads": -1,
"filename": "ods_tools-3.2.1-py3-none-any.whl",
"has_sig": false,
"md5_digest": "cadaaff50a5d298c4a82e61ff1c25a19",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.7",
"size": 109329,
"upload_time": "2024-02-29T13:54:39",
"upload_time_iso_8601": "2024-02-29T13:54:39.135267Z",
"url": "https://files.pythonhosted.org/packages/63/7a/550b420b0aeb80712d068f9da13f4cb00fc945722c949406bcf5b0e8061a/ods_tools-3.2.1-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "5f2156b78badcc8616ac2fc8c8ebbd0cf511124883b8dbc08c6ae7c5f53cd4d2",
"md5": "b46b714f42784e901b84b034940836bf",
"sha256": "6afff34d6ebd52295a5acb9c9595305508dd7c4ad643e3dd9ca3c54e1cf8cd79"
},
"downloads": -1,
"filename": "ods_tools-3.2.1.tar.gz",
"has_sig": false,
"md5_digest": "b46b714f42784e901b84b034940836bf",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.7",
"size": 29481,
"upload_time": "2024-02-29T13:54:36",
"upload_time_iso_8601": "2024-02-29T13:54:36.799597Z",
"url": "https://files.pythonhosted.org/packages/5f/21/56b78badcc8616ac2fc8c8ebbd0cf511124883b8dbc08c6ae7c5f53cd4d2/ods_tools-3.2.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-02-29 13:54:36",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "OasisLMF",
"github_project": "OpenDataStandards",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "ods-tools"
}
JSON
