# osemosys2iamc
[](https://doi.org/10.5281/zenodo.7473185)
Convert OSeMOSYS results to IAMC format
## Acknowledgements
This work was financially supported by:
- The European Union’s Horizon 2020 research and innovation programme under the grant agreement No 101022622 ([European Climate and Energy Modelling Forum ECEMF](https://doi.org/10.3030/101022622))
- The [IAM COMPACT](https://doi.org/10.3030/101056306) project has received funding from the European Union’s HORIZON EUROPE Research and Innovation Programme under grant agreement No 101056306
## Install from Github repository
pip install git+https://github.com/osemosys/osemosys2iamc@main#egg=osemosys2iamc
## Run the package
$ osemosys2iamc --help
osemosys2iamc <inputs_path> <results_path> <config_path> <output_path>
`inputs_path`: Path to a folder of csv files (OSeMOSYS inputs). File names should correspond to OSeMOSYS parameter names.
`results_path`: Path to a folder of csv files (OSeMOSYS results). File names should correspond to OSeMOSYS variable names.
`config_path`: Path to the configuration file (see below)
`output_path`: Path to the .xlsx file you wish to write out
## The IAMC format
The IAMC format was developed by the [Integrated Assessment Modeling Consortium (IAMC)](https://www.iamconsortium.org/)
and is used in many model comparison projects at the global and national level.
It can be used for integrated-assessment models, energy-systems scenarios
and analysis of specific sectors like transport, industry or buildings.
The format is a tabular structure with the columns *model*, *scenario*, *region*,
*variable*, *unit*, and a time domain. Each project defines "codelists"
to be used across modelling teams for comparison and analysis of results.
The most recent high-profile application of the IAMC format is the [AR6 Scenario Explorer](https://data.ece.iiasa.ac.at/ar6)
hosting the scenario ensemble supporting the quantitative assessment
in the contribution by Working Group III to the IPCC's Sixth Assessment Report (AR6).
Please refer to the Horizon 2020 project [openENTRANCE](https://github.com/openENTRANCE/openentrance#data-format-structure)
for more information about the format and its usage in that project.
## Writing a configuration file
Write a configuration file in YAML format. A simple configuration file with two result variables looks like this:
```yaml
model: OSeMBE v1.0.0
scenario: DIAG-C400-lin-ResidualFossil
region: iso2_x, iso3_x, from_csv, or a name of a country/region [substitute x with start, end, or a positive number]
results:
- iamc_variable: 'Carbon Capture|Biomass'
tech_emi: ['(?=^.{2}(BM))^.{4}(CS)']
emissions: [CO2]
unit: kt CO2/yr
transform: abs
osemosys_param: AnnualTechnologyEmission
- iamc_variable: 'Capital Investment'
capacity: ['^.*$']
unit: MUSD
osemosys_param: CapitalInvestment
```
The first section of the configuration file with the keys `model`, `scenario`, and `region` are used to define the metadata for
the IAMC template.
`model` and `scenario` are user-defined, taking any word or phrase and having no set format. `region` may be defined using 4 methods:
* `iso2_x` - using the ISO 3166-1 alpha-2 country codes located in the names of technologies/fuels/emissions (depending on the file) at the location given by the user.\*
* `iso3_x` - using the ISO 3166-1 alpha-3 country codes located in the names of technologies/fuels/emissions (depending on the file) at the location given by the user.\*
* `from_csv` - using the region as defined in the REGION column of the CSV file
* Any other word or phrase the user defines (other than an invalid iso option) will be used as the region for all variables. eg. Austria, Nepal, Ethiopia, Guyana, etc.
The `x` in the above ISO options is to be replaced by:
* `start`, if the codes are at the beginning of the names
* `end`, if the codes are at the end of the names, or
* a positive number indicating the position of the first letter of the code in the name. eg. iso2_5 will target the 'GH' in 'POWRGHSOL'
The second section, `results`, is where you describe each of the IAMC variables and provide instructions to osemosys2iamc on how
to compute the values.
`iamc_variable` - this should match one of the IAMC variable names
`unit` - provide the units of the OSeMOSYS results
`transform` - only `abs` is currently available. This returns the absolute value of the results
`osemosys_param` - provide the name of the result file from which the script should extract the result data
One or more of the following filter keys. These filter the
results by one or more columns found in OSeMOSYS results.
Following the fitering, the remaining columns except region
and year are discarded and rows are summed.
`tech_emi` - filter the results by TECHNOLOGY and EMISSION columns using the provide regular expression and an `emissions` entry
`emissions` - a list of emissions to filter the results by the EMISSION column
`fuel` - filter by the FUEL column
`capacity` - filter the TECHNOLOGY column
`primary_technology` - filter the TECHNOLOGY column (can be replaced by `capacity` key)
`excluded_prod_tech` - filter the TECHNOLOGY column (can be replaced by `capacity` key)
`el_prod_technology` - filter the TECHNOLOGY column (can be replaced by `capacity` key)
`demand` - filters by the FUEL column (final energy)
The value for each of these keys is a list of regular expressions. These regular expressions are used to filter the rows of data in the chosen column to those that match the
regular expression.
Writing regular expressions can be tricky, but there are [useful tools](https://regexr.com/) to help.
Below we provide some examples:
`^.{2}(WI)` match rows with any two characters followed by `WI`
`(?=^.{2}(HF))^((?!00).)*$` match rows with `HF` in 3rd and 4th position which do not include `00`
`(?=^.{2}(NG))^((?!(00)|(CS)).)*$` match rows with `NG` in 3rd and 4th position that do not include `00` or `CS`
`^.{6}(I0)` match rows which contain any 6 characters followed by `IO` in the 7th and 8th position
Putting this together, the following entry extracts results from the result file `ProductionByTechnologyAnnual.csv`, filters out the rows
by matching values in the TECHNOLOGY column with a list of 6 regular expressions (this is an OR operation)
and assigns the unit `PJ/yr` and adds the aggregated (summing over region and year) total under `Primary Energy` in the IAMC template format.
- iamc_variable: 'Primary Energy'
primary_technology: ['^.{6}(I0)','^.{6}(X0)','^.{2}(HY)','^.{2}(OC)','^.{2}(SO)','^.{2}(WI)']
unit: PJ/yr
osemosys_param: ProductionByTechnologyAnnual
## List of relevant IAMC variables for OSeMOSYS
### Primary Energy
Level | Commodity
---|---
Primary Energy
Primary Energy|Biomass
Primary Energy|Coal
Primary Energy|Fossil
Primary Energy|Gas
Primary Energy|Geothermal
Primary Energy|Hydro
Primary Energy|Non-Biomass Renewables
Primary Energy|Nuclear
Primary Energy|Ocean
Primary Energy|Oil
Primary Energy|Other
Primary Energy|Secondary Energy Trade
Primary Energy|Solar
Primary Energy|Wind
# Priority Variables
Level | Commodity | Fuel
---|---|---
Secondary Energy|Electricity|Coal, w/ and w/o CCS
Secondary Energy|Liquids|Coal
Secondary Energy|Gases|Coal
Secondary Energy|Solids|Coal
Secondary Energy|hydrogen|Coal, w/ and w/o CCS
Level | Commodity | Fuel
---|---|---
Final Energy|Solids|Coal
Final Energy|Industry|Solids|Coal
Final Energy|Residential and Commercial|Solids|Coal
Final Energy|Other Sector|Solids|Coal
Level | Commodity | Sector | Sector | Sector
---|---|---|---|---
Emissions|CO2|Energy and Industrial Processes
Emissions|CO2|Energy|Supply
Emissions|CO2|Energy|Supply|Electricity
Emissions|CO2|Energy|Supply|Other Sector
Emissions|CO2|Energy|Demand
Emissions|CO2|Energy|Demand|Industry
Emissions|CO2|Energy|Demand|Residential and Commercial
Emissions|CO2|Energy|Demand|Transportation
Emissions|CO2|Energy|Demand|Agriculture
Emissions|CO2|Energy|Demand|Other Sector
Emissions|CO2|Industrial Processes
Emissions|CO2|AFOLU
Level | Commodity | Technology | Type
---|---|---|---
Capacity|Electricity|Solar|PV
Capacity|Electricity|Solar|CSP
Capacity|Electricity|Wind
Capacity|Electricity|Wind|Offshore
Capacity|Electricity|Wind|Onshore
Carbon Sequestration|CCS
Carbon Sequestration|CCS|Biomass
Carbon Sequestration|CCS|Fossil
Carbon Sequestration|Direct Air Capture
Emissions|N2O
Emissions|CH4
Raw data
{
"_id": null,
"home_page": "https://github.com/osemosys/osemosys2iamc/",
"name": "osemosys2iamc",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": "",
"keywords": "",
"author": "Will Usher",
"author_email": "wusher@kth.se",
"download_url": "https://files.pythonhosted.org/packages/c7/3a/4b4f50e8590998d69956010979ee08ddf478715a6fce2a25f7e3c0e1274c/osemosys2iamc-0.2.1.tar.gz",
"platform": "any",
"description": "# osemosys2iamc\n\n[](https://doi.org/10.5281/zenodo.7473185)\n\nConvert OSeMOSYS results to IAMC format\n\n## Acknowledgements\n\nThis work was financially supported by:\n\n- The European Union\u2019s Horizon 2020 research and innovation programme under the grant agreement No 101022622 ([European Climate and Energy Modelling Forum ECEMF](https://doi.org/10.3030/101022622))\n- The [IAM COMPACT](https://doi.org/10.3030/101056306) project has received funding from the European Union\u2019s HORIZON EUROPE Research and Innovation Programme under grant agreement No 101056306\n\n## Install from Github repository\n\n pip install git+https://github.com/osemosys/osemosys2iamc@main#egg=osemosys2iamc\n\n## Run the package\n\n $ osemosys2iamc --help\n osemosys2iamc <inputs_path> <results_path> <config_path> <output_path>\n\n`inputs_path`: Path to a folder of csv files (OSeMOSYS inputs). File names should correspond to OSeMOSYS parameter names.\n`results_path`: Path to a folder of csv files (OSeMOSYS results). File names should correspond to OSeMOSYS variable names.\n`config_path`: Path to the configuration file (see below)\n`output_path`: Path to the .xlsx file you wish to write out\n\n## The IAMC format\n\nThe IAMC format was developed by the [Integrated Assessment Modeling Consortium (IAMC)](https://www.iamconsortium.org/)\nand is used in many model comparison projects at the global and national level.\nIt can be used for integrated-assessment models, energy-systems scenarios\nand analysis of specific sectors like transport, industry or buildings.\n\nThe format is a tabular structure with the columns *model*, *scenario*, *region*,\n*variable*, *unit*, and a time domain. Each project defines \"codelists\"\nto be used across modelling teams for comparison and analysis of results.\n\nThe most recent high-profile application of the IAMC format is the [AR6 Scenario Explorer](https://data.ece.iiasa.ac.at/ar6)\nhosting the scenario ensemble supporting the quantitative assessment\nin the contribution by Working Group III to the IPCC's Sixth Assessment Report (AR6).\n\nPlease refer to the Horizon 2020 project [openENTRANCE](https://github.com/openENTRANCE/openentrance#data-format-structure)\nfor more information about the format and its usage in that project.\n\n## Writing a configuration file\n\nWrite a configuration file in YAML format. A simple configuration file with two result variables looks like this:\n\n```yaml\nmodel: OSeMBE v1.0.0\nscenario: DIAG-C400-lin-ResidualFossil\nregion: iso2_x, iso3_x, from_csv, or a name of a country/region [substitute x with start, end, or a positive number]\nresults:\n- iamc_variable: 'Carbon Capture|Biomass'\n tech_emi: ['(?=^.{2}(BM))^.{4}(CS)']\n emissions: [CO2]\n unit: kt CO2/yr\n transform: abs\n osemosys_param: AnnualTechnologyEmission\n- iamc_variable: 'Capital Investment'\n capacity: ['^.*$']\n unit: MUSD\n osemosys_param: CapitalInvestment\n```\n\nThe first section of the configuration file with the keys `model`, `scenario`, and `region` are used to define the metadata for\nthe IAMC template.\n\n`model` and `scenario` are user-defined, taking any word or phrase and having no set format. `region` may be defined using 4 methods:\n\n* `iso2_x` - using the ISO 3166-1 alpha-2 country codes located in the names of technologies/fuels/emissions (depending on the file) at the location given by the user.\\*\n* `iso3_x` - using the ISO 3166-1 alpha-3 country codes located in the names of technologies/fuels/emissions (depending on the file) at the location given by the user.\\*\n* `from_csv` - using the region as defined in the REGION column of the CSV file\n* Any other word or phrase the user defines (other than an invalid iso option) will be used as the region for all variables. eg. Austria, Nepal, Ethiopia, Guyana, etc.\n\nThe `x` in the above ISO options is to be replaced by:\n* `start`, if the codes are at the beginning of the names\n* `end`, if the codes are at the end of the names, or\n* a positive number indicating the position of the first letter of the code in the name. eg. iso2_5 will target the 'GH' in 'POWRGHSOL'\n\nThe second section, `results`, is where you describe each of the IAMC variables and provide instructions to osemosys2iamc on how\nto compute the values.\n\n`iamc_variable` - this should match one of the IAMC variable names\n`unit` - provide the units of the OSeMOSYS results\n`transform` - only `abs` is currently available. This returns the absolute value of the results\n`osemosys_param` - provide the name of the result file from which the script should extract the result data\n\nOne or more of the following filter keys. These filter the\nresults by one or more columns found in OSeMOSYS results.\nFollowing the fitering, the remaining columns except region\n and year are discarded and rows are summed.\n\n`tech_emi` - filter the results by TECHNOLOGY and EMISSION columns using the provide regular expression and an `emissions` entry\n`emissions` - a list of emissions to filter the results by the EMISSION column\n`fuel` - filter by the FUEL column\n`capacity` - filter the TECHNOLOGY column\n`primary_technology` - filter the TECHNOLOGY column (can be replaced by `capacity` key)\n`excluded_prod_tech` - filter the TECHNOLOGY column (can be replaced by `capacity` key)\n`el_prod_technology` - filter the TECHNOLOGY column (can be replaced by `capacity` key)\n`demand` - filters by the FUEL column (final energy)\n\nThe value for each of these keys is a list of regular expressions. These regular expressions are used to filter the rows of data in the chosen column to those that match the\nregular expression.\n\nWriting regular expressions can be tricky, but there are [useful tools](https://regexr.com/) to help.\nBelow we provide some examples:\n\n`^.{2}(WI)` match rows with any two characters followed by `WI`\n\n`(?=^.{2}(HF))^((?!00).)*$` match rows with `HF` in 3rd and 4th position which do not include `00`\n\n`(?=^.{2}(NG))^((?!(00)|(CS)).)*$` match rows with `NG` in 3rd and 4th position that do not include `00` or `CS`\n\n`^.{6}(I0)` match rows which contain any 6 characters followed by `IO` in the 7th and 8th position\n\nPutting this together, the following entry extracts results from the result file `ProductionByTechnologyAnnual.csv`, filters out the rows\nby matching values in the TECHNOLOGY column with a list of 6 regular expressions (this is an OR operation)\nand assigns the unit `PJ/yr` and adds the aggregated (summing over region and year) total under `Primary Energy` in the IAMC template format.\n\n - iamc_variable: 'Primary Energy'\n primary_technology: ['^.{6}(I0)','^.{6}(X0)','^.{2}(HY)','^.{2}(OC)','^.{2}(SO)','^.{2}(WI)']\n unit: PJ/yr\n osemosys_param: ProductionByTechnologyAnnual\n\n## List of relevant IAMC variables for OSeMOSYS\n\n### Primary Energy\n\nLevel | Commodity\n---|---\nPrimary Energy\nPrimary Energy|Biomass\nPrimary Energy|Coal\nPrimary Energy|Fossil\nPrimary Energy|Gas\nPrimary Energy|Geothermal\nPrimary Energy|Hydro\nPrimary Energy|Non-Biomass Renewables\nPrimary Energy|Nuclear\nPrimary Energy|Ocean\nPrimary Energy|Oil\nPrimary Energy|Other\nPrimary Energy|Secondary Energy Trade\nPrimary Energy|Solar\nPrimary Energy|Wind\n\n# Priority Variables\n\nLevel | Commodity | Fuel\n---|---|---\nSecondary Energy|Electricity|Coal, w/ and w/o CCS\nSecondary Energy|Liquids|Coal\nSecondary Energy|Gases|Coal\nSecondary Energy|Solids|Coal\nSecondary Energy|hydrogen|Coal, w/ and w/o CCS\n\nLevel | Commodity | Fuel\n---|---|---\nFinal Energy|Solids|Coal\nFinal Energy|Industry|Solids|Coal\nFinal Energy|Residential and Commercial|Solids|Coal\nFinal Energy|Other Sector|Solids|Coal\n\nLevel | Commodity | Sector | Sector | Sector\n---|---|---|---|---\nEmissions|CO2|Energy and Industrial Processes\nEmissions|CO2|Energy|Supply\nEmissions|CO2|Energy|Supply|Electricity\nEmissions|CO2|Energy|Supply|Other Sector\nEmissions|CO2|Energy|Demand\nEmissions|CO2|Energy|Demand|Industry\nEmissions|CO2|Energy|Demand|Residential and Commercial\nEmissions|CO2|Energy|Demand|Transportation\nEmissions|CO2|Energy|Demand|Agriculture\nEmissions|CO2|Energy|Demand|Other Sector\nEmissions|CO2|Industrial Processes\nEmissions|CO2|AFOLU\n\nLevel | Commodity | Technology | Type\n---|---|---|---\nCapacity|Electricity|Solar|PV\nCapacity|Electricity|Solar|CSP\nCapacity|Electricity|Wind\nCapacity|Electricity|Wind|Offshore\nCapacity|Electricity|Wind|Onshore\nCarbon Sequestration|CCS\nCarbon Sequestration|CCS|Biomass\nCarbon Sequestration|CCS|Fossil\nCarbon Sequestration|Direct Air Capture\nEmissions|N2O\nEmissions|CH4\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "A utility package to convert OSeMOSYS results into IAMC format",
"version": "0.2.1",
"project_urls": {
"Homepage": "https://github.com/osemosys/osemosys2iamc/",
"Source": "https://github.com/osemosys/osemosys2iamc/",
"Tracker": "https://github.com/osemosys/osemosys2iamc/issues"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "91123b1bf601a6e88b62da8c0591356e66103e5d89e59f4edc4c0d74ea132d62",
"md5": "c16453ae2e94f20e1a8ef219d43c281a",
"sha256": "f723616d3bd8fd65390b6656ad46931bf9a028d12b48c9ca247f902dde9bae61"
},
"downloads": -1,
"filename": "osemosys2iamc-0.2.1-py3-none-any.whl",
"has_sig": false,
"md5_digest": "c16453ae2e94f20e1a8ef219d43c281a",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8",
"size": 11314,
"upload_time": "2023-07-12T08:32:46",
"upload_time_iso_8601": "2023-07-12T08:32:46.099080Z",
"url": "https://files.pythonhosted.org/packages/91/12/3b1bf601a6e88b62da8c0591356e66103e5d89e59f4edc4c0d74ea132d62/osemosys2iamc-0.2.1-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "c73a4b4f50e8590998d69956010979ee08ddf478715a6fce2a25f7e3c0e1274c",
"md5": "ed05710c8bcd8afc5818fdb97d1fb2ee",
"sha256": "1108b8a6fd1519916cfde6c5a898d698aaa3b7f1ed76e656c59373dff65bac23"
},
"downloads": -1,
"filename": "osemosys2iamc-0.2.1.tar.gz",
"has_sig": false,
"md5_digest": "ed05710c8bcd8afc5818fdb97d1fb2ee",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8",
"size": 29117,
"upload_time": "2023-07-12T08:32:47",
"upload_time_iso_8601": "2023-07-12T08:32:47.386494Z",
"url": "https://files.pythonhosted.org/packages/c7/3a/4b4f50e8590998d69956010979ee08ddf478715a6fce2a25f7e3c0e1274c/osemosys2iamc-0.2.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-07-12 08:32:47",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "osemosys",
"github_project": "osemosys2iamc",
"travis_ci": false,
"coveralls": true,
"github_actions": true,
"requirements": [
{
"name": "matplotlib",
"specs": []
},
{
"name": "pandas",
"specs": []
},
{
"name": "pyam-iamc",
"specs": [
[
">=",
"1.0"
]
]
},
{
"name": "iso3166",
"specs": []
}
],
"tox": true,
"lcname": "osemosys2iamc"
}
JSON
