<!-- SPDX-FileCopyrightText: 2024 German Aerospace Center <fame@dlr.de>
SPDX-License-Identifier: Apache-2.0 -->
[](https://badge.fury.io/py/fameio)
[](https://doi.org/10.21105/joss.04958)
[](https://doi.org/10.5281/zenodo.4314337)
[](https://badge.fury.io/py/fameio)
[](https://gitlab.com/fame-framework/fame-io/commits/main)
[](https://gitlab.com/fame-framework/fame-io/-/commits/main)
[](https://github.com/psf/black)
[](https://api.reuse.software/info/gitlab.com/fame-framework/fame-io)
[](https://common-changelog.org)
# FAME-Io
*Tools for input preparation and output digestion of FAME models*
FAME-Io compiles input for FAME models in protobuf format and extracts model outputs to human-readable files.
Please visit the [FAME-Wiki](https://gitlab.com/fame-framework/wiki/-/wikis/home) to get an explanation of FAME and its components.
# Installation
We recommend installing `fameio` using PyPI:
pip install fameio
You may also use `pipx`. For detailed information please refer to the
official `pipx` [documentation](https://github.com/pypa/pipx).
pipx install fameio
`fameio` is currently developed and tested for Python 3.8 or higher.
See the `pyproject.toml` for a complete listing of dependencies.
# Usage
FAME-Io currently offers two main scripts `makeFameRunConfig` and `convertFameResults`.
Both are automatically installed with the package.
The first one creates a protobuf file for FAME applications using YAML definition files and CSV files.
The latter one reads output files from FAME applications in protobuf format and converts them to CSV files.
You may use the [example data](https://gitlab.com/dlr-ve/esy/amiris/examples) provided for
the [AMIRIS](https://gitlab.com/dlr-ve/esy/amiris/amiris) model which can be used to simulate electricity markets
in [Germany](https://gitlab.com/dlr-ve/esy/amiris/examples/-/tree/main/Germany2019), [Austria](https://gitlab.com/dlr-ve/esy/amiris/examples/-/tree/main/Austria2019),
and a simple [proof-of-concept model](https://gitlab.com/dlr-ve/esy/amiris/examples/-/tree/main/Simple).
## Make a FAME run configuration
Digests configuration files in YAML format, combines them with CSV data files and creates a single input file for FAME
applications in protobuf format.
Call structure:
makeFameRunConfig -f <path/to/scenario.yaml>
You may also specify any of the following arguments:
| Command | Action |
|------------------------|------------------------------------------------------------------------------------------------------------------------------------------|
| `-l` or `--log` | Sets the logging level. Default is `info`. Options are `debug`, `info`, `warning`, `warn`, `error`, `critical`. |
| `-lf` or `--logfile` | Sets the logging file. Default is `None`. If `None` is provided, all logs get only printed to the console. |
| `-o` or `--output` | Sets the path of the compiled protobuf output file. Default is `config.pb`. |
| `-enc` or `--encoding` | Sets the encoding of all yaml files to the given one (e.g. 'utf8' or 'cp1252'. Default is `None`, i.e. your operating system's standard. |
This could look as follows:
makeFameRunConfig -f <path/to/scenario.yaml> -l debug -lf <path/to/scenario.log> -o <path/to/config.pb>
You may also call the configuration builder from any Python script with
```python
from fameio.scripts.make_config import Options, run as make_config
make_config({Options.FILE: "path/to/scenario.yaml", })
```
Similar to the console call you may also specify custom run config arguments and add it in a dictionary to the function
call.
```python
from fameio.scripts.make_config import Options, run as make_config
run_config = {Options.FILE: "path/to/scenario.yaml",
Options.LOG_LEVEL: "info",
Options.OUTPUT: "output.pb",
Options.LOG_FILE: "scenario.log",
}
make_config(run_config)
```
You can also use the associated argument parser, to extract the run_config dynamically from a string:
```python
from fameio.scripts.make_config import Options, run as make_config
from fameio.cli.make_config import handle_args
my_defaults = {Options.FILE: "path/to/scenario.yaml",
Options.LOG_LEVEL: "info",
Options.OUTPUT: "output.pb",
Options.LOG_FILE: "scenario.log",
}
my_arg_string = ['-f', 'my/other/scenario.yaml', '-l', 'error']
run_config = handle_args(my_arg_string, my_defaults)
make_config(run_config)
```
### Scenario YAML
The "scenario.yaml" file contains all configuration options for a FAME-based simulation.
It consists of the sections `Schema`, `GeneralProperties`, `Agents` and `Contracts`, and the optional
section `StringSets`.
All of them are described below.
#### Schema
The Schema describes a model's components such as its types of agents, their inputs, what data they exchange, etc.
It is also used to validate the model inputs provided in the `scenario.yaml`.
Since the Schema is valid until the model itself is changed, it is recommended to defined it in a separate file and
include the file here.
Currently, the schema specifies:
* which type of Agents can be created
* what type of input attributes an Agent uses
* what type of Products an Agent can send in Contracts, and
* the names of the Java packages for the classes corresponding to Agents, DataItems and Portables.
The Schema consists of the sections `JavaPackages` and `AgentTypes`.
##### JavaPackages
This section defines the name of the Java packages in which the model code is located.
A similar data set was formerly specified in the `fameSetup.yaml`, but is now specified in the schema.
Each of the three sections `Agents`, `DataItems`, and `Portables` contain a list of fully qualified java package names
of your model's classes.
Package names can occur in multiple lists and may overlap.
It is not necessary (but possible) to specify the nearest enclosing package for each Agent, DataItem or Portable.
Specifying any super-package will also work.
Also, package names occur on multiple lists for Agent, DataItem or Portable.
For example, for a project with all its
* Agent-derived java classes located in packages below the package named "agents",
* DataItem implementation classes in a subpackage named "msg",
* Portable implementation classes in a subpackages named "portableItems" and "otherPortables",
the corresponding section in the schema would look like this:
```yaml
JavaPackages:
Agents:
- "agents"
DataItems:
- "msg"
Portables:
- "portableItems"
- "otherPortables"
```
One can leave out the `DataItems` specifications, but `Agents` and `Portables` are required and must not be empty.
##### AgentTypes
Here, each type of agent that can be created in your FAME-based application is listed, its attributes and its available
Products for Contracts.
The structure of this section
```yaml
AgentTypes:
MyAgentType:
Attributes:
MyAttribute:
...
MyOtherAttribute:
...
Products: [ 'Product1', 'Product2', 'Product3' ]
Outputs: [ 'Column1', 'Column2', 'Column3' ]
Metadata:
Some: "Dict with Metadata that you would like to add"
MyOtherAgentWithoutProductsOrAttributes:
```
* `MyAgentType` Java's simple class name of the Agent type
* `Attributes` indicates that beginning of the attribute definition section for this Agent type
* `MyAttribute` Name of an attribute as specified in the corresponding Java source code of this Agent type (annotated
with "@Input")
* `MyOtherAttribute` Name of another attribute derived from Java source code
* `Products` list or dictionary of Products that this Agent can send in Contracts; derived from Java source code of this
Agent type (annotated with "@Product")
* `Outputs` list or dictionary of Output columns that this Agent can write to; derived from Java source code of this
Agent type (annotated with "@Output")
* `Metadata` dictionary with any content that is assigned to this Agent type as additional information
* `MyOtherAgentWithoutProductsOrAttributes` an Agent type that requires neither Attributes nor Products
Attributes, Products, Outputs and Metadata are optional - there may be useful Agents that require none of them.
Products and Outputs can both be lists of Strings, or dictionaries with additional Metadata.
For example, you could write the above in the following way:
```yaml
Products:
Product1:
Metadata:
Any: "information you would like to add to Product1 using a dictionary form"
Product2:
Product3:
Outputs:
Column1:
Column2:
ThisEntry: "is ignored, as it is not below the keyword: 'Metadata'"
Metadata:
My: "Metadata"
That: "will be saved to Column2"
Column3:
```
Here, "Product1" and "Column2" have additional, optional Metadata assigned to them (using the keyword "Metadata").
The other Products and Columns have no metadata assigned to them - which is also ok.
In the AgentType definition example above attribute definition was not shown explicitly (indicated by `...`).
The next example provides details on how to define an attribute:
```yaml
MySimpleAttribute:
AttributeType: enum
Mandatory: true
List: false
Values: [ 'AllowedValue1', 'AllowedValue2' ]
Default: 'AllowedValue1'
Help: 'My help text'
Metadata:
Go: "here"
MyComplexAttribute:
AttributeType: block
NestedAttributes:
InnerAttributeA:
AttributeType: integer
Values:
1:
Metadata:
Explain: "1 is a allowed value"
2:
Metadata:
Comment: "2 is also allowed, but consider using 1"
InnerAttributeB:
AttributeType: double
```
* `MySimpleAttribute`, `MyDoubleList`, `MyComplexAttribute` Names of the attributes as specified in the Java enum
annotated with "@Input"
* `AttributeType` (required) data type of the attribute; see options in table below
* `Mandatory` (optional - true by default) if true: the attribute is required for this agent and validation will fail if
the attribute is missing in the scenario **and** no default is provided
* `List` (optional - false by default)
* `AttributeType: time_series` cannot be true
* `AttributeType: block`
* if true: any nested element in the scenario must be part of a list element and thus can appear multiple times
* if false: any nested element in the scenario can only appear once
* any other AttributeType: the attribute is interpreted as list, i.e. multiple values can be assigned to this
attribute in the scenario
* `NestedAttributes` (required only if `AttributeType: block`, otherwise disallowed) starts an inner Attribute
definition block - defined Attributes are sub-elements of `MyComplexAttribute`
* `Values` (optional - None by default):
* if present, defines a list or dictionary of allowed values for this attribute
* if a dictionary is used, individual Metadata can be assigned to each allowed value using the `Metadata` keyword
* `Default` (optional - None by default):
* if present, defines a default value to be used if the scenario does not specify one
* must match one of the entries in `Values` in case those are defined
* can be a list if the attribute is a list
* `Help` (optional - None by default): if present, defines a help text for your Attribute
* `Metadata` (optional - None by default): if present, defines additional metadata assigned to the Attribute
| AttributeType | value |
|---------------|-------------------------------------------------------------------------------------------------------------------------|
| `integer` | a 32-bit integer value |
| `double` | a 64-bit floating-point value (integers also allowed) |
| `long` | a 64-bit integer value |
| `time_stamp` | either a FAME time stamp string or 64-bit integer value |
| `string` | any string |
| `string_set` | a string from a set of allowed `Values` defined in `StringSet` section in `scenario` |
| `enum` | a string from a set of allowed `Values` defined in `schema` |
| `time_series` | either a path to a .csv-file or a single 64-bit floating-point value; does not support `List: true` |
| `block` | this attribute has no value of its own but hosts a group of nested Attributes; implies `NestedAttributes` to be defined |
#### GeneralProperties
Specifies FAME-specific properties of the simulation. Structure:
```yaml
GeneralProperties:
RunId: 1
Simulation:
StartTime: 2011-12-31_23:58:00
StopTime: 2012-12-30_23:58:00
RandomSeed: 1
```
Parameters:
* `RunId` an ID that can be given to the simulation; use at your discretion
* `StartTime` time stamp in the format YYYY-MM-DD_hh:mm:ss; first moment of the simulation.
* `StopTime` time stamp in the format YYYY-MM-DD_hh:mm:ss; last moment of the simulation - i.e. simulation terminates
after passing that time stamp
* `RandomSeed` seed to initialise random number generation; each value leads to a unique series of random numbers.
#### Agents
Specifies all Agents to be created in the simulation in a list. Each Agent has its own entry.
Structure:
```yaml
Agents:
- Type: MyAgentWithInputs
Id: 1
Attributes:
MyEnum: SAME_SHARES
MyInteger: 2
MyDouble: 4.2
MyTimeSeries: "./path/to/time_series.csv"
Metadata:
Can: "also be assigned"
- Type: MyAgentWithoutInputs
Id: 2
```
Agent Parameters:
* `Type` Mandatory; Java's simple class name of the agent to be created
* `Id` Mandatory; simulation-unique id of this agent; if two agents have the same ID, the configuration process will
stop.
* `Attributes` Optional; if the agent has any attributes, specify them here in the format "AttributeName: value"; please
see attribute table above
* `Metadata` Optional; can be assigned to each instance of an Agent, as well as to each of its Attributes
The specified `Attributes` for each agent must match the specified `Attributes` options in the linked Schema (see above).
For better structure and readability of the `scenario.yaml`, `Attributes` may also be specified in a nested way as demonstrated below.
```yaml
Agents:
- Type: MyAgentWithInputs
Id: 1
Attributes:
Parent:
MyEnum: SAME_SHARES
MyInteger: 2
Parent2:
MyDouble: 4.2
Child:
MyTimeSeries: "./path/to/time_series.csv"
```
In case Attributes are defined with `List: true` option, lists are assigned to an Attribute or Group:
```yaml
Attributes:
MyDoubleList: [ 5.2, 4.5, 7, 9.9 ]
MyListGroup:
- IntValueA: 5
IntValueB: 42
- IntValueA: 7
IntValueB: 100
```
Here, `MyDoubleList` and `MyListGroup` need to specify `List: true` in the corresponding Schema.
The shorter `[]`-notation was used to assign a list of floating-point values to `MyDoubleList`.
Nested items `IntValueA` and `IntValueB` of `MyListGroup` are assigned within a list, allowing the specification of
these nested items several times.
##### Attribute Metadata
Metadata can be assigned to any value, list item, or superstructure.
To assign Metadata to a primitive value, create a dictionary from it, set the actual value with the inner keyword `Value` and add the keyword `Metadata` like this:
```yaml
ValueWithoutMetadata: 1
SameValueWithMetadata:
Value: 1
Metadata: # describe `SameValueWithMetadata` herein
```
You can assign Metadata to a list of primitive values using the keyword `Values` like this:
```yaml
ValueListWithoutMetadata: [1,2,3]
SameValueListWithListMetadata:
Values: [1,2,3]
Metadata: # describe the whole list of values with Metadata here
```
or specify Metadata for each (or just some) value individually, like this:
```yaml
ValueListWithoutMetadata: [1,2,3]
SameValueListWithMetadataAtEachElement:
- Value: 1
Metadata: # describe this specific value "1" with Metadata here
- Value: 2 # this value has no Metadata attached, but you can still use the keyword `Value`
- 3 # or use in the actual directly since this value has no Metadata anyway
```
or assign Metadata to both the list and any of its list entries, like this:
```yaml
ValueListWithoutMetadata: [1,2,3]
SameValueListWithAllMetadata:
Metadata: # Recommendation: place the Metadata of the list first if the list of values is extensive, as in this case
Values:
- Value: 1
Metadata: # describe this specific value "1" with Metadata here
- Value: 2
Metadata: # describe this specific value "2" with Metadata here
- Value: 3
Metadata: # describe this specific value "3" with Metadata here
```
You can assign Metadata directly to a nested element by adding the Metadata keyword:
```yaml
NestedItemWithoutMetadata:
A: 1
B: 2
SameNestedItemWithMetadata:
A: 1
B: 2
Metadata: # These Metadata describe `SameNestedItemWithMetadata`
```
Similar to lists of values, you can assign Metadata to a list of nested elements using the `Values` keyword, like this:
```yaml
ListOfNestedItemsWithoutMetadata:
- A: 1
B: 10
- A: 2
B: 20
SameListOfNestedItemsWithGeneralMetadata:
Values:
- A: 1
B: 10
- A: 2
B: 20
Metadata: # These Metadata describe `SameListOfNestedItemsWithGeneralMetadata` as a whole
```
and, similar to nested elements, you can assign Metadata directly to any list element, like this:
```yaml
ListOfNestedItemsWithoutMetadata:
- A: 1
B: 10
- A: 2
B: 20
SameListOfNestedItemsWithGeneralMetadata:
- A: 1
B: 10
Metadata: # These Metadata describe the first list item
- A: 2
B: 20
Metadata: # These Metadata describe the second list item
```
Again, you may apply both variants and apply Metadata to the list and each of its items if you wish.
#### Contracts
Specifies all Contracts, i.e. repetitive bilateral transactions in between agents.
Contracts are given as a list.
We recommend moving Contracts to separate files and to use the `!include` command to integrate them in the scenario.
```yaml
Contracts:
- SenderId: 1
ReceiverId: 2
ProductName: ProductOfAgent_1
FirstDeliveryTime: -25
DeliveryIntervalInSteps: 3600
Metadata:
Some: "additional information can go here"
- SenderId: 2
ReceiverId: 1
ProductName: ProductOfAgent_2
FirstDeliveryTime: -22
DeliveryIntervalInSteps: 3600
Attributes:
ProductAppendix: value
TimeOffset: 42
```
Contract Parameters:
* `SenderId` unique ID of agent sending the product
* `ReceiverId` unique ID of agent receiving the product
* `ProductName` name of the product to be sent
* `FirstDeliveryTime` first time of delivery in the format "seconds after the January 1st 2000, 00:00:00"
* `DeliveryIntervalInSteps` delay time in between deliveries in seconds
* `Metadata` can be assigned to add further helpful information about a Contract
* `Attributes` can be set to include additional information as `int`, `float`, `enum`, or `dict` data types
##### Definition of Multiple Similar Contracts
Often, scenarios contain multiple agents of similar type that also have similar chains of contracts.
Therefore, FAME-Io supports a compact definition of multiple similar contracts.
`SenderId` and `ReceiverId` can both be lists and support One-to-N, N-to-One and N-to-N relations like in the following
example:
```yaml
Contracts:
# effectively 3 similar contracts (0 -> 11), (0 -> 12), (0 -> 13)
# with otherwise identical ProductName, FirstDeliveryTime & DeliveryIntervalInSteps
- SenderId: 0
ReceiverId: [ 11, 12, 13 ]
ProductName: MyOtherProduct
FirstDeliveryTime: 100
DeliveryIntervalInSteps: 3600
# effectively 3 similar contracts (1 -> 10), (2 -> 10), (3 -> 10)
# with otherwise identical ProductName, FirstDeliveryTime & DeliveryIntervalInSteps
- SenderId: [ 1, 2, 3 ]
ReceiverId: 10
ProductName: MyProduct
FirstDeliveryTime: 100
DeliveryIntervalInSteps: 3600
# effectively 3 similar contracts (1 -> 11), (2 -> 12), (3 -> 13)
# with otherwise identical ProductName, FirstDeliveryTime & DeliveryIntervalInSteps
- SenderId: [ 1, 2, 3 ]
ReceiverId: [ 11, 12, 13 ]
ProductName: MyThirdProduct
FirstDeliveryTime: 100
DeliveryIntervalInSteps: 3600
```
Combined with YAML anchors complex contract chains can be easily reduced to a minimum of required configuration.
The following example is equivalent to the previous one and allows a quick extension of contracts to a new couple of
agents e.g. (4;14):
```yaml
Groups:
- &agentList1: [ 1,2,3 ]
- &agentList2: [ 11,12,13 ]
Contracts:
- SenderId: 0
ReceiverId: *agentList2
ProductName: MyOtherProduct
FirstDeliveryTime: 100
DeliveryIntervalInSteps: 3600
- SenderId: *agentList1
ReceiverId: 10
ProductName: MyProduct
FirstDeliveryTime: 100
DeliveryIntervalInSteps: 3600
- SenderId: *agentList1
ReceiverId: *agentList2
ProductName: MyThirdProduct
FirstDeliveryTime: 100
DeliveryIntervalInSteps: 3600
```
#### StringSets
This optional section defines values of type `string_set`.
In contrast to `enum` values, which are **statically** defined in the `Schema`, `string_set` values can be **dynamically
** defined in this section.
If an agent attribute is of type `string_set` and the attribute is set in the `scenario`, then
1. the section `StringSets` in the `scenario` must contain an entry named exactly like the attribute, and
2. the attribute value must be contained in the string set's `Values` declaration.
For instance:
In `schema`:
``` yaml
AgentTypes:
FuelsMarket:
Attributes:
FuelType:
AttributeType: string_set
```
In `scenario`:
``` yaml
StringSets:
FuelType:
Values: ['OIL', 'HARD_COAL', 'LIGNITE']
Agents:
- Type: FuelsMarket
Id: 1
Attributes:
FuelType: OIL
```
Important: If different types of Agents shall refer to the same StringSet, their attributes in schema must have the
**exact** same name.
### CSV files
TIME_SERIES inputs are not directly fed into the Scenario YAML file.
Instead, TIME_SERIES reference a CSV file that can be stored some place else.
These CSV files follow a specific structure:
* They should contain exactly two columns - any other columns are ignored.
A warning is raised if more than two non-empty columns are detected.
* The first column must be a time stamp in form `YYYY-MM-DD_hh:mm:ss`
* The second column must be a numerical value (either integer or floating-point)
* The separator of the two columns is a semicolon
* The data must **not** have headers, except for comments marked with `#`
You may add comments using `#`.
Exemplary content of a valid CSV file:
# If you want an optional header, you must use a comment
2012-01-01_00:00:00;400
2013-01-01_00:00:00;720.5
2014-01-01_00:00:00;650
2015-01-01_00:00:00;99.27772
2016-01-01_00:00:00;42 # optional comment on this particular data point
2017-01-01_00:00:00;0.1
Please refer also to the detailed article about `TimeStamps` in
the [FAME-Wiki](https://gitlab.com/fame-framework/wiki/-/wikis/TimeStamp).
### Split and join multiple YAML files
The user may include other YAML files into a YAML file to divide the content across files as convenient.
We explicitly recommend using this feature for the `Schema` and `Contracts` sections.
Otherwise, the scenario.yaml may become crowded.
#### Command: !Include
To hint YAML to load the content of another file use `!include "path/relative/to/including/yaml/file.yml"`.
You can concatenate !include commands and can use !include in the included file as well.
The path to the included file is always relative to the file using the !include command.
So with the following file structure
###### file-structure
```
a.yaml
folder/b.yaml
folder/c.yaml
folder/deeper_folder/d.yaml
```
the following !include commands work
###### in a.yaml
```
ToBe: !include "folder/b.yaml"
OrNot: !include "folder/deeper_folder/d.yaml"
```
###### in b.yaml
```
ThatIs: !include "c.yaml"
TheQuestion: !include "deeper_folder/d.yaml"
```
Provided that
###### in c.yaml
```
Or: maybe
```
###### d.yaml
```
not: "?"
```
the resulting file would look like this:
###### THe Joined file a.yaml
```
ToBe:
ThatIs:
Or: maybe
TheQuestion:
not: "?"
OrNot:
not: "?"
```
You may also specify absolute file paths if preferred by starting with a "/".
When specifying only a file path, the complete content of the file is assigned to the given key.
You always need a key to assign the !include command to.
However, you cannot combine the value returned from !include with other values in the same key.
Thus, the following combinations do not work:
###### caveats.yml
```
!include "file.yaml" # no key assigned
Key:
Some: OtherItem
!include "file.yaml" # cannot join with other named items
List:
- an: entry
!include "file.yaml" # cannot directly join with list items, even if !include returns a list
```
#### Integrate specific nodes of YAML files
Instead of including *all* content in the included file, you may also pick a specific node within that file.
For this use `!include [<relative/path/to/file.yaml>, Path:To:Field:In:Yaml]`.
Here, `:` is used in the node-specifying string to select a sequence of nodes to follow - with custom depth.
Consider the following two files:
###### file_to_be_included.yaml
```yaml
Set1:
Subset1:
Key: Value
Set2:
OtherKey: OtherValue
```
###### including_file.yaml
```yaml
- Type: MyAgentWithInputs
Id: 1
Attributes: !include_node [ file_to_be_included.yaml, Set1:Subset1 ]
```
Compiling "including_file.yaml" results in
###### resulting_file.yaml
```yaml
- Type: MyAgentWithInputs
Id: 1
Attributes:
Key: Value
```
#### Load multiple files
Using wildcards in the given path (e.g. "path/to/many/*.yaml") will lead to loading multiple files and assigning their
content to the same key.
You can make use of this feature with or without specifying a node selector.
However, the elements to be joined across multiple files must be lists.
These lists are then concatenated into a single list and then assigned to the key in the file calling !include.
This feature is especially useful for Contracts: You can split the Contracts list into several files and place them in a
separate folder.
Then use !include to re-integrate them into your configuration. An example:
###### my_contract1.yaml
```
Contracts:
- ContractA
- ContractB
```
###### my_contract2.yaml
```
Contracts:
- ContractC
- ContractD
- ContractE
```
###### including_file.yaml
```
Contracts: [!include "my_contract*.yaml", "Contracts"]
```
results in
###### result.yaml
```
Contracts:
- ContractA
- ContractB
- ContractC
- ContractD
- ContractE
```
#### Ignoring files
Files that have their name start with "IGNORE_" are not included with the !include command.
You will see a debug output to notify you that the file was ignored.
Use this to temporarily take files out ouf your configuration without deleting or moving them.
## Read FAME results
Takes an output file in protobuf format of FAME-based applications and converts it into files in CSV format.
An individual file for each type of Agent is created in a folder named after the protobuf input file.
Call structure:
convertFameResults -f <./path/to/protobuf_file.pb>
You may also specify any of the following arguments:
| Command | Action |
|-----------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `-l` or `--log` <option> | Sets the logging level. Default is `WARNING`. Options are `DEBUG`, `INFO`, `WARNING`, `ERROR`, `CRITICAL`. |
| `-lf` or `--logfile` <file> | Sets the logging file. Default is `None`. If `None` is provided, all logs get only printed to the console. |
| `-a` or `--agents` <list-of-agents> | If specified, only a subset of agents is extracted from the protobuf file. Default is to extract all agents. |
| `-o` or `--output` | Sets the path to where the generated output files are written to. If not specified, the folder's name is derived from the input file's name. Folder will be created if it does not exist. |
| `-se` or `--single-export` | Enables export of individual agents to individual files, when present. If not present (the default) one file per `AgentType` is created. |
| `-m` or `--memory-saving` | When specified, reduces memory usage profile at the cost of runtime. Use only when necessary. |
| `-cc` or `--complex-column` <option> | Defines how to deal with complex indexed output columns (if any). `IGNORE` ignores complex columns. `SPLIT` creates a separate file for each complex indexed output column. |
| `-t` or `--time` <option> | Option to define conversion of time steps to given format (default=`UTC`) by `-t/--time {UTC, INT, FAME}` |
| `--input-recovery` or `--no-input-recovery` | If True, all input data are recovered in addition to the outputs (default=False). |
| `-mt` or `--merge-times` <list-of-parameters> | Option to merge `TimeSteps` of a certain range of steps in the output files to associate multiple time steps with a common logical time in your simulation and reduce number of lines in output files |
The option `--merge-times` requires exactly three integer arguments separated by spaces:
| Position | Name | Meaning |
|----------|--------------|------------------------------------------------------------------------------------------|
| First | Focal point | TimeStep on which `steps-before` earlier and `steps-after` later TimeSteps are merged on |
| Second | Steps before | Range of TimeSteps before the `focal-point` they get merged to, must be Zero or positive |
| Third | Steps after | Range of TimeSteps after the `focal-point` they get merged to, must be Zero or positive |
This could look as follows:
convertFameResults -f <./path/to/protobuf_file.pb> -l debug -lf <path/to/output.log> -a AgentType1 AgentType2 -o myCsvFolder -m -cc SPLIT --merge-times 0 1799 1800
Make sure that in the range of time steps you specify for merging, there is only one value per column in the merged time range.
If multiple values per column are merged values will get concatenated and might yield unexpected results.
You may also call the conversion script from any Python script with:
```python
from fameio.scripts.convert_results import Options, run as convert_results
convert_results({Options.FILE: "./path/to/protobuf_file.pb"})
```
Similar to the console call you may also specify custom run config arguments and add it in a dictionary to the function
call.
```python
from fameio.scripts.convert_results import Options, run as convert_results
run_config = {Options.FILE: "./path/to/protobuf_file.pb",
Options.LOG_LEVEL: "info",
Options.LOG_FILE: "scenario.log",
Options.OUTPUT: "Output",
Options.AGENT_LIST: ['AgentType1', 'AgentType2'],
Options.MEMORY_SAVING: False,
Options.SINGLE_AGENT_EXPORT: False,
Options.RESOLVE_COMPLEX_FIELD: "SPLIT",
Options.TIME: "INT",
Options.TIME_MERGING: {},
}
convert_results(run_config)
```
You can also use the associated argument parser, to extract the run_config dynamically from a string:
```python
from fameio.scripts.convert_results import Options, run as convert_results
from fameio.cli.convert_results import handle_args
my_defaults = {Options.FILE: "./path/to/protobuf_file.pb",
Options.LOG_LEVEL: "info",
Options.LOG_FILE: "scenario.log",
Options.OUTPUT: "Output",
Options.AGENT_LIST: ['AgentType1', 'AgentType2'],
Options.MEMORY_SAVING: False,
Options.SINGLE_AGENT_EXPORT: False,
Options.RESOLVE_COMPLEX_FIELD: "SPLIT",
Options.TIME: "INT",
Options.TIME_MERGING: {},
}
my_arg_string = ['-f', 'my/other/scenario.yaml', '-l', 'error']
run_config = handle_args(my_arg_string, my_defaults)
convert_results(run_config)
```
## Cite FAME-Io
If you use FAME-Io for academic work, please cite as follows.
Bibtex entry:
```
@article{fameio2023joss,
author = {Felix Nitsch and Christoph Schimeczek and Ulrich Frey and Benjamin Fuchs},
title = {FAME-Io: Configuration tools for complex agent-based simulations},
journal = {Journal of Open Source Software},
year = {2023},
doi = {doi: https://doi.org/10.21105/joss.04958}
}
```
## Available Support
This is a purely scientific project by (at the moment) one research group.
Thus, there is no paid technical support available.
However, we will give our best to answer your questions and provide support.
If you experience any trouble with FAME-Io, you may contact the developers via [fame@dlr.de](mailto:fame@dlr.de).
Please report bugs and make feature requests by filing issues following the provided templates (see
also [Contribute](CONTRIBUTING.md)).
For substantial enhancements, we recommend that you contact us via [fame@dlr.de](mailto:fame@dlr.de) for working
together on the code in common projects or towards common publications and thus further develop FAME-Io.
Raw data
{
"_id": null,
"home_page": "https://gitlab.com/fame-framework/wiki/-/wikis/home",
"name": "fameio",
"maintainer": "Felix Nitsch",
"docs_url": null,
"requires_python": "<4.0,>=3.9",
"maintainer_email": "fame@dlr.de",
"keywords": "FAME, fameio, agent-based modelling, energy systems",
"author": "Felix Nitsch",
"author_email": "fame@dlr.de",
"download_url": "https://files.pythonhosted.org/packages/9e/bd/99a9c563e6cb22b6c5921da07b330a473943864aebbcaf780acf60313fd9/fameio-3.0.0.tar.gz",
"platform": null,
"description": "<!-- SPDX-FileCopyrightText: 2024 German Aerospace Center <fame@dlr.de>\n\nSPDX-License-Identifier: Apache-2.0 -->\n[](https://badge.fury.io/py/fameio)\n[](https://doi.org/10.21105/joss.04958)\n[](https://doi.org/10.5281/zenodo.4314337)\n[](https://badge.fury.io/py/fameio)\n[](https://gitlab.com/fame-framework/fame-io/commits/main)\n[](https://gitlab.com/fame-framework/fame-io/-/commits/main)\n[](https://github.com/psf/black)\n[](https://api.reuse.software/info/gitlab.com/fame-framework/fame-io)\n[](https://common-changelog.org)\n\n\n\n# FAME-Io\n\n*Tools for input preparation and output digestion of FAME models*\n\nFAME-Io compiles input for FAME models in protobuf format and extracts model outputs to human-readable files.\nPlease visit the [FAME-Wiki](https://gitlab.com/fame-framework/wiki/-/wikis/home) to get an explanation of FAME and its components.\n\n# Installation\n\nWe recommend installing `fameio` using PyPI:\n\n pip install fameio\n\nYou may also use `pipx`. For detailed information please refer to the\nofficial `pipx` [documentation](https://github.com/pypa/pipx).\n\n pipx install fameio\n\n`fameio` is currently developed and tested for Python 3.8 or higher.\nSee the `pyproject.toml` for a complete listing of dependencies.\n\n# Usage\n\nFAME-Io currently offers two main scripts `makeFameRunConfig` and `convertFameResults`.\nBoth are automatically installed with the package.\nThe first one creates a protobuf file for FAME applications using YAML definition files and CSV files.\nThe latter one reads output files from FAME applications in protobuf format and converts them to CSV files.\n\nYou may use the [example data](https://gitlab.com/dlr-ve/esy/amiris/examples) provided for\nthe [AMIRIS](https://gitlab.com/dlr-ve/esy/amiris/amiris) model which can be used to simulate electricity markets\nin [Germany](https://gitlab.com/dlr-ve/esy/amiris/examples/-/tree/main/Germany2019), [Austria](https://gitlab.com/dlr-ve/esy/amiris/examples/-/tree/main/Austria2019),\nand a simple [proof-of-concept model](https://gitlab.com/dlr-ve/esy/amiris/examples/-/tree/main/Simple).\n\n## Make a FAME run configuration\n\nDigests configuration files in YAML format, combines them with CSV data files and creates a single input file for FAME\napplications in protobuf format.\nCall structure:\n\n makeFameRunConfig -f <path/to/scenario.yaml>\n\nYou may also specify any of the following arguments:\n\n| Command | Action |\n|------------------------|------------------------------------------------------------------------------------------------------------------------------------------|\n| `-l` or `--log` | Sets the logging level. Default is `info`. Options are `debug`, `info`, `warning`, `warn`, `error`, `critical`. |\n| `-lf` or `--logfile` | Sets the logging file. Default is `None`. If `None` is provided, all logs get only printed to the console. |\n| `-o` or `--output` | Sets the path of the compiled protobuf output file. Default is `config.pb`. |\n| `-enc` or `--encoding` | Sets the encoding of all yaml files to the given one (e.g. 'utf8' or 'cp1252'. Default is `None`, i.e. your operating system's standard. |\n\nThis could look as follows:\n\n makeFameRunConfig -f <path/to/scenario.yaml> -l debug -lf <path/to/scenario.log> -o <path/to/config.pb>\n\nYou may also call the configuration builder from any Python script with\n\n```python\nfrom fameio.scripts.make_config import Options, run as make_config\n\nmake_config({Options.FILE: \"path/to/scenario.yaml\", })\n```\n\nSimilar to the console call you may also specify custom run config arguments and add it in a dictionary to the function\ncall.\n\n```python\nfrom fameio.scripts.make_config import Options, run as make_config\n\nrun_config = {Options.FILE: \"path/to/scenario.yaml\",\n Options.LOG_LEVEL: \"info\",\n Options.OUTPUT: \"output.pb\",\n Options.LOG_FILE: \"scenario.log\",\n }\n\nmake_config(run_config)\n```\n\nYou can also use the associated argument parser, to extract the run_config dynamically from a string:\n\n```python\nfrom fameio.scripts.make_config import Options, run as make_config\nfrom fameio.cli.make_config import handle_args\n\nmy_defaults = {Options.FILE: \"path/to/scenario.yaml\",\n Options.LOG_LEVEL: \"info\",\n Options.OUTPUT: \"output.pb\",\n Options.LOG_FILE: \"scenario.log\",\n }\nmy_arg_string = ['-f', 'my/other/scenario.yaml', '-l', 'error']\n\nrun_config = handle_args(my_arg_string, my_defaults)\nmake_config(run_config)\n```\n\n### Scenario YAML\n\nThe \"scenario.yaml\" file contains all configuration options for a FAME-based simulation.\nIt consists of the sections `Schema`, `GeneralProperties`, `Agents` and `Contracts`, and the optional\nsection `StringSets`.\nAll of them are described below.\n\n#### Schema\n\nThe Schema describes a model's components such as its types of agents, their inputs, what data they exchange, etc.\nIt is also used to validate the model inputs provided in the `scenario.yaml`.\nSince the Schema is valid until the model itself is changed, it is recommended to defined it in a separate file and\ninclude the file here.\n\nCurrently, the schema specifies:\n\n* which type of Agents can be created\n* what type of input attributes an Agent uses\n* what type of Products an Agent can send in Contracts, and\n* the names of the Java packages for the classes corresponding to Agents, DataItems and Portables.\n\nThe Schema consists of the sections `JavaPackages` and `AgentTypes`.\n\n##### JavaPackages\n\nThis section defines the name of the Java packages in which the model code is located.\nA similar data set was formerly specified in the `fameSetup.yaml`, but is now specified in the schema.\nEach of the three sections `Agents`, `DataItems`, and `Portables` contain a list of fully qualified java package names\nof your model's classes.\nPackage names can occur in multiple lists and may overlap.\nIt is not necessary (but possible) to specify the nearest enclosing package for each Agent, DataItem or Portable.\nSpecifying any super-package will also work.\nAlso, package names occur on multiple lists for Agent, DataItem or Portable.\n\nFor example, for a project with all its\n\n* Agent-derived java classes located in packages below the package named \"agents\",\n* DataItem implementation classes in a subpackage named \"msg\",\n* Portable implementation classes in a subpackages named \"portableItems\" and \"otherPortables\",\n\nthe corresponding section in the schema would look like this:\n\n```yaml\nJavaPackages:\n Agents:\n - \"agents\"\n DataItems:\n - \"msg\"\n Portables:\n - \"portableItems\"\n - \"otherPortables\"\n```\n\nOne can leave out the `DataItems` specifications, but `Agents` and `Portables` are required and must not be empty.\n\n##### AgentTypes\n\nHere, each type of agent that can be created in your FAME-based application is listed, its attributes and its available\nProducts for Contracts.\nThe structure of this section\n\n```yaml\nAgentTypes:\n MyAgentType:\n Attributes:\n MyAttribute:\n ...\n MyOtherAttribute:\n ...\n Products: [ 'Product1', 'Product2', 'Product3' ]\n Outputs: [ 'Column1', 'Column2', 'Column3' ]\n Metadata:\n Some: \"Dict with Metadata that you would like to add\"\n MyOtherAgentWithoutProductsOrAttributes:\n```\n\n* `MyAgentType` Java's simple class name of the Agent type\n* `Attributes` indicates that beginning of the attribute definition section for this Agent type\n* `MyAttribute` Name of an attribute as specified in the corresponding Java source code of this Agent type (annotated\n with \"@Input\")\n* `MyOtherAttribute` Name of another attribute derived from Java source code\n* `Products` list or dictionary of Products that this Agent can send in Contracts; derived from Java source code of this\n Agent type (annotated with \"@Product\")\n* `Outputs` list or dictionary of Output columns that this Agent can write to; derived from Java source code of this\n Agent type (annotated with \"@Output\")\n* `Metadata` dictionary with any content that is assigned to this Agent type as additional information\n* `MyOtherAgentWithoutProductsOrAttributes` an Agent type that requires neither Attributes nor Products\n\nAttributes, Products, Outputs and Metadata are optional - there may be useful Agents that require none of them.\nProducts and Outputs can both be lists of Strings, or dictionaries with additional Metadata.\nFor example, you could write the above in the following way:\n\n```yaml\nProducts:\n Product1:\n Metadata:\n Any: \"information you would like to add to Product1 using a dictionary form\"\n Product2:\n Product3:\nOutputs:\n Column1:\n Column2:\n ThisEntry: \"is ignored, as it is not below the keyword: 'Metadata'\"\n Metadata:\n My: \"Metadata\"\n That: \"will be saved to Column2\"\n Column3:\n```\n\nHere, \"Product1\" and \"Column2\" have additional, optional Metadata assigned to them (using the keyword \"Metadata\").\nThe other Products and Columns have no metadata assigned to them - which is also ok.\n\nIn the AgentType definition example above attribute definition was not shown explicitly (indicated by `...`).\nThe next example provides details on how to define an attribute:\n\n```yaml\nMySimpleAttribute:\n AttributeType: enum\n Mandatory: true\n List: false\n Values: [ 'AllowedValue1', 'AllowedValue2' ]\n Default: 'AllowedValue1'\n Help: 'My help text'\n Metadata:\n Go: \"here\"\n\nMyComplexAttribute:\n AttributeType: block\n NestedAttributes:\n InnerAttributeA:\n AttributeType: integer\n Values:\n 1:\n Metadata:\n Explain: \"1 is a allowed value\"\n 2:\n Metadata:\n Comment: \"2 is also allowed, but consider using 1\"\n InnerAttributeB:\n AttributeType: double\n```\n\n* `MySimpleAttribute`, `MyDoubleList`, `MyComplexAttribute` Names of the attributes as specified in the Java enum\n annotated with \"@Input\"\n* `AttributeType` (required) data type of the attribute; see options in table below\n* `Mandatory` (optional - true by default) if true: the attribute is required for this agent and validation will fail if\n the attribute is missing in the scenario **and** no default is provided\n* `List` (optional - false by default)\n * `AttributeType: time_series` cannot be true\n * `AttributeType: block`\n * if true: any nested element in the scenario must be part of a list element and thus can appear multiple times\n * if false: any nested element in the scenario can only appear once\n * any other AttributeType: the attribute is interpreted as list, i.e. multiple values can be assigned to this\n attribute in the scenario\n* `NestedAttributes` (required only if `AttributeType: block`, otherwise disallowed) starts an inner Attribute\n definition block - defined Attributes are sub-elements of `MyComplexAttribute`\n* `Values` (optional - None by default):\n * if present, defines a list or dictionary of allowed values for this attribute\n * if a dictionary is used, individual Metadata can be assigned to each allowed value using the `Metadata` keyword\n* `Default` (optional - None by default):\n * if present, defines a default value to be used if the scenario does not specify one\n * must match one of the entries in `Values` in case those are defined\n * can be a list if the attribute is a list\n* `Help` (optional - None by default): if present, defines a help text for your Attribute\n* `Metadata` (optional - None by default): if present, defines additional metadata assigned to the Attribute\n\n| AttributeType | value |\n|---------------|-------------------------------------------------------------------------------------------------------------------------|\n| `integer` | a 32-bit integer value |\n| `double` | a 64-bit floating-point value (integers also allowed) |\n| `long` | a 64-bit integer value |\n| `time_stamp` | either a FAME time stamp string or 64-bit integer value |\n| `string` | any string |\n| `string_set` | a string from a set of allowed `Values` defined in `StringSet` section in `scenario` |\n| `enum` | a string from a set of allowed `Values` defined in `schema` |\n| `time_series` | either a path to a .csv-file or a single 64-bit floating-point value; does not support `List: true` |\n| `block` | this attribute has no value of its own but hosts a group of nested Attributes; implies `NestedAttributes` to be defined |\n\n#### GeneralProperties\nSpecifies FAME-specific properties of the simulation. Structure:\n\n```yaml\nGeneralProperties:\n RunId: 1\n Simulation:\n StartTime: 2011-12-31_23:58:00\n StopTime: 2012-12-30_23:58:00\n RandomSeed: 1\n```\n\nParameters:\n\n* `RunId` an ID that can be given to the simulation; use at your discretion\n* `StartTime` time stamp in the format YYYY-MM-DD_hh:mm:ss; first moment of the simulation.\n* `StopTime` time stamp in the format YYYY-MM-DD_hh:mm:ss; last moment of the simulation - i.e. simulation terminates\n after passing that time stamp\n* `RandomSeed` seed to initialise random number generation; each value leads to a unique series of random numbers.\n\n#### Agents\n\nSpecifies all Agents to be created in the simulation in a list. Each Agent has its own entry.\nStructure:\n\n```yaml\nAgents:\n - Type: MyAgentWithInputs\n Id: 1\n Attributes:\n MyEnum: SAME_SHARES\n MyInteger: 2\n MyDouble: 4.2\n MyTimeSeries: \"./path/to/time_series.csv\"\n Metadata:\n Can: \"also be assigned\"\n\n - Type: MyAgentWithoutInputs\n Id: 2\n```\n\nAgent Parameters:\n\n* `Type` Mandatory; Java's simple class name of the agent to be created\n* `Id` Mandatory; simulation-unique id of this agent; if two agents have the same ID, the configuration process will\n stop.\n* `Attributes` Optional; if the agent has any attributes, specify them here in the format \"AttributeName: value\"; please\n see attribute table above\n* `Metadata` Optional; can be assigned to each instance of an Agent, as well as to each of its Attributes\n\nThe specified `Attributes` for each agent must match the specified `Attributes` options in the linked Schema (see above).\nFor better structure and readability of the `scenario.yaml`, `Attributes` may also be specified in a nested way as demonstrated below.\n\n```yaml\nAgents:\n - Type: MyAgentWithInputs\n Id: 1\n Attributes:\n Parent:\n MyEnum: SAME_SHARES\n MyInteger: 2\n Parent2:\n MyDouble: 4.2\n Child:\n MyTimeSeries: \"./path/to/time_series.csv\"\n```\n\nIn case Attributes are defined with `List: true` option, lists are assigned to an Attribute or Group:\n\n```yaml\nAttributes:\n MyDoubleList: [ 5.2, 4.5, 7, 9.9 ]\n MyListGroup:\n - IntValueA: 5\n IntValueB: 42\n - IntValueA: 7\n IntValueB: 100\n```\n\nHere, `MyDoubleList` and `MyListGroup` need to specify `List: true` in the corresponding Schema.\nThe shorter `[]`-notation was used to assign a list of floating-point values to `MyDoubleList`.\nNested items `IntValueA` and `IntValueB` of `MyListGroup` are assigned within a list, allowing the specification of\nthese nested items several times.\n\n##### Attribute Metadata\nMetadata can be assigned to any value, list item, or superstructure.\nTo assign Metadata to a primitive value, create a dictionary from it, set the actual value with the inner keyword `Value` and add the keyword `Metadata` like this:\n\n```yaml\nValueWithoutMetadata: 1\nSameValueWithMetadata:\n Value: 1\n Metadata: # describe `SameValueWithMetadata` herein\n```\n\nYou can assign Metadata to a list of primitive values using the keyword `Values` like this:\n\n```yaml\nValueListWithoutMetadata: [1,2,3]\nSameValueListWithListMetadata:\n Values: [1,2,3]\n Metadata: # describe the whole list of values with Metadata here\n```\n\nor specify Metadata for each (or just some) value individually, like this:\n\n```yaml\nValueListWithoutMetadata: [1,2,3]\nSameValueListWithMetadataAtEachElement:\n - Value: 1\n Metadata: # describe this specific value \"1\" with Metadata here\n - Value: 2 # this value has no Metadata attached, but you can still use the keyword `Value`\n - 3 # or use in the actual directly since this value has no Metadata anyway\n```\n\nor assign Metadata to both the list and any of its list entries, like this:\n\n```yaml\nValueListWithoutMetadata: [1,2,3]\nSameValueListWithAllMetadata:\n Metadata: # Recommendation: place the Metadata of the list first if the list of values is extensive, as in this case\n Values:\n - Value: 1\n Metadata: # describe this specific value \"1\" with Metadata here\n - Value: 2\n Metadata: # describe this specific value \"2\" with Metadata here\n - Value: 3\n Metadata: # describe this specific value \"3\" with Metadata here\n```\n\nYou can assign Metadata directly to a nested element by adding the Metadata keyword:\n\n```yaml\nNestedItemWithoutMetadata:\n A: 1\n B: 2\nSameNestedItemWithMetadata:\n A: 1\n B: 2\n Metadata: # These Metadata describe `SameNestedItemWithMetadata`\n```\n\nSimilar to lists of values, you can assign Metadata to a list of nested elements using the `Values` keyword, like this:\n\n```yaml\nListOfNestedItemsWithoutMetadata:\n - A: 1\n B: 10\n - A: 2\n B: 20\nSameListOfNestedItemsWithGeneralMetadata:\n Values:\n - A: 1\n B: 10\n - A: 2\n B: 20\n Metadata: # These Metadata describe `SameListOfNestedItemsWithGeneralMetadata` as a whole\n```\n\nand, similar to nested elements, you can assign Metadata directly to any list element, like this:\n\n```yaml\nListOfNestedItemsWithoutMetadata:\n - A: 1\n B: 10\n - A: 2\n B: 20\nSameListOfNestedItemsWithGeneralMetadata:\n - A: 1\n B: 10\n Metadata: # These Metadata describe the first list item\n - A: 2\n B: 20\n Metadata: # These Metadata describe the second list item\n```\n\nAgain, you may apply both variants and apply Metadata to the list and each of its items if you wish.\n\n#### Contracts\nSpecifies all Contracts, i.e. repetitive bilateral transactions in between agents.\nContracts are given as a list.\nWe recommend moving Contracts to separate files and to use the `!include` command to integrate them in the scenario.\n\n```yaml\nContracts:\n - SenderId: 1\n ReceiverId: 2\n ProductName: ProductOfAgent_1\n FirstDeliveryTime: -25\n DeliveryIntervalInSteps: 3600\n Metadata:\n Some: \"additional information can go here\"\n\n - SenderId: 2\n ReceiverId: 1\n ProductName: ProductOfAgent_2\n FirstDeliveryTime: -22\n DeliveryIntervalInSteps: 3600\n Attributes:\n ProductAppendix: value\n TimeOffset: 42\n```\n\nContract Parameters:\n\n* `SenderId` unique ID of agent sending the product\n* `ReceiverId` unique ID of agent receiving the product\n* `ProductName` name of the product to be sent\n* `FirstDeliveryTime` first time of delivery in the format \"seconds after the January 1st 2000, 00:00:00\"\n* `DeliveryIntervalInSteps` delay time in between deliveries in seconds\n* `Metadata` can be assigned to add further helpful information about a Contract\n* `Attributes` can be set to include additional information as `int`, `float`, `enum`, or `dict` data types\n\n##### Definition of Multiple Similar Contracts\nOften, scenarios contain multiple agents of similar type that also have similar chains of contracts.\nTherefore, FAME-Io supports a compact definition of multiple similar contracts.\n`SenderId` and `ReceiverId` can both be lists and support One-to-N, N-to-One and N-to-N relations like in the following\nexample:\n\n```yaml\nContracts:\n # effectively 3 similar contracts (0 -> 11), (0 -> 12), (0 -> 13)\n # with otherwise identical ProductName, FirstDeliveryTime & DeliveryIntervalInSteps\n - SenderId: 0\n ReceiverId: [ 11, 12, 13 ]\n ProductName: MyOtherProduct\n FirstDeliveryTime: 100\n DeliveryIntervalInSteps: 3600\n\n # effectively 3 similar contracts (1 -> 10), (2 -> 10), (3 -> 10)\n # with otherwise identical ProductName, FirstDeliveryTime & DeliveryIntervalInSteps\n - SenderId: [ 1, 2, 3 ]\n ReceiverId: 10\n ProductName: MyProduct\n FirstDeliveryTime: 100\n DeliveryIntervalInSteps: 3600\n\n # effectively 3 similar contracts (1 -> 11), (2 -> 12), (3 -> 13)\n # with otherwise identical ProductName, FirstDeliveryTime & DeliveryIntervalInSteps\n - SenderId: [ 1, 2, 3 ]\n ReceiverId: [ 11, 12, 13 ]\n ProductName: MyThirdProduct\n FirstDeliveryTime: 100\n DeliveryIntervalInSteps: 3600\n```\n\nCombined with YAML anchors complex contract chains can be easily reduced to a minimum of required configuration.\nThe following example is equivalent to the previous one and allows a quick extension of contracts to a new couple of\nagents e.g. (4;14):\n\n```yaml\nGroups:\n - &agentList1: [ 1,2,3 ]\n - &agentList2: [ 11,12,13 ]\n\nContracts:\n - SenderId: 0\n ReceiverId: *agentList2\n ProductName: MyOtherProduct\n FirstDeliveryTime: 100\n DeliveryIntervalInSteps: 3600\n\n - SenderId: *agentList1\n ReceiverId: 10\n ProductName: MyProduct\n FirstDeliveryTime: 100\n DeliveryIntervalInSteps: 3600\n\n - SenderId: *agentList1\n ReceiverId: *agentList2\n ProductName: MyThirdProduct\n FirstDeliveryTime: 100\n DeliveryIntervalInSteps: 3600\n```\n\n#### StringSets\n\nThis optional section defines values of type `string_set`.\nIn contrast to `enum` values, which are **statically** defined in the `Schema`, `string_set` values can be **dynamically\n** defined in this section.\nIf an agent attribute is of type `string_set` and the attribute is set in the `scenario`, then\n\n1. the section `StringSets` in the `scenario` must contain an entry named exactly like the attribute, and\n2. the attribute value must be contained in the string set's `Values` declaration.\n\nFor instance:\n\nIn `schema`:\n\n``` yaml\nAgentTypes:\n FuelsMarket:\n Attributes:\n FuelType:\n AttributeType: string_set\n```\n\nIn `scenario`:\n\n``` yaml\nStringSets:\n FuelType:\n Values: ['OIL', 'HARD_COAL', 'LIGNITE']\n\nAgents:\n - Type: FuelsMarket\n Id: 1\n Attributes:\n FuelType: OIL\n```\n\nImportant: If different types of Agents shall refer to the same StringSet, their attributes in schema must have the\n**exact** same name.\n\n### CSV files\n\nTIME_SERIES inputs are not directly fed into the Scenario YAML file.\nInstead, TIME_SERIES reference a CSV file that can be stored some place else.\nThese CSV files follow a specific structure:\n\n* They should contain exactly two columns - any other columns are ignored.\n A warning is raised if more than two non-empty columns are detected.\n* The first column must be a time stamp in form `YYYY-MM-DD_hh:mm:ss`\n* The second column must be a numerical value (either integer or floating-point)\n* The separator of the two columns is a semicolon\n* The data must **not** have headers, except for comments marked with `#`\n\nYou may add comments using `#`.\nExemplary content of a valid CSV file:\n\n # If you want an optional header, you must use a comment\n 2012-01-01_00:00:00;400\n 2013-01-01_00:00:00;720.5\n 2014-01-01_00:00:00;650\n 2015-01-01_00:00:00;99.27772\n 2016-01-01_00:00:00;42 # optional comment on this particular data point\n 2017-01-01_00:00:00;0.1\n\nPlease refer also to the detailed article about `TimeStamps` in\nthe [FAME-Wiki](https://gitlab.com/fame-framework/wiki/-/wikis/TimeStamp).\n\n### Split and join multiple YAML files\n\nThe user may include other YAML files into a YAML file to divide the content across files as convenient.\nWe explicitly recommend using this feature for the `Schema` and `Contracts` sections.\nOtherwise, the scenario.yaml may become crowded.\n\n#### Command: !Include\n\nTo hint YAML to load the content of another file use `!include \"path/relative/to/including/yaml/file.yml\"`.\nYou can concatenate !include commands and can use !include in the included file as well.\nThe path to the included file is always relative to the file using the !include command.\nSo with the following file structure\n\n###### file-structure\n\n```\na.yaml\nfolder/b.yaml\nfolder/c.yaml\nfolder/deeper_folder/d.yaml\n```\n\nthe following !include commands work\n\n###### in a.yaml\n\n```\nToBe: !include \"folder/b.yaml\"\nOrNot: !include \"folder/deeper_folder/d.yaml\"\n```\n\n###### in b.yaml\n\n```\nThatIs: !include \"c.yaml\"\nTheQuestion: !include \"deeper_folder/d.yaml\"\n```\n\nProvided that\n\n###### in c.yaml\n\n```\nOr: maybe\n```\n\n###### d.yaml\n\n```\nnot: \"?\"\n```\n\nthe resulting file would look like this:\n\n###### THe Joined file a.yaml\n\n```\nToBe:\n ThatIs:\n Or: maybe\n TheQuestion:\n not: \"?\"\nOrNot:\n not: \"?\"\n```\n\nYou may also specify absolute file paths if preferred by starting with a \"/\".\n\nWhen specifying only a file path, the complete content of the file is assigned to the given key.\nYou always need a key to assign the !include command to.\nHowever, you cannot combine the value returned from !include with other values in the same key.\nThus, the following combinations do not work:\n\n###### caveats.yml\n\n```\n!include \"file.yaml\" # no key assigned\n\nKey:\n Some: OtherItem\n !include \"file.yaml\" # cannot join with other named items\n\nList:\n - an: entry\n !include \"file.yaml\" # cannot directly join with list items, even if !include returns a list\n```\n\n#### Integrate specific nodes of YAML files\n\nInstead of including *all* content in the included file, you may also pick a specific node within that file.\nFor this use `!include [<relative/path/to/file.yaml>, Path:To:Field:In:Yaml]`.\nHere, `:` is used in the node-specifying string to select a sequence of nodes to follow - with custom depth.\nConsider the following two files:\n\n###### file_to_be_included.yaml\n\n```yaml\nSet1:\n Subset1:\n Key: Value\nSet2:\n OtherKey: OtherValue\n```\n\n###### including_file.yaml\n\n```yaml\n- Type: MyAgentWithInputs\n Id: 1\n Attributes: !include_node [ file_to_be_included.yaml, Set1:Subset1 ]\n```\n\nCompiling \"including_file.yaml\" results in\n\n###### resulting_file.yaml\n\n```yaml\n- Type: MyAgentWithInputs\n Id: 1\n Attributes:\n Key: Value\n```\n\n#### Load multiple files\n\nUsing wildcards in the given path (e.g. \"path/to/many/*.yaml\") will lead to loading multiple files and assigning their\ncontent to the same key.\nYou can make use of this feature with or without specifying a node selector.\nHowever, the elements to be joined across multiple files must be lists.\nThese lists are then concatenated into a single list and then assigned to the key in the file calling !include.\nThis feature is especially useful for Contracts: You can split the Contracts list into several files and place them in a\nseparate folder.\nThen use !include to re-integrate them into your configuration. An example:\n\n###### my_contract1.yaml\n\n```\nContracts:\n - ContractA\n - ContractB\n```\n\n###### my_contract2.yaml\n\n```\nContracts:\n - ContractC\n - ContractD\n - ContractE\n```\n\n###### including_file.yaml\n\n```\nContracts: [!include \"my_contract*.yaml\", \"Contracts\"]\n```\n\nresults in\n\n###### result.yaml\n\n```\nContracts:\n - ContractA\n - ContractB\n - ContractC\n - ContractD\n - ContractE\n```\n\n#### Ignoring files\n\nFiles that have their name start with \"IGNORE_\" are not included with the !include command.\nYou will see a debug output to notify you that the file was ignored.\nUse this to temporarily take files out ouf your configuration without deleting or moving them.\n\n## Read FAME results\n\nTakes an output file in protobuf format of FAME-based applications and converts it into files in CSV format.\nAn individual file for each type of Agent is created in a folder named after the protobuf input file.\nCall structure:\n\n convertFameResults -f <./path/to/protobuf_file.pb>\n\nYou may also specify any of the following arguments:\n\n| Command | Action |\n|-----------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|\n| `-l` or `--log` <option> | Sets the logging level. Default is `WARNING`. Options are `DEBUG`, `INFO`, `WARNING`, `ERROR`, `CRITICAL`. |\n| `-lf` or `--logfile` <file> | Sets the logging file. Default is `None`. If `None` is provided, all logs get only printed to the console. |\n| `-a` or `--agents` <list-of-agents> | If specified, only a subset of agents is extracted from the protobuf file. Default is to extract all agents. |\n| `-o` or `--output` | Sets the path to where the generated output files are written to. If not specified, the folder's name is derived from the input file's name. Folder will be created if it does not exist. |\n| `-se` or `--single-export` | Enables export of individual agents to individual files, when present. If not present (the default) one file per `AgentType` is created. |\n| `-m` or `--memory-saving` | When specified, reduces memory usage profile at the cost of runtime. Use only when necessary. |\n| `-cc` or `--complex-column` <option> | Defines how to deal with complex indexed output columns (if any). `IGNORE` ignores complex columns. `SPLIT` creates a separate file for each complex indexed output column. |\n| `-t` or `--time` <option> | Option to define conversion of time steps to given format (default=`UTC`) by `-t/--time {UTC, INT, FAME}` |\n| `--input-recovery` or `--no-input-recovery` | If True, all input data are recovered in addition to the outputs (default=False). |\n| `-mt` or `--merge-times` <list-of-parameters> | Option to merge `TimeSteps` of a certain range of steps in the output files to associate multiple time steps with a common logical time in your simulation and reduce number of lines in output files |\n\nThe option `--merge-times` requires exactly three integer arguments separated by spaces:\n\n| Position | Name | Meaning |\n|----------|--------------|------------------------------------------------------------------------------------------|\n| First | Focal point | TimeStep on which `steps-before` earlier and `steps-after` later TimeSteps are merged on |\n| Second | Steps before | Range of TimeSteps before the `focal-point` they get merged to, must be Zero or positive |\n| Third | Steps after | Range of TimeSteps after the `focal-point` they get merged to, must be Zero or positive |\n\n\nThis could look as follows:\n\n convertFameResults -f <./path/to/protobuf_file.pb> -l debug -lf <path/to/output.log> -a AgentType1 AgentType2 -o myCsvFolder -m -cc SPLIT --merge-times 0 1799 1800\n\nMake sure that in the range of time steps you specify for merging, there is only one value per column in the merged time range.\nIf multiple values per column are merged values will get concatenated and might yield unexpected results.\n\nYou may also call the conversion script from any Python script with:\n\n```python\nfrom fameio.scripts.convert_results import Options, run as convert_results\n\nconvert_results({Options.FILE: \"./path/to/protobuf_file.pb\"})\n```\n\nSimilar to the console call you may also specify custom run config arguments and add it in a dictionary to the function\ncall.\n\n```python\nfrom fameio.scripts.convert_results import Options, run as convert_results\n\nrun_config = {Options.FILE: \"./path/to/protobuf_file.pb\",\n Options.LOG_LEVEL: \"info\",\n Options.LOG_FILE: \"scenario.log\",\n Options.OUTPUT: \"Output\",\n Options.AGENT_LIST: ['AgentType1', 'AgentType2'],\n Options.MEMORY_SAVING: False,\n Options.SINGLE_AGENT_EXPORT: False,\n Options.RESOLVE_COMPLEX_FIELD: \"SPLIT\",\n Options.TIME: \"INT\",\n Options.TIME_MERGING: {},\n }\n\nconvert_results(run_config)\n```\n\nYou can also use the associated argument parser, to extract the run_config dynamically from a string:\n\n```python\nfrom fameio.scripts.convert_results import Options, run as convert_results\nfrom fameio.cli.convert_results import handle_args\n\nmy_defaults = {Options.FILE: \"./path/to/protobuf_file.pb\",\n Options.LOG_LEVEL: \"info\",\n Options.LOG_FILE: \"scenario.log\",\n Options.OUTPUT: \"Output\",\n Options.AGENT_LIST: ['AgentType1', 'AgentType2'],\n Options.MEMORY_SAVING: False,\n Options.SINGLE_AGENT_EXPORT: False,\n Options.RESOLVE_COMPLEX_FIELD: \"SPLIT\",\n Options.TIME: \"INT\",\n Options.TIME_MERGING: {},\n }\nmy_arg_string = ['-f', 'my/other/scenario.yaml', '-l', 'error']\n\nrun_config = handle_args(my_arg_string, my_defaults)\nconvert_results(run_config)\n```\n\n## Cite FAME-Io\n\nIf you use FAME-Io for academic work, please cite as follows.\n\nBibtex entry:\n\n```\n@article{fameio2023joss,\n author = {Felix Nitsch and Christoph Schimeczek and Ulrich Frey and Benjamin Fuchs},\n title = {FAME-Io: Configuration tools for complex agent-based simulations},\n journal = {Journal of Open Source Software},\n year = {2023},\n doi = {doi: https://doi.org/10.21105/joss.04958}\n}\n```\n\n## Available Support\n\nThis is a purely scientific project by (at the moment) one research group.\nThus, there is no paid technical support available.\nHowever, we will give our best to answer your questions and provide support.\n\nIf you experience any trouble with FAME-Io, you may contact the developers via [fame@dlr.de](mailto:fame@dlr.de).\nPlease report bugs and make feature requests by filing issues following the provided templates (see\nalso [Contribute](CONTRIBUTING.md)).\nFor substantial enhancements, we recommend that you contact us via [fame@dlr.de](mailto:fame@dlr.de) for working\ntogether on the code in common projects or towards common publications and thus further develop FAME-Io.\n",
"bugtrack_url": null,
"license": "Apache-2.0",
"summary": "Tools for input preparation and output digestion of FAME models",
"version": "3.0.0",
"project_urls": {
"Homepage": "https://gitlab.com/fame-framework/wiki/-/wikis/home",
"Repository": "https://gitlab.com/fame-framework/fame-io/"
},
"split_keywords": [
"fame",
" fameio",
" agent-based modelling",
" energy systems"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "4b58a016fe5704163bb9e77dc4b1c57560fcd34cb8001fcff24b9f92a7ae57e5",
"md5": "a8aa2c98e4fd0a7d24a3e20b87a67603",
"sha256": "2687dba5017ea43cf9d717d20a70534106b6be3f232bc6d40da6351e2244f3fe"
},
"downloads": -1,
"filename": "fameio-3.0.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "a8aa2c98e4fd0a7d24a3e20b87a67603",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": "<4.0,>=3.9",
"size": 98879,
"upload_time": "2024-12-02T14:56:51",
"upload_time_iso_8601": "2024-12-02T14:56:51.775335Z",
"url": "https://files.pythonhosted.org/packages/4b/58/a016fe5704163bb9e77dc4b1c57560fcd34cb8001fcff24b9f92a7ae57e5/fameio-3.0.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "9ebd99a9c563e6cb22b6c5921da07b330a473943864aebbcaf780acf60313fd9",
"md5": "1016fb437c3f731040607a5a6472d6e1",
"sha256": "11057f2ebe703ded5967d67bebbab1c69d1264d48e5fe85e4e184a1ded6b9172"
},
"downloads": -1,
"filename": "fameio-3.0.0.tar.gz",
"has_sig": false,
"md5_digest": "1016fb437c3f731040607a5a6472d6e1",
"packagetype": "sdist",
"python_version": "source",
"requires_python": "<4.0,>=3.9",
"size": 80748,
"upload_time": "2024-12-02T14:56:53",
"upload_time_iso_8601": "2024-12-02T14:56:53.684846Z",
"url": "https://files.pythonhosted.org/packages/9e/bd/99a9c563e6cb22b6c5921da07b330a473943864aebbcaf780acf60313fd9/fameio-3.0.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-12-02 14:56:53",
"github": false,
"gitlab": true,
"bitbucket": false,
"codeberg": false,
"gitlab_user": "fame-framework",
"gitlab_project": "wiki",
"lcname": "fameio"
}
JSON