fameio 3.4.0__py3-none-any.whl → 3.5.1__py3-none-any.whl

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (70) hide show
  1. fameio/__init__.py +2 -1
  2. fameio/cli/__init__.py +2 -0
  3. fameio/cli/convert_results.py +8 -0
  4. fameio/cli/make_config.py +2 -0
  5. fameio/cli/options.py +4 -0
  6. fameio/cli/parser.py +17 -1
  7. fameio/cli/reformat.py +2 -0
  8. fameio/input/__init__.py +2 -1
  9. fameio/input/loader/__init__.py +1 -0
  10. fameio/input/loader/controller.py +20 -6
  11. fameio/input/loader/loader.py +2 -0
  12. fameio/input/metadata.py +2 -0
  13. fameio/input/resolver.py +2 -0
  14. fameio/input/scenario/__init__.py +2 -0
  15. fameio/input/scenario/agent.py +2 -0
  16. fameio/input/scenario/attribute.py +2 -0
  17. fameio/input/scenario/contract.py +2 -0
  18. fameio/input/scenario/exception.py +2 -0
  19. fameio/input/scenario/fameiofactory.py +2 -0
  20. fameio/input/scenario/generalproperties.py +2 -0
  21. fameio/input/scenario/scenario.py +5 -3
  22. fameio/input/scenario/stringset.py +2 -0
  23. fameio/input/schema/__init__.py +1 -0
  24. fameio/input/schema/agenttype.py +2 -0
  25. fameio/input/schema/attribute.py +2 -0
  26. fameio/input/schema/java_packages.py +2 -0
  27. fameio/input/schema/schema.py +8 -3
  28. fameio/input/validator.py +2 -0
  29. fameio/input/writer.py +16 -0
  30. fameio/logs.py +2 -1
  31. fameio/output/__init__.py +1 -0
  32. fameio/output/agent_type.py +24 -12
  33. fameio/output/conversion.py +2 -0
  34. fameio/output/csv_writer.py +10 -36
  35. fameio/output/data_transformer.py +2 -0
  36. fameio/output/execution_dao.py +5 -0
  37. fameio/output/files.py +55 -0
  38. fameio/output/input_dao.py +59 -15
  39. fameio/output/metadata/__init__.py +10 -0
  40. fameio/output/metadata/compiler.py +75 -0
  41. fameio/output/metadata/json_writer.py +36 -0
  42. fameio/output/metadata/locator.py +242 -0
  43. fameio/output/metadata/oeo_template.py +93 -0
  44. fameio/output/metadata/template_reader.py +65 -0
  45. fameio/output/output_dao.py +2 -0
  46. fameio/output/reader.py +1 -0
  47. fameio/output/yaml_writer.py +3 -1
  48. fameio/scripts/REUSE.toml +6 -0
  49. fameio/scripts/__init__.py +4 -0
  50. fameio/scripts/convert_results.py +46 -12
  51. fameio/scripts/exception.py +1 -0
  52. fameio/scripts/reformat.py +25 -1
  53. fameio/series.py +16 -7
  54. fameio/time.py +1 -0
  55. fameio/tools.py +1 -0
  56. fameio-3.5.1.dist-info/LICENSES/CC-BY-ND-4.0.txt +392 -0
  57. fameio-3.5.1.dist-info/METADATA +99 -0
  58. fameio-3.5.1.dist-info/RECORD +65 -0
  59. fameio/scripts/__init__.py.license +0 -3
  60. fameio/scripts/convert_results.py.license +0 -3
  61. fameio/scripts/make_config.py.license +0 -3
  62. fameio/scripts/reformat.py.license +0 -3
  63. fameio-3.4.0.dist-info/METADATA +0 -990
  64. fameio-3.4.0.dist-info/RECORD +0 -60
  65. {fameio-3.4.0.dist-info → fameio-3.5.1.dist-info}/LICENSE.txt +0 -0
  66. {fameio-3.4.0.dist-info → fameio-3.5.1.dist-info}/LICENSES/Apache-2.0.txt +0 -0
  67. {fameio-3.4.0.dist-info → fameio-3.5.1.dist-info}/LICENSES/CC-BY-4.0.txt +0 -0
  68. {fameio-3.4.0.dist-info → fameio-3.5.1.dist-info}/LICENSES/CC0-1.0.txt +0 -0
  69. {fameio-3.4.0.dist-info → fameio-3.5.1.dist-info}/WHEEL +0 -0
  70. {fameio-3.4.0.dist-info → fameio-3.5.1.dist-info}/entry_points.txt +0 -0
@@ -1,990 +0,0 @@
1
- Metadata-Version: 2.3
2
- Name: fameio
3
- Version: 3.4.0
4
- Summary: Tools for input preparation and output digestion of FAME models
5
- License: Apache-2.0
6
- Keywords: FAME,fameio,agent-based modelling,energy systems
7
- Author: Felix Nitsch
8
- Author-email: fame@dlr.de
9
- Maintainer: Felix Nitsch
10
- Maintainer-email: fame@dlr.de
11
- Requires-Python: >=3.9,<4.0
12
- Classifier: Development Status :: 5 - Production/Stable
13
- Classifier: Environment :: Console
14
- Classifier: Intended Audience :: Science/Research
15
- Classifier: License :: OSI Approved :: Apache Software License
16
- Classifier: Operating System :: OS Independent
17
- Classifier: Programming Language :: Python :: 3
18
- Classifier: Programming Language :: Python :: 3.9
19
- Classifier: Programming Language :: Python :: 3.10
20
- Classifier: Programming Language :: Python :: 3.11
21
- Classifier: Programming Language :: Python :: 3.12
22
- Classifier: Programming Language :: Python :: 3.13
23
- Classifier: Topic :: Scientific/Engineering
24
- Requires-Dist: fameprotobuf (>=2.0.2,<3.0)
25
- Requires-Dist: pandas (>=1.0,<3.0)
26
- Requires-Dist: pyyaml (>=6.0,<7.0)
27
- Project-URL: Changelog, https://gitlab.com/fame-framework/fame-io/-/blob/main/CHANGELOG.md
28
- Project-URL: Homepage, https://helmholtz.software/software/fame
29
- Project-URL: Issue Tracking, https://gitlab.com/fame-framework/fame-io/-/issues
30
- Project-URL: Repository, https://gitlab.com/fame-framework/fame-io
31
- Description-Content-Type: text/markdown
32
-
33
- <!-- SPDX-FileCopyrightText: 2025 German Aerospace Center <fame@dlr.de>
34
-
35
- SPDX-License-Identifier: Apache-2.0 -->
36
-
37
- | | |
38
- |---------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
39
- | **Package** | ![PyPI - Python Version](https://img.shields.io/pypi/pyversions/fameio) ![PyPI - Version](https://img.shields.io/pypi/v/fameio) [![PyPI license](https://img.shields.io/pypi/l/fameio.svg)](https://badge.fury.io/py/fameio) [![REUSE status](https://api.reuse.software/badge/gitlab.com/fame-framework/fame-io)](https://api.reuse.software/info/gitlab.com/fame-framework/fame-io) |
40
- | **Test** | [![pipeline status](https://gitlab.com/fame-framework/fame-io/badges/main/pipeline.svg)](https://gitlab.com/fame-framework/fame-io/commits/main) [![coverage report](https://gitlab.com/fame-framework/fame-io/badges/main/coverage.svg)](https://gitlab.com/fame-framework/fame-io/-/commits/main) |
41
- | **Activity** | ![GitLab last commit](https://img.shields.io/gitlab/last-commit/fame-framework%2Ffame-io) ![GitLab closed issues by-label](https://img.shields.io/gitlab/issues/closed/fame-framework%2Ffame-io) |
42
- | **Style** | [![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black) [![log style - common changelog](https://img.shields.io/badge/log_style-common_changelog-blue)](https://common-changelog.org/) ![Static Badge](https://img.shields.io/badge/type%20checked-mypy-039dfc) [![linting: pylint](https://img.shields.io/badge/linting-pylint-green)](https://github.com/pylint-dev/pylint) |
43
- | **Reference** | [![JOSS](https://joss.theoj.org/papers/10.21105/joss.04958/status.svg)](https://doi.org/10.21105/joss.04958) [![Zenodo](https://zenodo.org/badge/DOI/10.5281/zenodo.4314337.svg)](https://doi.org/10.5281/zenodo.4314337) |
44
-
45
- # FAME-Io
46
-
47
- *Tools for input preparation and output digestion of FAME models*
48
-
49
- FAME-Io compiles input for FAME models in protobuf format and extracts model outputs to human-readable files.
50
- Please visit the [FAME-Wiki](https://gitlab.com/fame-framework/wiki/-/wikis/home) to get an explanation of FAME and its components.
51
-
52
- # Installation
53
-
54
- We recommend installing `fameio` using PyPI:
55
-
56
- pip install fameio
57
-
58
- You may also use `pipx`. For detailed information please refer to the official `pipx` [documentation](https://github.com/pypa/pipx).
59
-
60
- pipx install fameio
61
-
62
- `fameio` is currently developed and tested for Python 3.9 or higher.
63
- See the `pyproject.toml` for a complete listing of dependencies.
64
-
65
- # Usage
66
-
67
- FAME-Io currently offers two main scripts `makeFameRunConfig` and `convertFameResults`, plus a helper script `reformatTimeSeries`
68
- All are automatically installed with the package.
69
- The first one creates a protobuf file for FAME applications using YAML definition files and CSV files.
70
- The second one reads output files from FAME applications in protobuf format and converts them to CSV files.
71
- The third script reformats time series CSV files to FAME format.
72
-
73
- You may use the [example data](https://gitlab.com/dlr-ve/esy/amiris/examples) provided for the [AMIRIS](https://gitlab.com/dlr-ve/esy/amiris/amiris) model which can be used to simulate electricity markets in [Germany](https://gitlab.com/dlr-ve/esy/amiris/examples/-/tree/main/Germany2019), [Austria](https://gitlab.com/dlr-ve/esy/amiris/examples/-/tree/main/Austria2019), and a simple [proof-of-concept model](https://gitlab.com/dlr-ve/esy/amiris/examples/-/tree/main/Simple).
74
-
75
- ## Make a FAME run configuration
76
-
77
- Digests configuration files in YAML format, combines them with CSV data files and creates a single input file for FAME applications in protobuf format.
78
- Call structure:
79
-
80
- makeFameRunConfig -f <path/to/scenario.yaml>
81
-
82
- You may also specify any of the following arguments:
83
-
84
- | Command | Action |
85
- |------------------------|------------------------------------------------------------------------------------------------------------------------------------------|
86
- | `-l` or `--log` | Sets the logging level. Default is `info`. Options are `debug`, `info`, `warning`, `warn`, `error`, `critical`. |
87
- | `-lf` or `--logfile` | Sets the logging file. Default is `None`. If `None` is provided, all logs get only printed to the console. |
88
- | `-o` or `--output` | Sets the path of the compiled protobuf output file. Default is `config.pb`. |
89
- | `-enc` or `--encoding` | Sets the encoding of all yaml files to the given one (e.g. 'utf8' or 'cp1252'. Default is `None`, i.e. your operating system's standard. |
90
-
91
- This could look as follows:
92
-
93
- makeFameRunConfig -f <path/to/scenario.yaml> -l debug -lf <path/to/scenario.log> -o <path/to/config.pb>
94
-
95
- You may also call the configuration builder from any Python script with
96
-
97
- ```python
98
- from fameio.scripts.make_config import Options, run as make_config
99
-
100
- make_config({Options.FILE: "path/to/scenario.yaml", })
101
- ```
102
-
103
- Similar to the console call you may also specify custom run config arguments and add it in a dictionary to the function
104
- call.
105
-
106
- ```python
107
- from fameio.scripts.make_config import Options, run as make_config
108
-
109
- run_config = {Options.FILE: "path/to/scenario.yaml",
110
- Options.LOG_LEVEL: "info",
111
- Options.OUTPUT: "output.pb",
112
- Options.LOG_FILE: "scenario.log",
113
- }
114
-
115
- make_config(run_config)
116
- ```
117
-
118
- You can also use the associated argument parser, to extract the run_config dynamically from a string:
119
-
120
- ```python
121
- from fameio.scripts.make_config import Options, run as make_config
122
- from fameio.cli.make_config import handle_args
123
-
124
- my_defaults = {Options.FILE: "path/to/scenario.yaml",
125
- Options.LOG_LEVEL: "info",
126
- Options.OUTPUT: "output.pb",
127
- Options.LOG_FILE: "scenario.log",
128
- }
129
- my_arg_string = ['-f', 'my/other/scenario.yaml', '-l', 'error']
130
-
131
- run_config = handle_args(my_arg_string, my_defaults)
132
- make_config(run_config)
133
- ```
134
-
135
- ### Scenario YAML
136
-
137
- The "scenario.yaml" file contains all configuration options for a FAME-based simulation.
138
- It consists of the sections `Schema`, `GeneralProperties`, `Agents` and `Contracts`, and the optional
139
- section `StringSets`.
140
- All of them are described below.
141
-
142
- #### Schema
143
-
144
- The Schema describes a model's components such as its types of agents, their inputs, what data they exchange, etc.
145
- It is also used to validate the model inputs provided in the `scenario.yaml`.
146
- Since the Schema is valid until the model itself is changed, it is recommended to defined it in a separate file and
147
- include the file here.
148
-
149
- Currently, the schema specifies:
150
-
151
- * which type of Agents can be created
152
- * what type of input attributes an Agent uses
153
- * what type of Products an Agent can send in Contracts, and
154
- * the names of the Java packages for the classes corresponding to Agents, DataItems and Portables.
155
-
156
- The Schema consists of the sections `JavaPackages` and `AgentTypes`.
157
-
158
- ##### JavaPackages
159
-
160
- This section defines the name of the Java packages in which the model code is located.
161
- A similar data set was formerly specified in the `fameSetup.yaml`, but is now specified in the schema.
162
- Each of the three sections `Agents`, `DataItems`, and `Portables` contain a list of fully qualified java package names
163
- of your model's classes.
164
- Package names can occur in multiple lists and may overlap.
165
- It is not necessary (but possible) to specify the nearest enclosing package for each Agent, DataItem or Portable.
166
- Specifying any super-package will also work.
167
- Also, package names occur on multiple lists for Agent, DataItem or Portable.
168
-
169
- For example, for a project with all its
170
-
171
- * Agent-derived java classes located in packages below the package named "agents",
172
- * DataItem implementation classes in a subpackage named "msg",
173
- * Portable implementation classes in a subpackages named "portableItems" and "otherPortables",
174
-
175
- the corresponding section in the schema would look like this:
176
-
177
- ```yaml
178
- JavaPackages:
179
- Agents:
180
- - "agents"
181
- DataItems:
182
- - "msg"
183
- Portables:
184
- - "portableItems"
185
- - "otherPortables"
186
- ```
187
-
188
- One can leave out the `DataItems` specifications, but `Agents` and `Portables` are required and must not be empty.
189
-
190
- ##### AgentTypes
191
-
192
- Here, each type of agent that can be created in your FAME-based application is listed, its attributes and its available
193
- Products for Contracts.
194
- The structure of this section
195
-
196
- ```yaml
197
- AgentTypes:
198
- MyAgentType:
199
- Attributes:
200
- MyAttribute:
201
- ...
202
- MyOtherAttribute:
203
- ...
204
- Products: [ 'Product1', 'Product2', 'Product3' ]
205
- Outputs: [ 'Column1', 'Column2', 'Column3' ]
206
- Metadata:
207
- Some: "Dict with Metadata that you would like to add"
208
- MyOtherAgentWithoutProductsOrAttributes:
209
- ```
210
-
211
- * `MyAgentType` Java's simple class name of the Agent type
212
- * `Attributes` indicates that beginning of the attribute definition section for this Agent type
213
- * `MyAttribute` Name of an attribute as specified in the corresponding Java source code of this Agent type (annotated
214
- with "@Input")
215
- * `MyOtherAttribute` Name of another attribute derived from Java source code
216
- * `Products` list or dictionary of Products that this Agent can send in Contracts; derived from Java source code of this
217
- Agent type (annotated with "@Product")
218
- * `Outputs` list or dictionary of Output columns that this Agent can write to; derived from Java source code of this
219
- Agent type (annotated with "@Output")
220
- * `Metadata` dictionary with any content that is assigned to this Agent type as additional information
221
- * `MyOtherAgentWithoutProductsOrAttributes` an Agent type that requires neither Attributes nor Products
222
-
223
- Attributes, Products, Outputs and Metadata are optional - there may be useful Agents that require none of them.
224
- Products and Outputs can both be lists of Strings, or dictionaries with additional Metadata.
225
- For example, you could write the above in the following way:
226
-
227
- ```yaml
228
- Products:
229
- Product1:
230
- Metadata:
231
- Any: "information you would like to add to Product1 using a dictionary form"
232
- Product2:
233
- Product3:
234
- Outputs:
235
- Column1:
236
- Column2:
237
- ThisEntry: "is ignored, as it is not below the keyword: 'Metadata'"
238
- Metadata:
239
- My: "Metadata"
240
- That: "will be saved to Column2"
241
- Column3:
242
- ```
243
-
244
- Here, "Product1" and "Column2" have additional, optional Metadata assigned to them (using the keyword "Metadata").
245
- The other Products and Columns have no metadata assigned to them - which is also ok.
246
-
247
- In the AgentType definition example above attribute definition was not shown explicitly (indicated by `...`).
248
- The next example provides details on how to define an attribute:
249
-
250
- ```yaml
251
- MySimpleAttribute:
252
- AttributeType: enum
253
- Mandatory: true
254
- List: false
255
- Values: [ 'AllowedValue1', 'AllowedValue2' ]
256
- Default: 'AllowedValue1'
257
- Help: 'My help text'
258
- Metadata:
259
- Go: "here"
260
-
261
- MyComplexAttribute:
262
- AttributeType: block
263
- NestedAttributes:
264
- InnerAttributeA:
265
- AttributeType: integer
266
- Values:
267
- 1:
268
- Metadata:
269
- Explain: "1 is a allowed value"
270
- 2:
271
- Metadata:
272
- Comment: "2 is also allowed, but consider using 1"
273
- InnerAttributeB:
274
- AttributeType: double
275
- ```
276
-
277
- * `MySimpleAttribute`, `MyDoubleList`, `MyComplexAttribute` Names of the attributes as specified in the Java enum
278
- annotated with "@Input"
279
- * `AttributeType` (required) data type of the attribute; see options in table below
280
- * `Mandatory` (optional - true by default) if true: the attribute is required for this agent and validation will fail if
281
- the attribute is missing in the scenario **and** no default is provided
282
- * `List` (optional - false by default)
283
- * `AttributeType: time_series` cannot be true
284
- * `AttributeType: block`
285
- * if true: any nested element in the scenario must be part of a list element and thus can appear multiple times
286
- * if false: any nested element in the scenario can only appear once
287
- * any other AttributeType: the attribute is interpreted as list, i.e. multiple values can be assigned to this
288
- attribute in the scenario
289
- * `NestedAttributes` (required only if `AttributeType: block`, otherwise disallowed) starts an inner Attribute
290
- definition block - defined Attributes are sub-elements of `MyComplexAttribute`
291
- * `Values` (optional - None by default):
292
- * if present, defines a list or dictionary of allowed values for this attribute
293
- * if a dictionary is used, individual Metadata can be assigned to each allowed value using the `Metadata` keyword
294
- * `Default` (optional - None by default):
295
- * if present, defines a default value to be used if the scenario does not specify one
296
- * must match one of the entries in `Values` in case those are defined
297
- * can be a list if the attribute is a list
298
- * `Help` (optional - None by default): if present, defines a help text for your Attribute
299
- * `Metadata` (optional - None by default): if present, defines additional metadata assigned to the Attribute
300
-
301
- | AttributeType | value |
302
- |---------------|-------------------------------------------------------------------------------------------------------------------------|
303
- | `integer` | a 32-bit integer value |
304
- | `double` | a 64-bit floating-point value (integers also allowed) |
305
- | `long` | a 64-bit integer value |
306
- | `time_stamp` | either a FAME time stamp string or 64-bit integer value |
307
- | `string` | any string |
308
- | `string_set` | a string from a set of allowed `Values` defined in `StringSet` section in `scenario` |
309
- | `enum` | a string from a set of allowed `Values` defined in `schema` |
310
- | `time_series` | either a path to a .csv-file or a single 64-bit floating-point value; does not support `List: true` |
311
- | `block` | this attribute has no value of its own but hosts a group of nested Attributes; implies `NestedAttributes` to be defined |
312
-
313
- #### GeneralProperties
314
-
315
- Specifies FAME-specific properties of the simulation. Structure:
316
-
317
- ```yaml
318
- GeneralProperties:
319
- RunId: 1
320
- Simulation:
321
- StartTime: 2011-12-31_23:58:00
322
- StopTime: 2012-12-30_23:58:00
323
- RandomSeed: 1
324
- ```
325
-
326
- Parameters:
327
-
328
- * `RunId` an ID that can be given to the simulation; use at your discretion
329
- * `StartTime` time stamp in the format YYYY-MM-DD_hh:mm:ss; first moment of the simulation.
330
- * `StopTime` time stamp in the format YYYY-MM-DD_hh:mm:ss; last moment of the simulation - i.e. simulation terminates
331
- after passing that time stamp
332
- * `RandomSeed` seed to initialise random number generation; each value leads to a unique series of random numbers.
333
-
334
- #### Agents
335
-
336
- Specifies all Agents to be created in the simulation in a list. Each Agent has its own entry.
337
- Structure:
338
-
339
- ```yaml
340
- Agents:
341
- - Type: MyAgentWithInputs
342
- Id: 1
343
- Attributes:
344
- MyEnum: SAME_SHARES
345
- MyInteger: 2
346
- MyDouble: 4.2
347
- MyTimeSeries: "./path/to/time_series.csv"
348
- Metadata:
349
- Can: "also be assigned"
350
-
351
- - Type: MyAgentWithoutInputs
352
- Id: 2
353
- ```
354
-
355
- Agent Parameters:
356
-
357
- * `Type` Mandatory; Java's simple class name of the agent to be created
358
- * `Id` Mandatory; simulation-unique id of this agent; if two agents have the same ID, the configuration process will
359
- stop.
360
- * `Attributes` Optional; if the agent has any attributes, specify them here in the format "AttributeName: value"; please
361
- see attribute table above
362
- * `Metadata` Optional; can be assigned to each instance of an Agent, as well as to each of its Attributes
363
- * `Ext` Optional; Reserved key for parameters not used by fameio but its extensions, e.g., FAME-Gui
364
-
365
- A warning is logged for any other key at this level.
366
- The specified `Attributes` for each agent must match the specified `Attributes` options in the linked Schema (see
367
- above).
368
- For better structure and readability of the `scenario.yaml`, `Attributes` may also be specified in a nested way as
369
- demonstrated below.
370
-
371
- ```yaml
372
- Agents:
373
- - Type: MyAgentWithInputs
374
- Id: 1
375
- Attributes:
376
- Parent:
377
- MyEnum: SAME_SHARES
378
- MyInteger: 2
379
- Parent2:
380
- MyDouble: 4.2
381
- Child:
382
- MyTimeSeries: "./path/to/time_series.csv"
383
- ```
384
-
385
- In case Attributes are defined with `List: true` option, lists are assigned to an Attribute or Group:
386
-
387
- ```yaml
388
- Attributes:
389
- MyDoubleList: [ 5.2, 4.5, 7, 9.9 ]
390
- MyListGroup:
391
- - IntValueA: 5
392
- IntValueB: 42
393
- - IntValueA: 7
394
- IntValueB: 100
395
- ```
396
-
397
- Here, `MyDoubleList` and `MyListGroup` need to specify `List: true` in the corresponding Schema.
398
- The shorter `[]`-notation was used to assign a list of floating-point values to `MyDoubleList`.
399
- Nested items `IntValueA` and `IntValueB` of `MyListGroup` are assigned within a list, allowing the specification of
400
- these nested items several times.
401
-
402
- ##### Attribute Metadata
403
-
404
- Metadata can be assigned to any value, list item, or superstructure.
405
- To assign Metadata to a primitive value, create a dictionary from it, set the actual value with the inner
406
- keyword `Value` and add the keyword `Metadata` like this:
407
-
408
- ```yaml
409
- ValueWithoutMetadata: 1
410
- SameValueWithMetadata:
411
- Value: 1
412
- Metadata: # describe `SameValueWithMetadata` herein
413
- ```
414
-
415
- You can assign Metadata to a list of primitive values using the keyword `Values` like this:
416
-
417
- ```yaml
418
- ValueListWithoutMetadata: [ 1,2,3 ]
419
- SameValueListWithListMetadata:
420
- Values: [ 1,2,3 ]
421
- Metadata: # describe the whole list of values with Metadata here
422
- ```
423
-
424
- or specify Metadata for each (or just some) value individually, like this:
425
-
426
- ```yaml
427
- ValueListWithoutMetadata: [ 1,2,3 ]
428
- SameValueListWithMetadataAtEachElement:
429
- - Value: 1
430
- Metadata: # describe this specific value "1" with Metadata here
431
- - Value: 2 # this value has no Metadata attached, but you can still use the keyword `Value`
432
- - 3 # or use in the actual directly since this value has no Metadata anyway
433
- ```
434
-
435
- or assign Metadata to both the list and any of its list entries, like this:
436
-
437
- ```yaml
438
- ValueListWithoutMetadata: [ 1,2,3 ]
439
- SameValueListWithAllMetadata:
440
- Metadata: # Recommendation: place the Metadata of the list first if the list of values is extensive, as in this case
441
- Values:
442
- - Value: 1
443
- Metadata: # describe this specific value "1" with Metadata here
444
- - Value: 2
445
- Metadata: # describe this specific value "2" with Metadata here
446
- - Value: 3
447
- Metadata: # describe this specific value "3" with Metadata here
448
- ```
449
-
450
- You can assign Metadata directly to a nested element by adding the Metadata keyword:
451
-
452
- ```yaml
453
- NestedItemWithoutMetadata:
454
- A: 1
455
- B: 2
456
- SameNestedItemWithMetadata:
457
- A: 1
458
- B: 2
459
- Metadata: # These Metadata describe `SameNestedItemWithMetadata`
460
- ```
461
-
462
- Similar to lists of values, you can assign Metadata to a list of nested elements using the `Values` keyword, like this:
463
-
464
- ```yaml
465
- ListOfNestedItemsWithoutMetadata:
466
- - A: 1
467
- B: 10
468
- - A: 2
469
- B: 20
470
- SameListOfNestedItemsWithGeneralMetadata:
471
- Values:
472
- - A: 1
473
- B: 10
474
- - A: 2
475
- B: 20
476
- Metadata: # These Metadata describe `SameListOfNestedItemsWithGeneralMetadata` as a whole
477
- ```
478
-
479
- and, similar to nested elements, you can assign Metadata directly to any list element, like this:
480
-
481
- ```yaml
482
- ListOfNestedItemsWithoutMetadata:
483
- - A: 1
484
- B: 10
485
- - A: 2
486
- B: 20
487
- SameListOfNestedItemsWithGeneralMetadata:
488
- - A: 1
489
- B: 10
490
- Metadata: # These Metadata describe the first list item
491
- - A: 2
492
- B: 20
493
- Metadata: # These Metadata describe the second list item
494
- ```
495
-
496
- Again, you may apply both variants and apply Metadata to the list and each of its items if you wish.
497
-
498
- #### Contracts
499
-
500
- Specifies all Contracts, i.e. repetitive bilateral transactions in between agents.
501
- Contracts are given as a list.
502
- We recommend moving Contracts to separate files and to use the `!include` command to integrate them in the scenario.
503
-
504
- ```yaml
505
- Contracts:
506
- - SenderId: 1
507
- ReceiverId: 2
508
- ProductName: ProductOfAgent_1
509
- FirstDeliveryTime: -25
510
- Every: 3600
511
- Metadata:
512
- Some: "additional information can go here"
513
-
514
- - SenderId: 2
515
- ReceiverId: 1
516
- ProductName: ProductOfAgent_2
517
- FirstDeliveryTime: -22
518
- Every: 1 hour
519
- Attributes:
520
- ProductAppendix: value
521
- TimeOffset: 42
522
- ```
523
-
524
- Contract Parameters:
525
-
526
- * `SenderId` unique ID of agent sending the product
527
- * `ReceiverId` unique ID of agent receiving the product
528
- * `ProductName` name of the product to be sent
529
- * `FirstDeliveryTime` first time of delivery in the format "seconds after the January 1st 2000, 00:00:00"
530
- * `Every` delay time in between deliveries; either an integer value in seconds, or a qualified time span in the format "<integer> <TimeUnit>(s)", where TimeUnit is one of "second", "minute", "hour", "day", "week", "month", "year" - with an options "s" at the end; mind that week, month, and year refer to fixed-length intervals of 168, 730, and 8760 hours.
531
- * `DeliveryIntervalInSteps` deprecated; delay time in between deliveries in seconds (use instead of `Every`)
532
- * `Metadata` can be assigned to add further helpful information about a Contract
533
- * `Attributes` can be set to include additional information as `int`, `float`, `enum`, or `dict` data types
534
-
535
- ##### Definition of Multiple Similar Contracts
536
-
537
- Often, scenarios contain multiple agents of similar type that also have similar chains of contracts.
538
- Therefore, FAME-Io supports a compact definition of multiple similar contracts.
539
- `SenderId` and `ReceiverId` can both be lists and support One-to-N, N-to-One and N-to-N relations like in the following
540
- example:
541
-
542
- ```yaml
543
- Contracts:
544
- # effectively 3 similar contracts (0 -> 11), (0 -> 12), (0 -> 13)
545
- # with otherwise identical ProductName, FirstDeliveryTime, and Every
546
- - SenderId: 0
547
- ReceiverId: [ 11, 12, 13 ]
548
- ProductName: MyOtherProduct
549
- FirstDeliveryTime: 100
550
- Every: 1 hour
551
-
552
- # effectively 3 similar contracts (1 -> 10), (2 -> 10), (3 -> 10)
553
- # with otherwise identical ProductName, FirstDeliveryTime, and Every
554
- - SenderId: [ 1, 2, 3 ]
555
- ReceiverId: 10
556
- ProductName: MyProduct
557
- FirstDeliveryTime: 100
558
- Every: 1 hour
559
-
560
- # effectively 3 similar contracts (1 -> 11), (2 -> 12), (3 -> 13)
561
- # with otherwise identical ProductName, FirstDeliveryTime, and Every
562
- - SenderId: [ 1, 2, 3 ]
563
- ReceiverId: [ 11, 12, 13 ]
564
- ProductName: MyThirdProduct
565
- FirstDeliveryTime: 100
566
- Every: 1 hour
567
- ```
568
-
569
- When combined with YAML anchors, the complexity of extensive contract chains can be reduced.
570
- The following example is equivalent to the previous one, enabling contracts to be quickly extended to a new group of agents:
571
-
572
- ```yaml
573
- Groups:
574
- - &agentList1: [ 1, 2, 3 ]
575
- - &agentList2: [ 11, 12, 13 ]
576
-
577
- Contracts:
578
- - SenderId: 0
579
- ReceiverId: *agentList2
580
- ProductName: MyOtherProduct
581
- FirstDeliveryTime: 100
582
- Every: 1 hour
583
-
584
- - SenderId: *agentList1
585
- ReceiverId: 10
586
- ProductName: MyProduct
587
- FirstDeliveryTime: 100
588
- Every: 1 hour
589
-
590
- - SenderId: *agentList1
591
- ReceiverId: *agentList2
592
- ProductName: MyThirdProduct
593
- FirstDeliveryTime: 100
594
- Every: 1 hour
595
- ```
596
-
597
- Lists can be nested in senders and receivers as follows:
598
-
599
- ```
600
- Groups:
601
- - SenderId: 42
602
- ReceiverId: [1, [2, 3], [4, [5]]]
603
- ProductName: AnImportantProduct
604
- FirstDeliveryTime: 101
605
- Every: 30 minutes
606
- ```
607
-
608
- This feature should not be overused, as nested lists can become messy and difficult to read.
609
- Special care should be taken when using it with an N-to-N mapping, as it will be difficult to check whether senders and receivers are matched correctly.
610
-
611
- #### StringSets
612
-
613
- This optional section defines values of type `string_set`.
614
- In contrast to `enum` values, which are **statically** defined in the `Schema`, `string_set` values can be **dynamically
615
- ** defined in this section.
616
- If an agent attribute is of type `string_set` and the attribute is set in the `scenario`, then
617
-
618
- 1. the section `StringSets` in the `scenario` must contain an entry named exactly like the attribute, and
619
- 2. the attribute value must be contained in the string set's `Values` declaration.
620
-
621
- For instance:
622
-
623
- In `schema`:
624
-
625
- ``` yaml
626
- AgentTypes:
627
- FuelsMarket:
628
- Attributes:
629
- FuelType:
630
- AttributeType: string_set
631
- ```
632
-
633
- In `scenario`:
634
-
635
- ``` yaml
636
- StringSets:
637
- FuelType:
638
- Values: ['OIL', 'HARD_COAL', 'LIGNITE']
639
-
640
- Agents:
641
- - Type: FuelsMarket
642
- Id: 1
643
- Attributes:
644
- FuelType: OIL
645
- ```
646
-
647
- Important: If different types of Agents shall refer to the same StringSet, their attributes in schema must have the
648
- **exact** same name.
649
-
650
- ### CSV files
651
-
652
- TIME_SERIES inputs are not directly fed into the Scenario YAML file.
653
- Instead, TIME_SERIES reference a CSV file that can be stored some place else.
654
- These CSV files follow a specific structure:
655
-
656
- * They should contain exactly two columns - any other columns are ignored. A warning is raised if more than two non-empty columns are detected.
657
- * The first column must be a time stamp in form `YYYY-MM-DD_hh:mm:ss` or a [FAME-Timestamp](https://gitlab.com/fame-framework/wiki/-/wikis/architecture/decisions/TimeStamp) integer value.
658
- * The second column must be a numerical value (either integer or floating-point)
659
- * The separator of the two columns is a semicolon
660
- * The data must **not** have headers, except for comments marked with `#`
661
-
662
- You may add comments using `#`.
663
- Exemplary content of a valid CSV file:
664
-
665
- # If you want an optional header, you must use a comment
666
- 2012-01-01_00:00:00;400
667
- 2013-01-01_00:00:00;720.5
668
- 2014-01-01_00:00:00;650
669
- 2015-01-01_00:00:00;99.27772
670
- 2016-01-01_00:00:00;42 # optional comment on this particular data point
671
- 2017-01-01_00:00:00;0.1
672
-
673
- Please refer also to the detailed article about `TimeStamps` in the [FAME-Wiki](https://gitlab.com/fame-framework/wiki/-/wikis/TimeStamp).
674
- For large CSV files (with more than 20,000 rows) we recommend using the integer representation of FAME-Timestamps in the first column (instead of text representation) to improve conversion speed.
675
- A warning will be raised for very large files (exceeding 50,000 rows) that require time stamp conversion.
676
- Use `reformatTimeSeries` to convert one or multiple timeseries CSV files into FAME format to improve conversion speed and avoid this warning.
677
-
678
- ### Split and join multiple YAML files
679
-
680
- The user may include other YAML files into a YAML file to divide the content across files as convenient.
681
- We explicitly recommend using this feature for the `Schema` and `Contracts` sections.
682
- Otherwise, the scenario.yaml may become crowded.
683
-
684
- #### Command: !Include
685
-
686
- To hint YAML to load the content of another file use `!include "path/relative/to/including/yaml/file.yml"`.
687
- You can concatenate !include commands and can use !include in the included file as well.
688
- The path to the included file is always relative to the file using the !include command.
689
- So with the following file structure
690
-
691
- ###### file-structure
692
-
693
- ```
694
- a.yaml
695
- folder/b.yaml
696
- folder/c.yaml
697
- folder/deeper_folder/d.yaml
698
- ```
699
-
700
- the following !include commands work
701
-
702
- ###### in a.yaml
703
-
704
- ```
705
- ToBe: !include "folder/b.yaml"
706
- OrNot: !include "folder/deeper_folder/d.yaml"
707
- ```
708
-
709
- ###### in b.yaml
710
-
711
- ```
712
- ThatIs: !include "c.yaml"
713
- TheQuestion: !include "deeper_folder/d.yaml"
714
- ```
715
-
716
- Provided that
717
-
718
- ###### in c.yaml
719
-
720
- ```
721
- Or: maybe
722
- ```
723
-
724
- ###### d.yaml
725
-
726
- ```
727
- not: "?"
728
- ```
729
-
730
- the resulting file would look like this:
731
-
732
- ###### THe Joined file a.yaml
733
-
734
- ```
735
- ToBe:
736
- ThatIs:
737
- Or: maybe
738
- TheQuestion:
739
- not: "?"
740
- OrNot:
741
- not: "?"
742
- ```
743
-
744
- You may also specify absolute file paths if preferred by starting with a "/".
745
-
746
- When specifying only a file path, the complete content of the file is assigned to the given key.
747
- You always need a key to assign the !include command to.
748
- However, you cannot combine the value returned from !include with other values in the same key.
749
- Thus, the following combinations do not work:
750
-
751
- ###### caveats.yml
752
-
753
- ```
754
- !include "file.yaml" # no key assigned
755
-
756
- Key:
757
- Some: OtherItem
758
- !include "file.yaml" # cannot join with other named items
759
-
760
- List:
761
- - an: entry
762
- !include "file.yaml" # cannot directly join with list items, even if !include returns a list
763
- ```
764
-
765
- #### Integrate specific nodes of YAML files
766
-
767
- Instead of including *all* content in the included file, you may also pick a specific node within that file.
768
- For this use `!include [<relative/path/to/file.yaml>, Path:To:Field:In:Yaml]`.
769
- Here, `:` is used in the node-specifying string to select a sequence of nodes to follow - with custom depth.
770
- Consider the following two files:
771
-
772
- ###### file_to_be_included.yaml
773
-
774
- ```yaml
775
- Set1:
776
- Subset1:
777
- Key: Value
778
- Set2:
779
- OtherKey: OtherValue
780
- ```
781
-
782
- ###### including_file.yaml
783
-
784
- ```yaml
785
- - Type: MyAgentWithInputs
786
- Id: 1
787
- Attributes: !include_node [ file_to_be_included.yaml, Set1:Subset1 ]
788
- ```
789
-
790
- Compiling "including_file.yaml" results in
791
-
792
- ###### resulting_file.yaml
793
-
794
- ```yaml
795
- - Type: MyAgentWithInputs
796
- Id: 1
797
- Attributes:
798
- Key: Value
799
- ```
800
-
801
- #### Load multiple files
802
-
803
- Using wildcards in the given path (e.g. "path/to/many/*.yaml") will lead to loading multiple files and assigning their
804
- content to the same key.
805
- You can make use of this feature with or without specifying a node selector.
806
- However, the elements to be joined across multiple files must be lists.
807
- These lists are then concatenated into a single list and then assigned to the key in the file calling !include.
808
- This feature is especially useful for Contracts: You can split the Contracts list into several files and place them in a
809
- separate folder.
810
- Then use !include to re-integrate them into your configuration. An example:
811
-
812
- ###### my_contract1.yaml
813
-
814
- ```
815
- Contracts:
816
- - ContractA
817
- - ContractB
818
- ```
819
-
820
- ###### my_contract2.yaml
821
-
822
- ```
823
- Contracts:
824
- - ContractC
825
- - ContractD
826
- - ContractE
827
- ```
828
-
829
- ###### including_file.yaml
830
-
831
- ```
832
- Contracts: [!include "my_contract*.yaml", "Contracts"]
833
- ```
834
-
835
- results in
836
-
837
- ###### result.yaml
838
-
839
- ```
840
- Contracts:
841
- - ContractA
842
- - ContractB
843
- - ContractC
844
- - ContractD
845
- - ContractE
846
- ```
847
-
848
- #### Ignoring files
849
-
850
- Files that have their name start with "IGNORE_" are not included with the !include command.
851
- You will see a debug output to notify you that the file was ignored.
852
- Use this to temporarily take files out ouf your configuration without deleting or moving them.
853
-
854
- ## Read FAME results
855
-
856
- Takes an output file in protobuf format of FAME-based applications and converts it into files in CSV format.
857
- An individual file for each type of Agent is created in a folder named after the protobuf input file.
858
- Call structure:
859
-
860
- convertFameResults -f <./path/to/protobuf_file.pb>
861
-
862
- You may also specify any of the following arguments:
863
-
864
- | Command | Action |
865
- |-----------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
866
- | `-l` or `--log` <option> | Sets the logging level. Default is `WARNING`. Options are `DEBUG`, `INFO`, `WARNING`, `ERROR`, `CRITICAL`. |
867
- | `-lf` or `--logfile` <file> | Sets the logging file. Default is `None`. If `None` is provided, all logs get only printed to the console. |
868
- | `-a` or `--agents` <list-of-agents> | If specified, only a subset of agents is extracted from the protobuf file. Default is to extract all agents. |
869
- | `-o` or `--output` | Sets the path to where the generated output files are written to. If not specified, the folder's name is derived from the input file's name. Folder will be created if it does not exist. |
870
- | `-se` or `--single-export` | Enables export of individual agents to individual files, when present. If not present (the default) one file per `AgentType` is created. |
871
- | `-m` or `--memory-saving` | When specified, reduces memory usage profile at the cost of runtime. Use only when necessary. |
872
- | `-cc` or `--complex-column` <option> | Defines how to deal with complex indexed output columns (if any). `IGNORE` ignores complex columns. `SPLIT` creates a separate file for each complex indexed output column. |
873
- | `-t` or `--time` <option> | Option to define conversion of time steps to given format (default=`UTC`) by `-t/--time {UTC, INT, FAME}` |
874
- | `--input-recovery` or `--no-input-recovery` | If True, all input data are recovered in addition to the outputs (default=False). |
875
- | `-mt` or `--merge-times` <list-of-parameters> | Option to merge `TimeSteps` of a certain range of steps in the output files to associate multiple time steps with a common logical time in your simulation and reduce number of lines in output files |
876
-
877
- The option `--merge-times` requires exactly three integer arguments separated by spaces:
878
-
879
- | Position | Name | Meaning |
880
- |----------|--------------|------------------------------------------------------------------------------------------|
881
- | First | Focal point | TimeStep on which `steps-before` earlier and `steps-after` later TimeSteps are merged on |
882
- | Second | Steps before | Range of TimeSteps before the `focal-point` they get merged to, must be Zero or positive |
883
- | Third | Steps after | Range of TimeSteps after the `focal-point` they get merged to, must be Zero or positive |
884
-
885
- This could look as follows:
886
-
887
- convertFameResults -f <./path/to/protobuf_file.pb> -l debug -lf <path/to/output.log> -a AgentType1 AgentType2 -o myCsvFolder -m -cc SPLIT --merge-times 0 1799 1800
888
-
889
- Make sure that in the range of time steps you specify for merging, there is only one value per column in the merged time
890
- range.
891
- If multiple values per column are merged values will get concatenated and might yield unexpected results.
892
-
893
- You may also call the conversion script from any Python script with:
894
-
895
- ```python
896
- from fameio.scripts.convert_results import Options, run as convert_results
897
-
898
- convert_results({Options.FILE: "./path/to/protobuf_file.pb"})
899
- ```
900
-
901
- Similar to the console call you may also specify custom run config arguments and add it in a dictionary to the function
902
- call.
903
-
904
- ```python
905
- from fameio.scripts.convert_results import Options, run as convert_results
906
-
907
- run_config = {Options.FILE: "./path/to/protobuf_file.pb",
908
- Options.LOG_LEVEL: "info",
909
- Options.LOG_FILE: "scenario.log",
910
- Options.OUTPUT: "Output",
911
- Options.AGENT_LIST: ['AgentType1', 'AgentType2'],
912
- Options.MEMORY_SAVING: False,
913
- Options.SINGLE_AGENT_EXPORT: False,
914
- Options.RESOLVE_COMPLEX_FIELD: "SPLIT",
915
- Options.TIME: "INT",
916
- Options.TIME_MERGING: {},
917
- }
918
-
919
- convert_results(run_config)
920
- ```
921
-
922
- You can also use the associated argument parser, to extract the run_config dynamically from a string:
923
-
924
- ```python
925
- from fameio.scripts.convert_results import Options, run as convert_results
926
- from fameio.cli.convert_results import handle_args
927
-
928
- my_defaults = {Options.FILE: "./path/to/protobuf_file.pb",
929
- Options.LOG_LEVEL: "info",
930
- Options.LOG_FILE: "scenario.log",
931
- Options.OUTPUT: "Output",
932
- Options.AGENT_LIST: ['AgentType1', 'AgentType2'],
933
- Options.MEMORY_SAVING: False,
934
- Options.SINGLE_AGENT_EXPORT: False,
935
- Options.RESOLVE_COMPLEX_FIELD: "SPLIT",
936
- Options.TIME: "INT",
937
- Options.TIME_MERGING: {},
938
- }
939
- my_arg_string = ['-f', 'my/other/scenario.yaml', '-l', 'error']
940
-
941
- run_config = handle_args(my_arg_string, my_defaults)
942
- convert_results(run_config)
943
- ```
944
-
945
- ## Reformat time series
946
-
947
- Takes CSV time series files and reformats them into FAME time format.
948
- This improves speed of run configuration creation but also reduces readability of the CSV files' content.
949
- Thus, we recommend to apply this reformatting only for CSV time series files with more than 20,000 lines.
950
-
951
- Call structure:
952
-
953
- reformatTimeSeries -fp <file_or_file_pattern*.csv>
954
-
955
- You may also specify any of the following arguments:
956
-
957
- | Command | Action |
958
- |-------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
959
- | `-l` or `--log` <option> | Sets the logging level. Default is `WARNING`. Options are `DEBUG`, `INFO`, `WARNING`, `ERROR`, `CRITICAL`. |
960
- | `-lf` or `--logfile` <file> | Sets the logging file. Default is `None`. If `None` is provided, all logs get only printed to the console. |
961
- | `--replace` or `--no-replace` | If `--replace` is specified, existing csv files are replaced with the new content. Otherwise and by default, new files are created extending the original file name by "_reformatted". |
962
-
963
- ## Cite FAME-Io
964
-
965
- If you use FAME-Io for academic work, please cite as follows.
966
-
967
- Bibtex entry:
968
-
969
- ```
970
- @article{fameio2023joss,
971
- author = {Felix Nitsch and Christoph Schimeczek and Ulrich Frey and Benjamin Fuchs},
972
- title = {FAME-Io: Configuration tools for complex agent-based simulations},
973
- journal = {Journal of Open Source Software},
974
- year = {2023},
975
- doi = {doi: https://doi.org/10.21105/joss.04958}
976
- }
977
- ```
978
-
979
- ## Available Support
980
-
981
- This is a purely scientific project by (at the moment) one research group.
982
- Thus, there is no paid technical support available.
983
- However, we will give our best to answer your questions and provide support.
984
-
985
- If you experience any trouble with FAME-Io, you may contact the developers via [fame@dlr.de](mailto:fame@dlr.de).
986
- Please report bugs and make feature requests by filing issues following the provided templates (see
987
- also [Contribute](CONTRIBUTING.md)).
988
- For substantial enhancements, we recommend that you contact us via [fame@dlr.de](mailto:fame@dlr.de) for working
989
- together on the code in common projects or towards common publications and thus further develop FAME-Io.
990
-