fameio 3.0.0__tar.gz → 3.1.0__tar.gz

{fameio-3.0.0 → fameio-3.1.0}/CHANGELOG.md

@@ -1,9 +1,15 @@
-<!-- SPDX-FileCopyrightText: 2024 German Aerospace Center <fame@dlr.de>
+<!-- SPDX-FileCopyrightText: 2025 German Aerospace Center <fame@dlr.de>
 
 SPDX-License-Identifier: CC0-1.0 -->
 
+## [3.1.0](https://gitlab.com/fame-framework/fame-io/-/tags/v3.1.0) - 2025-01-29
+### Changed
+- Speed up of `makeFameRunConfig` for large CSV files #229 (@dlr-cjs, dlr_fn)
+- Improve testing of `tools.py` #227 (@dlr_fn)
+- Reorganize badges in tabular representation in `README.md` #226 (@dlr-cjs, dlr_fn)
+
 # Changelog
-## [3.0.0](https://gitlab.com/fame-framework/fame-io/-/tags/v3.0.0) - 2024-02-12
+## [3.0.0](https://gitlab.com/fame-framework/fame-io/-/tags/v3.0.0) - 2024-12-02
 ### Changed
 - **Breaking**: Update to fameprotobuf v2.0.2 #208, #215 (@dlr-cjs)
 - **Breaking**: Remove section `GeneralProperties.Output` in scenarios - any content there will be ignored #208 (@dlr-cjs)

{fameio-3.0.0 → fameio-3.1.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: fameio
-Version: 3.0.0
+Version: 3.1.0
 Summary: Tools for input preparation and output digestion of FAME models
 Home-page: https://gitlab.com/fame-framework/wiki/-/wikis/home
 License: Apache-2.0
@@ -27,27 +27,25 @@ Requires-Dist: pyyaml (>=6.0,<7.0)
 Project-URL: Repository, https://gitlab.com/fame-framework/fame-io/
 Description-Content-Type: text/markdown
 
-<!-- SPDX-FileCopyrightText: 2024 German Aerospace Center <fame@dlr.de>
+<!-- SPDX-FileCopyrightText: 2025 German Aerospace Center <fame@dlr.de>
 
 SPDX-License-Identifier: Apache-2.0 -->
-[![PyPI version](https://badge.fury.io/py/fameio.svg)](https://badge.fury.io/py/fameio)
-[![JOSS](https://joss.theoj.org/papers/10.21105/joss.04958/status.svg)](https://doi.org/10.21105/joss.04958)
-[![Zenodo](https://zenodo.org/badge/DOI/10.5281/zenodo.4314337.svg)](https://doi.org/10.5281/zenodo.4314337)
-[![PyPI license](https://img.shields.io/pypi/l/fameio.svg)](https://badge.fury.io/py/fameio)
-[![pipeline status](https://gitlab.com/fame-framework/fame-io/badges/main/pipeline.svg)](https://gitlab.com/fame-framework/fame-io/commits/main)
-[![coverage report](https://gitlab.com/fame-framework/fame-io/badges/main/coverage.svg)](https://gitlab.com/fame-framework/fame-io/-/commits/main)
-[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)
-[![REUSE status](https://api.reuse.software/badge/gitlab.com/fame-framework/fame-io)](https://api.reuse.software/info/gitlab.com/fame-framework/fame-io)
-[![Common Changelog](https://common-changelog.org/badge.svg)](https://common-changelog.org)
-![GitLab last commit](https://img.shields.io/gitlab/last-commit/fame-framework%2Ffame-io)
-![GitLab closed issues by-label](https://img.shields.io/gitlab/issues/closed/fame-framework%2Ffame-io)
+
+|               |                                                                                                                                                                                                                                                                                                                                                 |
+|---------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| **Package**   | [![PyPI version](https://badge.fury.io/py/fameio.svg)](https://badge.fury.io/py/fameio) [![PyPI license](https://img.shields.io/pypi/l/fameio.svg)](https://badge.fury.io/py/fameio) [![REUSE status](https://api.reuse.software/badge/gitlab.com/fame-framework/fame-io)](https://api.reuse.software/info/gitlab.com/fame-framework/fame-io)   |
+| **Tests**     | [![pipeline status](https://gitlab.com/fame-framework/fame-io/badges/main/pipeline.svg)](https://gitlab.com/fame-framework/fame-io/commits/main) [![coverage report](https://gitlab.com/fame-framework/fame-io/badges/main/coverage.svg)](https://gitlab.com/fame-framework/fame-io/-/commits/main)                                             |
+| **Activity**  | ![GitLab last commit](https://img.shields.io/gitlab/last-commit/fame-framework%2Ffame-io) ![GitLab closed issues by-label](https://img.shields.io/gitlab/issues/closed/fame-framework%2Ffame-io)                                                                                                                                                |
+| **Style**     | [![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black) [![Common Changelog](https://common-changelog.org/badge.svg)](https://common-changelog.org)                                                                                                                                    |
+| **Reference** | [![JOSS](https://joss.theoj.org/papers/10.21105/joss.04958/status.svg)](https://doi.org/10.21105/joss.04958) [![Zenodo](https://zenodo.org/badge/DOI/10.5281/zenodo.4314337.svg)](https://doi.org/10.5281/zenodo.4314337)                                                                                                                       |
 
 # FAME-Io
 
 *Tools for input preparation and output digestion of FAME models*
 
 FAME-Io compiles input for FAME models in protobuf format and extracts model outputs to human-readable files.
-Please visit the [FAME-Wiki](https://gitlab.com/fame-framework/wiki/-/wikis/home) to get an explanation of FAME and its components.
+Please visit the [FAME-Wiki](https://gitlab.com/fame-framework/wiki/-/wikis/home) to get an explanation of FAME and its
+components.
 
 # Installation
 
@@ -293,12 +291,12 @@ MyComplexAttribute:
 * `NestedAttributes` (required only if `AttributeType: block`, otherwise disallowed) starts an inner Attribute
   definition block - defined Attributes are sub-elements of `MyComplexAttribute`
 * `Values` (optional - None by default):
-  * if present, defines a list or dictionary of allowed values for this attribute
-  * if a dictionary is used, individual Metadata can be assigned to each allowed value using the `Metadata` keyword
+    * if present, defines a list or dictionary of allowed values for this attribute
+    * if a dictionary is used, individual Metadata can be assigned to each allowed value using the `Metadata` keyword
 * `Default` (optional - None by default):
-  * if present, defines a default value to be used if the scenario does not specify one
-  * must match one of the entries in `Values` in case those are defined
-  * can be a list if the attribute is a list
+    * if present, defines a default value to be used if the scenario does not specify one
+    * must match one of the entries in `Values` in case those are defined
+    * can be a list if the attribute is a list
 * `Help` (optional - None by default): if present, defines a help text for your Attribute
 * `Metadata` (optional - None by default): if present, defines additional metadata assigned to the Attribute
 
@@ -315,6 +313,7 @@ MyComplexAttribute:
 | `block`       | this attribute has no value of its own but hosts a group of nested Attributes; implies `NestedAttributes` to be defined |
 
 #### GeneralProperties
+
 Specifies FAME-specific properties of the simulation. Structure:
 
 ```yaml
@@ -364,8 +363,10 @@ Agent Parameters:
   see attribute table above
 * `Metadata` Optional; can be assigned to each instance of an Agent, as well as to each of its Attributes
 
-The specified `Attributes` for each agent must match the specified `Attributes` options in the linked Schema (see above).
-For better structure and readability of the `scenario.yaml`, `Attributes` may also be specified in a nested way as demonstrated below.
+The specified `Attributes` for each agent must match the specified `Attributes` options in the linked Schema (see
+above).
+For better structure and readability of the `scenario.yaml`, `Attributes` may also be specified in a nested way as
+demonstrated below.
 
 ```yaml
 Agents:
@@ -399,8 +400,10 @@ Nested items `IntValueA` and `IntValueB` of `MyListGroup` are assigned within a
 these nested items several times.
 
 ##### Attribute Metadata
+
 Metadata can be assigned to any value, list item, or superstructure.
-To assign Metadata to a primitive value, create a dictionary from it, set the actual value with the inner keyword `Value` and add the keyword `Metadata` like this:
+To assign Metadata to a primitive value, create a dictionary from it, set the actual value with the inner
+keyword `Value` and add the keyword `Metadata` like this:
 
 ```yaml
 ValueWithoutMetadata: 1
@@ -412,16 +415,16 @@ SameValueWithMetadata:
 You can assign Metadata to a list of primitive values using the keyword `Values` like this:
 
 ```yaml
-ValueListWithoutMetadata: [1,2,3]
+ValueListWithoutMetadata: [ 1,2,3 ]
 SameValueListWithListMetadata:
-  Values: [1,2,3]
+  Values: [ 1,2,3 ]
   Metadata: # describe the whole list of values with Metadata here
 ```
 
 or specify Metadata for each (or just some) value individually, like this:
 
 ```yaml
-ValueListWithoutMetadata: [1,2,3]
+ValueListWithoutMetadata: [ 1,2,3 ]
 SameValueListWithMetadataAtEachElement:
   - Value: 1
     Metadata: # describe this specific value "1" with Metadata here
@@ -432,7 +435,7 @@ SameValueListWithMetadataAtEachElement:
 or assign Metadata to both the list and any of its list entries, like this:
 
 ```yaml
-ValueListWithoutMetadata: [1,2,3]
+ValueListWithoutMetadata: [ 1,2,3 ]
 SameValueListWithAllMetadata:
   Metadata: # Recommendation: place the Metadata of the list first if the list of values is extensive, as in this case
   Values:
@@ -493,6 +496,7 @@ SameListOfNestedItemsWithGeneralMetadata:
 Again, you may apply both variants and apply Metadata to the list and each of its items if you wish.
 
 #### Contracts
+
 Specifies all Contracts, i.e. repetitive bilateral transactions in between agents.
 Contracts are given as a list.
 We recommend moving Contracts to separate files and to use the `!include` command to integrate them in the scenario.
@@ -528,6 +532,7 @@ Contract Parameters:
 * `Attributes` can be set to include additional information as `int`, `float`, `enum`, or `dict` data types
 
 ##### Definition of Multiple Similar Contracts
+
 Often, scenarios contain multiple agents of similar type that also have similar chains of contracts.
 Therefore, FAME-Io supports a compact definition of multiple similar contracts.
 `SenderId` and `ReceiverId` can both be lists and support One-to-N, N-to-One and N-to-N relations like in the following
@@ -636,7 +641,8 @@ These CSV files follow a specific structure:
 
 * They should contain exactly two columns - any other columns are ignored.
   A warning is raised if more than two non-empty columns are detected.
-* The first column must be a time stamp in form `YYYY-MM-DD_hh:mm:ss`
+* The first column must be a time stamp in form `YYYY-MM-DD_hh:mm:ss` or
+  a [FAME-Timestamp](https://gitlab.com/fame-framework/wiki/-/wikis/architecture/decisions/TimeStamp) integer value
 * The second column must be a numerical value (either integer or floating-point)
 * The separator of the two columns is a semicolon
 * The data must **not** have headers, except for comments marked with `#`
@@ -654,6 +660,8 @@ Exemplary content of a valid CSV file:
 
 Please refer also to the detailed article about `TimeStamps` in
 the [FAME-Wiki](https://gitlab.com/fame-framework/wiki/-/wikis/TimeStamp).
+For large CSV files (with more than 20,000 rows) we recommend using the integer representation of FAME-Timestamps in the
+first column (instead of text representation) to improve conversion speed.
 
 ### Split and join multiple YAML files
 
@@ -862,12 +870,12 @@ The option `--merge-times` requires exactly three integer arguments separated by
 | Second   | Steps before | Range of TimeSteps before the `focal-point` they get merged to, must be Zero or positive |
 | Third    | Steps after  | Range of TimeSteps after the `focal-point` they get merged to, must be Zero or positive  |
 
-
 This could look as follows:
 
     convertFameResults -f <./path/to/protobuf_file.pb> -l debug -lf <path/to/output.log> -a AgentType1 AgentType2 -o myCsvFolder -m -cc SPLIT --merge-times 0 1799 1800
 
-Make sure that in the range of time steps you specify for merging, there is only one value per column in the merged time range.
+Make sure that in the range of time steps you specify for merging, there is only one value per column in the merged time
+range.
 If multiple values per column are merged values will get concatenated and might yield unexpected results.
 
 You may also call the conversion script from any Python script with:

{fameio-3.0.0 → fameio-3.1.0}/README.md

@@ -1,24 +1,22 @@
-<!-- SPDX-FileCopyrightText: 2024 German Aerospace Center <fame@dlr.de>
+<!-- SPDX-FileCopyrightText: 2025 German Aerospace Center <fame@dlr.de>
 
 SPDX-License-Identifier: Apache-2.0 -->
-[![PyPI version](https://badge.fury.io/py/fameio.svg)](https://badge.fury.io/py/fameio)
-[![JOSS](https://joss.theoj.org/papers/10.21105/joss.04958/status.svg)](https://doi.org/10.21105/joss.04958)
-[![Zenodo](https://zenodo.org/badge/DOI/10.5281/zenodo.4314337.svg)](https://doi.org/10.5281/zenodo.4314337)
-[![PyPI license](https://img.shields.io/pypi/l/fameio.svg)](https://badge.fury.io/py/fameio)
-[![pipeline status](https://gitlab.com/fame-framework/fame-io/badges/main/pipeline.svg)](https://gitlab.com/fame-framework/fame-io/commits/main)
-[![coverage report](https://gitlab.com/fame-framework/fame-io/badges/main/coverage.svg)](https://gitlab.com/fame-framework/fame-io/-/commits/main)
-[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)
-[![REUSE status](https://api.reuse.software/badge/gitlab.com/fame-framework/fame-io)](https://api.reuse.software/info/gitlab.com/fame-framework/fame-io)
-[![Common Changelog](https://common-changelog.org/badge.svg)](https://common-changelog.org)
-![GitLab last commit](https://img.shields.io/gitlab/last-commit/fame-framework%2Ffame-io)
-![GitLab closed issues by-label](https://img.shields.io/gitlab/issues/closed/fame-framework%2Ffame-io)
+
+|               |                                                                                                                                                                                                                                                                                                                                                 |
+|---------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| **Package**   | [![PyPI version](https://badge.fury.io/py/fameio.svg)](https://badge.fury.io/py/fameio) [![PyPI license](https://img.shields.io/pypi/l/fameio.svg)](https://badge.fury.io/py/fameio) [![REUSE status](https://api.reuse.software/badge/gitlab.com/fame-framework/fame-io)](https://api.reuse.software/info/gitlab.com/fame-framework/fame-io)   |
+| **Tests**     | [![pipeline status](https://gitlab.com/fame-framework/fame-io/badges/main/pipeline.svg)](https://gitlab.com/fame-framework/fame-io/commits/main) [![coverage report](https://gitlab.com/fame-framework/fame-io/badges/main/coverage.svg)](https://gitlab.com/fame-framework/fame-io/-/commits/main)                                             |
+| **Activity**  | ![GitLab last commit](https://img.shields.io/gitlab/last-commit/fame-framework%2Ffame-io) ![GitLab closed issues by-label](https://img.shields.io/gitlab/issues/closed/fame-framework%2Ffame-io)                                                                                                                                                |
+| **Style**     | [![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black) [![Common Changelog](https://common-changelog.org/badge.svg)](https://common-changelog.org)                                                                                                                                    |
+| **Reference** | [![JOSS](https://joss.theoj.org/papers/10.21105/joss.04958/status.svg)](https://doi.org/10.21105/joss.04958) [![Zenodo](https://zenodo.org/badge/DOI/10.5281/zenodo.4314337.svg)](https://doi.org/10.5281/zenodo.4314337)                                                                                                                       |
 
 # FAME-Io
 
 *Tools for input preparation and output digestion of FAME models*
 
 FAME-Io compiles input for FAME models in protobuf format and extracts model outputs to human-readable files.
-Please visit the [FAME-Wiki](https://gitlab.com/fame-framework/wiki/-/wikis/home) to get an explanation of FAME and its components.
+Please visit the [FAME-Wiki](https://gitlab.com/fame-framework/wiki/-/wikis/home) to get an explanation of FAME and its
+components.
 
 # Installation
 
@@ -264,12 +262,12 @@ MyComplexAttribute:
 * `NestedAttributes` (required only if `AttributeType: block`, otherwise disallowed) starts an inner Attribute
   definition block - defined Attributes are sub-elements of `MyComplexAttribute`
 * `Values` (optional - None by default):
-  * if present, defines a list or dictionary of allowed values for this attribute
-  * if a dictionary is used, individual Metadata can be assigned to each allowed value using the `Metadata` keyword
+    * if present, defines a list or dictionary of allowed values for this attribute
+    * if a dictionary is used, individual Metadata can be assigned to each allowed value using the `Metadata` keyword
 * `Default` (optional - None by default):
-  * if present, defines a default value to be used if the scenario does not specify one
-  * must match one of the entries in `Values` in case those are defined
-  * can be a list if the attribute is a list
+    * if present, defines a default value to be used if the scenario does not specify one
+    * must match one of the entries in `Values` in case those are defined
+    * can be a list if the attribute is a list
 * `Help` (optional - None by default): if present, defines a help text for your Attribute
 * `Metadata` (optional - None by default): if present, defines additional metadata assigned to the Attribute
 
@@ -286,6 +284,7 @@ MyComplexAttribute:
 | `block`       | this attribute has no value of its own but hosts a group of nested Attributes; implies `NestedAttributes` to be defined |
 
 #### GeneralProperties
+
 Specifies FAME-specific properties of the simulation. Structure:
 
 ```yaml
@@ -335,8 +334,10 @@ Agent Parameters:
   see attribute table above
 * `Metadata` Optional; can be assigned to each instance of an Agent, as well as to each of its Attributes
 
-The specified `Attributes` for each agent must match the specified `Attributes` options in the linked Schema (see above).
-For better structure and readability of the `scenario.yaml`, `Attributes` may also be specified in a nested way as demonstrated below.
+The specified `Attributes` for each agent must match the specified `Attributes` options in the linked Schema (see
+above).
+For better structure and readability of the `scenario.yaml`, `Attributes` may also be specified in a nested way as
+demonstrated below.
 
 ```yaml
 Agents:
@@ -370,8 +371,10 @@ Nested items `IntValueA` and `IntValueB` of `MyListGroup` are assigned within a
 these nested items several times.
 
 ##### Attribute Metadata
+
 Metadata can be assigned to any value, list item, or superstructure.
-To assign Metadata to a primitive value, create a dictionary from it, set the actual value with the inner keyword `Value` and add the keyword `Metadata` like this:
+To assign Metadata to a primitive value, create a dictionary from it, set the actual value with the inner
+keyword `Value` and add the keyword `Metadata` like this:
 
 ```yaml
 ValueWithoutMetadata: 1
@@ -383,16 +386,16 @@ SameValueWithMetadata:
 You can assign Metadata to a list of primitive values using the keyword `Values` like this:
 
 ```yaml
-ValueListWithoutMetadata: [1,2,3]
+ValueListWithoutMetadata: [ 1,2,3 ]
 SameValueListWithListMetadata:
-  Values: [1,2,3]
+  Values: [ 1,2,3 ]
   Metadata: # describe the whole list of values with Metadata here
 ```
 
 or specify Metadata for each (or just some) value individually, like this:
 
 ```yaml
-ValueListWithoutMetadata: [1,2,3]
+ValueListWithoutMetadata: [ 1,2,3 ]
 SameValueListWithMetadataAtEachElement:
   - Value: 1
     Metadata: # describe this specific value "1" with Metadata here
@@ -403,7 +406,7 @@ SameValueListWithMetadataAtEachElement:
 or assign Metadata to both the list and any of its list entries, like this:
 
 ```yaml
-ValueListWithoutMetadata: [1,2,3]
+ValueListWithoutMetadata: [ 1,2,3 ]
 SameValueListWithAllMetadata:
   Metadata: # Recommendation: place the Metadata of the list first if the list of values is extensive, as in this case
   Values:
@@ -464,6 +467,7 @@ SameListOfNestedItemsWithGeneralMetadata:
 Again, you may apply both variants and apply Metadata to the list and each of its items if you wish.
 
 #### Contracts
+
 Specifies all Contracts, i.e. repetitive bilateral transactions in between agents.
 Contracts are given as a list.
 We recommend moving Contracts to separate files and to use the `!include` command to integrate them in the scenario.
@@ -499,6 +503,7 @@ Contract Parameters:
 * `Attributes` can be set to include additional information as `int`, `float`, `enum`, or `dict` data types
 
 ##### Definition of Multiple Similar Contracts
+
 Often, scenarios contain multiple agents of similar type that also have similar chains of contracts.
 Therefore, FAME-Io supports a compact definition of multiple similar contracts.
 `SenderId` and `ReceiverId` can both be lists and support One-to-N, N-to-One and N-to-N relations like in the following
@@ -607,7 +612,8 @@ These CSV files follow a specific structure:
 
 * They should contain exactly two columns - any other columns are ignored.
   A warning is raised if more than two non-empty columns are detected.
-* The first column must be a time stamp in form `YYYY-MM-DD_hh:mm:ss`
+* The first column must be a time stamp in form `YYYY-MM-DD_hh:mm:ss` or
+  a [FAME-Timestamp](https://gitlab.com/fame-framework/wiki/-/wikis/architecture/decisions/TimeStamp) integer value
 * The second column must be a numerical value (either integer or floating-point)
 * The separator of the two columns is a semicolon
 * The data must **not** have headers, except for comments marked with `#`
@@ -625,6 +631,8 @@ Exemplary content of a valid CSV file:
 
 Please refer also to the detailed article about `TimeStamps` in
 the [FAME-Wiki](https://gitlab.com/fame-framework/wiki/-/wikis/TimeStamp).
+For large CSV files (with more than 20,000 rows) we recommend using the integer representation of FAME-Timestamps in the
+first column (instead of text representation) to improve conversion speed.
 
 ### Split and join multiple YAML files
 
@@ -833,12 +841,12 @@ The option `--merge-times` requires exactly three integer arguments separated by
 | Second   | Steps before | Range of TimeSteps before the `focal-point` they get merged to, must be Zero or positive |
 | Third    | Steps after  | Range of TimeSteps after the `focal-point` they get merged to, must be Zero or positive  |
 
-
 This could look as follows:
 
     convertFameResults -f <./path/to/protobuf_file.pb> -l debug -lf <path/to/output.log> -a AgentType1 AgentType2 -o myCsvFolder -m -cc SPLIT --merge-times 0 1799 1800
 
-Make sure that in the range of time steps you specify for merging, there is only one value per column in the merged time range.
+Make sure that in the range of time steps you specify for merging, there is only one value per column in the merged time
+range.
 If multiple values per column are merged values will get concatenated and might yield unexpected results.
 
 You may also call the conversion script from any Python script with:

{fameio-3.0.0 → fameio-3.1.0}/pyproject.toml

@@ -7,7 +7,7 @@ build-backend = "poetry.core.masonry.api"
 
 [tool.poetry]
 name = "fameio"
-version = "3.0.0"
+version = "3.1.0"
 description = "Tools for input preparation and output digestion of FAME models"
 license = "Apache-2.0"
 authors = [

{fameio-3.0.0 → fameio-3.1.0}/src/fameio/series.py

@@ -101,12 +101,10 @@ class TimeSeriesManager:
         is_empty = additional_columns.dropna(how="all").empty
         if not is_empty:
             log().warning(self._WARN_DATA_IGNORED)
-        data = data.apply(
-            lambda r: [FameTime.convert_string_if_is_datetime(r[0]), self._assert_valid(r[1])],
-            axis=1,
-            result_type="expand",
-        )
-        return data.astype({0: "int64"})
+        if data.dtypes[0] != "int64":
+            data[0] = [FameTime.convert_string_if_is_datetime(time) for time in data[0]]
+        data[1] = [TimeSeriesManager._assert_valid(value) for value in data[1]]
+        return data
 
     @staticmethod
     def _assert_valid(value: Any) -> float: