mlonmcu 0.3.0.dev0__tar.gz → 0.4.0__tar.gz

{mlonmcu-0.3.0.dev0 → mlonmcu-0.4.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: mlonmcu
-Version: 0.3.0.dev0
+Version: 0.4.0
 Summary: This project contains research code related to the deployment of inferenceor learning applications on tiny micro-controllers.
 Home-page: https://github.com/tum-ei-eda/mlonmcu
 Author: TUM Department of Electrical and Computer Engineering - Chair of Electronic Design Automation
@@ -21,6 +21,7 @@ Classifier: Programming Language :: Python :: 3.10
 Classifier: Programming Language :: Python :: 3.11
 Requires-Python: >=3.7
 Description-Content-Type: text/markdown
+Provides-Extra: all-prod
 Provides-Extra: espidf
 Provides-Extra: etiss
 Provides-Extra: gvsoc_pulp
@@ -35,7 +36,6 @@ Provides-Extra: tvm-autotuning
 Provides-Extra: visualize
 Provides-Extra: visualize_tflite
 Provides-Extra: zephyr
-Provides-Extra: all-prod
 License-File: LICENSE
 License-File: NOTICE
 
@@ -43,14 +43,15 @@ License-File: NOTICE
 
 [![pypi package](https://badge.fury.io/py/mlonmcu.svg)](https://pypi.org/project/mlonmcu)
 [![readthedocs](https://readthedocs.org/projects/mlonmcu/badge/?version=latest)](https://mlonmcu.readthedocs.io/en/latest/?version=latest)
-![covergae](https://byob.yarr.is/tum-ei-eda/mlonmcu/coverage)
+![coverage](https://byob.yarr.is/tum-ei-eda/mlonmcu/coverage)
 [![GitHub license](https://img.shields.io/github/license/tum-ei-eda/mlonmcu.svg)](https://github.com/tum-ei-eda/mlonmcu/blob/main/LICENSE)
 
 [![cicd workflow](https://github.com/tum-ei-eda/mlonmcu/actions/workflows/cicd.yml/badge.svg)](https://github.com/tum-ei-eda/mlonmcu/actions/workflows/cicd.yml)
+[![lint workflow](https://github.com/tum-ei-eda/mlonmcu/actions/workflows/style.yml/badge.svg)](https://github.com/tum-ei-eda/mlonmcu/actions/workflows/style.yml)
 [![demo workflow](https://github.com/tum-ei-eda/mlonmcu/actions/workflows/demo.yml/badge.svg)](https://github.com/tum-ei-eda/mlonmcu/actions/workflows/demo.yml)
-[![container workflow](https://github.com/tum-ei-eda/mlonmcu/actions/workflows/container.yml/badge.svg)](https://github.com/tum-ei-eda/mlonmcu/actions/workflows/container.yml)
 [![bench workflow](https://github.com/tum-ei-eda/mlonmcu/actions/workflows/bench.yml/badge.svg)](https://github.com/tum-ei-eda/mlonmcu/actions/workflows/bench.yml)
 
+
 This project contains research code related to the deployment of inference or learning applications on tiny micro-controllers.
 
 
@@ -63,11 +64,7 @@ This project contains research code related to the deployment of inference or le
 
 - Highly configurable python package
 - Automatic resolution and installation of dependencies
-- Supporting a large combination of frameworks/backends/targets/features:
-  - Frameworks (Backends): TFLite Micro, MircoTVM
-  - Targets: Host (x86), ETISS (Pulpino), Corstone300 (ARM Cortex-M), QEMU (RISC-V), Spike (`riscv-isa-sim`),...
-  - Platforms: Zephyr, ESP-IDF,...
-  - Features: Autotuning, Debugging, RPC,...
+- Supporting a large combination of frameworks/backends/targets/features
 - Build-in parallel processing of large number of benchmarks
 - Isolated enironments (not interfering with other installations)
 - Command Line and Python Development Interfaces
@@ -88,10 +85,10 @@ First, a set of APT packages needs to be installed:
 sudo apt install python3-pip python3-venv
 
 # MLonMCU related
-sudo apt install libboost-system-dev libboost-filesystem-dev libboost-program-options-dev graphviz doxygen libtinfo-dev zlib1g-dev texinfo unzip device-tree-compiler tree g++
+sudo apt install libboost-all-dev graphviz doxygen libtinfo-dev zlib1g-dev texinfo unzip device-tree-compiler tree g++
 
 # Optional (depending on configuration)
-sudo apt install ninja-build
+sudo apt install ninja-build flex lsb-release libelf-dev
 ```
 
 Also make sure that your default Python is at least v3.7. If the `python` command is not available in your shell or points Python v2.7 check out `python-is-python3`.
@@ -121,10 +118,7 @@ source .venv/bin/activate
 
 **Warning:** It is recommended to have at least version 3.20 of CMake installed for full compatibility!
 
-# Install minimal dependencies
-pip install -r requirements_dev.txt
-
-# Optional (only for development)
+# Install ptional dependecies (only for development)
 pip install -r requirements_dev.txt
 pip install -r docs/requirements.txt
 
@@ -169,7 +163,7 @@ mlonmcu init
 
 Make sure to point the `MLONMCU_HOME` environment variable to the location of the previously initialied environment. (Alternative: use the `default` environment or `--home` argument on the command line)
 
-Next, generate a `requirements.txt` file inside the environment directory using `mlonmcu setup -g` which now be installed by running `pip install -r $MLONMCU_HOME/requirements.txt` inside the virtual Python environment.
+Next, generate a `requirements_addition.txt` file inside the environment directory using `mlonmcu setup -g` which now be installed by running `pip install -r $MLONMCU_HOME/requirements_addition.txt` inside the virtual Python environment.
 
 
 To use the created environment in a python program, a `MlonMcuContext` needs to be created as follows:
@@ -181,6 +175,14 @@ with mlonmcu.context.MlonMcuContext() as context:
     pass
 ```
 
+## List of interesting MLonMCU forks
+
+- MINRES TGC support: https://github.com/Minres/mlonmcu/tree/develop
+
+## List of existing MLonMCU extensions/plugins
+
+- ABC Example Plugin: coming soon!
+- MINRES TGC Support: coming soon!
 
 ## Development
 
@@ -200,18 +202,6 @@ Documentation is mainly generated automatically from doctrings (triggered via `m
 
 Regarding coding style, it is recommended to run `black` before every commit. The default line length should be given in the `setup.cfg` file.
 
-### Future Work
-
-- [ ] Finish beta version `v0.1.0`
-- [ ] Open Source Release
-  - [ ] Make repository public
-  - [ ] Release python package
-  - [ ] Publish docs automatically to https://readthedocs.org
-
-## Credits
-
-This is a research project proposed by the Chair of Design Automation of the Technical University of Munich.
-
 ### Developers
 
 - Rafael Stahl (TUM) [@rafzi]
@@ -222,6 +212,33 @@ This is a research project proposed by the Chair of Design Automation of the Tec
 
   - Came up with MLonMCU Python package
 
+## Publications
+
+- **MLonMCU: TinyML Benchmarking with Fast Retargeting** ([https://dl.acm.org/doi/10.1145/3637543.3652878](https://dl.acm.org/doi/abs/10.1145/3615338.3618128))
+
+  *CODAI '23: Proceedings of the 2023 Workshop on Compilers, Deployment, and Tooling for Edge AI*
+
+  BibTeX
+
+  ```bibtex
+  @inproceedings{10.1145/3615338.3618128,
+    author = {van Kempen, Philipp and Stahl, Rafael and Mueller-Gritschneder, Daniel and Schlichtmann, Ulf},
+    title = {MLonMCU: TinyML Benchmarking with Fast Retargeting},
+    year = {2024},
+    isbn = {9798400703379},
+    publisher = {Association for Computing Machinery},
+    address = {New York, NY, USA},
+    url = {https://doi.org/10.1145/3615338.3618128},
+    doi = {10.1145/3615338.3618128},
+    abstract = {While there exist many ways to deploy machine learning models on microcontrollers, it is non-trivial to choose the optimal combination of frameworks and targets for a given application. Thus, automating the end-to-end benchmarking flow is of high relevance nowadays. A tool called MLonMCU is proposed in this paper and demonstrated by benchmarking the state-of-the-art TinyML frameworks TFLite for Microcontrollers and TVM effortlessly with a large number of configurations in a low amount of time.},
+    booktitle = {Proceedings of the 2023 Workshop on Compilers, Deployment, and Tooling for Edge AI},
+    pages = {32–36},
+    numpages = {5},
+    keywords = {TinyML, neural networks, microcontrollers},
+    location = {Hamburg, Germany},
+    series = {CODAI '23}
+  }
+  ```
 
 ### Other
 This package was created with Cookiecutter_ and the `audreyr/cookiecutter-pypackage`_ project template. However most of the templates was manually changed to be in Markdown instead of reStructuredText.
@@ -230,6 +247,14 @@ This package was created with Cookiecutter_ and the `audreyr/cookiecutter-pypack
 - **`audreyr/cookiecutter-pypackage`:** https://github.com/audreyr/cookiecutter-pypackage
 
 
+## Acknowledgment
+
+<img src="./BMBF_gefoerdert_2017_en.jpg" alt="drawing" height="75" align="left" >
+
+This research is partially funded by the German Federal Ministry of Education and Research (BMBF) within
+the projects [Scale4Edge](https://www.edacentrum.de/scale4edge/) (grant number 16ME0127) and [MANNHEIM-FlexKI](https://www.edacentrum.de/projekte/MANNHEIM-FlexKI) (grant number 01IS22086L).
+
+
 # History
 
 ## 0.1.0 (2021-11-12)

{mlonmcu-0.3.0.dev0 → mlonmcu-0.4.0}/README.md

@@ -2,14 +2,15 @@
 
 [![pypi package](https://badge.fury.io/py/mlonmcu.svg)](https://pypi.org/project/mlonmcu)
 [![readthedocs](https://readthedocs.org/projects/mlonmcu/badge/?version=latest)](https://mlonmcu.readthedocs.io/en/latest/?version=latest)
-![covergae](https://byob.yarr.is/tum-ei-eda/mlonmcu/coverage)
+![coverage](https://byob.yarr.is/tum-ei-eda/mlonmcu/coverage)
 [![GitHub license](https://img.shields.io/github/license/tum-ei-eda/mlonmcu.svg)](https://github.com/tum-ei-eda/mlonmcu/blob/main/LICENSE)
 
 [![cicd workflow](https://github.com/tum-ei-eda/mlonmcu/actions/workflows/cicd.yml/badge.svg)](https://github.com/tum-ei-eda/mlonmcu/actions/workflows/cicd.yml)
+[![lint workflow](https://github.com/tum-ei-eda/mlonmcu/actions/workflows/style.yml/badge.svg)](https://github.com/tum-ei-eda/mlonmcu/actions/workflows/style.yml)
 [![demo workflow](https://github.com/tum-ei-eda/mlonmcu/actions/workflows/demo.yml/badge.svg)](https://github.com/tum-ei-eda/mlonmcu/actions/workflows/demo.yml)
-[![container workflow](https://github.com/tum-ei-eda/mlonmcu/actions/workflows/container.yml/badge.svg)](https://github.com/tum-ei-eda/mlonmcu/actions/workflows/container.yml)
 [![bench workflow](https://github.com/tum-ei-eda/mlonmcu/actions/workflows/bench.yml/badge.svg)](https://github.com/tum-ei-eda/mlonmcu/actions/workflows/bench.yml)
 
+
 This project contains research code related to the deployment of inference or learning applications on tiny micro-controllers.
 
 
@@ -22,11 +23,7 @@ This project contains research code related to the deployment of inference or le
 
 - Highly configurable python package
 - Automatic resolution and installation of dependencies
-- Supporting a large combination of frameworks/backends/targets/features:
-  - Frameworks (Backends): TFLite Micro, MircoTVM
-  - Targets: Host (x86), ETISS (Pulpino), Corstone300 (ARM Cortex-M), QEMU (RISC-V), Spike (`riscv-isa-sim`),...
-  - Platforms: Zephyr, ESP-IDF,...
-  - Features: Autotuning, Debugging, RPC,...
+- Supporting a large combination of frameworks/backends/targets/features
 - Build-in parallel processing of large number of benchmarks
 - Isolated enironments (not interfering with other installations)
 - Command Line and Python Development Interfaces
@@ -47,10 +44,10 @@ First, a set of APT packages needs to be installed:
 sudo apt install python3-pip python3-venv
 
 # MLonMCU related
-sudo apt install libboost-system-dev libboost-filesystem-dev libboost-program-options-dev graphviz doxygen libtinfo-dev zlib1g-dev texinfo unzip device-tree-compiler tree g++
+sudo apt install libboost-all-dev graphviz doxygen libtinfo-dev zlib1g-dev texinfo unzip device-tree-compiler tree g++
 
 # Optional (depending on configuration)
-sudo apt install ninja-build
+sudo apt install ninja-build flex lsb-release libelf-dev
 ```
 
 Also make sure that your default Python is at least v3.7. If the `python` command is not available in your shell or points Python v2.7 check out `python-is-python3`.
@@ -80,10 +77,7 @@ source .venv/bin/activate
 
 **Warning:** It is recommended to have at least version 3.20 of CMake installed for full compatibility!
 
-# Install minimal dependencies
-pip install -r requirements_dev.txt
-
-# Optional (only for development)
+# Install ptional dependecies (only for development)
 pip install -r requirements_dev.txt
 pip install -r docs/requirements.txt
 
@@ -128,7 +122,7 @@ mlonmcu init
 
 Make sure to point the `MLONMCU_HOME` environment variable to the location of the previously initialied environment. (Alternative: use the `default` environment or `--home` argument on the command line)
 
-Next, generate a `requirements.txt` file inside the environment directory using `mlonmcu setup -g` which now be installed by running `pip install -r $MLONMCU_HOME/requirements.txt` inside the virtual Python environment.
+Next, generate a `requirements_addition.txt` file inside the environment directory using `mlonmcu setup -g` which now be installed by running `pip install -r $MLONMCU_HOME/requirements_addition.txt` inside the virtual Python environment.
 
 
 To use the created environment in a python program, a `MlonMcuContext` needs to be created as follows:
@@ -140,6 +134,14 @@ with mlonmcu.context.MlonMcuContext() as context:
     pass
 ```
 
+## List of interesting MLonMCU forks
+
+- MINRES TGC support: https://github.com/Minres/mlonmcu/tree/develop
+
+## List of existing MLonMCU extensions/plugins
+
+- ABC Example Plugin: coming soon!
+- MINRES TGC Support: coming soon!
 
 ## Development
 
@@ -159,18 +161,6 @@ Documentation is mainly generated automatically from doctrings (triggered via `m
 
 Regarding coding style, it is recommended to run `black` before every commit. The default line length should be given in the `setup.cfg` file.
 
-### Future Work
-
-- [ ] Finish beta version `v0.1.0`
-- [ ] Open Source Release
-  - [ ] Make repository public
-  - [ ] Release python package
-  - [ ] Publish docs automatically to https://readthedocs.org
-
-## Credits
-
-This is a research project proposed by the Chair of Design Automation of the Technical University of Munich.
-
 ### Developers
 
 - Rafael Stahl (TUM) [@rafzi]
@@ -181,9 +171,44 @@ This is a research project proposed by the Chair of Design Automation of the Tec
 
   - Came up with MLonMCU Python package
 
+## Publications
+
+- **MLonMCU: TinyML Benchmarking with Fast Retargeting** ([https://dl.acm.org/doi/10.1145/3637543.3652878](https://dl.acm.org/doi/abs/10.1145/3615338.3618128))
+
+  *CODAI '23: Proceedings of the 2023 Workshop on Compilers, Deployment, and Tooling for Edge AI*
+
+  BibTeX
+
+  ```bibtex
+  @inproceedings{10.1145/3615338.3618128,
+    author = {van Kempen, Philipp and Stahl, Rafael and Mueller-Gritschneder, Daniel and Schlichtmann, Ulf},
+    title = {MLonMCU: TinyML Benchmarking with Fast Retargeting},
+    year = {2024},
+    isbn = {9798400703379},
+    publisher = {Association for Computing Machinery},
+    address = {New York, NY, USA},
+    url = {https://doi.org/10.1145/3615338.3618128},
+    doi = {10.1145/3615338.3618128},
+    abstract = {While there exist many ways to deploy machine learning models on microcontrollers, it is non-trivial to choose the optimal combination of frameworks and targets for a given application. Thus, automating the end-to-end benchmarking flow is of high relevance nowadays. A tool called MLonMCU is proposed in this paper and demonstrated by benchmarking the state-of-the-art TinyML frameworks TFLite for Microcontrollers and TVM effortlessly with a large number of configurations in a low amount of time.},
+    booktitle = {Proceedings of the 2023 Workshop on Compilers, Deployment, and Tooling for Edge AI},
+    pages = {32–36},
+    numpages = {5},
+    keywords = {TinyML, neural networks, microcontrollers},
+    location = {Hamburg, Germany},
+    series = {CODAI '23}
+  }
+  ```
 
 ### Other
 This package was created with Cookiecutter_ and the `audreyr/cookiecutter-pypackage`_ project template. However most of the templates was manually changed to be in Markdown instead of reStructuredText.
 
 - **Cookiecutter:** https://github.com/audreyr/cookiecutter
 - **`audreyr/cookiecutter-pypackage`:** https://github.com/audreyr/cookiecutter-pypackage
+
+
+## Acknowledgment
+
+<img src="./BMBF_gefoerdert_2017_en.jpg" alt="drawing" height="75" align="left" >
+
+This research is partially funded by the German Federal Ministry of Education and Research (BMBF) within
+the projects [Scale4Edge](https://www.edacentrum.de/scale4edge/) (grant number 16ME0127) and [MANNHEIM-FlexKI](https://www.edacentrum.de/projekte/MANNHEIM-FlexKI) (grant number 01IS22086L).

{mlonmcu-0.3.0.dev0 → mlonmcu-0.4.0}/mlonmcu/artifact.py

@@ -26,7 +26,6 @@ from mlonmcu.setup import utils
 # TODO: offer pack/unpack/flatten methods for mlf
 # TODO: implement restore methods
 # TODO: decide if inheritance based scheme would fit better
-# TODO: add artifact flags and lookup utility to find best match
 
 
 class ArtifactFormat(Enum):  # TODO: ArtifactType, ArtifactKind?
@@ -41,15 +40,18 @@ class ArtifactFormat(Enum):  # TODO: ArtifactType, ArtifactKind?
     DATA = 6
     NUMPY = 7
     PARAMS = 8
-    JSON = 9  # ?
+    JSON = 9  # TODO: how about YAML or more general: DICT?
     PATH = 10  # NOT A DIRECTORY?
     RAW = 11
     BIN = 11
     SHARED_OBJECT = 12  # Here: the parent tar archive
+    ARCHIVE = 13
 
 
 def lookup_artifacts(artifacts, name=None, fmt=None, flags=None, first_only=False):
     """Utility to get a matching artifact for a given set of properties."""
+    if isinstance(name, Path):
+        name = str(name)
     matches = []
     # Warning: if neither name, fmt nor flags is provided, the first artifact (first_only=True)
     # or all (first_only=False) are returned
@@ -112,7 +114,7 @@ class Artifact:
             assert self.content is not None
         elif self.fmt in [ArtifactFormat.RAW, ArtifactFormat.BIN]:
             assert self.raw is not None
-        elif self.fmt in [ArtifactFormat.MLF, ArtifactFormat.SHARED_OBJECT]:
+        elif self.fmt in [ArtifactFormat.MLF, ArtifactFormat.SHARED_OBJECT, ArtifactFormat.ARCHIVE]:
             assert self.raw is not None
         elif self.fmt in [ArtifactFormat.PATH]:
             assert self.path is not None
@@ -130,10 +132,12 @@ class Artifact:
             If archive: extract to destination.
 
         """
+        if not isinstance(dest, Path):
+            dest = Path(dest)
         if dest.is_dir():
-            filename = Path(dest) / self.name
+            filename = dest / self.name
         else:
-            filename = Path(dest)
+            filename = dest
         if self.fmt in [ArtifactFormat.TEXT, ArtifactFormat.SOURCE]:
             assert not extract, "extract option is only available for ArtifactFormat.MLF"
             with open(filename, "w", encoding="utf-8") as handle:
@@ -142,7 +146,7 @@ class Artifact:
             assert not extract, "extract option is only available for ArtifactFormat.MLF"
             with open(filename, "wb") as handle:
                 handle.write(self.raw)
-        elif self.fmt in [ArtifactFormat.MLF, ArtifactFormat.SHARED_OBJECT]:
+        elif self.fmt in [ArtifactFormat.MLF, ArtifactFormat.SHARED_OBJECT, ArtifactFormat.ARCHIVE]:
             with open(filename, "wb") as handle:
                 handle.write(self.raw)
             if extract:
@@ -164,7 +168,7 @@ class Artifact:
             print(self.content)
         elif self.fmt in [ArtifactFormat.RAW, ArtifactFormat.BIN]:
             print(f"Data Size: {len(self.raw)}B")
-        elif self.fmt in [ArtifactFormat.MLF, ArtifactFormat.SHARED_OBJECT]:
+        elif self.fmt in [ArtifactFormat.MLF, ArtifactFormat.SHARED_OBJECT, ArtifactFormat.ARCHIVE]:
             print(f"Archive Size: {len(self.raw)}B")
         elif self.fmt in [ArtifactFormat.PATH]:
             print(f"File Location: {self.path}")

{mlonmcu-0.3.0.dev0 → mlonmcu-0.4.0}/mlonmcu/cli/common.py

@@ -20,6 +20,7 @@ import os
 import sys
 import multiprocessing
 import logging
+import argparse
 
 from mlonmcu.platform import get_platforms
 from mlonmcu.session.postprocess import SUPPORTED_POSTPROCESSES
@@ -29,6 +30,8 @@ from .helper.parse import extract_config
 
 logger = get_logger()
 
+NUM_GEN_ARGS = 9
+
 
 def handle_logging_flags(args):
     if hasattr(args, "verbose") and hasattr(args, "quiet"):
@@ -90,15 +93,6 @@ def add_flow_options(parser):
         choices=get_available_feature_names(),
         help="Enabled features for target/framework/backend (choices: %(choices)s)",
     )
-    flow_parser.add_argument(
-        "--feature-gen",
-        dest="feature_gen",
-        type=str,
-        metavar="FEATURES",
-        nargs="+",
-        action="append",
-        help="Generator statement for features.",
-    )
     flow_parser.add_argument(
         "-c",
         "--config",
@@ -114,14 +108,37 @@ def add_flow_options(parser):
             "values are always treated as strings."
         ),
     )
-    flow_parser.add_argument(
-        "--config-gen",
-        dest="config_gen",
-        metavar="KEY=VALUE",
-        nargs="+",
-        action="append",
-        help="Generator statement for configs.",
-    )
+
+    def add_gen_args(parser, number):
+        for i in range(number):
+            suffix = str(i + 1) if i > 0 else ""
+            parser.add_argument(
+                "--feature-gen" + suffix,
+                dest="feature_gen" + suffix,
+                type=str,
+                metavar="FEATURES",
+                nargs="+",
+                action="append",
+                help=(
+                    f"Generator statement for features. (Also available: --feature-gen2 ... --feature-gen{number})"
+                    if i == 0
+                    else argparse.SUPPRESS
+                ),
+            )
+            flow_parser.add_argument(
+                "--config-gen" + suffix,
+                dest="config_gen" + suffix,
+                metavar="KEY=VALUE",
+                nargs="+",
+                action="append",
+                help=(
+                    f"Generator statement for configs. (Also available: --config-gen2 ... --config-gen{number})"
+                    if i == 0
+                    else argparse.SUPPRESS
+                ),
+            )
+
+    add_gen_args(flow_parser, NUM_GEN_ARGS)
     flow_parser.add_argument(
         "--parallel",
         metavar="THREADS",
@@ -213,15 +230,16 @@ def kickoff_runs(args, until, context):
         print_report = bool(config["print_report"])
     elif "print_report" in context.environment.vars:
         print_report = bool(context.environment.vars["print_report"])
-    success = session.process_runs(
-        until=until,
-        per_stage=per_stage,
-        print_report=print_report,
-        num_workers=args.parallel,
-        progress=args.progress,
-        context=context,
-        export=True,
-    )
+    with session:
+        success = session.process_runs(
+            until=until,
+            per_stage=per_stage,
+            print_report=print_report,
+            num_workers=args.parallel,
+            progress=args.progress,
+            context=context,
+            export=True,
+        )
     if not success:
         logger.error("At least one error occured!")
         sys.exit(1)

{mlonmcu-0.3.0.dev0 → mlonmcu-0.4.0}/mlonmcu/cli/helper/parse.py

@@ -16,6 +16,10 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 #
+import itertools
+from collections import ChainMap
+
+NUM_GEN_ARGS = 9
 
 
 def parse_var(s):
@@ -57,16 +61,32 @@ def extract_feature_names(args):
         features = args.feature
     else:
         features = []
-    if hasattr(args, "feature_gen") and args.feature_gen:
+
+    def helper(args, name):
         gen = []
-        for x in args.feature_gen:
-            if "_" in x:
-                assert len(x) == 1
-                gen.append([])
-            else:
-                gen.append(x)
-    else:
-        gen = [[]]
+        if hasattr(args, name):
+            feature_gen = getattr(args, name)
+            if not feature_gen:
+                gen = [[]]
+                return gen
+            for x in feature_gen:
+                if "_" in x:
+                    assert len(set(x)) == 1
+                    gen.append([])
+                else:
+                    gen.append(x)
+        else:
+            gen = [[]]
+        return gen
+
+    gens = []
+    for i in range(NUM_GEN_ARGS):
+        suffix = str(i + 1) if i > 0 else ""
+        gen = helper(args, "feature_gen" + suffix)
+        gens.append(gen)
+
+    gen = list(map(lambda x: sum(x, []), (itertools.product(*gens))))
+
     return features, gen
 
 
@@ -76,17 +96,32 @@ def extract_config(args):
         configs = parse_vars(configs)
     else:
         configs = {}
-    if hasattr(args, "config_gen") and args.config_gen:
+
+    def helper(args, name):
         gen = []
-        for x in args.config_gen:
-            if "_" in x:
-                assert len(x) == 1
-                gen.append({})
-            else:
-                c = parse_vars(x)
-                gen.append(c)
-    else:
-        gen = [{}]
+        if hasattr(args, name):
+            config_gen = getattr(args, name)
+            if not config_gen:
+                gen = [{}]
+                return gen
+            for x in config_gen:
+                if "_" in x:
+                    assert len(set(x)) == 1
+                    gen.append({})
+                else:
+                    c = parse_vars(x)
+                    gen.append(c)
+        else:
+            gen = [{}]
+        return gen
+
+    gens = []
+    for i in range(NUM_GEN_ARGS):
+        suffix = str(i + 1) if i > 0 else ""
+        gen = helper(args, "config_gen" + suffix)
+        gens.append(gen)
+
+    gen = list(map(lambda x: dict(ChainMap(*x)), (itertools.product(*gens))))
 
     return configs, gen
 

{mlonmcu-0.3.0.dev0 → mlonmcu-0.4.0}/mlonmcu/cli/main.py

@@ -81,6 +81,7 @@ def handle_docker(args):
 # def main(args):
 def main(args=None):
     """Console script for mlonmcu."""
+
     parser = argparse.ArgumentParser(
         description="ML on MCU Flow",
         formatter_class=argparse.ArgumentDefaultsHelpFormatter,

{mlonmcu-0.3.0.dev0 → mlonmcu-0.4.0}/mlonmcu/config.py

@@ -78,7 +78,14 @@ def filter_config(config, prefix, defaults, optionals, required_keys):
     ------
     AssertionError: If a required key is missing.
     """
-    cfg = remove_config_prefix(config, prefix, skip=required_keys + optionals)
+    if not isinstance(required_keys, set):
+        logger.warning(
+            "Deprecated: FEATURES, REQUIRED, OPTIONAL should now be sets, not lists for component %s", prefix
+        )
+        required_keys = set(required_keys)
+    if not isinstance(optionals, set):
+        optionals = set(optionals)
+    cfg = remove_config_prefix(config, prefix, skip=required_keys | optionals)
     for required in required_keys:
         value = None
         if required in cfg:
@@ -102,7 +109,7 @@ def filter_config(config, prefix, defaults, optionals, required_keys):
             cfg[key] = defaults[key]
 
     for key in cfg:
-        if key not in list(defaults.keys()) + required_keys:
+        if key not in set(defaults.keys()) | required_keys:
             # logger.warn("Component received an unknown config key: %s", key)
             pass
 
@@ -210,10 +217,30 @@ def str2bool(value, allow_none=False):
     if value is None:
         assert allow_none, "str2bool received None value while allow_none=False"
         return value
+    if isinstance(value, (int, bool)):
+        return bool(value)
     assert isinstance(value, str)
-    return bool(value) if isinstance(value, (int, bool)) else bool(distutils.util.strtobool(value))
+    return bool(distutils.util.strtobool(value))
 
 
-def str2dict(value):
+def str2dict(value, allow_none=False):
+    if value is None:
+        assert allow_none, "str2dict received None value while allow_none=False"
+        return value
+    if isinstance(value, dict):
+        return value
     assert isinstance(value, str)
+    # TODO: parse key=value,key2=value2 via regex
     return dict(ast.literal_eval(value))
+
+
+def str2list(value, allow_none=False):
+    if value is None:
+        assert allow_none, "str2list received None value while allow_none=False"
+        return value
+    if isinstance(value, (list, set, tuple)):
+        return list(value)
+    assert isinstance(value, str)
+    if value.startswith("["):
+        return list(ast.literal_eval(value))
+    return value.split(",")