simstack 2.0.1__py3-none-any.whl

simstack/static/docs/_sources/array_lists.rst.txt

@@ -0,0 +1,60 @@
+.. _array_lists:
+
+Array Lists
+===========
+
+SimStack II provides a specialized way to handle collections of arrays (typically numpy arrays) through the ``ArrayList`` and ``ArrayStorage`` models.
+
+ArrayStorage
+------------
+
+``ArrayStorage`` is the model used to store an individual array. It handles compression and serialization of the array data.
+
+**Key Features:**
+
+* **Compression**: Array data is automatically compressed before being stored in the database.
+* **Shape Preservation**: Automatically stores and restores the shape of the array.
+* **Property Access**: Use the ``array`` property to get or set the numpy array directly.
+
+**Usage:**
+
+.. code-block:: python
+
+   import numpy as np
+   from simstack.models.array_storage import ArrayStorage
+
+   storage = ArrayStorage(name="my_array")
+   storage.array = np.random.rand(10, 10)
+
+ArrayList
+---------
+
+``ArrayList`` is a top-level ``Model`` that stores a collection of ``ArrayStorage`` objects. Like ``FileListModel``, it inherits from ``ObjectListMixin``, providing a familiar list-like interface.
+
+**Usage:**
+
+.. code-block:: python
+
+   from simstack.models.array_list import ArrayList
+   from simstack.models.array_storage import ArrayStorage
+   import numpy as np
+
+   array_list = ArrayList()
+   
+   storage = ArrayStorage(name="first")
+   storage.array = np.zeros((5, 5))
+   
+   array_list.append(storage)
+
+API Reference
+-------------
+
+.. autoclass:: simstack.models.array_list.ArrayList
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+.. autoclass:: simstack.models.array_storage.ArrayStorage
+   :members:
+   :undoc-members:
+   :show-inheritance:

simstack/static/docs/_sources/ci_pipeline.rst.txt

@@ -0,0 +1,34 @@
+.. _ci_pipeline:
+
+CI Pipeline and Documentation Artifacts
+=======================================
+
+The SimStack II project uses GitHub Actions for Continuous Integration. One of the jobs in the pipeline is ``package-html-docs``, which automatically generates the API documentation and builds the Sphinx HTML documentation for the installed package.
+
+Where the HTML documentation lives
+----------------------------------
+
+The generated HTML documentation is owned by the ``simstack`` package and is stored under:
+
+.. code-block:: text
+
+   src/simstack/static/docs
+
+This directory is included as package data. Any application that installs ``simstack`` can serve the generated HTML from the installed package instead of rebuilding or copying documentation from this repository.
+
+How the CI job updates docs
+---------------------------
+
+On branch pushes and manual workflow runs, the ``package-html-docs`` job:
+
+1. Runs ``sphinx-apidoc`` to refresh generated API documentation sources.
+2. Builds Sphinx HTML into ``src/simstack/static/docs``.
+3. Uploads the generated HTML as the ``sphinx-docs`` artifact for inspection.
+4. Commits changed generated HTML back to the same branch.
+
+The auto-commit message contains ``[skip ci]`` so the generated-docs commit does not start another CI run.
+
+How other projects should consume docs
+--------------------------------------
+
+Other projects should not download CI artifacts or rebuild these docs. They should install the desired ``simstack`` revision and use/serve ``simstack/static/docs`` from the installed package.

simstack/static/docs/_sources/data.rst.txt

@@ -0,0 +1,175 @@
+.. _persisting_data:
+
+Persisting and Visualizing Data
+===============================
+
+.. note::
+    This section is intended for node-developers.
+
+Simstack persists data using a MongoDB database, which is exposed via the
+fast-api server to a web-frontend. Each user has a dedicated database
+on a MongoDB server which is configured in the configuration file
+(:ref:`configuration-file`).
+
+To facilitate access to the database in the python workflows, Simstack II uses the odmantic
+package to provide an object-oriented access to the data via odmantic
+Models. Odmantic Models are drived from pydantic (V2) classes and developers
+of nodes (:ref:`workflows`) can define their own classes deriving from Model.
+
+To customize the appearance of the models in the UI, the model definition
+should be decorated by ``@simstack_model``
+(:func:`simstack.core.simstack_model.simstack_model`)
+which provides ui-related functionality discussed below
+(:ref:`model-appearance`)
+
+Model Definitions
+~~~~~~~~~~~~~~~~~~
+
+For detailed information on odmantic models, consult the odmantic
+documentation.
+
+A basic model definition look like:
+
+.. code-block:: python
+
+   from typing import Optional, List
+   from odmantic import Model, EmbeddedModel, Field
+
+   @simstack_model
+   class OtherModel(Model):
+        observations: List[float]
+        max: float
+
+   @simstack_model
+   class SmallData(EmbeddedModel)
+        number: int
+
+   @simstack_model
+   class ComplexData(Model)
+        counter: int = Field(default=1)
+        limit: Optional[int] = None
+        other_model: OtherModel = Reference() # this defines a reference to the
+                                              # other table in the db
+                                              # loading ComplexData will load
+                                              # the other model too
+        small_data: SmallData # directly stored in the complex_data table
+
+
+
+.. important::
+   - The node developer does not have explicitly save models to the db, the
+     @node decorator will take care of this.
+   - There is **no Model inheritance**, which is a real pain.
+   - ODMantic Issue #484: https://github.com/art049/odmantic/issues/484
+
+.. note::
+    Even though its formally allowed do not use Dict[str,Any] in models,
+    because the UI behavior is unpredictable (see below).
+    Dict[str,known_type] is ok.
+
+**Standard Models**
+
+:mod:`simstack.models.base_types` provides basic models:
+
+- IntData
+- StrData
+- FloatData
+- BooleanData
+
+And :mod:`simstack.models.array_storage`:
+
+- ArrayStorage
+- ArrayList
+
+And the list mixins for creating custom list models (see :doc:`lists`):
+
+- GenericListMixin
+- ObjectListMixin
+
+And for file management (see :doc:`files`):
+
+- FileStack
+- FileList
+- FileListModel
+
+And the parallel execution utilities (see :doc:`mass_runner` and :doc:`fire_and_forget_runner`):
+
+- MassRunner
+- FireAndForgetRunner
+
+:class:`simstack.models.array_storage.ArrayStorage` is an example of a class which
+packs its content by serializing and deserializing it. This is the
+recommended procedure for all content which MongoDB cannot natively store. To
+interact with such classes, the developer has to provide access functions and
+a member function ``custom_model_dump`` which the fast-api server to convert
+the content of the class into a dict which can be visualized in the GUI.
+
+.. _model-appearance:
+
+Model Appearance in the GUI
+---------------------------
+
+The :class:`simstack.models.models.ModelTable` table in the database stores
+for each class a json-schema (derived from the pydantic json-schema) and a
+``ui_scheme`` which is created by
+:func:`simstack.core.model_table.make_model_table`.
+
+The UI used a package react-json-forms (rjsf) to interpret these schema to
+provide input and output representations of the model. When queried the
+routes of the fast-api server provide (get) or store (post) dicts that are
+compatible with the json-schema. 8
+
+**Class Methods**
+
++---------------+-------------------------------------+------------------+
+| Function Name | Purpose                             | default          |
++---------------+-------------------------------------+------------------+
+| json_schema   | json schema for rjsf                | pydantic-schema  |
++---------------+-------------------------------------+------------------+
+| ui_schema     | ui_schema dict for rjsf             | hide id          |
++---------------+-------------------------------------+------------------+
+| ui_make_title | title for the model in ui           | class name       |
++---------------+-------------------------------------+------------------+
+| from_dict     | create model instance from dict     | odmantic func    |
++---------------+-------------------------------------+------------------+
+| from_model    | copy constructor                    | odmantic default |
++---------------+-------------------------------------+------------------+
+
+**Instance Functions**
+
++-------------------+--------------------------------+------------------+
+| Function Name     | Purpose                        | default          |
++-------------------+--------------------------------+------------------+
+| custom_model_dump | convert model to dict          | model_dump       |
++-------------------+--------------------------------+------------------+
+
+Advanced Tools
+~~~~~~~~~~~~~~
+
+Conditional Schema
+
+.. code-block:: python
+
+   schema['properties']['test'] = {
+         "type": "object",
+         "oneOf": [
+             {
+                 "properties": {
+                     "lorem": {
+                         "type": "string"
+                     }
+                 },
+                 "required": [
+                     "lorem"
+                 ]
+             },
+             {
+                 "properties": {
+                     "ipsum": {
+                         "type": "string"
+                     }
+                 },
+                 "required": [
+                     "ipsum"
+                 ]
+             }

simstack/static/docs/_sources/database_migrations.rst.txt

@@ -0,0 +1,310 @@
+.. _database_migrations:
+
+Database Migrations
+===================
+
+Use a migration when stored MongoDB data must change: model updates, field
+backfills, key renames, embedded-model rewrites, index changes, or one-off data
+repairs.
+
+Add One
+-------
+
+1. Define the target state first: model code, collection shape, and recovery
+   flow.
+2. Add one file under ``server/migrations/`` named
+   ``YYYYMMDDHHMMSS_short_name.py``.
+3. Set ``name`` to the file stem and ``dependencies`` to the latest migration.
+4. Take a backup before running the migration on any real workspace database.
+5. Check ``CONNECTION_STRING`` before running:
+
+   - The current runner requires a MongoDB user with ``admin.root``.
+   - This applies to both ``--scope ALL`` and ``--scope DATABASES`` in the
+     current implementation.
+
+6. Decide rollout before writing code:
+
+   - **If old runners or old server code cannot read the new schema, stop
+     runners and stop writes until the migration is finished.**
+   - If a mixed schema must be tolerated for a short rollout window, ship
+     compatibility code first, run the migration second, remove compatibility
+     code last.
+
+7. Define recovery before merge:
+
+   - Prefer a real ``downgrade``.
+   - If downgrade would drop data or cannot fully restore the old state,
+     document a forward-only recovery flow:
+     backup, stop writers, rerun or restore.
+
+8. Keep the migration strict:
+
+   - Touch only the affected collections and fields.
+   - Fail fast on unsupported data.
+   - Log migration-specific counts or checkpoints. The runner already logs the
+     target database.
+   - If the migration is started again after a partial failure, it must not
+     duplicate or corrupt already migrated data.
+
+9. Test on one workspace DB first.
+
+   .. code-block:: bash
+
+      uv run python -m server.migrations.run_workspace_migrations \
+        --scope DATABASES \
+        --databases my_test_user_db \
+        show
+
+      uv run python -m server.migrations.run_workspace_migrations \
+        --scope DATABASES \
+        --databases my_test_user_db \
+        migrate
+
+      uv run python -m server.migrations.run_workspace_migrations \
+        --scope ALL \
+        migrate
+
+10. ``db_migrations.yml`` runs ``--scope ALL migrate`` by default. Use that
+    only after one workspace DB is verified.
+
+    .. code-block:: bash
+
+       docker compose -f db_migrations.yml run --rm simstack-migrations
+
+    To target one database from Docker, override the command:
+
+    .. code-block:: bash
+
+       docker compose -f db_migrations.yml run --rm simstack-migrations \
+         uv run python -m server.migrations.run_workspace_migrations \
+         --scope DATABASES \
+         --databases my_test_user_db \
+         migrate
+
+Example 1: Add Field - ResourceDefinition.is_enabled
+----------------------------------------------------
+
+Case: ``ResourceDefinition`` gets a new field ``is_enabled: bool = True`` and
+old resource documents must receive the default.
+
+Model change:
+
+.. code-block:: python
+
+   is_enabled: bool = True
+
+Create ``server/migrations/20260527120000_resource_definition_is_enabled.py``:
+
+.. code-block:: python
+
+   """Backfill ResourceDefinition.is_enabled with True when missing."""
+   from __future__ import annotations
+
+   import logging
+
+   name = "20260527120000_resource_definition_is_enabled"
+   dependencies = ["20260527110000_previous_migration"]
+
+   logger = logging.getLogger(__name__)
+   MISSING_IS_ENABLED_QUERY = {"is_enabled": {"$exists": False}}
+
+
+   def upgrade(db):
+       resources_collection = db["resource_definition"]
+       result = resources_collection.update_many(
+           MISSING_IS_ENABLED_QUERY,
+           {"$set": {"is_enabled": True}},
+       )
+
+       remaining_missing_count = resources_collection.count_documents(MISSING_IS_ENABLED_QUERY)
+       if remaining_missing_count != 0:
+           raise RuntimeError(
+               f"[{name}] Expected 0 resources without is_enabled, found {remaining_missing_count}."
+           )
+
+       logger.info("[%s] Completed database=%s backfilled_resources=%s", name, db.name, result.modified_count)
+
+
+   def downgrade(db):
+       resources_collection = db["resource_definition"]
+       result = resources_collection.update_many(
+           {},
+           {"$unset": {"is_enabled": ""}},
+       )
+       logger.info("[%s] Completed database=%s removed_is_enabled_field=%s", name, db.name, result.modified_count)
+
+Example 2: Edit Field - QMInput.print_level Enum Rewrite
+--------------------------------------------------------
+
+Case: ``QMInput.print_level`` changes from legacy integers ``0``-``4`` to the
+existing ``PrintLevel`` enum strings.
+
+Model change:
+
+.. code-block:: python
+
+   print_level: PrintLevel = Field(
+       PrintLevel.LOW,
+       json_schema_extra={"description": "Print level for the calculation"},
+   )
+
+Create ``server/migrations/20260527121000_qm_input_print_level_enum.py``:
+
+.. code-block:: python
+
+   """Convert QMInput.print_level from legacy integers to PrintLevel strings."""
+   from __future__ import annotations
+
+   import logging
+
+   name = "20260527121000_qm_input_print_level_enum"
+   dependencies = ["20260527120000_resource_definition_is_enabled"]
+
+   logger = logging.getLogger(__name__)
+
+   PRINT_LEVEL_BY_INT = {
+       0: "SILENT",
+       1: "LOW",
+       2: "MEDIUM",
+       3: "HIGH",
+       4: "EXTREME",
+   }
+
+
+   def upgrade(db):
+       qm_inputs = db["qm_input"]
+       migrated_count = 0
+
+       for old_value, new_value in PRINT_LEVEL_BY_INT.items():
+           result = qm_inputs.update_many(
+               {"print_level": old_value},
+               {"$set": {"print_level": new_value}},
+           )
+           migrated_count += result.modified_count
+
+       remaining_legacy_count = qm_inputs.count_documents({"print_level": {"$type": "int"}})
+       if remaining_legacy_count:
+           raise RuntimeError(
+               f"[{name}] Found {remaining_legacy_count} QMInput documents with unsupported integer print_level."
+           )
+
+       logger.info("[%s] database=%s migrated_print_levels=%s", name, db.name, migrated_count)
+
+
+   def downgrade(db):
+       qm_inputs = db["qm_input"]
+       migrated_count = 0
+
+       for old_value, new_value in PRINT_LEVEL_BY_INT.items():
+           result = qm_inputs.update_many(
+               {"print_level": new_value},
+               {"$set": {"print_level": old_value}},
+           )
+           migrated_count += result.modified_count
+
+       logger.info("[%s] database=%s downgraded_print_levels=%s", name, db.name, migrated_count)
+
+Example 3: Delete Field - QMInput.name
+--------------------------------------
+
+Case: ``QMInput`` drops the legacy ``name`` field because the node title is the
+single source of truth and old documents must stop storing the duplicate value.
+
+Model change:
+
+.. code-block:: python
+
+   # Remove this field from QMInput
+   name: Optional[str] = None
+
+Create ``server/migrations/20260527122000_qm_input_remove_name.py``:
+
+.. code-block:: python
+
+   """Remove the legacy QMInput.name field."""
+   from __future__ import annotations
+
+   import logging
+
+   name = "20260527122000_qm_input_remove_name"
+   dependencies = ["20260527121000_qm_input_print_level_enum"]
+
+   logger = logging.getLogger(__name__)
+   NAME_EXISTS_QUERY = {"name": {"$exists": True}}
+
+
+   def upgrade(db):
+       qm_inputs = db["qm_input"]
+       result = qm_inputs.update_many(NAME_EXISTS_QUERY, {"$unset": {"name": ""}})
+
+       remaining_name_count = qm_inputs.count_documents(NAME_EXISTS_QUERY)
+       if remaining_name_count != 0:
+           raise RuntimeError(f"[{name}] Expected 0 QMInput documents with legacy name, found {remaining_name_count}.")
+
+       logger.info("[%s] database=%s removed_legacy_name=%s", name, db.name, result.modified_count)
+
+
+   def downgrade(db):
+       raise RuntimeError(
+           f"[{name}] Forward-only migration. Restore QMInput.name from backup if rollback is required."
+       )
+
+Example 4: Edit Embedded Model - FileList to References
+-------------------------------------------------------
+
+Case: a parent document stores a full ``FileList`` object inside its own data,
+and each ``FileStack`` inside that list must be moved to its own collection.
+
+Partial migration shape:
+
+.. code-block:: python
+
+   def _migrate_embedded_file_stack(file_stack_collection, file_stack_data: dict):
+       file_stack_id = file_stack_data.setdefault("_id", ObjectId())
+       existing = file_stack_collection.find_one({"_id": file_stack_id})
+       if existing is None:
+           file_stack_collection.insert_one(file_stack_data)
+
+       return file_stack_id
+
+
+   def _migrate_file_list_data(file_stack_collection, file_list_data: dict) -> bool:
+       if "file_stacks" not in file_list_data:
+           return False
+
+       new_elements = []
+       for file_stack in file_list_data["file_stacks"]:
+           if isinstance(file_stack, dict):
+               file_stack_id = _migrate_embedded_file_stack(file_stack_collection, file_stack)
+               new_elements.append(file_stack_id)
+           else:
+               new_elements.append(file_stack)
+
+       del file_list_data["file_stacks"]
+       file_list_data["elements"] = new_elements
+       return True
+
+
+   def _migrate_parent_collection(parent_collection, parent_fields: list[str], file_stack_collection) -> int:
+       migrated_count = 0
+
+       for doc in parent_collection.find():
+           changed = False
+
+           for parent_field in parent_fields:
+               parent_value = doc.get(parent_field)
+               if isinstance(parent_value, dict) and _migrate_file_list_data(file_stack_collection, parent_value):
+                   changed = True
+
+           if changed:
+               parent_collection.replace_one({"_id": doc["_id"]}, doc)
+               migrated_count += 1
+
+       return migrated_count
+
+
+   def upgrade(db):
+       file_stack_collection = db["file_stack"]
+       node_registry_count = _migrate_parent_collection(db["node_registry"], ["info_files"], file_stack_collection)
+       file_list_io_count = _migrate_parent_collection(db["file_list_io"], ["file_list"], file_stack_collection)
+       logger.info("[%s] database=%s migrated_node_registry=%s migrated_file_list_io=%s", name, db.name, node_registry_count, file_list_io_count)

simstack/static/docs/_sources/datasets.rst.txt

@@ -0,0 +1,93 @@
+.. _datasets:
+
+Datasets
+========
+
+SimStack II uses a structured way to handle datasets through the ``DataSet``, ``DataSetSection``, and ``DataSetSelection`` models defined in ``simstack.models.dataset``.
+
+DataSet
+-------
+
+The ``DataSet`` model is the top-level container for data.
+
+
+It consists of:
+
+*   **metadata**: A ``DataSetMetadata`` object that defines the type and validation rules for the dataset.
+*   **sections**: A dictionary of ``DataSetSection`` objects, keyed by section name.
+
+It behaves like a dictionary of sections. Think of a dataset as an excel file, where each ``DataSetSection`` is a sheet.
+
+Metadata
+--------
+
+Metadata in SimStack II is handled by the ``DataSetMetadata`` and ``DataSetMetadataTemplate`` classes.
+
+- **DataSetMetadata**: This is an ``EmbeddedModel`` used within a ``DataSet``. It behaves like a dictionary and stores key-value pairs of information about the dataset (e.g., experimental conditions, parameters). It also ensures structural consistency of the data.
+- **DataSetMetadataTemplate**: This model defines the expected structure and schema for a specific type of dataset metadata. When a new ``DataSet`` is saved, its metadata is validated against the corresponding template.
+
+The metadata provides:
+*   **Validation**: Ensures that all datasets of the same type have consistent metadata keys and value types.
+*   **JSON Schema**: Automatically generates JSON schemas for the stored metadata.
+*   **Dict-like API**: Supports standard dictionary operations like ``__getitem__``, ``keys()``, ``items()``, and ``update()``.
+
+DataSetSection
+--------------
+
+A ``DataSetSection`` represents a collection of rows within a dataset. Each row is a dictionary of models, but only the model ids are stored when saving.
+
+
+Key features:
+*   **Type consistency**: All rows in a section must have the same model types for the same keys.
+*   **Lazy loading**: Models are cached and only loaded from the database when needed.
+*   **Table representation**: It can automatically generate column definitions and table entries for UI components (like ag-grid).
+
+To add data to a section, you can use the ``add_row`` method:
+
+.. code-block:: python
+
+    section.add_row({"model_a": instance_a, "model_b": instance_b})
+
+DataSetSelection
+----------------
+
+The ``DataSetSelection`` model is used to reference specific items within a ``DataSet``. Instead of duplicating data, it stores the ``dataset_id`` and a list of indices for each section.
+
+This is particularly useful for workflows where a user selects a subset of a dataset to be processed by a subsequent node.
+
+Example usage:
+
+.. code-block:: python
+
+    selection = DataSetSelection(dataset_id=my_dataset.id)
+    selection.dataset_selection_fields.append(
+        DataSetSelectionField(section_name="default", indices=[0, 2, 5])
+    )
+
+API Reference
+-------------
+
+.. autoclass:: simstack.models.dataset.DataSet
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+.. autoclass:: simstack.models.dataset.DataSetSection
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+.. autoclass:: simstack.models.dataset.DataSetSelection
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+.. autoclass:: simstack.models.dataset_metadata.DataSetMetadata
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+.. autoclass:: simstack.models.dataset_metadata.DataSetMetadataTemplate
+   :members:
+   :undoc-members:
+   :show-inheritance:

simstack/static/docs/_sources/developer_guide.rst.txt

@@ -0,0 +1,25 @@
+.. _developer_guide:
+
+Developer Guide
+===============
+
+In this section we compiled a list of how-to's to help developers to use simstack efficiently.
+
+.. toctree::
+   :maxdepth: 1
+
+   data
+   datasets
+   lists
+   array_lists
+   tables
+   files
+   resource_config
+   mass_runner
+   fire_and_forget_runner
+   docker_nodes
+   testing_complex_nodes
+   database_migrations
+   ci_pipeline
+   misc
+