simstack 2.0.1__py3-none-any.whl

simstack/static/docs/_sources/lists.rst.txt

@@ -0,0 +1,111 @@
+.. _lists:
+
+List Mixins
+===========
+
+Simstack provides two main mixin classes to simplify the creation of models that behave like lists of other models: ``GenericListMixin`` and ``ObjectListMixin``. These mixins are located in ``simstack.models.base_lists``.
+
+GenericListMixin
+----------------
+
+The ``GenericListMixin`` is intended for lists of items that are stored directly within the model, such as ``EmbeddedModel`` instances or basic Python types (strings, floats, etc.). It provides standard Python list-like behavior.
+
+Requirements:
+~~~~~~~~~~~~~
+
+To use ``GenericListMixin``, your class must:
+
+1. Inherit from ``GenericListMixin[T]``.
+2. Have an attribute named ``elements`` of type ``List[T]``.
+
+Example:
+~~~~~~~~
+
+.. code-block:: python
+
+   from typing import List
+   from odmantic import Model, Field
+   from simstack.models import simstack_model
+   from simstack.models.base_lists import GenericListMixin
+
+   @simstack_model
+   class StringList(Model, GenericListMixin[str]):
+       field_name: str = "string_list"
+       elements: List[str] = Field(default_factory=list)
+
+Available Methods:
+~~~~~~~~~~~~~~~~~~
+
+- ``append(element: T)``: Adds an element to the list.
+- ``extend(elements: Union[List[T], GenericListMixin[T]])``: Extends the list with another list or another mixin-based list.
+- ``insert(index: int, element: T)``: Inserts an element at a specific index.
+- ``remove(element: T)``: Removes the first occurrence of an element.
+- ``pop(index: int = -1)``: Removes and returns the element at the given index.
+- ``clear()``: Removes all elements.
+- Standard list methods: ``index``, ``count``, ``reverse``, ``sort``, ``copy``.
+- Support for standard operators: ``len()``, ``[]`` (getitem/setitem/delitem), ``iter()``, ``in``, ``bool()``.
+
+Search and Filter:
+~~~~~~~~~~~~~~~~~~
+
+The mixin also provides convenience methods for searching elements if they have a ``name`` or ``size`` attribute:
+
+- ``find(pattern: str)``: Returns the first element where ``element.name == pattern``.
+- ``find_all(pattern: str)``: Returns all elements where ``element.name`` matches the regex ``pattern``.
+- ``filter_by_size(min_size, max_size)``: Returns elements with a ``size`` attribute within the specified range.
+- ``filter_by_property(property_name, value)``: Returns elements where ``getattr(element, property_name) == value``.
+- ``sort_by_name()``, ``sort_by_size()``: Sorts the list based on these attributes.
+
+
+ObjectListMixin
+---------------
+
+The ``ObjectListMixin`` is designed for lists of ``Model`` instances. Instead of storing the full objects, it stores only their ``ObjectId``. This is useful for keeping the parent model small and avoiding deep nesting in MongoDB. Interaction with the objects is done asynchronously through the database.
+
+Requirements:
+~~~~~~~~~~~~~
+
+To use ``ObjectListMixin``, your class must:
+
+1. Inherit from ``ObjectListMixin[T]`` where ``T`` is an odmantic ``Model``.
+2. Have an attribute named ``elements`` of type ``List[ObjectId]``.
+
+Example:
+~~~~~~~~
+
+.. code-block:: python
+
+   from typing import List
+   from odmantic import Model, Field, ObjectId
+   from simstack.models import simstack_model, StringData
+   from simstack.models.base_lists import ObjectListMixin
+
+   @simstack_model
+   class StringDataList(Model, ObjectListMixin[StringData]):
+       field_name: str = "string_data_list"
+       elements: List[ObjectId] = Field(default_factory=list)
+
+Concrete Implementations
+------------------------
+
+Several core SimStack models utilize these mixins:
+
+*   :doc:`files`: ``FileList`` and ``FileListModel`` use ``ObjectListMixin[FileStack]``.
+*   :doc:`array_lists`: ``ArrayList`` uses ``ObjectListMixin[ArrayStorage]``.
+*   :doc:`tables`: ``SimpleTable`` (while not using list mixins directly, it's a related data structure).
+
+Available Methods (mostly Async):
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+- ``await append(element: T)``: Saves the ``Model`` instance to the database and adds its ID to the list.
+- ``await extend(elements: Union[List[T], ObjectListMixin[T]])``: Extends the list, saving any new models to the database.
+- ``await get(index: int) -> T``: Retrieves the full model from the database at the specified index.
+- ``await find(pattern: str) -> Optional[T]``: Searches for a model with a matching ``name`` attribute in the database.
+- ``await find_all(pattern: str) -> List[T]``: Searches for all models matching the regex ``pattern``.
+- ``async for element in my_list``: Support for asynchronous iteration over the full objects.
+- ``len(my_list)``: Returns the number of IDs in the list.
+- ``my_list[index]``: Returns the ``ObjectId`` at the specified index (sync).
+- ``element in my_list``: Checks if a ``Model`` or ``ObjectId`` is in the list (sync).
+
+.. note::
+   Interaction with ``ObjectListMixin`` requires an active database engine context, which is typically provided during node execution.

simstack/static/docs/_sources/mass_runner.rst.txt

@@ -0,0 +1,91 @@
+MassRunner
+==========
+
+.. autoclass:: simstack.methods.mass_runner.MassRunner
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+The ``MassRunner`` is a specialized ``NodeRunner`` utility designed to execute a large number of independent node tasks in parallel. It handles task orchestration, concurrency control, and result persistence within a ``DataSet``.
+
+Key Features
+------------
+
+* **Parallel Execution**: Leverages ``asyncio`` to run multiple nodes concurrently.
+* **Concurrency Control**: Supports an optional ``max_concurrency`` parameter using an internal semaphore.
+* **Automatic Persistence**: Stores results (both successful and failed) in a ``DataSet`` model.
+* **Result Recovery**: Can recover results from previous runs of the same or different nodes using ``recover_orphaned_datasets``.
+* **Hashing and Caching**: Computes hashes for arguments to skip redundant executions.
+
+Usage
+-----
+
+The ``MassRunner`` is typically used within an ``async with`` block inside a parent node.
+
+.. code-block:: python
+
+    from applications.general.mass_runner import MassRunner
+    from simstack.models import IntData
+
+    @node
+    async def my_parallel_master(argument: IntData, **kwargs):
+        # Initialize MassRunner with the target node to be executed in parallel
+        async with MassRunner(target_node, max_concurrency=10, **kwargs) as runner:
+            for i in range(argument.value):
+                # Create tasks for individual inputs
+                runner.create_tasks(IntData(value=i))
+        
+        # After the block, runner.dataset contains all results
+        return runner.dataset
+
+
+The dataset will comprise a single ``DataSetSection`` called "tasks". Each row contains all inputs and outputs of the
+node that has been called, success and error flags. The name of the row is the arg-hash of the arguments of the node.
+
+.. :note:
+   The rows returned by MassRunner may contain much more data than is actually needed in the subsequent calculation.
+   It may therefore be better to immideatly create a derived dataset with only the relevent data.
+
+.. :note:
+   MassRunner is meant for jobs which are expected to mostly suceed and which take roughly the same amount of time
+
+
+
+Detailed Flow in MassRunner Tests
+---------------------------------
+
+The ``mass_runner_test.py`` file demonstrates how ``MassRunner`` handles task execution, failures, and recovery.
+
+1. **Failure Demonstration (mass_runner_test_master_node)**
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The test first calls ``mass_runner_test_master_node``, which uses ``MassRunner`` to call ``sad_node``.
+
+*   **sad_node logic**: Fails if the input integer is even (``argument.value % 2 == 0``).
+*   **Execution**: For inputs ``0, 1, 2``, inputs ``0`` and ``2`` will fail, while ``1`` will succeed.
+*   **Dataset State**: The resulting ``DataSet`` contains a "tasks" section where each entry records the success status and any error messages.
+
+2. **Recovery and Fix (mass_runner_test_fix_master_node)**
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The second part of the test demonstrates how to "fix" a failed workflow by reusing successful results and replacing failed ones with results from a different node (or a fixed version of the same node).
+
+*   **Context Setup**: It passes the ``previous_task_id`` and sets ``previous_node_name`` in ``kwargs``.
+*   **MassRunner Initialization**: It initializes ``MassRunner`` with ``happy_node`` (which always succeeds).
+*   **Orphan Recovery**: Calling ``await result.recover_orphaned_datasets()`` searches the database for the dataset generated in the first step.
+*   **Selective Execution**:
+    - When ``create_tasks`` is called, ``MassRunner`` checks the existing dataset for a matching ``arg_hash``.
+    - If a **successful** result is found for that hash, it is copied into the new dataset and the node execution is skipped.
+    - If no result or a **failed** result is found, it executes the new node (``happy_node``) and stores the new success.
+*   **Result**: The final dataset contains the reused success from the first run (for input ``1``) and the new successful results from ``happy_node`` (for inputs ``0`` and ``2``).
+
+Logging and Metadata
+--------------------
+
+``MassRunner`` automatically attaches metadata to the ``DataSet``, including:
+*   The original node name.
+*   The task ID of the master node.
+*   The argument hash used for grouping.
+*   The call path within the workflow.
+
+This metadata is crucial for the ``recover_orphaned_datasets`` method to correctly identify related data from previous executions.

simstack/static/docs/_sources/misc.rst.txt

@@ -0,0 +1,106 @@
+Miscellaneous
+==============
+
+
+BUGS:
+-----
+
+Optional fields in embedded models are not handled correctly
+
+https://github.com/art049/odmantic/issues/370
+https://github.com/art049/odmantic/issues/484
+
+References
+----------
+
+
+
+The challenge in Simstack II is to exploit the features of Next-JS frontend
+without forcing the developer to write Next-JS or server components. In
+Simstack II each user has his/her own MongoDB database, there is an
+additional database used exclusively for authentication.
+
+.. note:: Only one instance of a mongodb database server, of the fastapi
+   server and of the next-js frontend is required per installation.
+
+
+The workflow developer customizes the appearance of workflows for submission,
+data analysis and data visualization using three main mechanisms:
+
+* `React JSON Schema Form <https://rjsf-team.github.io/react-jsonschema-form/>`_
+* `AG-Grid <https://www.ag-grid.com/>`_ is used to generate interactive tables (via Artifacts)
+* `AG-Charts <https://charts.ag-grid.com/>`_ is used to generate interactive graphs (via Artifacts)
+
+Customizing Simstack Models
+---------------------------
+
+.. note:: General information about Simstack models
+
+Simstack models are built on top of `odmantic`, which uses `pydantic` for data validation.
+By decorating a class with `@simstack_model`, you enable additional UI-related features.
+You can customize how a model appears in the GUI by implementing `json_schema` and `ui_schema` methods.
+Refer to :ref:`persisting_data` for more details.
+
+
+
+Running the above code as a workflow we would like to achieve the following:
+
+* code can be executed on remote resources (possibly in parallel)
+* users can inspect the results in a GUI
+* results can be reused when the workflow failed or in other workflows
+* the workflow can be run from a GUI
+* (workflows can be created from existing components in a GUI)
+
+To address the first points, all data must be serializable. This can be achieved by pickling the inputs/outputs but
+then the data is hidden in the pickled string and not searchable. Here we have chosen odmantic classes to persist the
+data in a database. See: :ref:`persisting-results-section`. This has the charm that routes to access this data can be
+automatically generated.
+
+Overall we need four "servers" to run the workflow:
+
+* The "workflow server" that manages and executes workflows
+* The "runners" that execute parts of workflows on remote resources
+* The "database server" provides an API to the database for the GUI
+* The webserver that serves the GUI
+
+
+.. code-block:: python
+
+    @node(parameters=Parameters(resource="my_resource", queue="slurm-queue"))
+    def my_node(arg1: Model1, arg2: Model2, **kwargs) -> SimstackResult:
+        # do something with inputs
+        return outputs
+
+
+
+.. note::
+  - Each node has a unique task_id which is the ObjectId of entry in the
+    node_registry table in the database
+  - Before control is passed to a node, the calling function will create
+    a directory as workdir/node_name/node_id which is unique to the node.
+  - all files generated by the node should be written into this directory.
+  - nodes can by synchronous or async functions.
+
+
+**Allowed Arguments**
+
+- All **positional** arguments of the wrapped function (my_node) must be
+  **registered** odmantic Models. These arguments are automatically persisted
+  in the database.
+- The my_node function must have \*\*kwargs even if they are not used in the
+  function.
+- \*\*kwargs is used to pass information to process the node, but
+  should not be used to pass arguments that affect the result of the
+  function. Presently, Model-type arguments in the \*\*kwargs are not persisted
+  in the database which precludes the execution of such nodes on resources
+  which are different from the function is called.
+
+.. important::
+  Only positional arguments should affect the outcome of the function execution
+
+**Return Types**
+
+While it is easy to check whether the variables
+passed to nodes can be persisted in the database, this can be checked for
+results only in post-processing.
+

simstack/static/docs/_sources/overview.rst.txt

@@ -0,0 +1,91 @@
+Overview
+========
+
+SimStack II
+~~~~~~~~~~~
+
+SimStack II is a modern, Python-based workflow system with a web-based graphical user interface (GUI). The GUI is generated automatically from your Python workflow code and supports interactive inspection of inputs, outputs, tables, and plots.
+
+Key features
+^^^^^^^^^^^^
+
+* **GUI-driven execution:** Workflows are submitted via a :ref:`graphical user interface <using-gui-section>`, which is auto-generated from the :ref:`Python code <writing-gui-section>`.
+* **Persistent results:** Inputs and results are stored automatically in a non-SQL database so they can be reused and re-analyzed.
+* **Flexible workflows:** Workflows can be generated dynamically at runtime, nested, and executed across different remote resources.
+* **Artifacts for analysis:** Tables, plots, and other artifacts can be created during or after workflow execution to support data analysis.
+* **Simple authoring model:** :doc:`Workflows <workflows>` are written as decorated plain Python functions.
+
+Roadmap
+^^^^^^^
+
+Future versions are planned to (roughly in this order):
+
+* integrate research data management
+* provide tools to wrap other workflow systems
+* implement artifact generation via the GUI
+* enable data aggregation via the GUI
+* enable workflow development via the GUI
+
+For a quick start, see :doc:`installation` and start using SimStack II via the :ref:`graphical user interface <using-gui-section>`.
+
+Motivation
+~~~~~~~~~~
+
+Computational scientists produce data that ultimately becomes figures and tables in publications.
+To aid this process many workflow environments already exist:
+`Wikipedia <https://en.wikipedia.org/wiki/Workflow_management_system>`_ lists a wide range of systems
+such as Airflow, Luigi, Nextflow, and Snakemake. Many widely used tools are file-based:
+a workflow is defined in a static configuration file and executed by an engine.
+This approach can be robust and has been very successful,
+but it often becomes limiting when workflows must be assembled dynamically or adapted during execution.
+
+In parallel, the AI/ML ecosystem has accelerated the development of powerful Python-native workflow
+systems such as
+`covalent <https://github.com/AgnostiqHQ/covalent>`_,
+`prefect <https://github.com/PrefectHQ/prefect>`_, and
+`pyiron <https://github.com/pyiron/pyiron>`_.
+These systems embrace Python directly, enabling more dynamic and expressive computation pipelines.
+
+Our experience with SimStack I suggests there are two only partially overlapping communities:
+
+* expert workflow developers (in python or otherwise) who struggle to develop user-interfaces
+  make these workflows accessible to end-users
+* a larger group of workflow users, but may not have the software engineering background
+  to develop the workflows on their own.
+
+Bridging developers and users
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+SimStack II aims to bridge this gap by making workflow execution and result exploration accessible
+through a GUI, while keeping workflow implementation in Python. This enables rapid development of
+workflows using the full power of the python ecosystem. We believe this approach is much more
+powerful than attempting to construct complex workflows in a graphical user interface.
+
+Developers who can construct workflows via a GUI typically have the ability to
+write them directly in Python. A GUI for *workflow development* thus adds overhead
+without providing much benefit. A GUI for *workflow execution and analysis*,
+however, can substantially lower the barrier for many users.
+
+In developing a workflow system for scientific applications, we also need to consider another
+issue: Workflows in IT settings (e.g., server or database maintenance) tend to be stable:
+the goal is an environment where everything works all the time.
+
+Scientific workflows often operate in a different environment: methods, hypotheses, and
+analysis requirements evolve continuously. A workflow may succeed technically while
+producing results that invalidate the motivating hypothesis—triggering
+new analysis and iteration.
+
+This reality makes persistence, traceability, and interactive re-analysis essential.
+
+Design goals
+^^^^^^^^^^^^
+
+SimStack II is designed with the following priorities:
+
+* Workflows are initialized, monitored, and managed through a GUI on a variety of compute resources.
+* Results are persisted and accessible for re-analysis, ideally through the GUI.
+* Workflow components are implemented in Python, with minimal coding overhead.
+* Workflows expose results in a UI that is (to a large extent)
+  automatically generated on the basis of the python code additional frontend development.
+
+.. include:: architecture.rst

simstack/static/docs/_sources/recovery.rst.txt

@@ -0,0 +1,38 @@
+Recovery and Repair of Nodes
+===========================
+
+This document explains how to implement recovery mechanisms for failed nodes in SimStack.
+The example `recovery_test.py` demonstrates how to handle nodes that might fail after producing some output, and how to recover that output using a recovery node.
+
+Overview
+--------
+
+The recovery process involves three main components:
+
+1. **The Failing Node**: A node that performs some work, writes output to a file, but then fails (e.g., due to an exception).
+2. **The Repair Function**: A helper function that knows how to locate the output of a failed node and "repair" the result by collecting the files and marking it as successful.
+3. **The Recovery Node**: A node that queries the database for failed instances of specific nodes and applies the repair function to them.
+
+Implementation Details
+----------------------
+
+The example implementation can be found in `examples/testing/recovery_test.py`.
+
+.. literalinclude:: ../../../examples/testing/recovery_test.py
+   :language: python
+   :linenos:
+
+How it Works
+------------
+
+1. **sxt_failing_node**:
+   This node simulates a failure. Even though it fails, it ensures that its progress (in this case, `result.txt`) is saved in its working directory.
+
+2. **stx_repair_node**:
+   This function takes a `NodeRegistry` entry. It constructs the path to the original node's working directory using `context.config.workdir`. If it finds the expected output files, it uses `node_runner.succeed()` to provide the missing data.
+
+3. **sxt_recovery_node**:
+   This is the orchestrator for recovery. It uses `context.db.find` to search for `NodeRegistry` entries with a `failed` status and a specific `call_path`. For each failed node found, it calls `stx_repair_node`. If repair is successful, it updates the original node's status in the database.
+
+4. **Main Execution**:
+   The `main` function first runs several instances of `sxt_failing_node`, which all fail. Then, it runs `sxt_recovery_node` to automatically fix these failures and recover the data.

simstack/static/docs/_sources/resource_config.rst.txt

@@ -0,0 +1,141 @@
+Program Execution
+=================
+
+When developers write new nodes that execute external programs they cannot know the specifics of the resources on which these nodes are to be run. 
+Simstack separates the program execution logic from the specifics of a particular resource. The ``ResourceConfig`` class handles resource-specific 
+configurations from a ``config.toml`` file. It provides methods to retrieve program parameters, setup scripts, 
+and run commands in a controlled environment (optionally using temporary directories).
+
+
+The config toml is structured by resources, i.e. if you have resources hpc1 and small-computer and my-laptop sections starting with [hpc1], [small-computer], [my-laptop]
+contain the data for each of these nodes. The convenience functions return the data for the actual resource. In principle you are totally free 
+which information you store in ``config.toml`` for your own nodes, but the convenience functions implement often used features, such as
+
+- using scratch space for program execution and where that is 
+- startup scripts 
+- the actual command to execute to run the programs
+
+.. note::
+   An instance of ``ResourceConfig`` is provided by the context
+
+
+The documentation below explains the current defaults. Standard nodes are then simply executed by:
+
+
+
+.. code-block:: python
+
+   from pathlib import Path
+   from simstack.core.context import context
+
+   input_files: List[str | FileStack] = [...]
+   output_files: List[str | FileStack] = [...] 
+   result = context.resource_config.run("program-name", input_files, output_files) 
+
+
+
+
+ResourceConfig Class
+--------------------
+
+The ``ResourceConfig`` class is located in ``simstack.util.resource_config``.
+
+.. autoclass:: simstack.util.resource_config.ResourceConfig
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+Initialization
+~~~~~~~~~~~~~~
+
+The ``ResourceConfig`` is initialized with a path to the configuration file (or directory containing ``config.toml``) and the name of the resource.
+
+.. code-block:: python
+
+   from pathlib import Path
+   from simstack.util.resource_config import ResourceConfig
+
+   config = ResourceConfig(Path("/path/to/project"), "local")
+   config.program = "orca"
+   
+
+Methods
+~~~~~~~
+
+*   ``os``: Property that returns the operating system of the resource (defaults to "linux").
+*   ``program``: Property to get or set the current program name.
+*   ``setup(node_runner=None)``: Executes the setup scripts defined for the resource.
+*   ``run(program_name=None, input_files=None, output_files=None, node_runner=None)``: Executes the program command, handling file transfers and temporary directories based on configuration.
+*   ``get_program()``: Retrieves configuration for the current program.
+*   ``get_setup_params()``: Retrieves setup configuration for the current resource.
+*   ``get_postprocessing_params()``: Retrieves post-processing configuration.
+
+Integration with NodeRunner
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+If a ``node_runner`` instance is passed to ``setup()`` or ``run()``, it will use ``node_runner.subprocess()`` to execute commands, which provides better logging and integration with SimStack's task management.
+
+Config.toml Structure
+---------------------
+
+The ``config.toml`` file is organized by resource name. Each resource can have several sections:
+
+General Resource Settings
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. code-block:: toml
+
+   [resource_name]
+   os = "linux"  # optional, defaults to "linux"
+
+Setup Section
+~~~~~~~~~~~~~
+
+Contains scripts to be executed before any program run and configuration for temporary directories.
+
+.. code-block:: toml
+
+   [resource_name.setup]
+   scripts = [
+       "module load orca",
+       "export MY_VAR=value"
+   ]
+   tmp_base_dir = "/path/to/scratch" # Can also be a command like "set TMP_BASE_DIR=..."
+
+Post-processing Section
+~~~~~~~~~~~~~~~~~~~~~~~
+
+Contains settings for cleanup after execution.
+
+.. code-block:: toml
+
+   [resource_name.post-processing] # or [resource_name.postprocessing]
+   scratch_cleanup = true
+
+Program Section
+~~~~~~~~~~~~~~~
+
+Defines configuration for specific applications on the resource.
+
+.. code-block:: toml
+
+   [resource_name.program.orca]
+   use_tmp = true
+   run_command = "orca orca.inp > orca.out"
+ 
+
+Example config.toml
+~~~~~~~~~~~~~~~~~~~
+
+.. code-block:: toml
+
+   [local]
+   os = "windows"
+
+   [local.setup]
+   scripts = ["set USER=%USERNAME%"]
+   tmp_base_dir = "C:\\Temp"
+
+   [local.program.orca]
+   use_tmp = true
+   run_command = "orca orca.inp > orca.out"

simstack/static/docs/_sources/simstack/modules/modules.rst.txt

@@ -0,0 +1,7 @@
+simstack
+========
+
+.. toctree::
+   :maxdepth: 4
+
+   simstack