simstack 2.0.1__py3-none-any.whl

simstack/static/docs/_sources/docker_nodes.rst.txt

@@ -0,0 +1,149 @@
+.. _docker_nodes:
+
+Creating Docker Containers for SimStack Nodes
+=============================================
+
+This document explains the mechanism for creating Docker containers for SimStack nodes, using the `psi4_node` as a practical example.
+
+Mechanism Overview
+------------------
+
+SimStack nodes can be containerized to ensure a consistent execution environment, especially when complex dependencies like quantum chemistry codes (e.g., Psi4) are involved. 
+
+The general approach involves:
+1. Creating a `Dockerfile` that sets up the environment and installs necessary software.
+2. Including SimStack and its core running mechanism in the container.
+3. Defining an `ENTRYPOINT` that calls the SimStack node runner.
+
+Example: Psi4 Node
+------------------
+
+The `psi4_node` is part of the `molecular_qm_psi4` package. It requires `psi4` and `crest` which are best installed via `micromamba` (a fast alternative to conda).
+
+Node Implementation
+^^^^^^^^^^^^^^^^^^^
+
+The node is defined in `molecular_qm_psi4/psi4_node.py`. It uses the `@node` decorator and interacts with SimStack's `node_runner`.
+
+.. code-block:: python
+
+    import logging
+    try:
+        import psi4
+    except ImportError:
+        psi4 = None
+
+    from simstack.core.node import node
+    from simstack.core.simstack_result import SimstackResult
+    from molecular_qm_models import QMInput, QMResult, Molecule, Atom, MoleculeList
+
+    logger = logging.getLogger(__name__)
+
+    def qminput_to_psi4_molecule(molecule: Molecule, charge: int, multiplicity: int) -> str:
+        """Converts a Simstack Molecule to a Psi4 molecule string."""
+        mol_str = f"{charge} {multiplicity}\n"
+        for atom in molecule.atoms:
+            mol_str += f"{atom.element} {atom.x} {atom.y} {atom.z}\n"
+        return mol_str
+
+    @node
+    async def psi4_calculator(qm_input: QMInput, **kwargs) -> SimstackResult:
+        """
+        Psi4 node implementation using Python bindings.
+        """
+        node_runner = kwargs.get("node_runner")
+        
+        if psi4 is None:
+            return node_runner.fail("Psi4 is not installed in the current environment.")
+
+        try:
+            # Set up Psi4 molecule
+            mol_str = qminput_to_psi4_molecule(qm_input.molecule, qm_input.charge, qm_input.multiplicity)
+            psi4.geometry(mol_str)
+            
+            # ... (configuration and execution logic) ...
+            
+            # Execute calculation
+            if qm_input.optimization:
+                energy, wfn = psi4.optimize(method, return_wfn=True)
+                # ...
+            else:
+                energy, wfn = psi4.energy(method, return_wfn=True)
+                
+            qm_result = QMResult()
+            qm_result.final_energy = energy
+            # ...
+            
+            node_runner.info("Psi4 calculation finished successfully")
+            node_runner.psi4_result = qm_result
+            return node_runner.succeed()
+
+        except Exception as e:
+            logger.error(f"Psi4 calculation failed: {str(e)}")
+            return node_runner.fail(f"Psi4 execution failed: {str(e)}")
+        finally:
+            if psi4 is not None:
+                psi4.core.clean()
+
+Dockerfile Configuration
+^^^^^^^^^^^^^^^^^^^^^^^^
+
+The `Dockerfile` for this node is located at `molecular_qm_psi4/Dockerfile`. It uses `micromamba` for environment management and `uv` for fast Python package installation.
+
+.. code-block:: dockerfile
+
+    # Use a base image with micromamba (conda replacement)
+    FROM mambaorg/micromamba:latest
+
+    USER root
+
+    # Install git and other build essentials
+    RUN apt-get update && apt-get install -y --no-install-recommends \
+        git \
+        ca-certificates \
+        curl \
+        build-essential \
+        && rm -rf /var/lib/apt/lists/*
+
+    # Fix for _distutils_hack error: explicitly install setuptools
+    RUN micromamba install -y -n base -c conda-forge setuptools && \
+        micromamba clean --all --yes
+
+    # Install uv
+    RUN curl -LsSf https://astral.sh/uv/install.sh | sh
+    ENV PATH="/opt/conda/bin:/root/.local/bin:$PATH"
+
+    # Set up work directory
+    WORKDIR /app
+
+    # Install Psi4 and crest from conda
+    RUN micromamba install -y -n base -c conda-forge psi4 crest python=3.12 && \
+        micromamba clean --all --yes
+
+    # Copy the repositories/directories
+    COPY molecular_qm_models /app/molecular_qm_models
+    COPY molecular_qm_psi4 /app/molecular_qm_psi4
+    COPY molecular_qm_psi4/README.md /app/README.md
+    COPY simstack.toml /app/simstack.toml
+
+    ENV PYTHONPATH="/app:/app/molecular_qm_psi4"
+    ENV UV_PYTHON=/opt/conda/bin/python
+    ENV UV_PROJECT_ENVIRONMENT=/opt/conda
+    ENV PATH="/opt/conda/bin:/root/.local/bin:$PATH"
+
+    # Install dependencies directly
+    RUN uv pip install --system "simstack @ git+https://github.com/simstack/simstack.git@feature-test-generation" "setuptools>=80.9.0"
+
+    # Entrypoint: call the module directly
+    ENTRYPOINT ["python", "-m", "simstack.core.run_node"]
+
+Key Components for Containerization
+-----------------------------------
+
+1. **Base Image**: Use a lightweight and appropriate base image (e.g., `micromamba` for scientific software).
+2. **Environment Setup**: Install all non-Python dependencies (compilers, libraries, etc.).
+3. **Python Environment**: Install Python and required packages. The use of `uv` is recommended for performance.
+4. **Code Integration**: Copy your node code and its local dependencies into the container.
+5. **PYTHONPATH**: Ensure that the `PYTHONPATH` is set correctly so that SimStack can find your node and models.
+6. **SimStack Installation**: Install the `simstack` package itself.
+7. **Entrypoint**: The `ENTRYPOINT ["python", "-m", "simstack.core.run_node"]` is crucial as it tells SimStack how to execute the node within the container.

simstack/static/docs/_sources/files.rst.txt

@@ -0,0 +1,159 @@
+File Management
+===============
+
+SimStack II provides a robust way to handle files within workflows. The core model for file management is ``FileStack``, which manages the storage and retrieval of individual files across different computational resources. For collections of files, ``FileList`` and ``FileListModel`` are provided.
+
+FileStack
+---------
+
+A ``FileStack`` represents a single file that can be stored in the database (in-memory) or tracked on various computational resources.
+
+**Key Features:**
+
+* **Storage Flexibility:** Files can be stored directly in MongoDB (compressed) or kept on disk at various locations.
+* **Automatic Transfer:** When a node requires a ``FileStack`` that is not locally available, SimStack automatically handles the transfer.
+* **Hashing:** Supports content-based hashing for verification and caching.
+
+**Usage:**
+
+To create a ``FileStack`` from a local file:
+
+.. code-block:: python
+
+   from simstack.models.files import FileStack
+
+   # Create a FileStack and store its content in the database
+   file_stack = FileStack.from_local_file("path/to/my_file.txt", in_memory=True)
+
+To create a ``FileStack`` from a string:
+
+.. code-block:: python
+
+   file_stack = FileStack.from_string("Hello SimStack!", "hello.txt")
+
+To retrieve the file in a node:
+
+.. code-block:: python
+
+   @node
+   def process_file(input_file: FileStack, **kwargs):
+       # This ensures the file is available locally
+       local_path = input_file.get()
+       
+       with open(local_path, "r") as f:
+           content = f.read()
+       ...
+
+FileList and FileListModel
+--------------------------
+
+When dealing with multiple files, you can use ``FileList`` or ``FileListModel``. Both models now inherit from ``ObjectListMixin``, providing a consistent and powerful interface for managing collections of files.
+
+Key features of both:
+* **Dict/List-like API**: Supports standard operations like ``append()``, ``extend()``, ``insert()``, ``remove()``, ``pop()``, and slicing.
+* **Regex Search**: Use ``find(pattern)`` or ``find_all(pattern)`` to search for files by their names using regular expressions.
+* **Database Integration**: Handles saving and loading of ``FileStack`` references automatically.
+
+FileList (Embedded)
+~~~~~~~~~~~~~~~~~~~
+
+``FileList`` is an ``EmbeddedModel``. It is suitable for smaller collections of files where you want to keep them directly within the parent model.
+
+.. code-block:: python
+
+   from simstack.models.file_list import FileList
+   from simstack.models.files import FileStack
+
+   file_list = FileList()
+   file_list.append(FileStack.from_local_file("file1.txt"))
+   file_list.append(FileStack.from_local_file("file2.txt"))
+
+   # Search for a file
+   config_file = file_list.find(r".*\.conf")
+
+FileListModel (Referenced)
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+``FileListModel`` is a top-level ``Model``. This is preferred for large collections or when ``FileStack`` objects need to be shared across multiple models.
+
+.. code-block:: python
+
+   from simstack.models.file_list import FileListModel
+
+   file_list_model = FileListModel()
+   # Note: saving the parent model will automatically save new FileStack objects
+   file_list_model.append(FileStack.from_local_file("large_file.dat"))
+
+Methods and Mixins
+~~~~~~~~~~~~~~~~~~
+
+Both ``FileList`` and ``FileListModel`` provide a rich set of methods for managing the collection. For more details on the underlying list behaviors provided by ``ObjectListMixin``, see :doc:`lists`.
+
+API Reference
+-------------
+
+.. autoclass:: simstack.models.file_list.FileList
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+.. autoclass:: simstack.models.file_list.FileListModel
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+External and Internal Files and Resources
+-----------------------------------------
+
+Workflows need access to external resources, such as files, databases, and URLs.
+External resources are only input and otherwise not managed. Internal resources, mostly
+files, are managed by the workflow by certain specfications.
+
+External resources differ from internal resources in that we cannot decide based
+on the data describing the resource, whether its "value" or content has changed
+since the last call.
+
+
+Input Data
+----------
+
+Files can (mostly) be hashed. For all other use cases an intermediate storage (electron) can be implemented that
+stores the "query" and the "result" of the query (assuming that the query and the result can be hashed).
+This service electron returns the result with a flag whether the result has changed.
+
+
+.. code-block:: python
+
+    def complex_compute(data, reuse=True):
+        # complex computation based on data
+        return result
+
+    def compute_electron(url):
+        result, changed = get_external(url)
+        if changed:
+            result = complex_compute(url)
+            electron.set(hash, result)
+        return result
+
+    def get_external(query):
+        hash = hash(query)
+        result, changed = get(hash)
+        if changed:
+            result = query_data(query)
+            electron.set(hash, result)
+        return result
+
+
+Output Data
+-----------
+
+Only Output Data is generated in SimStack II. Because "human readable" data is core concept of the new version all
+digestible output should be parsed to JSON. The question is how to handle large-scale data, which is costly to
+generate and which may have un-anticipated uses later on. An example would be MD trajectories.
+
+This data should be handled as part of the research data management plan, which we plan to implement as an integral
+part of the WF environment anyway. Every WANO will upload all inputs and outputs to the RDM storage anyway.
+The JSON output should thus contain records of all files that have been uploaded.
+
+The issue arising with large data is whether it makes sense to pass only the remote info
+When a WANO is re-executed

simstack/static/docs/_sources/fire_and_forget_runner.rst.txt

@@ -0,0 +1,67 @@
+FireAndForgetRunner
+====================
+
+.. autoclass:: simstack.methods.fire_and_forget_runner.FireAndForgetRunner
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+The ``FireAndForgetRunner`` is a specialized ``NodeRunner`` utility designed to execute multiple independent node tasks in parallel, similar to ``MassRunner``. However, unlike ``MassRunner`` which aggregates all results into a single ``DataSet``, the ``FireAndForgetRunner`` persists each individual result immediately to the database as a ``FireAndForgetResult`` record.
+
+Key Features
+------------
+
+* **Parallel Execution**: Leverages ``asyncio`` to run multiple nodes concurrently.
+* **Concurrency Control**: Supports an optional ``max_concurrency`` parameter using an internal semaphore.
+* **Immediate Persistence**: Saves a ``FireAndForgetResult`` to the database as soon as each individual node call finishes, without waiting for other tasks.
+* **Result Mapping**: Automatically captures and stores input arguments and return values (supporting ``Model``, ``SimstackResult``, and lists of ``Model``) into a dictionary.
+* **Call Path Tracking**: Records the full call path for each task, facilitating traceability.
+
+Usage
+-----
+
+The ``FireAndForgetRunner`` is typically used within an ``async with`` block inside a parent node.
+
+.. code-block:: python
+
+    from simstack.methods.fire_and_forget_runner import FireAndForgetRunner
+    from simstack.models import IntData
+
+    @node
+    async def my_parallel_manager(count: IntData, **kwargs):
+        # Initialize FireAndForgetRunner with the target node
+        async with FireAndForgetRunner(target_node, max_concurrency=5, **kwargs) as runner:
+            for i in range(count.value):
+                # Create tasks for individual inputs
+                runner.create_tasks(IntData(value=i))
+        
+        # After the block, all tasks are finished and persisted individually
+        return True
+
+Each call to a node via ``FireAndForgetRunner`` creates a ``FireAndForgetResult`` entry in the database.
+
+FireAndForgetResult Model
+-------------------------
+
+Each result is stored using the ``FireAndForgetResult`` model:
+
+.. autoclass:: simstack.models.fire_and_forget_result.FireAndForgetResult
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+Fields:
+^^^^^^^
+
+* **call_path**: A string representing the concatenated path of the runner and the node (e.g., ``/my_parallel_manager/target_node``).
+* **models**: A dictionary containing all input arguments (prefixed with ``arg_``) and output results (prefixed with ``result_``).
+* **success**: A boolean flag indicating if the node call was successful.
+* **next_step**: A boolean flag indicating if this result triggers a next step.
+
+Comparison with MassRunner
+--------------------------
+
+While both runners facilitate parallel execution, they serve different persistence needs:
+
+* **MassRunner**: Best for batch jobs where you want a single consolidated ``DataSet`` at the end. It supports result recovery and caching based on argument hashes.
+* **FireAndForgetRunner**: Best for scenarios where you want immediate visibility of results as they arrive, or when the number of tasks is extremely large and you want to avoid holding all results in memory/aggregated model before saving. It does not currently support the same recovery/caching mechanisms as ``MassRunner``.

simstack/static/docs/_sources/index.rst.txt

@@ -0,0 +1,25 @@
+.. SimStack II documentation master file, created by
+   sphinx-quickstart on Fri Feb  7 10:13:05 2025.
+   You can adapt this file completely to your liking, but it should at least
+   contain the root `toctree` directive.
+
+SimStack II Documentation
+=========================
+.. toctree::
+   :maxdepth: 3
+   :caption: Contents:
+
+   overview
+   workflows
+   ui_documentation
+
+   developer_guide
+
+   installation
+   simstack/modules/modules
+
+Indices and tables
+==================
+* :ref:`genindex`
+* :ref:`modindex`
+* :ref:`search`

simstack/static/docs/_sources/installation.rst.txt

@@ -0,0 +1,213 @@
+Simstack II Setup
+=================
+
+UI Setup
+--------
+
+1. Register at ``simstack.int.kit.edu`` (inside the KIT network). This will create an **inactive** user.
+2. Email the maintainer so your account can be activated and you can receive:
+
+   * a database name
+   * a database password
+
+   (This is not automated yet.)
+3. Go to your profile and upload your resources configuration.
+
+Example resources configuration
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. code-block:: json
+
+   [
+     {
+       "id": "<object-id>",
+       "resource_str": "local-home",
+       "hostname": "localhost",
+       "workdir": "$HOME/simstack",
+       "python_paths": [],
+       "environment_start": "",
+       "ssh_key": "$HOME/.ssh/id_rsa",
+       "routes": [
+         "int-nano"
+       ],
+       "queue": "default"
+     },
+     {
+       "id": "<object-id>",
+       "resource_str": "int-nano",
+       "hostname": "int-nano.int.kit.edu",
+       "workdir": "$HOME/simstack",
+       "python_paths": [],
+       "environment_start": "",
+       "ssh_key": "$HOME/.ssh/id_rsa",
+       "routes": [],
+       "queue": "default"
+     },
+     {
+       "id": "<object-id>",
+       "resource_str": "justus",
+       "hostname": "justus2.uni-ulm.de",
+       "workdir": "$HOME/simstack",
+       "python_paths": [],
+       "environment_start": "",
+       "ssh_key": "$HOME/.ssh/id_rsa",
+       "routes": [
+         "int-nano",
+         "local-home"
+       ],
+       "queue": "slurm-queue"
+     }
+   ]
+
+
+Installation of Runners
+-----------------------
+
+Prerequisites
+~~~~~~~~~~~~~
+
+* Install ``uv``:
+  https://docs.astral.sh/uv/getting-started/installation/
+* Create directories:
+
+  * ``$HOME/simstack``
+  * ``$HOME/projects``
+
+Clone the repository
+~~~~~~~~~~~~~~~~~~~~
+
+In ``$HOME/projects`` clone the repository:
+
+* SSH (if your SSH key is added to GitLab):
+
+  .. code-block:: bash
+
+     git clone git@gitlab.kit.edu:kit/ag_wenzel/simstack-model.git
+
+* HTTPS (if you have a token):
+
+  .. code-block:: bash
+
+     git clone https://gitlab.kit.edu/kit/ag_wenzel/simstack-model.git
+
+Then:
+
+.. code-block:: bash
+
+   cd simstack-model
+
+
+Host-specific environment notes
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+On ``int-nano`` you may need to override the C/C++ toolchain because the default compiler is too old.
+Add the following to your ``~/.bashrc``:
+
+.. code-block:: bash
+
+   export PATH=/path/to/gcc-12.3/bin:$PATH
+   export LD_LIBRARY_PATH=/path/to/gcc-12.3/lib64:$LD_LIBRARY_PATH
+   export CC=/path/to/gcc-12.3/bin/gcc
+   export CXX=/path/to/gcc-12.3/bin/g++
+
+Set the paths to the location where gcc-12.3 is installed (e.g. ``/shared/user/ww`` or ``/home/ws/<user>``).
+It is unclear what you need on other systems.
+
+On ``justus`` you may need:
+
+.. code-block:: bash
+
+   export LD_LIBRARY_PATH=$HOME/local/lib:$MKLROOT/lib/intel64:$LD_LIBRARY_PATH
+
+
+Sync dependencies
+~~~~~~~~~~~~~~~~~
+
+.. code-block:: bash
+
+   source ~/.bashrc
+   uv sync --locked
+
+Note: This will install all required dependencies, including ``mongomock`` and ``mongomock-motor``, which are used for in-memory database support during testing.
+
+
+.. _configuration-file:
+
+Create ``simstack.toml``
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+Create a ``simstack.toml`` file (placeholders shown below):
+
+.. code-block:: toml
+
+   [parameters]
+   [parameters.general]
+   use_db = true
+   workdir_self = "<PATH_TO_SIMSTACK_DATA_DIR>"
+   # these are parameters for one user for all hosts
+
+   [parameters.db]
+   database = "<NAME>_data"
+   test_database = "<NAME>_test"
+   connection_string = "mongodb://<USER>:<PASSWORD>@<HOST>:27017/"
+   mongodump_path = "<PATH_TO_MONGODUMP_EXE>"
+
+Where:
+
+* ``<PATH_TO_SIMSTACK_DATA_DIR>`` is the path to the data directory created above (e.g. ``$HOME/simstack``)
+* ``<NAME>`` is your database name (often your first name in lower case)
+* ``<PASSWORD>`` is the database password
+* ``<PATH_TO_MONGODUMP_EXE>`` is the path to the ``mongodump`` executable (required for database backups)
+
+
+Initialize the system
+~~~~~~~~~~~~~~~~~~~~~
+
+This will happen automatically when the default runner starts.
+
+.. code-block:: bash
+
+   uv run create_model_table --dir examples --dir applications
+   uv run create_node_table --dir examples --dir applications
+
+Note: this may crash if your database is very old.
+
+
+
+
+Configure Git identity (required)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Because the runner performs Git operations, ensure you have:
+
+.. code-block:: bash
+
+   git config --global user.email "<YOUR_EMAIL_ADDRESS>"
+   git config --global user.name "<YOUR_NAME>"
+
+
+Start the runner
+~~~~~~~~~~~~~~~~
+
+From ``$HOME/projects/simstack-model``:
+
+.. code-block:: bash
+
+   nohup uv run simstack_runner --resource <RESOURCE_NAME> &
+
+Where ``<RESOURCE_NAME>`` is one of the resources you defined in the UI.
+
+
+
+Ignore after this line
+----------------------
+
+Notes / scratch commands:
+
+.. code-block:: bash
+
+   git submodule add -b new-init https://git@github.com/simstack/simstack.git simstack
+
+.. code-block:: bash
+
+   uv lock --upgrade-package <package-name>