simstack 2.0.1__py3-none-any.whl

simstack/static/docs/_sources/simstack/modules/simstack.util.rst.txt

@@ -0,0 +1,381 @@
+simstack.util package
+=====================
+
+Submodules
+----------
+
+simstack.util.async\_zip\_utils module
+--------------------------------------
+
+.. automodule:: simstack.util.async_zip_utils
+   :members:
+   :show-inheritance:
+   :undoc-members:
+
+simstack.util.b64mixin module
+-----------------------------
+
+.. automodule:: simstack.util.b64mixin
+   :members:
+   :show-inheritance:
+   :undoc-members:
+
+simstack.util.cleaned\_json\_schema module
+------------------------------------------
+
+.. automodule:: simstack.util.cleaned_json_schema
+   :members:
+   :show-inheritance:
+   :undoc-members:
+
+simstack.util.config\_reader module
+-----------------------------------
+
+.. automodule:: simstack.util.config_reader
+   :members:
+   :show-inheritance:
+   :undoc-members:
+
+simstack.util.custom\_model\_dump module
+----------------------------------------
+
+.. automodule:: simstack.util.custom_model_dump
+   :members:
+   :show-inheritance:
+   :undoc-members:
+
+simstack.util.database\_information module
+------------------------------------------
+
+.. automodule:: simstack.util.database_information
+   :members:
+   :show-inheritance:
+   :undoc-members:
+
+simstack.util.db module
+-----------------------
+
+.. automodule:: simstack.util.db
+   :members:
+   :show-inheritance:
+   :undoc-members:
+
+simstack.util.db\_logger module
+-------------------------------
+
+.. automodule:: simstack.util.db_logger
+   :members:
+   :show-inheritance:
+   :undoc-members:
+
+simstack.util.default\_from\_dict module
+----------------------------------------
+
+.. automodule:: simstack.util.default_from_dict
+   :members:
+   :show-inheritance:
+   :undoc-members:
+
+simstack.util.directory\_iterator module
+----------------------------------------
+
+.. automodule:: simstack.util.directory_iterator
+   :members:
+   :show-inheritance:
+   :undoc-members:
+
+simstack.util.docstring\_parser module
+--------------------------------------
+
+.. automodule:: simstack.util.docstring_parser
+   :members:
+   :show-inheritance:
+   :undoc-members:
+
+simstack.util.file\_hashing module
+----------------------------------
+
+.. automodule:: simstack.util.file_hashing
+   :members:
+   :show-inheritance:
+   :undoc-members:
+
+simstack.util.file\_transfer\_client module
+-------------------------------------------
+
+.. automodule:: simstack.util.file_transfer_client
+   :members:
+   :show-inheritance:
+   :undoc-members:
+
+simstack.util.files module
+--------------------------
+
+.. automodule:: simstack.util.files
+   :members:
+   :show-inheritance:
+   :undoc-members:
+
+simstack.util.find\_python\_files module
+----------------------------------------
+
+.. automodule:: simstack.util.find_python_files
+   :members:
+   :show-inheritance:
+   :undoc-members:
+
+simstack.util.generate\_ui\_schema module
+-----------------------------------------
+
+.. automodule:: simstack.util.generate_ui_schema
+   :members:
+   :show-inheritance:
+   :undoc-members:
+
+simstack.util.generic\_list\_mixin module
+-----------------------------------------
+
+.. automodule:: simstack.util.generic_list_mixin
+   :members:
+   :show-inheritance:
+   :undoc-members:
+
+simstack.util.geometry\_utils module
+------------------------------------
+
+.. automodule:: simstack.util.geometry_utils
+   :members:
+   :show-inheritance:
+   :undoc-members:
+
+simstack.util.get\_module\_path module
+--------------------------------------
+
+.. automodule:: simstack.util.get_module_path
+   :members:
+   :show-inheritance:
+   :undoc-members:
+
+simstack.util.git\_pull module
+------------------------------
+
+.. automodule:: simstack.util.git_pull
+   :members:
+   :show-inheritance:
+   :undoc-members:
+
+simstack.util.git\_repository\_status module
+--------------------------------------------
+
+.. automodule:: simstack.util.git_repository_status
+   :members:
+   :show-inheritance:
+   :undoc-members:
+
+simstack.util.import\_module module
+-----------------------------------
+
+.. automodule:: simstack.util.import_module
+   :members:
+   :show-inheritance:
+   :undoc-members:
+
+simstack.util.importer module
+-----------------------------
+
+.. automodule:: simstack.util.importer
+   :members:
+   :show-inheritance:
+   :undoc-members:
+
+simstack.util.init\_data\_source module
+---------------------------------------
+
+.. automodule:: simstack.util.init_data_source
+   :members:
+   :show-inheritance:
+   :undoc-members:
+
+simstack.util.make\_graph module
+--------------------------------
+
+.. automodule:: simstack.util.make_graph
+   :members:
+   :show-inheritance:
+   :undoc-members:
+
+simstack.util.make\_table module
+--------------------------------
+
+.. automodule:: simstack.util.make_table
+   :members:
+   :show-inheritance:
+   :undoc-members:
+
+simstack.util.mappings module
+-----------------------------
+
+.. automodule:: simstack.util.mappings
+   :members:
+   :show-inheritance:
+   :undoc-members:
+
+simstack.util.minimal\_route\_finder module
+-------------------------------------------
+
+.. automodule:: simstack.util.minimal_route_finder
+   :members:
+   :show-inheritance:
+   :undoc-members:
+
+simstack.util.mock\_db module
+-----------------------------
+
+.. automodule:: simstack.util.mock_db
+   :members:
+   :show-inheritance:
+   :undoc-members:
+
+simstack.util.module\_path\_checker module
+------------------------------------------
+
+.. automodule:: simstack.util.module_path_checker
+   :members:
+   :show-inheritance:
+   :undoc-members:
+
+simstack.util.object\_list\_mixin module
+----------------------------------------
+
+.. automodule:: simstack.util.object_list_mixin
+   :members:
+   :show-inheritance:
+   :undoc-members:
+
+simstack.util.path\_manager module
+----------------------------------
+
+.. automodule:: simstack.util.path_manager
+   :members:
+   :show-inheritance:
+   :undoc-members:
+
+simstack.util.project\_root\_finder module
+------------------------------------------
+
+.. automodule:: simstack.util.project_root_finder
+   :members:
+   :show-inheritance:
+   :undoc-members:
+
+simstack.util.queue\_watcher module
+-----------------------------------
+
+.. automodule:: simstack.util.queue_watcher
+   :members:
+   :show-inheritance:
+   :undoc-members:
+
+simstack.util.resource\_config module
+-------------------------------------
+
+.. automodule:: simstack.util.resource_config
+   :members:
+   :show-inheritance:
+   :undoc-members:
+
+simstack.util.route\_finder module
+----------------------------------
+
+.. automodule:: simstack.util.route_finder
+   :members:
+   :show-inheritance:
+   :undoc-members:
+
+simstack.util.runner\_util module
+---------------------------------
+
+.. automodule:: simstack.util.runner_util
+   :members:
+   :show-inheritance:
+   :undoc-members:
+
+simstack.util.runner\_utils module
+----------------------------------
+
+.. automodule:: simstack.util.runner_utils
+   :members:
+   :show-inheritance:
+   :undoc-members:
+
+simstack.util.safe\_code\_executor module
+-----------------------------------------
+
+.. automodule:: simstack.util.safe_code_executor
+   :members:
+   :show-inheritance:
+   :undoc-members:
+
+simstack.util.sample\_config module
+-----------------------------------
+
+.. automodule:: simstack.util.sample_config
+   :members:
+   :show-inheritance:
+   :undoc-members:
+
+simstack.util.sensitive\_content\_filter module
+-----------------------------------------------
+
+.. automodule:: simstack.util.sensitive_content_filter
+   :members:
+   :show-inheritance:
+   :undoc-members:
+
+simstack.util.setup\_logging module
+-----------------------------------
+
+.. automodule:: simstack.util.setup_logging
+   :members:
+   :show-inheritance:
+   :undoc-members:
+
+simstack.util.submit\_to\_watchdog module
+-----------------------------------------
+
+.. automodule:: simstack.util.submit_to_watchdog
+   :members:
+   :show-inheritance:
+   :undoc-members:
+
+simstack.util.toml\_reader module
+---------------------------------
+
+.. automodule:: simstack.util.toml_reader
+   :members:
+   :show-inheritance:
+   :undoc-members:
+
+simstack.util.transform\_file\_name module
+------------------------------------------
+
+.. automodule:: simstack.util.transform_file_name
+   :members:
+   :show-inheritance:
+   :undoc-members:
+
+simstack.util.ui\_tools module
+------------------------------
+
+.. automodule:: simstack.util.ui_tools
+   :members:
+   :show-inheritance:
+   :undoc-members:
+
+Module contents
+---------------
+
+.. automodule:: simstack.util
+   :members:
+   :show-inheritance:
+   :undoc-members:

simstack/static/docs/_sources/submit.rst.txt

@@ -0,0 +1,141 @@
+
+Submitting New Workflows
+------------------------
+
+The Submit Panel is the main interface for creating and submitting computational jobs in the system. It provides a comprehensive workflow that guides users through model selection, parameter configuration, and input data specification.
+To submit a new workflow:
+
+1. Navigate to the submission page
+2. Select the workflow type from the available options
+3. Fill in the required parameters in the form
+   - Forms are dynamically generated based on the workflow type
+   - Required fields are marked with an asterisk (*)
+   - Help text provides guidance for each parameter
+4. Click "Submit" to start the workflow
+5. You'll be redirected to the dashboard where you can monitor the task's progress
+
+The submission forms use React JSON Schema Form, which provides:
+
+* **Validation**: Immediate feedback on invalid inputs
+* **Conditional Fields**: Fields that appear based on other selections
+* **File Uploads**: For workflows that require input files
+* **Advanced Controls**: Sliders, dropdowns, and other interactive elements
+* **Special Fields**: for complex objects, e.g. molecular data
+
+The Submit Panel consists of several key sections organized in a logical workflow:
+
+1. **Model Selection** (CascadingNodeSelector)
+2. **Job Configuration** (Custom Name & Category)
+3. **Parameter Settings** (ParametersAccordion)
+4. **Input Data Configuration** (Input Mappings)
+5. **Template Management** (TemplateSelector)
+6. **Job Submission**
+
+
+Node Selector
+~~~~~~~~~~~~~
+
+The CascadingNodeSelector is the primary interface for browsing and selecting computational models from the available node registry.
+
+.. figure:: resources/node_selector.png
+   :alt: Node Selector Interface
+   :align: center
+   :width: 700px
+
+   *Hierarchical model browser with favorites and tree structure*
+
+**Hierarchical Model Organization**
+   Models are organized in a tree structure based on their function mapping, making it easy to navigate through different categories and modules.
+
+.. figure:: resources/model-tree-structure.png
+   :alt: Model Tree Structure
+   :align: center
+   :width: 500px
+
+   *Example of hierarchical model organization which mirrors the directory structure of your project*
+
+**Favorites System**
+   Users can mark frequently used models as favorites for quick access:
+
+   - Star icon checkbox for marking/unmarking favorites
+   - Dedicated "Favorites" section at the top of the tree
+   - Persistent favorite status across sessions
+
+
+Once a model is selected, the ParametersAccordion and Input sections dynamically update to reflect the chosen model's requirements.
+
+Parameter Settings
+~~~~~~~~~~~~~~~~~~
+
+The ParametersAccordion section provides comprehensive job parameter configuration options.
+
+.. figure:: resources/job-parameters.png
+   :alt: Parameters Accordion
+   :align: center
+   :width: 700px
+
+   *Complete parameters configuration interface (for a slurm-queue job)*
+
+**Resource Selection**
+   Configure computational resources for job execution:
+
+   - Dropdown selection from allowed resources
+   - Default option available if no specific resource needed
+   - Dynamic loading of available resources from the system
+
+**Queue Configuration**
+   Specify execution queue for job scheduling:
+
+   - Selection from allowed queues list
+   - When the SLURM queue is selected, additional SLURM parameters become available
+
+**SLURM Parameters** (when using "slurm-queue")
+   Additional SLURM-specific configuration options become available when the SLURM queue is selected, including:
+
+   - Job allocation parameters
+   - Resource requirements
+   - Time limits
+   - Node specifications
+
+**Execution Options**
+   Control job execution behavior:
+
+   - **Force Rerun**: Force execution even if cached results exist
+   - **Recompute Artifacts**: Regenerate all output artifacts regardless of cache status
+
+**Custom Name & Category Editor**
+   Provides job identification and organization:
+
+   **Custom Name Field**
+      - User-defined name for the job
+      - Defaults to "submitted-job"
+      - Automatically saved to localStorage per operation
+
+   **Category Field**
+      - Organizational category for the job
+      - Defaults to "submission"
+      - Persistent storage with operation-specific recall
+
+Model Input Section
+~~~~~~~~~~~~~~~~~~~
+
+The Input section handles data configuration for each input mapping required by the selected model.
+For each input mapping defined by the selected model:
+
+.. figure:: resources/sample-input.png
+   :alt: Input Mapping Card
+   :align: center
+   :width: 650px
+
+   *Individual input mapping configuration card*
+
+**Dynamic Form Generation**
+   - Automatically generated forms based on input mapping schema
+   - Real-time validation and data binding
+   - Memoized form components for optimal performance
+
+**Template Management**
+   - Load existing input templates
+   - Save current configuration as template
+   - Update existing templates on submission
+   - Template-specific parameter inheritance

simstack/static/docs/_sources/tables.rst.txt

@@ -0,0 +1,50 @@
+.. _tables:
+
+Tables
+======
+
+SimStack II provides a ``SimpleTable`` model to handle and display tabular data, especially for UI representation in ag-grid.
+
+SimpleTable
+-----------
+
+The ``SimpleTable`` model allows you to define a table with headings, data types for columns, and rows of data.
+
+**Key Features:**
+
+* **UI Ready**: Designed to be used with a specialized UI field (``SimpleTableField``) that renders the data in an ag-grid component.
+* **Flexible Rows**: Rows are stored as dictionaries, making it easy to map data to column headers.
+* **Type Information**: Allows specifying the type of each column for better UI rendering or validation.
+
+**Usage:**
+
+.. code-block:: python
+
+   from simstack.models.simple_table import SimpleTable
+
+   table = SimpleTable(name="Experimental Results")
+   
+   # Add columns with their types
+   table.add_column("Material", "string")
+   table.add_column("Temperature", "number")
+   table.add_column("Pressure", "number")
+   
+   # Add rows as dictionaries
+   table.add_row({
+       "Material": "Silicon",
+       "Temperature": 300,
+       "Pressure": 1.0
+   })
+   table.add_row({
+       "Material": "Germanium",
+       "Temperature": 350,
+       "Pressure": 1.2
+   })
+
+API Reference
+-------------
+
+.. autoclass:: simstack.models.simple_table.SimpleTable
+   :members:
+   :undoc-members:
+   :show-inheritance:

simstack/static/docs/_sources/testing_complex_nodes.rst.txt

@@ -0,0 +1,92 @@
+Testing Complex Nodes
+=====================
+
+This document explains the testing strategy for complex nodes in SimStack, focusing on the test generation and re-execution mechanism.
+
+Overview
+--------
+
+In SimStack, some nodes perform complex tasks that might be difficult to reproduce in a standard unit test environment due to external dependencies, large data volumes, or non-deterministic behavior. To address this, we use a strategy of capturing the state of a successful node execution and using it to generate a repeatable test case.
+
+The two main components of this strategy are:
+1. `simstack/src/simstack/methods/generate_test.py`: A tool to extract node data from the database and work directory.
+2. `examples/testing/testing_test_generation.py`: An example script demonstrating how to use generated tests and patch complex executions.
+
+Test Generation (`generate_test.py`)
+------------------------------------
+
+The `generate_test` method (and its CLI wrapper) allows developers to create a test case from an existing `NodeRegistry` entry.
+
+Workflow:
+1. **Identify Node**: The user provides a `node_id` (ObjectId of a `NodeRegistry` entry).
+2. **Setup Target**: A target directory is created, organized by node name and argument hash: `target_base / node_name / arg_hash`.
+3. **Capture Files**: All files from the node's work directory (`workdir / node_name / node_id`) are copied to the target directory.
+4. **Serialize Data**:
+   - Input models are loaded from the database based on `input_references` and serialized to `inputs.json`.
+   - Output models (results) are loaded based on `results_references` and serialized to `outputs.json`.
+   - This is for convenience and not really needed for the test
+
+This results in a self-contained directory containing everything needed to verify the node's behavior for a specific set of inputs.
+
+Testing with Patches (`testing_test_generation.py`)
+--------------------------------------------------
+
+Once a test case is generated, it can be used to verify that changes to the node's logic don't break existing functionality, or to debug complex failures.
+
+The example script `testing_test_generation.py` demonstrates a pattern for testing nodes that call external functions or have complex side effects.
+
+### The Problem: Complex Side Effects
+Some nodes call functions that perform operations (like submitting a job to a cluster) which cannot be easily duplicated in testing.
+The idea is to store all of the resulting inout and output files and also serialized versions of all costly intermediate results.
+During a test run, we don't want to actually submit a new job; instead, we want to simulate the previous execution and
+return the *original* `task_id` so that SimStack can find the associated files and results.
+
+### The Solution: Patching and Re-execution
+The strategy involves patching the "inner" execution function during the test run.
+
+1. **Setup**: The node is first executed normally (or a previous execution is identified). All of the complex and costly computations are performed
+   in a function like `complex_task_execution` which **must** have the signature *args, **kwargs.
+    - The *args are used during normal execution to generate the costly data and serialize it.
+    - The function returns data needed to locally construct the output models. Generating the input files or arsing the output files of a job, which is
+      part of a test, should be outside `complex_task_execution`
+    - The goal is: Do all the stuff that can be locally and that needs to be testes outside of `complex_task_execution
+2. **Generation**: `generate_test` is called to create the test data.
+3. **Patching**: Using `unittest.mock.patch`, the function responsible for the side effect (e.g., `complex_task_execution`) is replaced with
+    a `patched_complex_task_execution`. This function also gets the *args and **kwargs. It needs the latter to automatically find
+    the test directory with respect to a test-directory root.
+4. **Mock Logic**: The patched function:
+   - Identifies the correct test directory using `arg_hash` and node name (provided in `kwargs`).
+   - The file can compare the input files generated in the test run with the input files in the test directory
+   - Restores the necessary files (e.g., `test.txt`) from the test directory to the current execution environment.
+   - Deserialize all important information to locally build the outputs of the nodes. In the test this is only the original `task_id`.
+
+5. **Re-execution**: The node is called again with `force_rerun=True`.
+6. **Verification**: The results of the re-execution are compared against the original results to ensure consistency.
+
+Example Patch Implementation
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. code-block:: python
+
+    def patched_complex_task_execution(arg_patched, **kwargs_patched):
+        # Locate the test directory
+        test_dir = local_target_base / node_name / kwargs_patched["arg_hash"]
+        
+        # Restore state
+        shutil.copy2(test_dir / "test.txt", Path.cwd() / "test.txt")
+        
+        # Retrieve original ID
+        with open("test.txt", "r") as f:
+            original_task_id = f.readlines()[1].strip()
+            
+        return original_task_id
+
+    with patch("__main__.complex_task_execution", side_effect=patched_complex_task_execution):
+        test_result = complex_task_for_testing(arg, parameters=Parameters(force_rerun=True))
+        # Assertions...
+
+Benefits
+--------
+- **Reproducibility**: Easily reproduce complex node executions in a local environment.
+- **Isolation**: Test node logic without triggering expensive or external side effects.
+- **Verification**: Ensure that data processing remains consistent over time.