gemstone-engine 1.2.0__tar.gz

gemstone_engine-1.2.0/LICENSE

@@ -0,0 +1,12 @@
+Creative Commons Attribution–NoDerivatives 4.0 International (CC BY-ND 4.0)
+Copyright © 2025 Your Name
+
+This work is licensed under the Creative Commons Attribution–NoDerivatives 4.0 International License.
+You may share — copy and redistribute the material in any medium or format — under the following terms:
+
+Attribution — You must give appropriate credit, provide a link to the license, and indicate if changes were made.
+
+No Derivatives — If you remix, transform, or build upon the material, you may not distribute the modified material.
+
+To view a copy of this license, visit:
+https://creativecommons.org/licenses/by-nd/4.0/

gemstone_engine-1.2.0/PKG-INFO

@@ -0,0 +1,109 @@
+Metadata-Version: 2.4
+Name: gemstone_engine
+Version: 1.2.0
+Summary: GEMSTONe: Generic Engine for Modular Software Task Orchestration and Negotiation
+Author-email: Milo Fryling <milo.fryling@duke.edu>
+License: CC BY-ND 4.0
+Classifier: Programming Language :: Python :: 3
+Requires-Python: >=3.8
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Dynamic: license-file
+
+# GEMSTONe: Generic Engine for Modular Software Task Orchestration and Negotiation
+
+## Overview
+
+GEMSTONe is an engine that runs customizable modular workflows. We use the term "module" in the usual sense — a standalone process that takes inputs and produces outputs. 
+When modules are linked together in a workflow, GEMSTONe refers to each one as a "stage"; otherwise terms "module" and "stage" can be used interchangeably.  
+Each stage is a standalone processing step: it reads input files, does its job, and writes outputs. A workflow links these stages together in sequence. GEMSTONe takes care of running them in order, passing data along the way.
+
+Your role is to make each stage GEMSTONe-ready so the engine can:
+
+- Start it with the right parameters  
+- Handle all file paths and connections between stages automatically
+
+## Install
+
+```bash
+pip install gemstone-engine
+```
+
+## Design Goals
+
+- **No proprietary language.** Simple and obvious human-readable syntax for all configuration files (`specs.txt`, `iofiles.txt`, `stageflows.txt`, etc.)
+
+- **Boilerplate code eliminated.** Parameter validation, path management, file compatibility, and chaining are handled by the engine.
+
+- **Zero coupling.** Module code and engine code are 100% independent — changes to one never require changes to the other.
+
+## Quick Start
+
+To plug a stage into GEMSTONe, just include these three files:
+
+- `specs.txt`: lists user input parameters and any defaults or constraints  
+- `iofiles.txt`: defines the stage’s inputs and outputs in table format  
+- `run.py`: contains `def run(params, paths):`, which launches your code  
+- Your module’s own implementation lives in a `runtime/` subdirectory inside the stage folder.
+
+To connect stages into a workflow, add:
+
+- `stageflows.txt`: shows how outputs from one stage become inputs to the next  
+- `standards.txt`: lists the types of files your workflow uses  
+
+That’s it. With these files in place, GEMSTONe handles path management and input validation automatically — no need to write custom code for filenames, directories, or parameter parsing.
+
+## Example Usage
+
+**Launching a workflow**
+
+A GEMSTONe workflow is started from the command line through the standard entry point:
+
+```python
+from gemstone.cli import main
+
+if __name__ == "__main__":
+    main()
+```
+
+This accepts:
+
+```
+<param file>
+[<jobid override>]
+[--testmode]
+```
+
+**Inside a stage (`run.py`)**
+
+Each stage defines:
+
+```python
+def run(params: Params, paths: Paths):
+    param_file = paths.get_parameter_file_path()
+    write_param_file(param_file, params)
+    args = make_args(params, param_file, paths)
+
+    subprocess.run(
+        args,
+        cwd=paths.stage,
+        capture_output=True,
+        text=True,
+        check=True
+    )
+```
+
+The engine supplies both `params` and `paths`; your code focuses only on launching the underlying executable.
+
+## Project Status
+
+GEMSTONe is under active development and already in use for production research pipelines. Interfaces shown here are stable, but minor refinements are expected as the documentation and tooling expand.
+
+## Documentation
+
+Full integration documentation will be released with the public source repository.  
+Until then, concise usage notes and guidance are available—please reach out through the project’s PyPI page or institutional contact channels if you need assistance integrating a module or workflow.
+
+## License
+
+Creative Commons BY‑ND 4.0

gemstone_engine-1.2.0/README.md

@@ -0,0 +1,101 @@
+## Name
+DukeSim 3 (provisional name)
+
+## Purpose (objectives & intended audience/users)
+This provides a generic, extensible framework for running very loosely coupled software components as a pipeline.
+It provides an engine for modular orchestration, and an app library containing templates and a sample application
+
+## OS Platform, Language, dependencies
+The software runs on Linux and is written in Python.  
+
+## How to build, install and/or publish
+Use directly from a clone of the repo. There is no build or publishing procedure.  
+To reset or switch the active pipeline, rerun `use_app_stages.py` with a new app name.
+
+## User Documentation
+
+**For full instructions and detailed information, see:**  
+
+### `src/USERGUIDE.md`  
+
+---
+
+### Quick Overview (For Reference Only):  
+
+#### Summary:  
+- `src/app_library`: Holds the catalog of pipelines.  
+- `src/stages`: Holds the working copy of the active app.  
+- `src/params`: Holds parameter files.  
+- `stages.py` and `dirs.py` (at app level): Define a pipeline.  
+- `specs.txt` and `run.py` (in each component): Define the parameters and launch interface.  
+- `src/engine`: *Must not* be modified by the user.  
+
+#### Common Commands:  
+- **Select an app to activate:**  
+    ```bash
+    python3 src/use_app_stages.py <app_name>  # App name is the directory in app_library
+    ```
+- **Run the numerous unit tests in `src/tests`:**  
+    ```bash
+    src/do_tests.sh
+    ```
+- **Run a pipeline from the command line:**  
+    ```bash
+    python3 run.py params/<param_file_name>
+    # Optional: override auto-generated Job ID
+    python3 run.py params/<param_file_name> <jobid_override>
+    ```
+
+- **Activate app and run pipeline in one step (convenience wrapper):**  
+    ```bash
+    ./run_app.sh <app_name> <param_file> [jobid_override] [--testmode]
+    ```
+## App directory structure
+App Repos Will:
+
+- Be Git repos named like G-Recon, G-VIT, etc.
+- Be self-contained, not installed via pip, but invoked from repo root.
+- Use the name stages/ for the directory containing stage code.
+- Place stageflows.txt inside the stages/ directory.
+- Place params_example.txt and run_app.py at the repo root.
+- Place job outputs in output/ (created by the engine), located at the repo root.
+
+## Packaging the engine and building apps
+App Dev with GEMSTONe Clone on Same Server
+- git clone git@gitlab.com:gemstone-world/GEMSTONe.git
+- cd GEMSTONe
+- pip install -e .
+
+- cd ~/repos
+- git clone git@gitlab.com:gemstone-world/G-Recon.git
+- cd G-Recon
+- python run_app.py params_example.txt
+
+    To update engine:
+        cd ~/repos/GEMSTONe
+        git pull
+
+App Dev in the Field (pip install only)
+- pip install gemstone_engine
+
+- cd ~/repos
+- git clone git@gitlab.com:gemstone-world/G-Recon.git
+- cd G-Recon
+- python run_app.py params_example.txt
+
+    To update engine:
+        pip install --upgrade gemstone_engine
+
+Engine Publishing (per release)
+- cd GEMSTONe
+- git pull
+- (bump version in pyproject.toml)
+- python -m build
+- twine upload dist/*
+
+    Then users can do:
+        pip install gemstone_engine
+        pip install --upgrade gemstone_engine
+
+## License details
+No licensing has been considered yet.

gemstone_engine-1.2.0/README_PyPI.md

@@ -0,0 +1,97 @@
+# GEMSTONe: Generic Engine for Modular Software Task Orchestration and Negotiation
+
+## Overview
+
+GEMSTONe is an engine that runs customizable modular workflows. We use the term "module" in the usual sense — a standalone process that takes inputs and produces outputs. 
+When modules are linked together in a workflow, GEMSTONe refers to each one as a "stage"; otherwise terms "module" and "stage" can be used interchangeably.  
+Each stage is a standalone processing step: it reads input files, does its job, and writes outputs. A workflow links these stages together in sequence. GEMSTONe takes care of running them in order, passing data along the way.
+
+Your role is to make each stage GEMSTONe-ready so the engine can:
+
+- Start it with the right parameters  
+- Handle all file paths and connections between stages automatically
+
+## Install
+
+```bash
+pip install gemstone-engine
+```
+
+## Design Goals
+
+- **No proprietary language.** Simple and obvious human-readable syntax for all configuration files (`specs.txt`, `iofiles.txt`, `stageflows.txt`, etc.)
+
+- **Boilerplate code eliminated.** Parameter validation, path management, file compatibility, and chaining are handled by the engine.
+
+- **Zero coupling.** Module code and engine code are 100% independent — changes to one never require changes to the other.
+
+## Quick Start
+
+To plug a stage into GEMSTONe, just include these three files:
+
+- `specs.txt`: lists user input parameters and any defaults or constraints  
+- `iofiles.txt`: defines the stage’s inputs and outputs in table format  
+- `run.py`: contains `def run(params, paths):`, which launches your code  
+- Your module’s own implementation lives in a `runtime/` subdirectory inside the stage folder.
+
+To connect stages into a workflow, add:
+
+- `stageflows.txt`: shows how outputs from one stage become inputs to the next  
+- `standards.txt`: lists the types of files your workflow uses  
+
+That’s it. With these files in place, GEMSTONe handles path management and input validation automatically — no need to write custom code for filenames, directories, or parameter parsing.
+
+## Example Usage
+
+**Launching a workflow**
+
+A GEMSTONe workflow is started from the command line through the standard entry point:
+
+```python
+from gemstone.cli import main
+
+if __name__ == "__main__":
+    main()
+```
+
+This accepts:
+
+```
+<param file>
+[<jobid override>]
+[--testmode]
+```
+
+**Inside a stage (`run.py`)**
+
+Each stage defines:
+
+```python
+def run(params: Params, paths: Paths):
+    param_file = paths.get_parameter_file_path()
+    write_param_file(param_file, params)
+    args = make_args(params, param_file, paths)
+
+    subprocess.run(
+        args,
+        cwd=paths.stage,
+        capture_output=True,
+        text=True,
+        check=True
+    )
+```
+
+The engine supplies both `params` and `paths`; your code focuses only on launching the underlying executable.
+
+## Project Status
+
+GEMSTONe is under active development and already in use for production research pipelines. Interfaces shown here are stable, but minor refinements are expected as the documentation and tooling expand.
+
+## Documentation
+
+Full integration documentation will be released with the public source repository.  
+Until then, concise usage notes and guidance are available—please reach out through the project’s PyPI page or institutional contact channels if you need assistance integrating a module or workflow.
+
+## License
+
+Creative Commons BY‑ND 4.0

gemstone_engine-1.2.0/pyproject.toml

@@ -0,0 +1,24 @@
+[build-system]
+requires = ["setuptools", "wheel"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "gemstone_engine"         # distribution name (pip install gemstone_engine)
+version = "1.2.0"                # bump on each release
+description = "GEMSTONe: Generic Engine for Modular Software Task Orchestration and Negotiation"
+authors = [
+    { name="Milo Fryling", email="milo.fryling@duke.edu" }
+]
+readme = "README_PyPI.md"
+license = { text = "CC BY-ND 4.0" }
+requires-python = ">=3.8"
+classifiers = [ "Programming Language :: Python :: 3" ]
+
+# what gets installed as the importable package
+[tool.setuptools.packages.find]
+include = ["gemstone"]
+where = ["src"]
+
+
+[project.scripts]
+gemstone = "gemstone.cli:main"

gemstone_engine-1.2.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

gemstone_engine-1.2.0/setup.py

@@ -0,0 +1,15 @@
+# setup.py
+from setuptools import setup, find_packages
+
+setup(
+    name="gemstone_engine",
+    version="1.0.0",
+    packages=find_packages(where="src"),
+    package_dir={"": "src"},
+    install_requires=[],
+    entry_points={
+        "console_scripts": [
+            "gemstone = gemstone.cli:main",
+        ],
+    },
+)

gemstone_engine-1.2.0/src/gemstone/cli.py

@@ -0,0 +1,149 @@
+import argparse
+import os
+import sys
+import logging
+import subprocess
+from datetime import datetime
+from importlib.metadata import version as pkg_version, PackageNotFoundError
+from gemstone.pipeliner import run_pipeline
+import gemstone.dir as Dir
+
+class ExitCode:
+    """Standardized exit codes for pipeline errors."""
+    MISSING_FILE = 99
+    CONFIG_ERROR = 2
+    PERMISSION_ERROR = 3
+    KEY_ERROR = 4
+    RUNTIME_ERROR = 1
+    UNKNOWN_ERROR = 100
+
+def get_version() -> str:
+    """Returns GEMSTONe version from package metadata (installed) or git (dev)."""
+    try:
+        return pkg_version("gemstone_engine")  # Use actual package name
+    except PackageNotFoundError:
+        try:
+            return subprocess.check_output(
+                ["git", "describe", "--tags", "--dirty", "--always"],
+                stderr=subprocess.DEVNULL,
+            ).decode().strip()
+        except Exception:
+            return "unknown"
+
+def display_version(terse:bool = False):
+    """Displays the pipeline version (from git describe)."""
+    version = get_version()
+
+    if terse:
+        print(f" GEMSTONe {version}")
+    else:
+        print('===========================')
+        print(f" GEMSTONe {version}")
+        print('===========================')
+
+def show_error(msg, exit_code):
+    """Prints and logs an error message, then exits."""
+    print(msg)
+    logging.error(msg)
+    sys.exit(exit_code)
+
+def main():
+    """Entry point to start the pipeline."""
+    parser = argparse.ArgumentParser(description="Run a GEMSTONe pipeline.")
+    parser.add_argument(
+        "params",
+        type=str,
+        nargs="?",  # Make this argument optional
+        help="Path to the param file."
+    )
+    parser.add_argument(
+        "jobid_override",
+        type=str,
+        nargs="?",  # Optional positional
+        help="Optional: override the auto-generated job ID, e.g. to restart a job."
+    )
+    parser.add_argument(
+        "--test_mode",
+        action="store_true",
+        help="Optional: run in test mode (create outputs without executing real code)."
+    )
+    parser.add_argument(
+        "--version", 
+        action="store_true", 
+        help="Display GEMSTONe version and exit.")
+
+    args = parser.parse_args()
+    if args.version:
+        display_version(True)
+        return
+    
+    display_version(False)
+    setup_logging()
+
+    if args.test_mode:
+        logging.warning("TEST MODE ENABLED — no real execution will occur.")
+
+    try:
+        Dir.set_root(os.getcwd())
+        run_pipeline(args.params, args.jobid_override, test_mode=args.test_mode)
+
+    except FileNotFoundError as e:
+        show_error(f"Error: {e}", ExitCode.MISSING_FILE)
+
+    except ValueError as e:
+        show_error(f"Configuration error: {e}", ExitCode.CONFIG_ERROR)
+
+    except PermissionError as e:
+        show_error(f"Permission error: {e}", ExitCode.PERMISSION_ERROR)
+
+    except RuntimeError as e:
+        show_error(f"Runtime error: {e}", ExitCode.RUNTIME_ERROR)
+
+    except KeyError as e:
+        show_error(f"Param key error: {e}", ExitCode.KEY_ERROR)
+
+    except Exception as e:
+        logging.exception("Unexpected error:")
+        show_error("An unexpected error occurred. Check the log for details.", ExitCode.UNKNOWN_ERROR)
+
+class DotMillisecondsFormatter(logging.Formatter):
+    def formatTime(self, record, datefmt=None):
+        t = datetime.fromtimestamp(record.created)
+        return t.strftime("%Y-%m-%d %H:%M:%S.%f")[:-3]  # Keep milliseconds only
+
+def setup_logging():
+    """Set up logging with consistent format and millisecond timestamps."""
+    log_file = "GEMSTONe.log"
+
+    root_logger = logging.getLogger()
+    root_logger.setLevel(logging.INFO)
+
+    # Remove any existing handlers
+    for handler in root_logger.handlers[:]:
+        root_logger.removeHandler(handler)
+
+    formatter = DotMillisecondsFormatter(
+        "%(asctime)s - %(levelname)s - %(message)s"
+    )
+
+    file_handler = logging.FileHandler(log_file)
+    file_handler.setLevel(logging.INFO)
+    file_handler.setFormatter(formatter)
+
+    console_handler = logging.StreamHandler(sys.stdout)
+    console_handler.setLevel(logging.INFO)
+    console_handler.setFormatter(formatter)
+
+    root_logger.addHandler(file_handler)
+    root_logger.addHandler(console_handler)
+
+    # Ensure submodules propagate to root
+    logging.getLogger('subprocess').propagate = True
+    logging.getLogger('gemstone').propagate = True
+    logging.getLogger('stages').propagate = True
+
+    logging.info(f"Logging to file: {log_file}")
+
+
+if __name__ == "__main__":
+    main()

gemstone_engine-1.2.0/src/gemstone/dir.py

@@ -0,0 +1,22 @@
+# dir.py - absolute path directory layout constants in deployed environment
+import os
+
+root = "."
+stages = "."
+output = "."
+
+def set_root(root_path):
+	global root, stages, output
+	root = os.path.abspath(root_path)
+	stages = os.path.join(root, "stages")
+	output = os.path.join(root, "output")
+
+def validate(subdirs=None):
+    if root in (".", None):
+        raise RuntimeError("Dir.set_root() must be called before validate()")
+
+    check = subdirs or ["stages", "output"]
+    for d in check:
+        full = os.path.join(root, d)
+        if not os.path.isdir(full):
+            raise RuntimeError(f"Missing required directory: {full}")