svc-processing 0.1.1__tar.gz → 0.1.3__tar.gz

{svc_processing-0.1.1 → svc_processing-0.1.3}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: svc-processing
-Version: 0.1.1
+Version: 0.1.3
 Summary: Pure-Python SVC HR-1024i hyperspectral .sig processing pipeline
 Author: Gold Lab (GrapeSPEC project), Cornell University
 Author-email: Cole Regnier <nr466@cornell.edu>
@@ -65,6 +65,20 @@ Generated outputs (gitignored): `pipeline_outputs/sig_processed/<run>/`, `pipeli
 > New to the pipeline? [`TUTORIAL.md`](TUTORIAL.md) starts with the interactive
 > Jupyter notebook (the gentle path) and has an advanced track for the CLI below.
 
+**Installed from PyPI, in a new project folder** (no clone of this repo needed):
+
+```bash
+python3.11 -m venv .venv
+source .venv/bin/activate
+python -m pip install svc-processing   # package name
+
+svc-pipeline --init-config   # command name — different from the package name above
+# Edit config/config.json — replace "<PATH_TO_SIG_INPUT_ROOT>" with your data path.
+svc-pipeline
+```
+
+**Working from a clone of this repo** (config/config.json already exists):
+
 ```bash
 python3.11 -m venv .venv
 source .venv/bin/activate
@@ -112,8 +126,10 @@ Outputs land under `pipeline_outputs/` by default. See [`config/README.md`](conf
    └────────────────────────────────────┘
 ```
 
-The `svc-pipeline` console script glues Stages 1 and 2 together through
-`pipeline.cli`; Stage 3 is invoked from notebooks against the Stage 2 output.
+The `svc-pipeline` console script runs Stages 1 and 2 through `pipeline.cli`,
+and Stage 3 too once a `groups_csv` is configured (or passed via
+`--groups-csv`) — see [`config/README.md`](config/README.md#grouping--stage-3-optional).
+Stage 3 is also available directly from notebooks against the Stage 2 output.
 The pipeline is **single-pass and idempotent per input directory** — previous
 processed `.sig` files in the target directory are deleted at the start of each
 run.
@@ -132,7 +148,9 @@ svc-pipeline [config] [options]
 |---|---|
 | `config` | Run-config JSON (positional, optional; default `config/config.json`). Bare names resolve under `config/`, so `config.json`, `config`, and `config/config.json` are equivalent. See [`config/README.md`](config/README.md) for schema. |
 | `--input-dir <path>` | Override `sig_input_dir` and process only this directory. |
-| `--step {1,2,all}` | `1` = process + summary CSV only; `2` = resample only (requires Stage 1 to have been run); `all` = both (default). |
+| `--step {1,2,3,all}` | `1` = process + summary CSV only; `2` = resample only (requires Stage 1); `3` = group + average only (requires Stage 2 and `groups_csv`); `all` = every stage that's configured (default). |
+| `--groups-csv <path>` | Override the config's `groups_csv` to group repeat scans and average them. |
+| `--group-method {mean,median,sum,min,max}` | Override the config's `group_agg_method` for Stage 3 (default `mean`). |
 | `--verbose` | Print INFO/DEBUG logs before and after each stage. |
 
 ### Expected output layout
@@ -143,10 +161,11 @@ pipeline_outputs/
 │   ├── <truncated *.sig files>
 │   └── <input_dir_name>_processed_sig_summary.csv
 └── sig_resampled/<input_dir_name>/
-    └── <input_dir_name>_merged_spectra.csv     # 2101 columns (400–2500 nm)
+    ├── <input_dir_name>_merged_spectra.csv     # 2101 columns (400–2500 nm)
+    └── <input_dir_name>_grouped_spectra.csv    # only if groups_csv is configured
 ```
 
-`summary_csv_name` and `merged_csv_name` in the run config are suffixes; the real filenames are prefixed with the input directory name. Verify your config in [`config/README.md`](config/README.md).
+`summary_csv_name`, `merged_csv_name`, and `grouped_csv_name` in the run config are suffixes; the real filenames are prefixed with the input directory name. Verify your config in [`config/README.md`](config/README.md).
 
 ---
 

{svc_processing-0.1.1 → svc_processing-0.1.3}/README.md

@@ -32,6 +32,20 @@ Generated outputs (gitignored): `pipeline_outputs/sig_processed/<run>/`, `pipeli
 > New to the pipeline? [`TUTORIAL.md`](TUTORIAL.md) starts with the interactive
 > Jupyter notebook (the gentle path) and has an advanced track for the CLI below.
 
+**Installed from PyPI, in a new project folder** (no clone of this repo needed):
+
+```bash
+python3.11 -m venv .venv
+source .venv/bin/activate
+python -m pip install svc-processing   # package name
+
+svc-pipeline --init-config   # command name — different from the package name above
+# Edit config/config.json — replace "<PATH_TO_SIG_INPUT_ROOT>" with your data path.
+svc-pipeline
+```
+
+**Working from a clone of this repo** (config/config.json already exists):
+
 ```bash
 python3.11 -m venv .venv
 source .venv/bin/activate
@@ -79,8 +93,10 @@ Outputs land under `pipeline_outputs/` by default. See [`config/README.md`](conf
    └────────────────────────────────────┘
 ```
 
-The `svc-pipeline` console script glues Stages 1 and 2 together through
-`pipeline.cli`; Stage 3 is invoked from notebooks against the Stage 2 output.
+The `svc-pipeline` console script runs Stages 1 and 2 through `pipeline.cli`,
+and Stage 3 too once a `groups_csv` is configured (or passed via
+`--groups-csv`) — see [`config/README.md`](config/README.md#grouping--stage-3-optional).
+Stage 3 is also available directly from notebooks against the Stage 2 output.
 The pipeline is **single-pass and idempotent per input directory** — previous
 processed `.sig` files in the target directory are deleted at the start of each
 run.
@@ -99,7 +115,9 @@ svc-pipeline [config] [options]
 |---|---|
 | `config` | Run-config JSON (positional, optional; default `config/config.json`). Bare names resolve under `config/`, so `config.json`, `config`, and `config/config.json` are equivalent. See [`config/README.md`](config/README.md) for schema. |
 | `--input-dir <path>` | Override `sig_input_dir` and process only this directory. |
-| `--step {1,2,all}` | `1` = process + summary CSV only; `2` = resample only (requires Stage 1 to have been run); `all` = both (default). |
+| `--step {1,2,3,all}` | `1` = process + summary CSV only; `2` = resample only (requires Stage 1); `3` = group + average only (requires Stage 2 and `groups_csv`); `all` = every stage that's configured (default). |
+| `--groups-csv <path>` | Override the config's `groups_csv` to group repeat scans and average them. |
+| `--group-method {mean,median,sum,min,max}` | Override the config's `group_agg_method` for Stage 3 (default `mean`). |
 | `--verbose` | Print INFO/DEBUG logs before and after each stage. |
 
 ### Expected output layout
@@ -110,10 +128,11 @@ pipeline_outputs/
 │   ├── <truncated *.sig files>
 │   └── <input_dir_name>_processed_sig_summary.csv
 └── sig_resampled/<input_dir_name>/
-    └── <input_dir_name>_merged_spectra.csv     # 2101 columns (400–2500 nm)
+    ├── <input_dir_name>_merged_spectra.csv     # 2101 columns (400–2500 nm)
+    └── <input_dir_name>_grouped_spectra.csv    # only if groups_csv is configured
 ```
 
-`summary_csv_name` and `merged_csv_name` in the run config are suffixes; the real filenames are prefixed with the input directory name. Verify your config in [`config/README.md`](config/README.md).
+`summary_csv_name`, `merged_csv_name`, and `grouped_csv_name` in the run config are suffixes; the real filenames are prefixed with the input directory name. Verify your config in [`config/README.md`](config/README.md).
 
 ---
 

{svc_processing-0.1.1 → svc_processing-0.1.3}/pipeline/cli.py

@@ -28,7 +28,13 @@ def _configure_logging(verbose: bool) -> logging.Logger:
 
 
 def _parse_args() -> argparse.Namespace:
-    parser = argparse.ArgumentParser(description="Run the SIG processing pipeline")
+    parser = argparse.ArgumentParser(
+        description="Run the SIG processing pipeline",
+        epilog=(
+            "Full README, TUTORIAL, and config reference: "
+            "https://github.com/regs08/SVCProcessingPipeline"
+        ),
+    )
     parser.add_argument(
         "config",
         nargs="?",
@@ -41,15 +47,36 @@ def _parse_args() -> argparse.Namespace:
     )
     parser.add_argument(
         "--step",
-        choices=["1", "2", "all"],
+        choices=["1", "2", "3", "all"],
         default="all",
-        help="Which step to run: 1=process+summary CSV, 2=Python resampling only, all=run both steps.",
+        help=(
+            "Which step to run: 1=process+summary CSV, 2=Python resampling only, "
+            "3=group+average only (requires groups_csv; step 2 must have already run), "
+            "all=every step that's configured (step 3 only if groups_csv is set)."
+        ),
     )
     parser.add_argument(
         "--verbose",
         action="store_true",
         help="Enable verbose logging before and after each processing step",
     )
+    parser.add_argument(
+        "--init-config",
+        action="store_true",
+        help="Write a starter config/config.json in the current directory and exit.",
+    )
+    parser.add_argument(
+        "--groups-csv",
+        help=(
+            "Override the config's groups_csv: group repeat scans of the same sample "
+            "and average them (a CSV with scans/scan_id + name columns)."
+        ),
+    )
+    parser.add_argument(
+        "--group-method",
+        choices=["mean", "median", "sum", "min", "max"],
+        help="Override the config's group_agg_method for step 3 (default: mean).",
+    )
     return parser.parse_args()
 
 
@@ -61,6 +88,13 @@ def main() -> None:
     # working directory so the installed `svc-pipeline` command works from any
     # directory. Absolute paths in the config are always honoured as-is.
     base_dir = Path.cwd()
+
+    if args.init_config:
+        target = RunConfig.write_starter_config(base_dir)
+        print(f"Wrote starter config: {target.resolve()}")
+        print('Edit "sig_input_dir" to point at your .sig data, then run: svc-pipeline')
+        return
+
     run_config = RunConfig.load(base_dir, str(args.config), logger)
     run_config.ensure_no_placeholder(args.input_dir)
 
@@ -73,9 +107,16 @@ def main() -> None:
     # warning for any non-default value before the per-directory work begins.
     run_config.processing_params()
 
+    groups_csv_override = Path(args.groups_csv).expanduser() if args.groups_csv else None
+
     overall_results: list[tuple[Path, dict[str, Path | None]]] = []
     for input_dir in input_dirs:
-        settings = run_config.settings_for(input_dir, verbose=args.verbose)
+        settings = run_config.settings_for(
+            input_dir,
+            verbose=args.verbose,
+            groups_csv_override=groups_csv_override,
+            group_method_override=args.group_method,
+        )
         results = Pipeline(settings, logger).run(args.step)
         overall_results.append((input_dir, results))
 

{svc_processing-0.1.1 → svc_processing-0.1.3}/pipeline/run_config.py

@@ -34,6 +34,24 @@ _PARITY_DEFAULTS: dict[str, Any] = {
 
 _PLACEHOLDER = "<PATH_TO_SIG_INPUT_ROOT>"
 
+# Starter template written by `svc-pipeline --init-config`. Mirrors the
+# repo's own config/config.json — keep the two in sync if either changes.
+_STARTER_CONFIG_TEMPLATE: dict[str, Any] = {
+    "sig_input_dir": _PLACEHOLDER,
+    "process_all_subdirs": True,
+    "processed_dir": "sig_processed",
+    "resampled_dir": "sig_resampled",
+    "output_dir": "pipeline_outputs",
+    "summary_csv_name": "processed_sig_summary.csv",
+    "merged_csv_name": "merged_spectra.csv",
+    "end_line_overrides": {},
+    "instrument": {
+        "bronze": {"end_line": "2520.4", "serial": "2212118"},
+        "silver": {"end_line": "2517.9", "serial": "1202103"},
+    },
+    "processing": dict(_PARITY_DEFAULTS),
+}
+
 
 def _resolve_under(base: Path, value: str) -> Path:
     """Resolve ``value`` against ``base`` unless it is already absolute."""
@@ -58,6 +76,9 @@ class PipelineSettings:
     processing_params: dict
     correction_types: dict[str, str]
     instrument_numbers: dict[str, str]
+    groups_csv: Path | None
+    group_agg_method: str
+    grouped_csv_name: str
 
 
 class RunConfig:
@@ -87,6 +108,25 @@ class RunConfig:
         logger.info("Using config: %s", path)
         return cls(data, path=path, base_dir=base_dir, logger=logger)
 
+    @classmethod
+    def write_starter_config(cls, base_dir: Path, *, filename: str = "config.json") -> Path:
+        """Write a starter ``config/<filename>`` under ``base_dir`` and return its path.
+
+        Refuses to overwrite an existing file, so re-running this never clobbers
+        edits you've already made.
+        """
+        target = base_dir / "config" / filename
+        if target.exists():
+            raise SystemExit(
+                f"{target} already exists — not overwriting.\n"
+                f"  Edit it directly, or remove it first to regenerate the starter template."
+            )
+        target.parent.mkdir(parents=True, exist_ok=True)
+        with target.open("w") as handle:
+            json.dump(_STARTER_CONFIG_TEMPLATE, handle, indent=2)
+            handle.write("\n")
+        return target
+
     @staticmethod
     def _resolve_path(base_dir: Path, value: str) -> Path:
         """Resolve a run-config argument with friendly fallbacks.
@@ -262,7 +302,14 @@ class RunConfig:
             self._logger.info("Instrument serials from config: %s", serials)
         return correction_types, instrument_numbers
 
-    def settings_for(self, input_dir: Path, *, verbose: bool) -> PipelineSettings:
+    def settings_for(
+        self,
+        input_dir: Path,
+        *,
+        verbose: bool,
+        groups_csv_override: str | Path | None = None,
+        group_method_override: str | None = None,
+    ) -> PipelineSettings:
         """Build the fully-resolved :class:`PipelineSettings` for one input directory."""
         input_dir = Path(input_dir).expanduser()
         source_name = input_dir.name or "sig_input"
@@ -288,6 +335,11 @@ class RunConfig:
             if key is not None
         }
 
+        groups_csv_value = groups_csv_override or self.data.get("groups_csv")
+        groups_csv = _resolve_under(self.base_dir, str(groups_csv_value)) if groups_csv_value else None
+        group_agg_method = str(group_method_override or self.data.get("group_agg_method", "mean"))
+        grouped_csv_name = f"{source_name}_{self.data.get('grouped_csv_name', 'grouped_spectra.csv')}"
+
         return PipelineSettings(
             source_name=source_name,
             input_dir=input_dir,
@@ -300,4 +352,7 @@ class RunConfig:
             processing_params=self.processing_params(),
             correction_types=correction_types,
             instrument_numbers=instrument_numbers,
+            groups_csv=groups_csv,
+            group_agg_method=group_agg_method,
+            grouped_csv_name=grouped_csv_name,
         )

{svc_processing-0.1.1 → svc_processing-0.1.3}/pipeline/runner.py

@@ -1,4 +1,4 @@
-"""Execution of the two SIG processing stages.
+"""Execution of the three SIG processing stages.
 
 :class:`Pipeline` runs, for a single resolved
 :class:`~pipeline.run_config.PipelineSettings`:
@@ -7,6 +7,9 @@
   the calibration end-line and write a per-run summary CSV.
 * Stage 2 (:meth:`Pipeline.resample`) — run the pure-Python resampler over the
   processed files.
+* Stage 3 (:meth:`Pipeline.group_and_average`) — optional; only runs when the
+  config supplies a ``groups_csv``. Groups repeat scans of the same sample and
+  averages them into one spectrum per group.
 
 Config loading that produces the ``PipelineSettings`` lives in
 ``pipeline/run_config.py``; the CLI that wires them together lives in
@@ -20,6 +23,7 @@ import logging
 from pathlib import Path
 from typing import Iterable
 
+from .processor import GroupSpec, SigSpectraAverager
 from .resampler import resample_spectra
 from .run_config import PipelineSettings
 from .sig_processor import SigFileProcessor
@@ -38,12 +42,14 @@ class Pipeline:
         """Run the requested step(s) and report the produced artifacts.
 
         ``step``: ``"1"`` = Stage 1 only, ``"2"`` = Stage 2 only (Stage 1 must
-        have run previously), ``"all"`` = both.
+        have run previously), ``"3"`` = Stage 3 only (Stage 2 must have run
+        previously; requires ``groups_csv`` to be configured), ``"all"`` = every
+        stage that's configured to run (Stage 3 only if ``groups_csv`` is set).
         """
-        summary_csv: Path | None
+        summary_csv: Path | None = None
         if step in {"1", "all"}:
             summary_csv = self.process_sig_files()
-        else:
+        elif step == "2":
             summary_csv = self.settings.summary_csv if self.settings.summary_csv.exists() else None
             if summary_csv is None:
                 self.logger.error(
@@ -54,8 +60,17 @@ class Pipeline:
         merged_csv: Path | None = None
         if step in {"2", "all"}:
             merged_csv = self.resample(summary_csv)
+        elif step == "3":
+            merged_path = self.settings.resampled_dir / self.settings.merged_csv_name
+            merged_csv = merged_path if merged_path.exists() else None
+            if merged_csv is None:
+                self.logger.error("Merged CSV not found at %s (run --step 2 first).", merged_path.resolve())
 
-        return {"summary_csv": summary_csv, "merged_csv": merged_csv}
+        grouped_csv: Path | None = None
+        if step in {"3", "all"} and self.settings.groups_csv is not None:
+            grouped_csv = self.group_and_average(merged_csv)
+
+        return {"summary_csv": summary_csv, "merged_csv": merged_csv, "grouped_csv": grouped_csv}
 
     # ── Stage 1: process raw .sig files ──────────────────────────────────────
 
@@ -194,6 +209,44 @@ class Pipeline:
         logger.warning("Merged spectra file was not created at %s", merged_path)
         return None
 
+    # ── Stage 3: group & average (optional) ──────────────────────────────────
+
+    def group_and_average(self, merged_csv: Path | None) -> Path | None:
+        """Average repeat scans of the same sample into one spectrum per group.
+
+        Requires ``settings.groups_csv`` (checked by the caller) and a merged
+        CSV from Stage 2. Groups are read via :meth:`GroupSpec.from_csv`
+        (``scans``/``scan_id`` + ``name`` columns) and applied with
+        :class:`SigSpectraAverager`.
+        """
+        settings = self.settings
+        logger = self.logger
+        if merged_csv is None:
+            logger.error("Skipping grouping because no merged CSV is available.")
+            return None
+
+        groups_csv = settings.groups_csv
+        if groups_csv is None or not groups_csv.exists():
+            logger.error("Groups CSV not found: %s", groups_csv.resolve() if groups_csv else groups_csv)
+            return None
+
+        groups = GroupSpec.from_csv(groups_csv)
+        if not groups:
+            logger.warning("No groups found in %s; nothing to average.", groups_csv.resolve())
+            return None
+
+        if settings.verbose:
+            logger.info("Grouping %s using %s (%s)", merged_csv, groups_csv.resolve(), settings.group_agg_method)
+
+        averager = SigSpectraAverager.from_csv(merged_csv)
+        grouped_df = averager.aggregate(groups, method=settings.group_agg_method)
+
+        output_path = settings.resampled_dir / settings.grouped_csv_name
+        output_path.parent.mkdir(parents=True, exist_ok=True)
+        grouped_df.to_csv(output_path, index=False)
+        logger.info("Grouped/averaged spectra written to %s", output_path.resolve())
+        return output_path
+
     # ── helpers ──────────────────────────────────────────────────────────────
 
     @staticmethod

{svc_processing-0.1.1 → svc_processing-0.1.3}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "svc-processing"
-version = "0.1.1"
+version = "0.1.3"
 description = "Pure-Python SVC HR-1024i hyperspectral .sig processing pipeline"
 readme = "README.md"
 requires-python = ">=3.11"

{svc_processing-0.1.1 → svc_processing-0.1.3}/svc_processing.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: svc-processing
-Version: 0.1.1
+Version: 0.1.3
 Summary: Pure-Python SVC HR-1024i hyperspectral .sig processing pipeline
 Author: Gold Lab (GrapeSPEC project), Cornell University
 Author-email: Cole Regnier <nr466@cornell.edu>
@@ -65,6 +65,20 @@ Generated outputs (gitignored): `pipeline_outputs/sig_processed/<run>/`, `pipeli
 > New to the pipeline? [`TUTORIAL.md`](TUTORIAL.md) starts with the interactive
 > Jupyter notebook (the gentle path) and has an advanced track for the CLI below.
 
+**Installed from PyPI, in a new project folder** (no clone of this repo needed):
+
+```bash
+python3.11 -m venv .venv
+source .venv/bin/activate
+python -m pip install svc-processing   # package name
+
+svc-pipeline --init-config   # command name — different from the package name above
+# Edit config/config.json — replace "<PATH_TO_SIG_INPUT_ROOT>" with your data path.
+svc-pipeline
+```
+
+**Working from a clone of this repo** (config/config.json already exists):
+
 ```bash
 python3.11 -m venv .venv
 source .venv/bin/activate
@@ -112,8 +126,10 @@ Outputs land under `pipeline_outputs/` by default. See [`config/README.md`](conf
    └────────────────────────────────────┘
 ```
 
-The `svc-pipeline` console script glues Stages 1 and 2 together through
-`pipeline.cli`; Stage 3 is invoked from notebooks against the Stage 2 output.
+The `svc-pipeline` console script runs Stages 1 and 2 through `pipeline.cli`,
+and Stage 3 too once a `groups_csv` is configured (or passed via
+`--groups-csv`) — see [`config/README.md`](config/README.md#grouping--stage-3-optional).
+Stage 3 is also available directly from notebooks against the Stage 2 output.
 The pipeline is **single-pass and idempotent per input directory** — previous
 processed `.sig` files in the target directory are deleted at the start of each
 run.
@@ -132,7 +148,9 @@ svc-pipeline [config] [options]
 |---|---|
 | `config` | Run-config JSON (positional, optional; default `config/config.json`). Bare names resolve under `config/`, so `config.json`, `config`, and `config/config.json` are equivalent. See [`config/README.md`](config/README.md) for schema. |
 | `--input-dir <path>` | Override `sig_input_dir` and process only this directory. |
-| `--step {1,2,all}` | `1` = process + summary CSV only; `2` = resample only (requires Stage 1 to have been run); `all` = both (default). |
+| `--step {1,2,3,all}` | `1` = process + summary CSV only; `2` = resample only (requires Stage 1); `3` = group + average only (requires Stage 2 and `groups_csv`); `all` = every stage that's configured (default). |
+| `--groups-csv <path>` | Override the config's `groups_csv` to group repeat scans and average them. |
+| `--group-method {mean,median,sum,min,max}` | Override the config's `group_agg_method` for Stage 3 (default `mean`). |
 | `--verbose` | Print INFO/DEBUG logs before and after each stage. |
 
 ### Expected output layout
@@ -143,10 +161,11 @@ pipeline_outputs/
 │   ├── <truncated *.sig files>
 │   └── <input_dir_name>_processed_sig_summary.csv
 └── sig_resampled/<input_dir_name>/
-    └── <input_dir_name>_merged_spectra.csv     # 2101 columns (400–2500 nm)
+    ├── <input_dir_name>_merged_spectra.csv     # 2101 columns (400–2500 nm)
+    └── <input_dir_name>_grouped_spectra.csv    # only if groups_csv is configured
 ```
 
-`summary_csv_name` and `merged_csv_name` in the run config are suffixes; the real filenames are prefixed with the input directory name. Verify your config in [`config/README.md`](config/README.md).
+`summary_csv_name`, `merged_csv_name`, and `grouped_csv_name` in the run config are suffixes; the real filenames are prefixed with the input directory name. Verify your config in [`config/README.md`](config/README.md).
 
 ---
 

svc_processing-0.1.3/tests/test_cli.py

@@ -0,0 +1,184 @@
+from __future__ import annotations
+
+import json
+import sys
+from pathlib import Path
+
+import pytest
+
+from pipeline.cli import _parse_args, main
+
+
+def _write_config(path: Path, data: dict) -> Path:
+    path.write_text(json.dumps(data))
+    return path
+
+
+def test_help_links_to_github_repo(
+    monkeypatch: pytest.MonkeyPatch,
+    capsys: pytest.CaptureFixture[str],
+) -> None:
+    monkeypatch.setattr(sys, "argv", ["svc-pipeline", "--help"])
+
+    with pytest.raises(SystemExit):
+        _parse_args()
+
+    assert "github.com/regs08/SVCProcessingPipeline" in capsys.readouterr().out
+
+
+def test_parse_args_step_and_input_dir_default_to_all_and_none(monkeypatch: pytest.MonkeyPatch) -> None:
+    monkeypatch.setattr(sys, "argv", ["svc-pipeline", "config.json"])
+
+    args = _parse_args()
+
+    assert args.config == "config.json"
+    assert args.step == "all"
+    assert args.input_dir is None
+    assert args.verbose is False
+
+
+def test_parse_args_accepts_step_input_dir_and_verbose(monkeypatch: pytest.MonkeyPatch) -> None:
+    monkeypatch.setattr(
+        sys,
+        "argv",
+        ["svc-pipeline", "myconfig", "--step", "2", "--input-dir", "/some/dir", "--verbose"],
+    )
+
+    args = _parse_args()
+
+    assert args.step == "2"
+    assert args.input_dir == "/some/dir"
+    assert args.verbose is True
+
+
+def test_parse_args_accepts_step_three_and_group_overrides(monkeypatch: pytest.MonkeyPatch) -> None:
+    monkeypatch.setattr(
+        sys,
+        "argv",
+        ["svc-pipeline", "--step", "3", "--groups-csv", "naming_ids/groups.csv", "--group-method", "median"],
+    )
+
+    args = _parse_args()
+
+    assert args.step == "3"
+    assert args.groups_csv == "naming_ids/groups.csv"
+    assert args.group_method == "median"
+
+
+def test_main_input_dir_override_bypasses_config_input_and_placeholder(
+    tmp_path: Path,
+    monkeypatch: pytest.MonkeyPatch,
+    capsys: pytest.CaptureFixture[str],
+) -> None:
+    # sig_input_dir is left as the unedited placeholder; --input-dir must
+    # still work without tripping the placeholder guard.
+    config_path = _write_config(
+        tmp_path / "config.json",
+        {
+            "sig_input_dir": "<PATH_TO_SIG_INPUT_ROOT>",
+            "processed_dir": "sig_processed",
+            "resampled_dir": "sig_resampled",
+            "output_dir": str(tmp_path / "out"),
+            "summary_csv_name": "summary.csv",
+            "merged_csv_name": "merged.csv",
+        },
+    )
+    empty_input = tmp_path / "empty_input"
+    empty_input.mkdir()
+
+    monkeypatch.setattr(
+        sys,
+        "argv",
+        ["svc-pipeline", str(config_path), "--input-dir", str(empty_input), "--step", "1"],
+    )
+
+    main()
+
+    captured = capsys.readouterr()
+    assert str(empty_input.resolve()) in captured.out
+    assert "not produced" in captured.out
+
+
+def test_main_step_three_groups_existing_merged_csv(
+    tmp_path: Path,
+    monkeypatch: pytest.MonkeyPatch,
+    capsys: pytest.CaptureFixture[str],
+) -> None:
+    groups_csv = tmp_path / "groups.csv"
+    groups_csv.write_text("scan_id,name\n1,leaf_a\n2,leaf_a\n")
+
+    config_path = _write_config(
+        tmp_path / "config.json",
+        {
+            "sig_input_dir": "<PATH_TO_SIG_INPUT_ROOT>",
+            "processed_dir": "sig_processed",
+            "resampled_dir": "sig_resampled",
+            "output_dir": str(tmp_path / "out"),
+            "summary_csv_name": "summary.csv",
+            "merged_csv_name": "merged.csv",
+            "groups_csv": str(groups_csv),
+        },
+    )
+    input_dir = tmp_path / "myinput"
+    input_dir.mkdir()
+
+    resampled_dir = tmp_path / "out" / "sig_resampled" / "myinput"
+    resampled_dir.mkdir(parents=True)
+    merged_csv = resampled_dir / "myinput_merged.csv"
+    merged_csv.write_text("sample_name,400\nleaf.0001,1.0\nleaf.0002,3.0\n")
+
+    monkeypatch.setattr(
+        sys,
+        "argv",
+        ["svc-pipeline", str(config_path), "--input-dir", str(input_dir), "--step", "3"],
+    )
+
+    main()
+
+    grouped_csv = resampled_dir / "myinput_grouped_spectra.csv"
+    assert grouped_csv.exists()
+    captured = capsys.readouterr()
+    assert "grouped_csv" in captured.out
+    assert str(grouped_csv.resolve()) in captured.out
+
+
+def test_main_init_config_writes_starter_and_exits(
+    tmp_path: Path,
+    monkeypatch: pytest.MonkeyPatch,
+    capsys: pytest.CaptureFixture[str],
+) -> None:
+    monkeypatch.chdir(tmp_path)
+    monkeypatch.setattr(sys, "argv", ["svc-pipeline", "--init-config"])
+
+    main()
+
+    written = tmp_path / "config" / "config.json"
+    assert written.is_file()
+    assert json.loads(written.read_text())["sig_input_dir"] == "<PATH_TO_SIG_INPUT_ROOT>"
+    assert str(written.resolve()) in capsys.readouterr().out
+
+
+def test_main_init_config_refuses_to_overwrite_existing(
+    tmp_path: Path,
+    monkeypatch: pytest.MonkeyPatch,
+) -> None:
+    monkeypatch.chdir(tmp_path)
+    existing = tmp_path / "config" / "config.json"
+    existing.parent.mkdir()
+    existing.write_text('{"already": "here"}')
+    monkeypatch.setattr(sys, "argv", ["svc-pipeline", "--init-config"])
+
+    with pytest.raises(SystemExit, match="already exists"):
+        main()
+
+
+def test_main_exits_on_placeholder_without_input_dir_override(
+    tmp_path: Path,
+    monkeypatch: pytest.MonkeyPatch,
+) -> None:
+    config_path = _write_config(tmp_path / "config.json", {"sig_input_dir": "<PATH_TO_SIG_INPUT_ROOT>"})
+
+    monkeypatch.setattr(sys, "argv", ["svc-pipeline", str(config_path)])
+
+    with pytest.raises(SystemExit, match="placeholder"):
+        main()

{svc_processing-0.1.1 → svc_processing-0.1.3}/tests/test_run_config.py

@@ -1,5 +1,6 @@
 from __future__ import annotations
 
+import json
 import logging
 from pathlib import Path
 
@@ -21,6 +22,26 @@ def test_resolve_path_finds_bare_config_name(tmp_path: Path) -> None:
     assert RunConfig._resolve_path(tmp_path, "demo") == config_path
 
 
+def test_write_starter_config_creates_config_json_with_placeholder(tmp_path: Path) -> None:
+    target = RunConfig.write_starter_config(tmp_path)
+
+    assert target == tmp_path / "config" / "config.json"
+    data = json.loads(target.read_text())
+    assert data["sig_input_dir"] == "<PATH_TO_SIG_INPUT_ROOT>"
+    assert data["processing"]["band_min"] == 400
+
+
+def test_write_starter_config_refuses_to_overwrite_existing(tmp_path: Path) -> None:
+    existing = tmp_path / "config" / "config.json"
+    existing.parent.mkdir()
+    existing.write_text('{"already": "here"}')
+
+    with pytest.raises(SystemExit, match="already exists"):
+        RunConfig.write_starter_config(tmp_path)
+
+    assert json.loads(existing.read_text()) == {"already": "here"}
+
+
 def test_placeholder_guard_exits_with_guidance(tmp_path: Path) -> None:
     config_path = tmp_path / "config.json"
     config = RunConfig(
@@ -103,3 +124,58 @@ def test_settings_for_resolves_output_paths(tmp_path: Path) -> None:
     assert settings.merged_csv_name == "example_input_merged_spectra.csv"
     assert settings.correction_types["silver"] == "2517.9"
     assert settings.instrument_numbers["silver"] == "1202103"
+
+
+def test_settings_for_grouping_defaults_when_unconfigured(tmp_path: Path) -> None:
+    data = {
+        "processed_dir": "sig_processed",
+        "resampled_dir": "sig_resampled",
+        "summary_csv_name": "summary.csv",
+        "merged_csv_name": "merged.csv",
+    }
+    config = RunConfig(data, path=tmp_path / "config.json", base_dir=tmp_path, logger=_logger())
+
+    settings = config.settings_for(Path("example_input"), verbose=False)
+
+    assert settings.groups_csv is None
+    assert settings.group_agg_method == "mean"
+    assert settings.grouped_csv_name == "example_input_grouped_spectra.csv"
+
+
+def test_settings_for_resolves_groups_csv_from_config(tmp_path: Path) -> None:
+    data = {
+        "processed_dir": "sig_processed",
+        "resampled_dir": "sig_resampled",
+        "summary_csv_name": "summary.csv",
+        "merged_csv_name": "merged.csv",
+        "groups_csv": "naming_ids/groups.csv",
+        "group_agg_method": "median",
+    }
+    config = RunConfig(data, path=tmp_path / "config.json", base_dir=tmp_path, logger=_logger())
+
+    settings = config.settings_for(Path("example_input"), verbose=False)
+
+    assert settings.groups_csv == tmp_path / "naming_ids/groups.csv"
+    assert settings.group_agg_method == "median"
+
+
+def test_settings_for_cli_overrides_take_priority_over_config(tmp_path: Path) -> None:
+    data = {
+        "processed_dir": "sig_processed",
+        "resampled_dir": "sig_resampled",
+        "summary_csv_name": "summary.csv",
+        "merged_csv_name": "merged.csv",
+        "groups_csv": "naming_ids/groups.csv",
+        "group_agg_method": "mean",
+    }
+    config = RunConfig(data, path=tmp_path / "config.json", base_dir=tmp_path, logger=_logger())
+
+    settings = config.settings_for(
+        Path("example_input"),
+        verbose=False,
+        groups_csv_override=tmp_path / "override_groups.csv",
+        group_method_override="max",
+    )
+
+    assert settings.groups_csv == tmp_path / "override_groups.csv"
+    assert settings.group_agg_method == "max"

svc_processing-0.1.3/tests/test_runner.py

@@ -0,0 +1,136 @@
+from __future__ import annotations
+
+import logging
+from pathlib import Path
+
+from pipeline.run_config import PipelineSettings
+from pipeline.runner import Pipeline
+
+
+def _settings(
+    tmp_path: Path,
+    input_dir: Path,
+    *,
+    groups_csv: Path | None = None,
+    group_agg_method: str = "mean",
+) -> PipelineSettings:
+    return PipelineSettings(
+        source_name=input_dir.name or "input",
+        input_dir=input_dir,
+        processed_dir=tmp_path / "processed",
+        resampled_dir=tmp_path / "resampled",
+        summary_csv=tmp_path / "processed" / "summary.csv",
+        merged_csv_name="merged.csv",
+        end_line_overrides={},
+        verbose=False,
+        processing_params={
+            "band_min": 400,
+            "band_max": 410,
+            "resample_fwhm_nm": 10.0,
+            "splice_interp_wvl": (5.0, 2.0),
+            "fixed_sensor": 2,
+        },
+        correction_types={"bronze": "2520.4", "silver": "2517.9"},
+        instrument_numbers={"bronze": "2212118", "silver": "1202103"},
+        groups_csv=groups_csv,
+        group_agg_method=group_agg_method,
+        grouped_csv_name="grouped.csv",
+    )
+
+
+def test_process_sig_files_returns_none_for_missing_input(tmp_path: Path) -> None:
+    pipeline = Pipeline(_settings(tmp_path, tmp_path / "missing"), logging.getLogger("tests.runner"))
+
+    assert pipeline.process_sig_files() is None
+
+
+def test_process_sig_files_returns_none_for_empty_input(tmp_path: Path) -> None:
+    input_dir = tmp_path / "empty"
+    input_dir.mkdir()
+    pipeline = Pipeline(_settings(tmp_path, input_dir), logging.getLogger("tests.runner"))
+
+    assert pipeline.process_sig_files() is None
+
+
+def test_run_step_two_without_summary_does_not_resample(tmp_path: Path) -> None:
+    input_dir = tmp_path / "input"
+    input_dir.mkdir()
+    pipeline = Pipeline(_settings(tmp_path, input_dir), logging.getLogger("tests.runner"))
+
+    result = pipeline.run("2")
+
+    assert result == {"summary_csv": None, "merged_csv": None, "grouped_csv": None}
+
+
+def test_run_all_skips_grouping_when_groups_csv_not_configured(tmp_path: Path) -> None:
+    input_dir = tmp_path / "input"
+    input_dir.mkdir()
+    pipeline = Pipeline(_settings(tmp_path, input_dir), logging.getLogger("tests.runner"))
+
+    result = pipeline.run("all")
+
+    assert result["grouped_csv"] is None
+
+
+def test_group_and_average_writes_grouped_csv(tmp_path: Path) -> None:
+    resampled_dir = tmp_path / "resampled"
+    resampled_dir.mkdir(parents=True)
+    merged_csv = resampled_dir / "merged.csv"
+    merged_csv.write_text("sample_name,400,401\nleaf.0001,1.0,2.0\nleaf.0002,3.0,4.0\n")
+
+    groups_csv = tmp_path / "groups.csv"
+    groups_csv.write_text("scan_id,name\n1,leaf_a\n2,leaf_a\n")
+
+    settings = _settings(tmp_path, tmp_path / "input", groups_csv=groups_csv)
+    pipeline = Pipeline(settings, logging.getLogger("tests.runner"))
+
+    output = pipeline.group_and_average(merged_csv)
+
+    assert output == resampled_dir / "grouped.csv"
+    written = output.read_text()
+    assert "leaf_a" in written
+    assert "2.0" in written  # mean(1.0, 3.0)
+    assert "3.0" in written  # mean(2.0, 4.0)
+
+
+def test_group_and_average_missing_groups_file_returns_none(tmp_path: Path) -> None:
+    resampled_dir = tmp_path / "resampled"
+    resampled_dir.mkdir(parents=True)
+    merged_csv = resampled_dir / "merged.csv"
+    merged_csv.write_text("sample_name,400\nleaf.0001,1.0\n")
+
+    settings = _settings(tmp_path, tmp_path / "input", groups_csv=tmp_path / "missing_groups.csv")
+    pipeline = Pipeline(settings, logging.getLogger("tests.runner"))
+
+    assert pipeline.group_and_average(merged_csv) is None
+
+
+def test_run_step_three_without_merged_csv_logs_error(tmp_path: Path) -> None:
+    input_dir = tmp_path / "input"
+    input_dir.mkdir()
+    groups_csv = tmp_path / "groups.csv"
+    groups_csv.write_text("scan_id,name\n1,leaf_a\n")
+    pipeline = Pipeline(_settings(tmp_path, input_dir, groups_csv=groups_csv), logging.getLogger("tests.runner"))
+
+    result = pipeline.run("3")
+
+    assert result == {"summary_csv": None, "merged_csv": None, "grouped_csv": None}
+
+
+def test_run_step_three_groups_existing_merged_csv(tmp_path: Path) -> None:
+    input_dir = tmp_path / "input"
+    input_dir.mkdir()
+    resampled_dir = tmp_path / "resampled"
+    resampled_dir.mkdir(parents=True)
+    merged_csv = resampled_dir / "merged.csv"
+    merged_csv.write_text("sample_name,400\nleaf.0001,1.0\nleaf.0002,3.0\n")
+    groups_csv = tmp_path / "groups.csv"
+    groups_csv.write_text("scan_id,name\n1,leaf_a\n2,leaf_a\n")
+
+    pipeline = Pipeline(_settings(tmp_path, input_dir, groups_csv=groups_csv), logging.getLogger("tests.runner"))
+
+    result = pipeline.run("3")
+
+    assert result["merged_csv"] == merged_csv
+    assert result["grouped_csv"] == resampled_dir / "grouped.csv"
+    assert result["grouped_csv"].exists()

svc_processing-0.1.1/tests/test_cli.py

@@ -1,85 +0,0 @@
-from __future__ import annotations
-
-import json
-import sys
-from pathlib import Path
-
-import pytest
-
-from pipeline.cli import _parse_args, main
-
-
-def _write_config(path: Path, data: dict) -> Path:
-    path.write_text(json.dumps(data))
-    return path
-
-
-def test_parse_args_step_and_input_dir_default_to_all_and_none(monkeypatch: pytest.MonkeyPatch) -> None:
-    monkeypatch.setattr(sys, "argv", ["svc-pipeline", "config.json"])
-
-    args = _parse_args()
-
-    assert args.config == "config.json"
-    assert args.step == "all"
-    assert args.input_dir is None
-    assert args.verbose is False
-
-
-def test_parse_args_accepts_step_input_dir_and_verbose(monkeypatch: pytest.MonkeyPatch) -> None:
-    monkeypatch.setattr(
-        sys,
-        "argv",
-        ["svc-pipeline", "myconfig", "--step", "2", "--input-dir", "/some/dir", "--verbose"],
-    )
-
-    args = _parse_args()
-
-    assert args.step == "2"
-    assert args.input_dir == "/some/dir"
-    assert args.verbose is True
-
-
-def test_main_input_dir_override_bypasses_config_input_and_placeholder(
-    tmp_path: Path,
-    monkeypatch: pytest.MonkeyPatch,
-    capsys: pytest.CaptureFixture[str],
-) -> None:
-    # sig_input_dir is left as the unedited placeholder; --input-dir must
-    # still work without tripping the placeholder guard.
-    config_path = _write_config(
-        tmp_path / "config.json",
-        {
-            "sig_input_dir": "<PATH_TO_SIG_INPUT_ROOT>",
-            "processed_dir": "sig_processed",
-            "resampled_dir": "sig_resampled",
-            "output_dir": str(tmp_path / "out"),
-            "summary_csv_name": "summary.csv",
-            "merged_csv_name": "merged.csv",
-        },
-    )
-    empty_input = tmp_path / "empty_input"
-    empty_input.mkdir()
-
-    monkeypatch.setattr(
-        sys,
-        "argv",
-        ["svc-pipeline", str(config_path), "--input-dir", str(empty_input), "--step", "1"],
-    )
-
-    main()
-
-    captured = capsys.readouterr()
-    assert str(empty_input.resolve()) in captured.out
-    assert "not produced" in captured.out
-
-
-def test_main_exits_on_placeholder_without_input_dir_override(
-    tmp_path: Path,
-    monkeypatch: pytest.MonkeyPatch,
-) -> None:
-    config_path = _write_config(tmp_path / "config.json", {"sig_input_dir": "<PATH_TO_SIG_INPUT_ROOT>"})
-
-    monkeypatch.setattr(sys, "argv", ["svc-pipeline", str(config_path)])
-
-    with pytest.raises(SystemExit, match="placeholder"):
-        main()

svc_processing-0.1.1/tests/test_runner.py

@@ -1,53 +0,0 @@
-from __future__ import annotations
-
-import logging
-from pathlib import Path
-
-from pipeline.run_config import PipelineSettings
-from pipeline.runner import Pipeline
-
-
-def _settings(tmp_path: Path, input_dir: Path) -> PipelineSettings:
-    return PipelineSettings(
-        source_name=input_dir.name or "input",
-        input_dir=input_dir,
-        processed_dir=tmp_path / "processed",
-        resampled_dir=tmp_path / "resampled",
-        summary_csv=tmp_path / "processed" / "summary.csv",
-        merged_csv_name="merged.csv",
-        end_line_overrides={},
-        verbose=False,
-        processing_params={
-            "band_min": 400,
-            "band_max": 410,
-            "resample_fwhm_nm": 10.0,
-            "splice_interp_wvl": (5.0, 2.0),
-            "fixed_sensor": 2,
-        },
-        correction_types={"bronze": "2520.4", "silver": "2517.9"},
-        instrument_numbers={"bronze": "2212118", "silver": "1202103"},
-    )
-
-
-def test_process_sig_files_returns_none_for_missing_input(tmp_path: Path) -> None:
-    pipeline = Pipeline(_settings(tmp_path, tmp_path / "missing"), logging.getLogger("tests.runner"))
-
-    assert pipeline.process_sig_files() is None
-
-
-def test_process_sig_files_returns_none_for_empty_input(tmp_path: Path) -> None:
-    input_dir = tmp_path / "empty"
-    input_dir.mkdir()
-    pipeline = Pipeline(_settings(tmp_path, input_dir), logging.getLogger("tests.runner"))
-
-    assert pipeline.process_sig_files() is None
-
-
-def test_run_step_two_without_summary_does_not_resample(tmp_path: Path) -> None:
-    input_dir = tmp_path / "input"
-    input_dir.mkdir()
-    pipeline = Pipeline(_settings(tmp_path, input_dir), logging.getLogger("tests.runner"))
-
-    result = pipeline.run("2")
-
-    assert result == {"summary_csv": None, "merged_csv": None}