pytesprocess 0.1.1__tar.gz

pytesprocess-0.1.1/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2022 Samuel Watkins
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

pytesprocess-0.1.1/PKG-INFO

@@ -0,0 +1,211 @@
+Metadata-Version: 2.4
+Name: pytesprocess
+Version: 0.1.1
+Summary: Detector data processing for pytesdaqx acquisitions
+Author-email: Bruno Serfass <serfass@berkeley.edu>, Samuel Watkins <samwatkins@berkeley.edu>
+License: MIT License
+        
+        Copyright (c) 2022 Samuel Watkins
+        
+        Permission is hereby granted, free of charge, to any person obtaining a copy
+        of this software and associated documentation files (the "Software"), to deal
+        in the Software without restriction, including without limitation the rights
+        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+        copies of the Software, and to permit persons to whom the Software is
+        furnished to do so, subject to the following conditions:
+        
+        The above copyright notice and this permission notice shall be included in all
+        copies or substantial portions of the Software.
+        
+        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+        SOFTWARE.
+        
+Project-URL: Homepage, https://github.com/spice-herald/pytesprocess
+Project-URL: Repository, https://github.com/spice-herald/pytesprocess
+Keywords: detector,TES,DAQ,signal processing
+Requires-Python: >=3.11
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: numpy>=1.26
+Requires-Dist: scipy
+Requires-Dist: matplotlib
+Requires-Dist: PyYAML
+Requires-Dist: qetpy>=1.8.6
+Requires-Dist: pandas
+Requires-Dist: pytesdaqx
+Requires-Dist: humanfriendly
+Requires-Dist: vaex
+Requires-Dist: pyarrow
+Requires-Dist: lmfit
+Requires-Dist: cloudpickle
+Provides-Extra: dev
+Requires-Dist: build; extra == "dev"
+Requires-Dist: pytest; extra == "dev"
+Dynamic: license-file
+
+# pytesprocess
+
+`pytesprocess` is the detector-processing layer used with raw acquisitions written
+by `pytesdaqx`. It provides random-event selection, software triggering,
+feature extraction, salting, IV/dIdV processing, noise/filter generation, and
+persistence of processed event products as Vaex HDF5 dataframes.
+
+The current code targets Python 3.11+ and the modern `pytesdaqx` acquisition
+model (`acquisition` + `stream`).
+
+## Installation
+
+For a development checkout:
+
+```bash
+pip install -e .
+```
+
+Runtime dependencies and the installed CLI entry point are defined entirely in
+`pyproject.toml`.
+
+## Command-line interface
+
+Installing the package provides:
+
+```bash
+pytesprocess --help
+```
+
+The normal event-processing workflow is selected directly from the top-level
+command:
+
+```bash
+pytesprocess ACQUISITION \
+    --config processing.yaml \
+    --steps randoms trigger feature
+```
+
+`--steps` is mandatory. It accepts either whitespace-separated or
+comma-separated values, including mixed input:
+
+```bash
+--steps randoms trigger feature
+--steps randoms,trigger,feature
+--steps randoms,trigger feature
+```
+
+All are equivalent. The execution order is normalized internally to:
+
+```text
+randoms -> salting -> trigger -> feature
+```
+
+The separate detector-characterization/filter workflows remain explicit:
+
+```bash
+pytesprocess filter ACQUISITION --config processing.yaml
+pytesprocess ivsweep ACQUISITION
+```
+
+Example:
+
+```bash
+pytesprocess acquisition_I2_D20260807_T170432.zarr \
+    --config continuous_data_processing_v2.yaml \
+    --steps randoms,trigger,feature \
+    --nrandoms 300 \
+    --nevents 300 \
+    --ncores 4
+```
+
+See [docs/user/cli.md](docs/user/cli.md) for the current CLI behavior.
+
+## Configuration
+
+New processing configuration uses `config_version: 2` and separates workflow
+sections explicitly:
+
+```yaml
+config_version: 2
+
+resources:
+  filter_file: /path/to/filterdata.hdf5
+
+trigger:
+  global: {}
+  channels: {}
+
+feature:
+  global:
+    trace_length_msec: 20
+    pretrigger_length_msec: 10
+  presets: {}
+  channels: {}
+
+salting:
+  global: {}
+  channels: {}
+
+filter: {}
+```
+
+Feature trace lengths are specified in milliseconds in v2 configuration. The
+resolver converts them to samples using the selected acquisition sample rate.
+Channel selectors include `all`, shell-style globs such as `Z1P*`, comma groups
+for applying one block to several independent channels, and explicit
+multi-channel expressions such as `A|B`.
+
+See [docs/user/configuration.md](docs/user/configuration.md).
+
+## Processed dataframe identity
+
+Each Vaex processing product has one dataframe-group identity:
+
+```text
+dataframe_group_name
+dataframe_group_id
+dataframe_group_number
+dataframe_file_index
+processing_label
+```
+
+For example:
+
+```text
+trigger_I2_D20260908_T123456/
+  trigger_I2_D20260908_T123456_F0001.hdf5
+  trigger_I2_D20260908_T123456_F0002.hdf5
+```
+
+`F####` is a processing task/output shard index, not a second stream or series
+identifier. Raw provenance uses canonical stream fields such as `stream_id`,
+`stream_number`, and `stream_trigger_index`.
+
+See [docs/user/outputs.md](docs/user/outputs.md).
+
+## Package structure
+
+The current package layout intentionally separates reusable analysis objects
+from acquisition-processing workflows:
+
+```text
+pytesprocess/
+  core/       reusable analysis/data objects and algorithms
+  process/    raw/dataframe processing executors
+  config/     YAML loading, selection, resolution, validation
+  salting/    salt metadata generation and waveform injection
+  workflows/  multi-step orchestration used by the CLI
+  cli/        command-line parsing and dispatch
+  utils/      shared utilities and HDF5/dataframe helpers
+```
+
+See [docs/developer/architecture.md](docs/developer/architecture.md) for the
+current developer-oriented architecture.
+
+## Notes on multiprocessing
+
+The existing Vaex/PyArrow and numerical-library thread limits are intentional.
+They were introduced to avoid thread oversubscription and unstable/slower
+multicore processing. They should not be removed or relocated without dedicated
+multicore benchmarking.

pytesprocess-0.1.1/README.md

@@ -0,0 +1,161 @@
+# pytesprocess
+
+`pytesprocess` is the detector-processing layer used with raw acquisitions written
+by `pytesdaqx`. It provides random-event selection, software triggering,
+feature extraction, salting, IV/dIdV processing, noise/filter generation, and
+persistence of processed event products as Vaex HDF5 dataframes.
+
+The current code targets Python 3.11+ and the modern `pytesdaqx` acquisition
+model (`acquisition` + `stream`).
+
+## Installation
+
+For a development checkout:
+
+```bash
+pip install -e .
+```
+
+Runtime dependencies and the installed CLI entry point are defined entirely in
+`pyproject.toml`.
+
+## Command-line interface
+
+Installing the package provides:
+
+```bash
+pytesprocess --help
+```
+
+The normal event-processing workflow is selected directly from the top-level
+command:
+
+```bash
+pytesprocess ACQUISITION \
+    --config processing.yaml \
+    --steps randoms trigger feature
+```
+
+`--steps` is mandatory. It accepts either whitespace-separated or
+comma-separated values, including mixed input:
+
+```bash
+--steps randoms trigger feature
+--steps randoms,trigger,feature
+--steps randoms,trigger feature
+```
+
+All are equivalent. The execution order is normalized internally to:
+
+```text
+randoms -> salting -> trigger -> feature
+```
+
+The separate detector-characterization/filter workflows remain explicit:
+
+```bash
+pytesprocess filter ACQUISITION --config processing.yaml
+pytesprocess ivsweep ACQUISITION
+```
+
+Example:
+
+```bash
+pytesprocess acquisition_I2_D20260807_T170432.zarr \
+    --config continuous_data_processing_v2.yaml \
+    --steps randoms,trigger,feature \
+    --nrandoms 300 \
+    --nevents 300 \
+    --ncores 4
+```
+
+See [docs/user/cli.md](docs/user/cli.md) for the current CLI behavior.
+
+## Configuration
+
+New processing configuration uses `config_version: 2` and separates workflow
+sections explicitly:
+
+```yaml
+config_version: 2
+
+resources:
+  filter_file: /path/to/filterdata.hdf5
+
+trigger:
+  global: {}
+  channels: {}
+
+feature:
+  global:
+    trace_length_msec: 20
+    pretrigger_length_msec: 10
+  presets: {}
+  channels: {}
+
+salting:
+  global: {}
+  channels: {}
+
+filter: {}
+```
+
+Feature trace lengths are specified in milliseconds in v2 configuration. The
+resolver converts them to samples using the selected acquisition sample rate.
+Channel selectors include `all`, shell-style globs such as `Z1P*`, comma groups
+for applying one block to several independent channels, and explicit
+multi-channel expressions such as `A|B`.
+
+See [docs/user/configuration.md](docs/user/configuration.md).
+
+## Processed dataframe identity
+
+Each Vaex processing product has one dataframe-group identity:
+
+```text
+dataframe_group_name
+dataframe_group_id
+dataframe_group_number
+dataframe_file_index
+processing_label
+```
+
+For example:
+
+```text
+trigger_I2_D20260908_T123456/
+  trigger_I2_D20260908_T123456_F0001.hdf5
+  trigger_I2_D20260908_T123456_F0002.hdf5
+```
+
+`F####` is a processing task/output shard index, not a second stream or series
+identifier. Raw provenance uses canonical stream fields such as `stream_id`,
+`stream_number`, and `stream_trigger_index`.
+
+See [docs/user/outputs.md](docs/user/outputs.md).
+
+## Package structure
+
+The current package layout intentionally separates reusable analysis objects
+from acquisition-processing workflows:
+
+```text
+pytesprocess/
+  core/       reusable analysis/data objects and algorithms
+  process/    raw/dataframe processing executors
+  config/     YAML loading, selection, resolution, validation
+  salting/    salt metadata generation and waveform injection
+  workflows/  multi-step orchestration used by the CLI
+  cli/        command-line parsing and dispatch
+  utils/      shared utilities and HDF5/dataframe helpers
+```
+
+See [docs/developer/architecture.md](docs/developer/architecture.md) for the
+current developer-oriented architecture.
+
+## Notes on multiprocessing
+
+The existing Vaex/PyArrow and numerical-library thread limits are intentional.
+They were introduced to avoid thread oversubscription and unstable/slower
+multicore processing. They should not be removed or relocated without dedicated
+multicore benchmarking.

pytesprocess-0.1.1/pyproject.toml

@@ -0,0 +1,49 @@
+[build-system]
+requires = ["setuptools>=68", "wheel"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "pytesprocess"
+dynamic = ["version"]
+description = "Detector data processing for pytesdaqx acquisitions"
+readme = "README.md"
+requires-python = ">=3.11"
+license = {file = "LICENSE"}
+authors = [
+  {name = "Bruno Serfass", email = "serfass@berkeley.edu"},
+  {name = "Samuel Watkins", email = "samwatkins@berkeley.edu"},
+]
+keywords = ["detector", "TES", "DAQ", "signal processing"]
+dependencies = [
+  "numpy>=1.26",
+  "scipy",
+  "matplotlib",
+  "PyYAML",
+  "qetpy>=1.8.6",
+  "pandas",
+  "pytesdaqx",
+  "humanfriendly",
+  "vaex",
+  "pyarrow",
+  "lmfit",
+  "cloudpickle",
+]
+
+[project.optional-dependencies]
+dev = [
+  "build",
+  "pytest",
+]
+
+[project.urls]
+Homepage = "https://github.com/spice-herald/pytesprocess"
+Repository = "https://github.com/spice-herald/pytesprocess"
+
+[project.scripts]
+pytesprocess = "pytesprocess.cli.main:main"
+
+[tool.setuptools.dynamic]
+version = {attr = "pytesprocess._version.__version__"}
+
+[tool.setuptools.packages.find]
+include = ["pytesprocess*"]

pytesprocess-0.1.1/pytesprocess/__init__.py

@@ -0,0 +1,9 @@
+from . import process
+from .process import *
+from . import core
+from .core import *
+from .salting import *
+from .utils import *
+from ._version import __version__
+
+from .config import *

pytesprocess-0.1.1/pytesprocess/_version.py

@@ -0,0 +1,2 @@
+# special file for defining the current version of the package
+__version__ = "0.1.1"

pytesprocess-0.1.1/pytesprocess/cli/__init__.py

@@ -0,0 +1 @@
+from .main import main

pytesprocess-0.1.1/pytesprocess/cli/commands/__init__.py

@@ -0,0 +1,5 @@
+"""Command-specific parser/run helpers for the pytesprocess CLI."""
+
+from . import event, filter, ivsweep
+
+__all__ = ["event", "filter", "ivsweep"]

pytesprocess-0.1.1/pytesprocess/cli/commands/event.py

@@ -0,0 +1,66 @@
+"""Default event-processing CLI arguments."""
+
+from pytesprocess.workflows.processing import run_processing_workflow
+
+_PROCESS_STEPS = ("randoms", "salting", "trigger", "feature")
+
+
+def configure(parser):
+    parser.add_argument(
+        "--steps",
+        nargs="+",
+        required=True,
+        metavar="STEP[,STEP...]",
+        help=(
+            "Processing steps to run. Values may be space-separated, "
+            "comma-separated, or mixed (for example '--steps randoms trigger "
+            "feature' or '--steps randoms,trigger,feature'). User order is "
+            "ignored; execution order is always randoms -> salting -> trigger "
+            "-> feature. Valid steps: " + ", ".join(_PROCESS_STEPS) + "."
+        ),
+    )
+
+    # Trigger/feature options
+    parser.add_argument("--ntriggers", type=int, default=-1)
+    parser.add_argument("--nevents", type=int, default=-1)
+    parser.add_argument(
+        "--measurement-type",
+        default="background",
+        choices=["background", "calibration", "threshold"],
+        help=(
+            "Raw measurement type for trigger/feature processing. Threshold "
+            "data may be processed directly by the feature step without a "
+            "trigger dataframe."
+        ),
+    )
+    parser.add_argument(
+        "--trigger-dataframe",
+        help="Existing trigger dataframe group/file for feature processing.",
+    )
+    parser.add_argument(
+        "--salting-dataframe",
+        help=(
+            "Existing salting dataframe to inject during trigger/feature "
+            "processing. Do not combine with the salting generation step."
+        ),
+    )
+    parser.add_argument("--external-features")
+    parser.add_argument(
+        "--partition-duration",
+        type=float,
+        default=10.0,
+        dest="partition_duration",
+    )
+
+    # Random-generation options. They are validated only when randoms is
+    # requested, so the same top-level command works for other step sets.
+    count = parser.add_mutually_exclusive_group(required=False)
+    count.add_argument("--nrandoms", type=int)
+    count.add_argument("--random-rate", type=float)
+    parser.add_argument("--min-separation-msec", type=float, default=0.0)
+    parser.add_argument("--edge-exclusion-msec", type=float, default=0.0)
+    parser.add_argument("--random-seed", type=int)
+
+
+def run(args):
+    return run_processing_workflow(args)

pytesprocess-0.1.1/pytesprocess/cli/commands/filter.py

@@ -0,0 +1,17 @@
+from pytesprocess.process import FilterDataProcessing
+
+
+def configure(parser):
+    parser.add_argument("--channels", nargs="+")
+
+
+def run(args):
+    proc = FilterDataProcessing(
+        args.acquisition, streams=args.streams, restricted=args.restricted,
+        config_file=args.config, verbose=not args.quiet,
+    )
+    proc.process(
+        channels=args.channels, processing_label=args.processing_label,
+        lgc_output=False, lgc_save=True,
+        save_file_path=args.output, ncores=args.ncores,
+    )

pytesprocess-0.1.1/pytesprocess/cli/commands/ivsweep.py

@@ -0,0 +1,29 @@
+from pytesprocess.process import IVSweepProcessing
+
+
+def configure(parser):
+    parser.add_argument("--channels", nargs="+")
+    parser.add_argument("--iv-only", action="store_true")
+    parser.add_argument("--didv-only", action="store_true")
+    parser.add_argument("--trace-length-msec", type=float, default=100.0)
+    parser.add_argument("--nrandoms-iv", type=int)
+    parser.add_argument("--random-seed", type=int)
+
+
+def run(args):
+    if args.iv_only and args.didv_only:
+        raise ValueError("Use only one of --iv-only or --didv-only")
+    proc = IVSweepProcessing(
+        args.acquisition, processing_label=args.processing_label,
+        verbose=not args.quiet,
+    )
+    proc.process(
+        channels=args.channels,
+        enable_iv=not args.didv_only,
+        enable_didv=not args.iv_only,
+        trace_length_iv_msec=args.trace_length_msec,
+        nrandoms_iv=args.nrandoms_iv,
+        random_seed=args.random_seed,
+        lgc_output=False, lgc_save=True,
+        save_path=args.output, ncores=args.ncores,
+    )

pytesprocess-0.1.1/pytesprocess/cli/common.py

@@ -0,0 +1,86 @@
+"""Shared helpers for the pytesprocess command-line interface."""
+
+from __future__ import annotations
+
+from pathlib import Path
+
+from pytesdaqx.io import AcquisitionCatalog
+
+from pytesprocess.config import ProcessingConfig
+
+
+def add_common_acquisition_args(parser, *, config=False, ncores=True):
+    parser.add_argument("acquisition", help="Raw pytesdaqx acquisition path")
+    if config:
+        parser.add_argument("--config", required=True, help="Processing YAML file")
+    parser.add_argument("--streams", nargs="+", help="Optional stream IDs")
+    parser.add_argument("--output", help="Base output directory")
+    parser.add_argument("--processing-label", help="Optional human-readable processing label")
+    if ncores:
+        parser.add_argument("--ncores", type=int, default=1, help="Number of local worker processes")
+    parser.add_argument("--restricted", action="store_true", help="Process restricted background data only")
+    parser.add_argument("--quiet", action="store_true", help="Reduce informational output")
+
+
+def catalog_for(args, measurement_types=None):
+    measurement = measurement_types
+    if isinstance(measurement, (list, tuple, set)):
+        unique = {str(item) for item in measurement}
+        background_only = unique == {"background"}
+    else:
+        background_only = (measurement is None or str(measurement) == "background")
+
+    if args.restricted and not background_only:
+        raise ValueError(
+            "--restricted is only valid for background data. It selects "
+            "restricted background data only, not open + restricted data."
+        )
+
+    catalog = AcquisitionCatalog(args.acquisition, verbose=not args.quiet)
+    return catalog.filter(
+        streams=args.streams,
+        measurement_types=measurement_types,
+        restricted=bool(args.restricted) if background_only else False,
+    )
+
+
+def processing_config(args, *, workflows, measurement_types="background", check_resources=True):
+    catalog = catalog_for(args, measurement_types=measurement_types)
+    config = ProcessingConfig(
+        args.config,
+        catalog.record_channels,
+        sample_rate=catalog.sample_rate_hz,
+        verbose=not args.quiet,
+    )
+    config.validate(workflows, check_resources=check_resources, display=not args.quiet)
+    return config, catalog
+
+
+def default_output_base(acquisition):
+    path = Path(acquisition)
+    parent = path.parent
+    if parent.name == "raw":
+        return str(parent.parent / "processed")
+    return str(parent / "processed")
+
+
+def output_base(args):
+    return args.output or default_output_base(args.acquisition)
+
+
+def trigger_edge_parameters(config, catalog):
+    """Return trigger edge exclusion and legacy-HDF5 livetime."""
+    from pytesprocess.core import FilterData
+    from pytesprocess.utils import get_trigger_template_info
+
+    trigger_config = config.get_config("trigger")
+    filter_file = trigger_config["overall"]["filter_file"]
+    filter_data = FilterData()
+    filter_data.load_hdf5(filter_file, overwrite=True)
+    info = get_trigger_template_info(trigger_config, filter_data)
+    edge_msec = float(info["max_edge_exclusion"])
+    livetime = None
+    if catalog.storage_format == "hdf5":
+        nsegments = sum(int(entry.get("n_segments") or 0) for entry in catalog.entries)
+        livetime = float(catalog.get_duration()) - nsegments * 2.0 * edge_msec * 1e-3
+    return edge_msec, livetime