scistack-db 0.1.9.dev0__tar.gz

scistack_db-0.1.9.dev0/.gitignore

@@ -0,0 +1,16 @@
+.venv
+.DS_Store
+*/*/__pycache__
+others_projects/sciforge
+scistack-gui/extension/node_modules/
+scistack-gui/frontend/node_modules/__pycache__/ 
+*.pyc 
+__pycache__/ 
+*.pyc
+*.pyo
+
+# Generated database artifacts (DuckDB data/lineage + write-ahead logs)
+*.duckdb
+*.duckdb.wal
+*.wal
+scistack-gui/frontend/node_modules/

scistack_db-0.1.9.dev0/PKG-INFO

@@ -0,0 +1,128 @@
+Metadata-Version: 2.4
+Name: scistack-db
+Version: 0.1.9.dev0
+Summary: A lightweight database framework for scientific computing with versioning and provenance tracking
+License: MIT
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Science/Research
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Database
+Classifier: Topic :: Scientific/Engineering
+Requires-Python: >=3.9
+Requires-Dist: duckdb>=0.9.0
+Requires-Dist: numpy>=1.20.0
+Requires-Dist: pandas>=1.3.0
+Requires-Dist: scicanonicalhash>=0.1.0
+Requires-Dist: sciduckdb>=0.1.0
+Requires-Dist: scifor>=0.1.0
+Requires-Dist: scipathgen>=0.1.0
+Requires-Dist: scistacklog>=0.1.0
+Provides-Extra: dev
+Requires-Dist: matplotlib; extra == 'dev'
+Requires-Dist: pytest-cov>=4.0.0; extra == 'dev'
+Requires-Dist: pytest>=7.0.0; extra == 'dev'
+Description-Content-Type: text/markdown
+
+# SciDB
+
+Database operations layer for SciStack. Provides abstractions for defining typed variables, configuring the database, and saving/loading data by metadata.
+
+## Named Filters (value-based, reusable)
+
+`Filter` objects are first-class values — assign them to a variable and reuse
+them in any `where=` clause, or compose them with `&` / `|` / `~`:
+
+```python
+# Define once (this IS a ColumnFilter — no special wrapper needed)
+clean_gr = GAITRiteLoadedCycle["StepLengths_GR"] != 0
+
+# Use anywhere
+for_each(mean_change_from_reference,
+         inputs={"baseline": Fixed(GAITRiteLoadedCycle["StepLengths_GR"], session="BL"),
+                 "value":    GAITRiteLoadedCycle["StepLengths_GR"]},
+         outputs=[DeltaStepLength],
+         where=clean_gr & (UAStartFoot() == "A"))
+
+# Compose named filters
+clean_and_unilateral = clean_gr & (UAStartFoot() == "U")
+for_each(..., where=clean_and_unilateral)
+```
+
+MATLAB equivalent:
+
+```matlab
+clean_gr = GAITRiteLoadedCycle("StepLengths_GR") ~= 0;
+scidb.for_each(@meanChangeFromReference, ..., where=clean_gr & (UAStartFoot() == "A"));
+```
+
+## Permanent Schema-Level Exclusions
+
+For data that should be excluded from **every** analysis (e.g., a failed
+recording session), use the exclusion registry instead of per-call filters.
+
+```python
+# Mark a specific trial as excluded (persisted in the database)
+scidb.exclude_schema(subject=1, trial=2,
+                     reason="equipment malfunction during recording")
+
+# Exclude an entire subject (trial omitted = wildcard)
+scidb.exclude_schema(subject=3, reason="participant withdrew")
+
+# Inspect currently-excluded combinations
+exclusions_df = scidb.list_exclusions()
+
+# Re-include (logged; the original exclusion row is preserved)
+scidb.include_schema(subject=1, trial=2,
+                     reason="re-reviewed video, recording was valid")
+```
+
+MATLAB equivalent:
+
+```matlab
+scidb.exclude_schema("equipment malfunction", 'subject', 1, 'trial', 2)
+scidb.include_schema("re-reviewed, recording was valid", 'subject', 1, 'trial', 2)
+tbl = scidb.list_exclusions()
+```
+
+Exclusions are applied automatically by `for_each` before the iteration loop.
+The full exclusion table is hashed and stored in `version_keys`
+(`__schema_overrides_hash`) so that adding or removing an exclusion
+invalidates cached results.
+
+```python
+from scidb import configure_database, BaseVariable
+import numpy as np
+
+db = configure_database("experiment.duckdb", ["subject", "session"])
+
+class RawSignal(BaseVariable):
+    schema_version = 1
+
+RawSignal.save(np.array([1, 2, 3]), subject=1, session="A")
+(raw,) = RawSignal.load(subject=1, session="A")  # generator; unpack for single result
+all_versions = list(RawSignal.load(subject=1, session="A", version="all"))
+```
+## Logging
+
+Every scidb operation logs through the shared [`scistacklog`](../scistacklog/README.md) facade. `configure_database()` points the file sink at `scidb.log` next to the database file and writes a run-context header (package versions, Python version, pid), so each log file is self-describing.
+
+Two sinks with independent levels, both defaulting to INFO:
+
+- **console** (stderr): the pipeline narrative — for_each banner, periodic progress, run summary with failure reasons, `[timing]` summaries.
+- **file** (`scidb.log`): the same narrative, with date + millisecond timestamps and the originating layer (`[scidb]`, `[scifor]`, `[matlab]`, …) on every line; one record per line.
+
+For debugging, raise a sink to DEBUG to capture the full execution trace (named internal steps with durations, per-iteration `[run]`/`[skip]` lines, per-phase `[timing]` tables, column/dtype dumps):
+
+```python
+from scidb import Log
+
+Log.set_level("DEBUG", sink="file")   # full detail in scidb.log only
+Log.set_level("DEBUG")                # both sinks
+```
+
+or set the `SCIDB_LOG_LEVEL` environment variable, or pass `-v` to the `scidb` CLI.

scistack_db-0.1.9.dev0/README.md

@@ -0,0 +1,98 @@
+# SciDB
+
+Database operations layer for SciStack. Provides abstractions for defining typed variables, configuring the database, and saving/loading data by metadata.
+
+## Named Filters (value-based, reusable)
+
+`Filter` objects are first-class values — assign them to a variable and reuse
+them in any `where=` clause, or compose them with `&` / `|` / `~`:
+
+```python
+# Define once (this IS a ColumnFilter — no special wrapper needed)
+clean_gr = GAITRiteLoadedCycle["StepLengths_GR"] != 0
+
+# Use anywhere
+for_each(mean_change_from_reference,
+         inputs={"baseline": Fixed(GAITRiteLoadedCycle["StepLengths_GR"], session="BL"),
+                 "value":    GAITRiteLoadedCycle["StepLengths_GR"]},
+         outputs=[DeltaStepLength],
+         where=clean_gr & (UAStartFoot() == "A"))
+
+# Compose named filters
+clean_and_unilateral = clean_gr & (UAStartFoot() == "U")
+for_each(..., where=clean_and_unilateral)
+```
+
+MATLAB equivalent:
+
+```matlab
+clean_gr = GAITRiteLoadedCycle("StepLengths_GR") ~= 0;
+scidb.for_each(@meanChangeFromReference, ..., where=clean_gr & (UAStartFoot() == "A"));
+```
+
+## Permanent Schema-Level Exclusions
+
+For data that should be excluded from **every** analysis (e.g., a failed
+recording session), use the exclusion registry instead of per-call filters.
+
+```python
+# Mark a specific trial as excluded (persisted in the database)
+scidb.exclude_schema(subject=1, trial=2,
+                     reason="equipment malfunction during recording")
+
+# Exclude an entire subject (trial omitted = wildcard)
+scidb.exclude_schema(subject=3, reason="participant withdrew")
+
+# Inspect currently-excluded combinations
+exclusions_df = scidb.list_exclusions()
+
+# Re-include (logged; the original exclusion row is preserved)
+scidb.include_schema(subject=1, trial=2,
+                     reason="re-reviewed video, recording was valid")
+```
+
+MATLAB equivalent:
+
+```matlab
+scidb.exclude_schema("equipment malfunction", 'subject', 1, 'trial', 2)
+scidb.include_schema("re-reviewed, recording was valid", 'subject', 1, 'trial', 2)
+tbl = scidb.list_exclusions()
+```
+
+Exclusions are applied automatically by `for_each` before the iteration loop.
+The full exclusion table is hashed and stored in `version_keys`
+(`__schema_overrides_hash`) so that adding or removing an exclusion
+invalidates cached results.
+
+```python
+from scidb import configure_database, BaseVariable
+import numpy as np
+
+db = configure_database("experiment.duckdb", ["subject", "session"])
+
+class RawSignal(BaseVariable):
+    schema_version = 1
+
+RawSignal.save(np.array([1, 2, 3]), subject=1, session="A")
+(raw,) = RawSignal.load(subject=1, session="A")  # generator; unpack for single result
+all_versions = list(RawSignal.load(subject=1, session="A", version="all"))
+```
+## Logging
+
+Every scidb operation logs through the shared [`scistacklog`](../scistacklog/README.md) facade. `configure_database()` points the file sink at `scidb.log` next to the database file and writes a run-context header (package versions, Python version, pid), so each log file is self-describing.
+
+Two sinks with independent levels, both defaulting to INFO:
+
+- **console** (stderr): the pipeline narrative — for_each banner, periodic progress, run summary with failure reasons, `[timing]` summaries.
+- **file** (`scidb.log`): the same narrative, with date + millisecond timestamps and the originating layer (`[scidb]`, `[scifor]`, `[matlab]`, …) on every line; one record per line.
+
+For debugging, raise a sink to DEBUG to capture the full execution trace (named internal steps with durations, per-iteration `[run]`/`[skip]` lines, per-phase `[timing]` tables, column/dtype dumps):
+
+```python
+from scidb import Log
+
+Log.set_level("DEBUG", sink="file")   # full detail in scidb.log only
+Log.set_level("DEBUG")                # both sinks
+```
+
+or set the `SCIDB_LOG_LEVEL` environment variable, or pass `-v` to the `scidb` CLI.

scistack_db-0.1.9.dev0/pyproject.toml

@@ -0,0 +1,71 @@
+[build-system]
+requires = ["hatchling", "hatch-vcs"]
+build-backend = "hatchling.build"
+
+[tool.hatch.version]
+source = "vcs"
+raw-options = { search_parent_directories = true, local_scheme = "no-local-version" }
+
+[project]
+name = "scistack-db"
+dynamic = ["version"]
+description = "A lightweight database framework for scientific computing with versioning and provenance tracking"
+readme = "README.md"
+license = {text = "MIT"}
+requires-python = ">=3.9"
+classifiers = [
+    "Development Status :: 3 - Alpha",
+    "Intended Audience :: Science/Research",
+    "License :: OSI Approved :: MIT License",
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3.10",
+    "Programming Language :: Python :: 3.11",
+    "Programming Language :: Python :: 3.12",
+    "Programming Language :: Python :: 3.13",
+    "Topic :: Database",
+    "Topic :: Scientific/Engineering",
+]
+dependencies = [
+    "numpy>=1.20.0",
+    "pandas>=1.3.0",
+    "duckdb>=0.9.0",
+    "scipathgen>=0.1.0",
+    "scicanonicalhash>=0.1.0",
+    "sciduckdb>=0.1.0",
+    "scifor>=0.1.0",
+    "scistacklog>=0.1.0",
+]
+
+[project.optional-dependencies]
+dev = [
+    "pytest>=7.0.0",
+    "pytest-cov>=4.0.0",
+    "matplotlib",
+]
+
+[project.scripts]
+scidb = "scidb.inspect.cli:main"
+
+[tool.hatch.build.targets.sdist]
+include = ["/src"]
+
+[tool.hatch.build.targets.wheel]
+packages = ["src/scidb"]
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]
+python_files = ["test_*.py"]
+python_functions = ["test_*"]
+addopts = "-v --tb=short"
+
+[tool.coverage.run]
+source = ["src/scidb"]
+branch = true
+
+[tool.coverage.report]
+exclude_lines = [
+    "pragma: no cover",
+    "def __repr__",
+    "raise NotImplementedError",
+    "if TYPE_CHECKING:",
+]

scistack_db-0.1.9.dev0/src/scidb/__init__.py

@@ -0,0 +1,144 @@
+"""
+SciStack: Scientific Data Versioning Framework
+
+A lightweight database framework for scientific computing that provides:
+- Type-safe serialization of numpy arrays, DataFrames, and custom types
+- Automatic content-based versioning
+- Flexible metadata-based addressing
+- DuckDB storage for data and lineage (via SciDuck)
+
+Example:
+    from scidb import configure_database, BaseVariable
+    import numpy as np
+
+    # Setup (single DuckDB file for data + lineage, auto-registers types)
+    db = configure_database("experiment.duckdb", ["subject", "session"])
+
+    class RawSignal(BaseVariable):
+        schema_version = 1
+
+    # Save/load
+    RawSignal.save(np.array([1, 2, 3]), subject=1, session="A")
+    raw = RawSignal.load(subject=1, session="A")
+"""
+
+# From scifor (Layer 1)
+from scifor import Col, PathInput, PathOutput, get_schema, set_schema
+
+from .across_variants import AcrossVariants
+from .artifact_stamp import read_artifact_stamp, stamp_artifact
+from .colname import ColName
+from .column_selection import ColumnSelection
+from .constant import Constant, constant
+from .database import configure_database, get_database, get_user_id
+from .discover import (
+    DiscoveryResult,
+    ModuleError,
+    ModuleExports,
+    PackageResult,
+    discover_module,
+    scan_package,
+    scan_project,
+)
+from .each_of import EachOf
+from .exceptions import (
+    AmbiguousParamError,
+    AmbiguousVersionError,
+    DatabaseLockedError,
+    DatabaseNotConfiguredError,
+    NotFoundError,
+    NotRegisteredError,
+    PipelineCycleError,
+    ReservedMetadataKeyError,
+    SciStackError,
+)
+from .exclusions import exclude_schema, include_schema, list_exclusions
+from .filters import raw_sql, schema_key
+from .fixed import Fixed
+
+# Batch execution (Layer 2 — DB-backed, no lineage)
+from .foreach import for_each
+from .foreach_config import ForEachConfig
+from .lineage_save import save
+from .log import Log
+from .merge import Merge
+from .pipeline import Pipeline, PipelineBinding, Step, active_pipeline, scistack
+from .state import (
+    check_combo_state,
+    check_multiple_nodes_state,
+    check_node_state,
+    check_pathinput_node_state,
+)
+from .variable import BaseVariable
+from .variant import Variant, branch_param
+
+__version__ = "0.1.0"
+
+__all__ = [
+    # Core classes
+    "BaseVariable",
+    "Constant",
+    "constant",
+    # Discovery
+    "scan_project",
+    "scan_package",
+    "discover_module",
+    "DiscoveryResult",
+    "PackageResult",
+    "ModuleExports",
+    "ModuleError",
+    # Configuration
+    "configure_database",
+    "get_database",
+    "get_user_id",
+    # Save
+    "save",
+    # Pipeline node staleness
+    "check_combo_state",
+    "check_node_state",
+    "check_pathinput_node_state",
+    "check_multiple_nodes_state",
+    # Logging
+    "Log",
+    # Batch execution
+    "for_each",
+    "scistack",
+    "Pipeline",
+    "PipelineBinding",
+    "Step",
+    "active_pipeline",
+    "Fixed",
+    "Variant",
+    "AcrossVariants",
+    "stamp_artifact",
+    "read_artifact_stamp",
+    "branch_param",
+    "Merge",
+    "ColumnSelection",
+    "ColName",
+    "EachOf",
+    "ForEachConfig",
+    "PathInput",
+    "PathOutput",
+    # Standalone / DataFrame support
+    "Col",
+    "set_schema",
+    "get_schema",
+    # Filter utilities
+    "raw_sql",
+    "schema_key",
+    # Schema exclusions
+    "exclude_schema",
+    "include_schema",
+    "list_exclusions",
+    # Exceptions
+    "SciStackError",
+    "NotRegisteredError",
+    "NotFoundError",
+    "DatabaseNotConfiguredError",
+    "ReservedMetadataKeyError",
+    "AmbiguousVersionError",
+    "AmbiguousParamError",
+    "DatabaseLockedError",
+    "PipelineCycleError",
+]

scistack_db-0.1.9.dev0/src/scidb/across_variants.py

@@ -0,0 +1,97 @@
+"""AcrossVariants pooling wrapper: opt IN to cross-variant aggregation."""
+
+from typing import Any
+
+
+class AcrossVariants:
+    """
+    Wrapper to deliberately pool ALL branch_param variants of an input into one
+    aggregated table, with each row's upstream branch_params attached as columns.
+
+    Aggregation-mode ``for_each`` (iterating a strict subset of the schema keys)
+    **auto-splits** by upstream branch_param signature by default: one call per
+    variant group, as if the user had written
+    ``EachOf(Variant(X, ...), Variant(X, ...))``. That default is right for
+    ordinary aggregation and for endpoint (``plot_``/``stat_``) functions — a
+    pooled t-test double-counts, and a figure mixing two filter settings is
+    wrong.
+
+    ``AcrossVariants`` is the explicit opt-out for **multiverse /
+    specification-curve analysis**: the one legitimate cross-variant use case,
+    where the analysis deliberately spans pipeline decisions to quantify
+    robustness. The pooled rows keep their variant identity — each namespaced
+    branch_param key (e.g. ``bandpass.low_hz``) becomes an ordinary DataFrame
+    column so the function can group by specification::
+
+        def robustness(df):
+            return df.groupby("bandpass.low_hz")["value"].mean()
+
+        for_each(robustness, {"df": AcrossVariants(Filtered)}, [Spec],
+                 subject=["S01", "S02"])
+
+    Composition:
+
+    - May wrap a variable type, a ``Fixed``, or a ``Variant``
+      (``AcrossVariants(Variant(X, low_hz=20))`` pins some params and pools
+      whatever variants remain).
+    - ``AcrossVariants(ColumnSelection)`` is an ERROR: the column selection
+      would drop the attached branch_param columns, silently defeating the
+      pooling contract.
+    - ``AcrossVariants(Merge(...))`` is an ERROR (mirrors ``Variant`` /
+      ``Fixed``): pool per constituent instead.
+    - ``AcrossVariants(EachOf(...))`` is an ERROR: ``EachOf`` must stay the
+      outermost wrapper.
+
+    In full-iteration mode (all schema keys iterated) each combo already sees
+    exactly one variant, so pooling is meaningless: the input behaves as if
+    unwrapped and a warning is logged.
+    """
+
+    def __init__(self, var_type: Any):
+        from .column_selection import ColumnSelection
+        from .each_of import EachOf
+        from .merge import Merge
+
+        if isinstance(var_type, Merge):
+            raise TypeError(
+                "AcrossVariants cannot wrap a Merge. branch_params are namespaced "
+                "per producing function, so pooling must happen per constituent: "
+                "Merge(AcrossVariants(A), B)."
+            )
+        if isinstance(var_type, EachOf):
+            raise TypeError(
+                "AcrossVariants cannot wrap an EachOf. EachOf must stay the "
+                "outermost wrapper."
+            )
+        if isinstance(var_type, ColumnSelection):
+            raise TypeError(
+                "AcrossVariants cannot wrap a ColumnSelection: the column "
+                "selection would drop the attached branch_param columns. "
+                "Select columns inside your function instead."
+            )
+        if isinstance(var_type, AcrossVariants):
+            var_type = var_type.var_type  # idempotent
+
+        self.var_type = var_type
+
+    def to_key(self) -> str:
+        """Return a canonical string for use as a version key.
+
+        A pooled run and an auto-split run of the same function must not
+        collide: this key lands in the ``__inputs`` config key (same mechanism
+        as ``Variant.to_key()``).
+        """
+        if hasattr(self.var_type, "to_key"):
+            inner_key = self.var_type.to_key()
+        elif isinstance(self.var_type, type):
+            inner_key = self.var_type.__name__
+        else:
+            inner_key = repr(self.var_type)
+        return f"AcrossVariants({inner_key})"
+
+    @property
+    def __name__(self) -> str:
+        """Display name for format_inputs and error messages."""
+        from .foreach import _input_type_name
+
+        return f"AcrossVariants({_input_type_name(self.var_type)})"