scistackplotdb 0.1.26__tar.gz

scistackplotdb-0.1.26/.gitignore

@@ -0,0 +1,28 @@
+.venv
+.DS_Store
+*/*/__pycache__
+others_projects/sciforge
+scistack-gui/extension/node_modules/
+scistack-gui/frontend/node_modules/__pycache__/ 
+*.pyc 
+__pycache__/ 
+*.pyc
+*.pyo
+*.egg-info/
+# Python package build output (rebuilt fresh by CI on every publish; a
+# committed dist/ gets deleted by `rm -rf dist` mid-build and dirties the
+# tree for hatch-vcs). Excludes scistack-gui/extension/dist/, which is an
+# intentionally committed compiled bundle, not a Python build artifact.
+/dist/
+/*/dist/
+# Generated database artifacts (DuckDB data/lineage + write-ahead logs)
+*.duckdb
+*.duckdb.wal
+*.wal
+scistack-gui/frontend/node_modules/
+# mkdocs build output (regenerated by `mkdocs build`)
+/site/
+# Runtime output of code_export_service (pipeline-to-code export) — timestamped
+# per-run files, not source.
+/exports/
+*.log

scistackplotdb-0.1.26/PKG-INFO

@@ -0,0 +1,142 @@
+Metadata-Version: 2.5
+Name: scistackplotdb
+Version: 0.1.26
+Summary: scidb-backed plotting: load variables into long tables and plot them with scistackplot
+Author: SciStack Contributors
+License-Expression: MIT
+Keywords: plotting,provenance,scidb,scistack,visualization
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Science/Research
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Scientific/Engineering :: Visualization
+Classifier: Typing :: Typed
+Requires-Python: >=3.10
+Requires-Dist: scistack-db>=0.1.0
+Requires-Dist: scistackplot>=0.1.0
+Provides-Extra: dev
+Requires-Dist: matplotlib>=3.6; extra == 'dev'
+Requires-Dist: pytest-cov>=4.0; extra == 'dev'
+Requires-Dist: pytest>=7.0; extra == 'dev'
+Requires-Dist: seaborn>=0.12; extra == 'dev'
+Description-Content-Type: text/markdown
+
+# scistackplotdb
+
+## Plot what's in the database
+
+`scistackplotdb` loads SciDB variables into the long format
+[`scistackplot`](../scistackplot/README.md) consumes, and generates pipeline
+endpoints from a finished plot spec.
+
+```bash
+pip install scistackplotdb
+```
+
+```python
+from scidb import configure_database
+from scistackplot import PlotSpec, Role, PlotKind, render
+from scistackplotdb import ScidbSource
+
+db = configure_database("experiment.duckdb", ["subject", "session", "trial"])
+source = ScidbSource(db)
+
+table = source.get_table(["StepLength"])
+spec = PlotSpec(
+    measures=["StepLength"],
+    roles={"session": Role.X, "subject": Role.FREE, "trial": Role.FREE},
+    kind=PlotKind.BOX,
+)
+figure = render(table, spec)
+```
+
+## What this layer actually solves
+
+The long format is nearly free — schema keys are already columns once a
+variable is joined to `_schema`, the same shape `stat_` functions receive. The
+real work is the four things a flat CSV never had.
+
+**Shape classification.** Scalar, 1-D, or 2-D, decided from observed values
+rather than declared SQL type names, and cached. It determines which plot kinds
+are offered at all.
+
+**Joins across schema depth.** Plotting trial-level `Speed` against
+subject-level `Mass` broadcasts the shallower variable down the hierarchy:
+
+```python
+source.joinable_with("StepLength")     # -> ["Mass"]  (Signal is 1-D: no x axis)
+table = source.get_table(["StepLength", "Mass"])   # one Mass value per trial row
+```
+
+Because the dataset schema is an ordered, contiguous hierarchy, one variable's
+levels are always a prefix of the other's or the two cannot be joined — and
+`join_frames` refuses the latter with a message saying why.
+
+**Variants are factors — this one is a correctness trap.** A variable produced
+at two filter cutoffs has *two records per schema combination*. Treating those
+branch params as ordinary columns silently plots two pipelines' results as if
+they were replicates of one:
+
+```python
+spec = PlotSpec(measures=["Scaled"], roles={"session": Role.X})
+validate(spec, table)
+# RoleError: Variant factor(s) ['scale.factor'] would be pooled: their levels
+# are different pipeline variants, not replicates... Assign them
+# 'color'/'facet'/'iterate', select the variants you want with
+# PlotSpec.variant_sets, or — to pool them deliberately — set them to
+# 'aggregate' or 'free' yourself.
+```
+
+**A transport budget.** 1-D data across hundreds of trials is megabytes.
+`resolve(..., max_points=N)` downsamples for the interactive panel; export
+never does.
+
+## From spec to pipeline endpoint
+
+```python
+from scistackplotdb import generate_endpoint
+
+code = generate_endpoint(spec, table, input_variable="StepLength")
+print(code.source)
+```
+
+```python
+def plot_steplength(df, filename):
+    ...
+    return g.figure
+
+for_each(
+    plot_steplength,
+    inputs={
+        "df": StepLength,
+        "filename": PathOutput("plots/steplength_{subject}.png"),
+    },
+    outputs=[StepLengthFigure],
+    as_table=['df'],
+    finalized=True,
+    subject=[],
+)
+```
+
+The one translation that has to be exactly right is `Role.ITERATE` → a
+`for_each` iteration keyword. Interactively, ITERATE fans out through a pandas
+`groupby`; in the pipeline it fans out through `for_each` + `PathOutput`. If
+those disagree, the exported pipeline is not what you previewed —
+`tests/test_fanout_parity.py` runs both paths against the same database and
+compares the figure sets.
+
+Everything about *recording* the figure — `finalized`, artifact stamping,
+`skip_computed`, `scidb report` — is SciDB's existing endpoint machinery and is
+untouched.
+
+## Ordering
+
+Factor levels are ordered by SciDB's declared `schema_key_types`, not by
+pandas' default: a key declared `numeric` sorts numerically, and everything
+else goes through a natural sort so zero-padded IDs land as
+`01, 02, … 10` instead of `01, 10, 02`.
+
+See [`docs/claude/plotting-library-design.md`](../docs/claude/plotting-library-design.md).

scistackplotdb-0.1.26/README.md

@@ -0,0 +1,116 @@
+# scistackplotdb
+
+## Plot what's in the database
+
+`scistackplotdb` loads SciDB variables into the long format
+[`scistackplot`](../scistackplot/README.md) consumes, and generates pipeline
+endpoints from a finished plot spec.
+
+```bash
+pip install scistackplotdb
+```
+
+```python
+from scidb import configure_database
+from scistackplot import PlotSpec, Role, PlotKind, render
+from scistackplotdb import ScidbSource
+
+db = configure_database("experiment.duckdb", ["subject", "session", "trial"])
+source = ScidbSource(db)
+
+table = source.get_table(["StepLength"])
+spec = PlotSpec(
+    measures=["StepLength"],
+    roles={"session": Role.X, "subject": Role.FREE, "trial": Role.FREE},
+    kind=PlotKind.BOX,
+)
+figure = render(table, spec)
+```
+
+## What this layer actually solves
+
+The long format is nearly free — schema keys are already columns once a
+variable is joined to `_schema`, the same shape `stat_` functions receive. The
+real work is the four things a flat CSV never had.
+
+**Shape classification.** Scalar, 1-D, or 2-D, decided from observed values
+rather than declared SQL type names, and cached. It determines which plot kinds
+are offered at all.
+
+**Joins across schema depth.** Plotting trial-level `Speed` against
+subject-level `Mass` broadcasts the shallower variable down the hierarchy:
+
+```python
+source.joinable_with("StepLength")     # -> ["Mass"]  (Signal is 1-D: no x axis)
+table = source.get_table(["StepLength", "Mass"])   # one Mass value per trial row
+```
+
+Because the dataset schema is an ordered, contiguous hierarchy, one variable's
+levels are always a prefix of the other's or the two cannot be joined — and
+`join_frames` refuses the latter with a message saying why.
+
+**Variants are factors — this one is a correctness trap.** A variable produced
+at two filter cutoffs has *two records per schema combination*. Treating those
+branch params as ordinary columns silently plots two pipelines' results as if
+they were replicates of one:
+
+```python
+spec = PlotSpec(measures=["Scaled"], roles={"session": Role.X})
+validate(spec, table)
+# RoleError: Variant factor(s) ['scale.factor'] would be pooled: their levels
+# are different pipeline variants, not replicates... Assign them
+# 'color'/'facet'/'iterate', select the variants you want with
+# PlotSpec.variant_sets, or — to pool them deliberately — set them to
+# 'aggregate' or 'free' yourself.
+```
+
+**A transport budget.** 1-D data across hundreds of trials is megabytes.
+`resolve(..., max_points=N)` downsamples for the interactive panel; export
+never does.
+
+## From spec to pipeline endpoint
+
+```python
+from scistackplotdb import generate_endpoint
+
+code = generate_endpoint(spec, table, input_variable="StepLength")
+print(code.source)
+```
+
+```python
+def plot_steplength(df, filename):
+    ...
+    return g.figure
+
+for_each(
+    plot_steplength,
+    inputs={
+        "df": StepLength,
+        "filename": PathOutput("plots/steplength_{subject}.png"),
+    },
+    outputs=[StepLengthFigure],
+    as_table=['df'],
+    finalized=True,
+    subject=[],
+)
+```
+
+The one translation that has to be exactly right is `Role.ITERATE` → a
+`for_each` iteration keyword. Interactively, ITERATE fans out through a pandas
+`groupby`; in the pipeline it fans out through `for_each` + `PathOutput`. If
+those disagree, the exported pipeline is not what you previewed —
+`tests/test_fanout_parity.py` runs both paths against the same database and
+compares the figure sets.
+
+Everything about *recording* the figure — `finalized`, artifact stamping,
+`skip_computed`, `scidb report` — is SciDB's existing endpoint machinery and is
+untouched.
+
+## Ordering
+
+Factor levels are ordered by SciDB's declared `schema_key_types`, not by
+pandas' default: a key declared `numeric` sorts numerically, and everything
+else goes through a natural sort so zero-padded IDs land as
+`01, 02, … 10` instead of `01, 10, 02`.
+
+See [`docs/claude/plotting-library-design.md`](../docs/claude/plotting-library-design.md).

scistackplotdb-0.1.26/pyproject.toml

@@ -0,0 +1,60 @@
+[build-system]
+requires = ["hatchling", "hatch-vcs"]
+build-backend = "hatchling.build"
+
+[tool.hatch.version]
+source = "vcs"
+raw-options = { search_parent_directories = true, local_scheme = "no-local-version" }
+
+[project]
+name = "scistackplotdb"
+dynamic = ["version"]
+description = "scidb-backed plotting: load variables into long tables and plot them with scistackplot"
+readme = "README.md"
+license = "MIT"
+requires-python = ">=3.10"
+authors = [
+    { name = "SciStack Contributors" }
+]
+keywords = [
+    "plotting",
+    "visualization",
+    "scidb",
+    "scistack",
+    "provenance",
+]
+classifiers = [
+    "Development Status :: 4 - Beta",
+    "Intended Audience :: Science/Research",
+    "License :: OSI Approved :: MIT License",
+    "Operating System :: OS Independent",
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3.11",
+    "Programming Language :: Python :: 3.12",
+    "Topic :: Scientific/Engineering :: Visualization",
+    "Typing :: Typed",
+]
+dependencies = [
+    "scistackplot>=0.1.0",
+    "scistack-db>=0.1.0",
+]
+
+[project.optional-dependencies]
+dev = [
+    "pytest>=7.0",
+    "pytest-cov>=4.0",
+    "matplotlib>=3.6",
+    "seaborn>=0.12",
+]
+
+[tool.hatch.build.targets.sdist]
+include = [
+    "/src",
+]
+
+[tool.hatch.build.targets.wheel]
+packages = ["src/scistackplotdb"]
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]
+pythonpath = ["src"]

scistackplotdb-0.1.26/src/scistackplotdb/__init__.py

@@ -0,0 +1,80 @@
+"""
+scistackplotdb — plot scidb variables.
+
+Loads variables into the long format ``scistackplot`` consumes, adds the four
+things a flat table never needed (shape classification, schema-depth joins,
+variants as factors, and a transport budget), and generates pipeline endpoints
+from a finished spec.
+
+::
+
+    from scidb import configure_database
+    from scistackplot import PlotSpec, Role, PlotKind, render
+    from scistackplotdb import ScidbSource
+
+    db = configure_database("experiment.duckdb", ["subject", "session", "trial"])
+    source = ScidbSource(db)
+
+    table = source.get_table(["StepLength"])
+    spec = PlotSpec(
+        measures=["StepLength"],
+        roles={"session": Role.X, "subject": Role.FREE, "trial": Role.FREE},
+        kind=PlotKind.BOX,
+    )
+    figure = render(table, spec)
+
+Everything about recording a figure — ``finalized``, artifact stamping,
+``scidb report`` — belongs to scidb's existing ``plot_`` endpoint machinery and
+is unchanged. See ``docs/claude/plotting-library-design.md``.
+"""
+
+from __future__ import annotations
+
+from .endpoint import (
+    EndpointCode,
+    default_output_variable,
+    default_path_template,
+    generate_endpoint,
+)
+from .hierarchy import join_frames, join_kind, joinable, joined_levels
+from .load import (
+    LATEST_COLUMN,
+    MISSING_VERSION_LEVEL,
+    VERSION_FACTOR_PREFIX,
+    VariableFrame,
+    attach_variants,
+    data_columns_for,
+    load_variable,
+    registered_variables,
+    sample_value,
+    schema_keys,
+)
+from .source import ScidbSource
+from .variants import selection_for, variant_graph, variant_set
+
+__all__ = [
+    "ScidbSource",
+    "variant_set",
+    "variant_graph",
+    "selection_for",
+    "VariableFrame",
+    "VERSION_FACTOR_PREFIX",
+    "MISSING_VERSION_LEVEL",
+    "LATEST_COLUMN",
+    "load_variable",
+    "attach_variants",
+    "registered_variables",
+    "schema_keys",
+    "data_columns_for",
+    "sample_value",
+    "join_kind",
+    "joinable",
+    "join_frames",
+    "joined_levels",
+    "generate_endpoint",
+    "EndpointCode",
+    "default_output_variable",
+    "default_path_template",
+]
+
+__version__ = "0.1.0"

scistackplotdb-0.1.26/src/scistackplotdb/endpoint.py

@@ -0,0 +1,257 @@
+"""
+Turning a ``PlotSpec`` into a pipeline endpoint.
+
+The GUI's "Add to pipeline" produces two things: a ``plot_`` function (generated
+by ``scistackplot.codegen`` — literal seaborn, no runtime dependency on this
+package) and the ``for_each`` call that runs it. This module owns the second
+half, and with it the one translation that has to be exactly right:
+
+    Role.ITERATE  ->  a for_each iteration keyword
+
+Interactively, ITERATE fans out through a pandas ``groupby`` inside
+``resolve()``. In the pipeline it fans out through ``for_each`` + ``PathOutput``.
+If those two ever disagree, the exported pipeline is not what the user
+previewed — the worst failure mode this layer has, and what
+``tests/test_fanout_parity.py`` exists to prevent.
+
+Everything else the endpoint needs already exists in scidb: ``finalized``,
+artifact stamping, ``skip_computed`` and ``scidb report`` are untouched.
+"""
+
+from __future__ import annotations
+
+import re
+from dataclasses import dataclass, field
+
+from scistacklog import Log
+from scistackplot import LongTable, PlotSpec, default_function_name
+from scistackplot import generate_plot_function
+from scistackplot.codegen import group_param, variant_params
+from scistackplot.roles import fanout_keys
+from scistackplot.variants import defined_sets
+
+from .load import LATEST_COLUMN
+
+LAYER = "scistackplotdb"
+
+
+@dataclass
+class EndpointCode:
+    """Generated source for one plotting endpoint."""
+
+    function_name: str
+    function_source: str
+    foreach_source: str
+    iterate_keys: list[str] = field(default_factory=list)
+    path_template: str = ""
+    output_variable: str = ""
+
+    @property
+    def source(self) -> str:
+        """Function and call together, ready to append to a pipeline module."""
+        return f"{self.function_source}\n\n{self.foreach_source}"
+
+
+def generate_endpoint(
+    spec: PlotSpec,
+    table: LongTable,
+    *,
+    input_variable: str,
+    function_name: str | None = None,
+    output_variable: str | None = None,
+    path_template: str | None = None,
+    finalized: bool = True,
+    x_variable: str | None = None,
+) -> EndpointCode:
+    """
+    Generate the ``plot_`` function and its ``for_each`` call.
+
+    ``input_variable`` is the scidb variable type supplying the data;
+    ``x_variable`` is the optional second measure for an x–y scatter.
+    """
+    name = function_name or default_function_name(spec)
+    # NOT spec.iterate_factors: the fan-out includes schema keys promoted
+    # because a nested key iterates, and runs in schema order. The preview and
+    # the generated for_each have to agree on both — that is what
+    # tests/test_fanout_parity.py checks.
+    iterate_keys = fanout_keys(spec, table)
+    output = output_variable or default_output_variable(input_variable)
+    template = path_template or default_path_template(name, iterate_keys)
+
+    function_source = generate_plot_function(spec, table, function_name=name)
+    foreach_source = _foreach_call(
+        spec=spec,
+        function_name=name,
+        input_variable=input_variable,
+        x_variable=x_variable,
+        output_variable=output,
+        path_template=template,
+        iterate_keys=iterate_keys,
+        finalized=finalized,
+    )
+
+    Log.info(
+        "generated endpoint %s: input=%s output=%s iterate=%s finalized=%s",
+        name,
+        input_variable,
+        output,
+        iterate_keys or "none",
+        finalized,
+        layer=LAYER,
+    )
+    return EndpointCode(
+        function_name=name,
+        function_source=function_source,
+        foreach_source=foreach_source,
+        iterate_keys=iterate_keys,
+        path_template=template,
+        output_variable=output,
+    )
+
+
+def default_output_variable(input_variable: str) -> str:
+    """``StepLength`` -> ``StepLengthFigure``."""
+    return f"{input_variable}Figure"
+
+
+def default_path_template(function_name: str, iterate_keys: list[str]) -> str:
+    """
+    Build a PathOutput template that cannot collide.
+
+    Every ITERATE key goes into the filename. Omitting one would make two
+    figures write the same file; for schema keys scidb treats that as
+    pre-existing overwrite behavior (no error), and for variants its collision
+    guard raises before anything renders. Including them all avoids both.
+    """
+    slug = re.sub(r"^plot_", "", function_name)
+    parts = "".join(f"_{{{key}}}" for key in iterate_keys)
+    return f"plots/{slug}{parts}.png"
+
+
+def variant_expression(input_variable: str, variant_set) -> str:
+    """A ``Variant(...)`` call selecting one named variant's records.
+
+    The inverse of :func:`~scistackplotdb.variants.selection_for`, and the thing
+    that makes a variant figure reproducible by hand: what the popup's
+    checkboxes and version dropdowns produced comes back out as the same
+    wrapper a scientist would have typed.
+
+    Selections spanning two producing functions **nest** rather than resorting
+    to dotted-string kwargs::
+
+        Variant(Variant(EMG, fn="loadEMG", code_version="v1"), fn="bandpass", low_hz=20)
+
+    Nesting is documented, merges the two filters (``scidb.variant.Variant``
+    handles it explicitly), and keeps every keyword readable — where
+    ``**{"__code__.loadEMG": "v1"}`` would leak a reserved namespace into code a
+    user is meant to edit.
+    """
+    from scistackplot import CODE_FACTOR_PREFIX, LATEST
+
+    by_function: dict[str, dict[str, object]] = {}
+    for column, value in (variant_set.selection or {}).items():
+        if column.startswith(CODE_FACTOR_PREFIX):
+            fn_name = column[len(CODE_FACTOR_PREFIX) :]
+            by_function.setdefault(fn_name, {})["code_version"] = value
+        elif column == LATEST_COLUMN:
+            # The source's "these are the current records" recommendation. scidb
+            # spells the same thing `code_version="latest"`, resolved per schema
+            # location by the same rule — not "the highest ordinal".
+            by_function.setdefault(None, {})["code_version"] = LATEST
+        else:
+            fn_name, _, param = column.rpartition(".")
+            by_function.setdefault(fn_name or None, {})[param] = value
+
+    expression = input_variable
+    for fn_name in sorted(by_function, key=lambda n: (n is None, n or "")):
+        arguments = []
+        if fn_name:
+            arguments.append(f"fn={fn_name!r}")
+        arguments.extend(f"{key}={value!r}" for key, value in by_function[fn_name].items())
+        expression = f"Variant({expression}, {', '.join(arguments)})"
+    return expression
+
+
+def _single_variant_expression(input_variable: str, spec) -> str:
+    """The lone variant's pin, or the bare variable when nothing is selected.
+
+    The row may name its own variable, in which case that is what the endpoint
+    loads — ``input_variable`` is only the default.
+    """
+    sets = defined_sets(spec.variant_sets)
+    if len(sets) != 1:
+        return input_variable
+    return variant_expression(sets[0].variable or input_variable, sets[0])
+
+
+def _foreach_call(
+    *,
+    spec: PlotSpec,
+    function_name: str,
+    input_variable: str,
+    x_variable: str | None,
+    output_variable: str,
+    path_template: str,
+    iterate_keys: list[str],
+    finalized: bool,
+) -> str:
+    variant_inputs = variant_params(spec)
+    if variant_inputs:
+        # One input per named variant, each loaded through its own pin. See
+        # `codegen.variant_params` for why this cannot be a single `df`.
+        # Zipped against `defined_sets`, not `spec.variant_sets`: unfilled rows
+        # produce no input, so indexing the raw list would pair a parameter with
+        # the wrong variant's selection.
+        inputs = [
+            f'        "{generated.param}": '
+            f"{variant_expression(generated.variable, variant)},"
+            for generated, variant in zip(
+                variant_inputs, defined_sets(spec.variant_sets), strict=True
+            )
+        ]
+        table_inputs = [generated.param for generated in variant_inputs]
+    else:
+        pinned = _single_variant_expression(input_variable, spec)
+        inputs = [f'        "df": {pinned},']
+        table_inputs = ["df"]
+    if x_variable:
+        inputs.append(f'        "df_x": {x_variable},')
+        table_inputs.append("df_x")
+    for group in spec.factor_variables:
+        # A grouping variable arrives as its own input and is merged onto the
+        # data inside the function: `as_table` hands a function schema keys and
+        # data columns only, so a subject-level Condition cannot ride along on
+        # the measure's frame.
+        inputs.append(f'        "{group_param(group)}": {group},')
+        table_inputs.append(group_param(group))
+    inputs.append(f'        "filename": PathOutput("{path_template}"),')
+
+    lines = [
+        "for_each(",
+        f"    {function_name},",
+        "    inputs={",
+        *inputs,
+        "    },",
+        f"    outputs=[{output_variable}],",
+        # A plot_ function receives the long-format table (schema keys as
+        # columns) — as_table defaults ON only for stat_, so say it explicitly.
+        f"    as_table={table_inputs!r},",
+        f"    finalized={finalized},",
+    ]
+    for key in iterate_keys:
+        # [] means "every value present" — the same all-values resolution
+        # scifor applies to an empty iteration list.
+        lines.append(f"    {key}=[],")
+    lines.append(")")
+    return "\n".join(lines)
+
+
+def required_declarations(code: EndpointCode) -> list[str]:
+    """
+    Variable types the generated call needs that may not exist yet.
+
+    The GUI declares these through the normal entity-declaration path before
+    writing the code, so a generated endpoint never references an undeclared
+    type.
+    """
+    return [code.output_variable]