views-frames 1.0.0__py3-none-any.whl

views_frames-1.0.0.dist-info/METADATA

@@ -0,0 +1,624 @@
+Metadata-Version: 2.4
+Name: views-frames
+Version: 1.0.0
+Summary: The VIEWS platform data-contract layer: immutable array+identifier frames (numpy only, root of the dependency DAG).
+Project-URL: Homepage, https://github.com/views-platform/views-frames
+Project-URL: Repository, https://github.com/views-platform/views-frames
+Project-URL: Changelog, https://github.com/views-platform/views-frames/blob/main/CHANGELOG.md
+Author-email: Simon Polichinel von der Maase <simmaa@prio.org>
+License-Expression: MIT
+License-File: LICENSE
+Requires-Python: >=3.10
+Requires-Dist: numpy<3,>=1.26
+Provides-Extra: arrow
+Requires-Dist: pyarrow<20,>=14; extra == 'arrow'
+Description-Content-Type: text/markdown
+
+# views-frames
+
+> The VIEWS platform's **data-contract layer**: small, stable, abstract array
+> containers (`FeatureFrame`, `PredictionFrame`, and their anticipated siblings)
+> that every other repo depends on and that depends on nothing internal.
+>
+> **Status:** **frozen — v1.0.0** (API freeze, ADR-018). This README is the design
+> bible; the contract it specifies is realised in `src/views_frames/` (index, frames,
+> io, conformance suite) plus the `src/views_frames_summarize/` sibling package
+> (sample-axis summarization — `collapse`/MAP/HDI/quantiles + cross-level
+> aggregation; ADR-017). The blocking design decisions are resolved (§13a) and
+> ratified as ADRs 011–018; two consumer-review rounds (`perspectives/`) validated
+> the design. Consumer adoption (re-export shims, pandas migration) is the owner's
+> migration, not this repo's.
+> If the code and this README disagree, that is a bug — reconcile before merging.
+
+---
+
+## 0. One-paragraph thesis
+
+DataFrames (pandas/geopandas/polars) are a **boundary/interop and analysis**
+format, not an internal data-handling representation. They do not belong as the
+canonical transport type inside the VIEWS pipeline. The canonical transport is an
+**array + spatiotemporal identifiers** value object — what we call a *frame*.
+Two frames already exist, **duplicated and diverging** across repos
+(`PredictionFrame` in views-pipeline-core, `FeatureFrame` in views-datafactory).
+`views-frames` unifies them into one **leaf package** at the root of the
+dependency graph: maximally stable, maximally abstract, numpy-only, depended on
+by everyone, depending on nothing internal. It is the keystone that
+de-duplicates the frames, breaks cross-repo dependency cycles, removes pandas
+from internal transport, and gives arrays the label-alignment that today forces
+pandas back into the hot path.
+
+---
+
+## 0a. Quickstart
+
+Build a frame, summarize its sample axis, serialize it, and run the published
+contract check. The full runnable script is [`examples/quickstart.py`](examples/quickstart.py)
+(`uv run examples/quickstart.py`):
+
+```python
+import numpy as np
+from views_frames import PredictionFrame, SpatialLevel, SpatioTemporalIndex
+from views_frames.conformance import assert_frame_contract
+from views_frames_summarize import collapse, hdi, map_estimate
+
+index = SpatioTemporalIndex(
+    time=np.array([1, 1, 2], dtype=np.int64),
+    unit=np.array([10, 11, 10], dtype=np.int32),
+    level=SpatialLevel.PGM,
+)
+pf = PredictionFrame(np.random.default_rng(0).gamma(2.0, 1.0, (3, 500)).astype("f4"), index)
+
+mean = collapse(pf, np.mean)      # (N, S) -> (N, 1) frame, statistic injected
+mode = map_estimate(pf)           # per-row MAP -> (N, 1) frame
+band = hdi(pf, mass=0.9)          # per-row 90% HDI -> (N, 2) index-aligned array
+
+pf.save("/tmp/pf"); reloaded = PredictionFrame.load("/tmp/pf")
+assert_frame_contract(pf)         # the check a consumer runs in its own CI
+```
+
+The leaf (`views_frames`) owns the immutable array+identifier contract and
+alignment; the sibling (`views_frames_summarize`) owns the sample-axis statistics.
+Both are numpy-only. For the subtler cm↔pgm surface — a time-varying
+`(time, unit)→country` mapping, `cross_level_align`, and conservation-correct
+`aggregate_distributions` (`HDI(sum) ≠ sum(HDI)`) — see
+[`examples/cross_level.py`](examples/cross_level.py).
+
+---
+
+## 1. Why this package exists (the problems it kills)
+
+Concrete, current pain — each item is a real, observed defect this package is
+designed to resolve (register IDs are from views-pipeline-core's technical risk
+register):
+
+- **Duplicated, diverging twins.** `PredictionFrame`
+  (`views-pipeline-core/views_pipeline_core/data/prediction_frame.py`) and
+  `FeatureFrame`
+  (`views-datafactory/src/datafactory_adapters/feature_frame.py`) share a core
+  (`values: ndarray` + `identifiers: {time, unit}` + `save/load`) but are **not
+  near-1:1**: they diverge on ≥6 axes — sample-axis position, `feature_names` /
+  `metadata`, identifier NaN-check, `collapse` / `mmap`, save footprint, and
+  `PredictionFrame` still imports pandas. They have two owners, two release
+  cadences, and **no shared base**. They will drift. (REP violation — reused
+  together, released apart.) The fix unifies the *shared index + protocols*, not
+  the classes — see §5 (Option C) and §13a.
+- **Circular package dependency.** views-pipeline-core ↔ views-reporting form a
+  cycle (one direction declared, the other hidden behind `try/except ImportError`).
+  See views-reporting issue #113. A neutral leaf package both sides route their
+  data contract *through* breaks the cycle (ADP).
+- **pandas leaks into internal transport.** The evaluation boundary still takes
+  `actual: pd.DataFrame, predictions: List[pd.DataFrame]`
+  (`modules/validation/adapter.py`); ingest returns `pd.DataFrame`; the
+  list-in-cell parquet encoding causes a measured ~33× memory blow-up (C-40,
+  C-66). The frame + a flat columnar disk format fix the scaling; a `TargetFrame`
+  fixes the eval boundary.
+- **Observed in production (#181) — the thesis, measured.** A HydraNet eval run
+  (`main.py -r calibration -t -e -re`) is **OOM-killed (exit 137, ~16–18 GB)** in
+  the report tail; dropping the report flag → 2.4 GB (~7× less). A synthetic
+  micro-benchmark line-isolated it: the report builds **object-dtype** DataFrames
+  (list-in-cell `pred_{target}` + per-cell `np.array` actuals) over the **full
+  grid × full timeline** — **~50–160×** the dense float32 cost (~200–650 B/row vs
+  4). The dense numpy compute is *small* (~0.3 GB); the cost is the object
+  representation. It scales with `n_posterior_samples` (the collapse step is what
+  first materializes the full-sample tensor). This is C-40/C-66 firing for real —
+  pipeline-core **C-186**, the **first observed-in-production member** of the
+  Data-Contract Gap cluster, and the live use-case that motivates this package.
+  A dense, collapsed array frame is the fix. See
+  `perspectives/from_views-pipeline-core_perspective.md`.
+- **God-class data handler with leaked internals.** `_ViewsDataset`
+  (`data/handlers.py`, ~950 LOC, C-36) is consumed across three repos by reaching
+  into its **private** members (`_time_id`, `_entity_id`, `_get_entity_index`,
+  `.dataframe`, `.to_tensor`) at ~56 sites (C-135), and views-reporting even
+  **mutates** a core object across the repo boundary
+  (`pg_dataset.reconciled_dataframe = ...`,
+  `views-reporting/reconciliation/dataset_export.py:103,122`; C-184). Frames are
+  immutable value objects with a *published* interface — the opposite of this.
+- **Evaluation outputs scattered, then mis-read.** A model's evaluation metrics
+  are written to a local `eval_*.parquet` *and* logged to wandb, with no typed
+  output container. views-reporting's evaluation report scrapes them back out of
+  wandb (`get_latest_run().summary`) and — because that returns the latest
+  *created* run, not the latest run *with* metrics — renders the wrong run:
+  **22/25 constituents showed "not calculated"** in a real ensemble report while
+  the scores sat in an earlier run (views-reporting's own register, C-48). A
+  first-class **`MetricFrame`** (§4.2) is the typed output form the report *could
+  adopt* instead of re-deriving from a mutable mirror — but `MetricFrame` is
+  **out of this leaf** (it is keyed `(target, step, unit)`, not a `(time, unit)`
+  frame; views-evaluation owns eval-output vocab). What this package provides is
+  the **substrate** for that cure (the typed, conformance-checked frame contract +
+  the extensible `FrameMetadata` header), not the cure itself. *(Exploratory; §4.2,
+  §13a.6.)*
+- **Stable package, zero abstractions.** views-pipeline-core's `data/` is its
+  most depended-on (most stable) package yet contains no protocols/ABCs (C-165,
+  C-48). A stable component must be abstract (SAP). This package *is* the
+  abstraction.
+
+**The product is not "a numpy wrapper." The product is the identifier/alignment
+contract** — the shared, versioned definition of "an array aligned to (time,
+unit)" that every model, evaluator, reconciler, and report agrees on.
+
+---
+
+## 2. Position in the dependency graph (the whole point)
+
+```
+                        ┌───────────────────────┐
+                        │     views-frames      │  ← leaf / root of the DAG
+                        │  (numpy only, stable, │     stable + abstract (SDP+SAP)
+                        │   abstract protocols)  │     depends on NOTHING internal
+                        └───────────▲───────────┘
+            ┌───────────────┬───────┴────────┬────────────────┐
+            │               │                │                │
+   views-pipeline-core  views-datafactory  views-evaluation  model repos
+   (orchestration)      (data production)  (metrics)         (hydranet, bayesian,
+            │               │                                  stepshifter, r2darts2,
+            ▼               ▼                                  baseline, lab00)
+   views-reporting / views-postprocessing (consumers, downstream)
+```
+
+**Rule:** every internal arrow points *toward* `views-frames`. `views-frames`
+imports **no** `views_*` package, ever. If it ever needs to, the boundary is
+wrong. This is what makes it impossible to participate in a cycle (ADP) and what
+makes it safe to depend on from everywhere (SDP).
+
+> **Consumer perspectives.** A downstream repo's detailed view of how it uses
+> these frames lives in `perspectives/from_<repo>_perspective.md`. The first is
+> `perspectives/from_views-reporting_perspective.md` — the presentation layer that
+> *consumes* `PredictionFrame`, `TargetFrame`, and `MetricFrame` and routes its
+> data contract through this leaf (which is what breaks the
+> views-pipeline-core ↔ views-reporting cycle, reporting issue **#113**).
+>
+> `perspectives/from_views-pipeline-core_perspective.md` is the **origin/orchestration**
+> repo's view — not a pure downstream consumer but the repo that *owns these types
+> today* (`PredictionFrame`, `_ViewsDataset`, the converter) and hands the contract
+> off to this leaf. It carries the worked failure mode (#181 report-stage OOM,
+> C-186) and the migration mechanics (it does most of README §10).
+
+---
+
+## 3. Hard constraints (non-negotiable; reject PRs that break these)
+
+1. **Dependencies:** `numpy` only, in the core. Optional extras may add
+   serialization deps **behind `io/` submodules** (`pyarrow` for the columnar
+   format), never in the core frame classes. **Never** import `pandas`,
+   `geopandas`, `polars`, `wandb`, `viewser`, `torch`, or any `views_*` package
+   from the core. (CRP: a model that wants a `PredictionFrame` must not
+   transitively install the pandas/reporting world.)
+2. **No application logic.** No fetching, no model code, no report rendering, no
+   reconciliation math, no wandb, no disk-path conventions beyond `save/load` of
+   the frame itself. Those are *adapters* and live in the consumer repos.
+3. **Immutable value objects.** A frame is validated at construction and then
+   treated as read-only. Operations (`collapse`, `select`, `with_metadata`)
+   **return new frames**; they never mutate in place. (Directly forbids the
+   C-184 cross-repo-mutation anti-pattern.) **Copy-vs-view:** structural and
+   metadata-only operations (`with_metadata`, contiguous `select`) return frames
+   that **share** the underlying `values` buffer (numpy view / zero-copy), and a
+   `mmap`-backed frame stays `mmap`-backed — a new frame must never copy a
+   multi-GB `values` buffer (that would reintroduce the §7 blow-up). Only a
+   reducing op (`collapse`) allocates, and only the reduced array. Pinned in the
+   conformance suite.
+4. **Fail loud at construction.** All invariants are checked in `__init__` and
+   raise `ValueError`/`TypeError` immediately — never return a half-valid object,
+   never log-and-continue. (Matches the platform's "Fail Loud and Proud" rule.)
+5. **dtype discipline.** `values` are `float32` (contiguous); identifier arrays
+   are integer dtype; **no `object` dtype, ever** (object/list-in-cell is the
+   thing that doesn't scale). Identifiers are complete (no NaN). The guarantee is
+   **structural, not temporal**: the leaf validates integer / length-N / no-NaN,
+   but `time` is an **opaque integer** — month_id epoch, range, and monotonicity
+   are a producer-adapter concern, never the leaf's (the leaf is epoch-agnostic).
+6. **One concept per file.** See §6. Multiple classes in one file is the
+   exception, justified only by genuine tight coupling.
+
+---
+
+## 4. The frame family
+
+A *frame* = a numeric array whose first axis is **N rows**, each row carrying a
+complete set of **spatiotemporal identifiers** `{time, unit}`, optionally with a
+trailing **sample axis S** (posterior draws / ensemble members) and, for
+multi-channel frames, a **feature/channel axis**.
+
+### 4.1 Existing (unify these first)
+
+| Frame | Array shape | Extra fields | Semantics | Lives today in |
+|---|---|---|---|---|
+| **`FeatureFrame`** | `y_features: (N, F)` or `(N, F, S)` | `feature_names: list[str]` | model **inputs** (X) | views-datafactory |
+| **`PredictionFrame`** | `y_pred: (N, S)` | — | model **outputs** (ŷ samples) | views-pipeline-core |
+
+Existing `PredictionFrame` contract (preserve on migration): `float32`;
+`REQUIRED_IDENTIFIERS = {"time", "unit"}`; validates 2D, `n_rows > 0`,
+`sample_count >= 1`, identifiers present + length-N + no NaN; properties
+`n_rows`, `sample_count`, `identifier_keys`; `collapse(method="arithmetic_mean")`
+→ new `(N, 1)` frame; `save(dir)` → `y_pred.npy` + `identifiers.npz`;
+`load(dir, mmap=False)`. Existing `FeatureFrame` adds `feature_names`,
+`metadata`, `n_features`, `is_sample`.
+
+**Sample axis convention (decided, §13a).** The sample axis **S** is **always an
+explicit trailing axis** (`S ≥ 1`): `PredictionFrame` is `(N, S)`, `FeatureFrame`
+is `(N, F, S)`, `TargetFrame` is `(N, 1)`. `is_sample` is `S > 1`; `collapse`
+reduces the trailing axis. One shape contract across the family — no `ndim`
+branching. A corollary: relocating `PredictionFrame` is a **numpy-only rewrite of
+its identifier validation, not a verbatim move** — today it imports pandas and
+uses `pd.isna` for the NaN-check (§10.2).
+
+### 4.2 Anticipated (design the base so these drop in via OCP, don't build all now)
+
+| Frame | Array shape | Why we already know we need it | Priority |
+|---|---|---|---|
+| **`TargetFrame`** (a.k.a. `ActualsFrame`) | `y_true: (N, 1)` | The **evaluation boundary** still takes pandas actuals (`adapter.py`). A target frame makes eval array-native and kills that pandas dependency. Structurally `PredictionFrame` with `S=1`. | **next** |
+| **`WeightFrame`** | `w: (N,)` or `(N, S)` | Weighted losses / weighted metrics. Same identifiers, different `values` meaning. | when weighting lands |
+| **`MaskFrame`** | `mask: (N,)` bool | Partial-data / sparse-actuals evaluation (C-26 silent truncation). Marks which (time, unit) cells are present. | when partial eval lands |
+| **`MetricFrame`** (a.k.a. `ScoreFrame`) | `(K, …)` keyed by `(target, step, unit)` | Evaluation **outputs** are currently scattered into wandb summaries + parquet. First-class array form. **views-reporting's eval report is the consumer of record** — today it scrapes wandb and renders the wrong run (its C-48; see `perspectives/from_views-reporting_perspective.md`). | exploratory |
+
+**Already exists externally — do NOT rebuild:** `EvaluationFrame` lives in
+`views-evaluation` (aligned pred×actual×(origin, step)). `views-frames` should
+define the **identifier/index protocol it conforms to**, and views-evaluation
+should adopt that protocol — not have its frame re-implemented here.
+
+### 4.3 The real shared primitive: `SpatioTemporalIndex`
+
+Every frame is **array + identifiers**. The identifiers — `{time, unit}` (plus
+the cm/pgm `SpatialLevel`) — and the **alignment/join logic over them** are the
+genuinely reused core. Build this once:
+
+- Fields: `time: int[N]`, `unit: int[N]`, `level: SpatialLevel` (cm/pgm), all
+  numpy, integer dtype, no NaN, length N.
+- **Same-level operations (owned here, pure-numpy, no pandas):** `intersect`,
+  `reindex`, `is_superset_of`, `argsort`, `searchsorted`-based joins over
+  `(time, unit)` **at a single `SpatialLevel`**. **This is the label-alignment
+  that today drags pandas back in** — pred↔actual join, partial-overlap
+  evaluation, same-level reindex. This alignment logic lives in the leaf
+  unconditionally.
+- **Cross-level operations (`cross_level_align`) — protocol here, data injected.**
+  The cm↔pgm **cross-level join** (country↔grid) is **not** a same-axis set op; it
+  is a one-to-many lookup against a `priogrid→country` mapping that is **injected**
+  by the consumer and **not embedded in the leaf** — the mapping is external,
+  viewser-sourced, and **time-varying** (a cell's country assignment changes by
+  month). The leaf owns only the operation signature `cross_level_align(index,
+  mapping)`. The alignment logic stays in the leaf; the alignment data (the
+  mapping) is supplied by the consumer (or a separate reference package the leaf
+  does not depend on), never fetched or versioned here — embedding versioned domain
+  data would make the leaf change for data reasons and break §8 maximal stability.
+  This resolves the falsified "domain-free cross-level" claim
+  (`critiqus/critique_02.md`); faoapi's producer-materialised metadata is the
+  existence proof (`perspectives/from_views-faoapi_perspective.md` §8.3).
+- `SpatialLevel` (currently `views-pipeline-core/domain/spatial.py`) should move
+  here — it is a tiny, stable value object that *is* part of the identifier
+  vocabulary (it defines `index_names` and `entity_column`: cm→`country_id`,
+  pgm→`priogrid_id`). It carries the *labels*, never the cross-level *mapping*.
+  Owning it here ends the bare-string `"cm"`/`"pgm"` sprawl (C-38) and the
+  `_ViewsDataset` private `_entity_id` reads (C-135). Relocate it with the C-65
+  reversed index-tuple (must be time-first `(month_id, entity)`) and the
+  `priogrid_gid`/`priogrid_id` inconsistency **fixed, not ported**.
+
+> Design heuristic: if two consumers disagree about how `(time, unit)` align **at
+> the same level**, that disagreement belongs **here**, resolved once. If they
+> disagree about *which country a cell belongs to*, that is domain reference data
+> — it belongs to the consumer / producer, never the leaf.
+
+---
+
+## 5. Abstractions / Protocols (DIP, ISP, SAP, LSP)
+
+The package exports **Protocols first, concretes second.** Consumers type against
+the protocols (DIP); a concrete frame is an implementation detail.
+
+Segregate the surface so no consumer depends on methods it does not use (ISP):
+
+- **`SpatioTemporalIndexed`** — `identifiers`, `n_rows`, `index: SpatioTemporalIndex`.
+  (What a reconciler/aligner needs.)
+- **`Sampled`** — `sample_count`, `is_sample` (the *structural* sample-axis facts).
+  Reduction over the sample axis lives in `views_frames_summarize`, not here (ADR-017).
+- **`Persistable`** — `save(dir)`, `load(dir, mmap)`.
+  (What I/O needs — and *only* I/O.)
+- **`Frame`** = the small composition the math layer needs: `values`, `index`,
+  `n_rows`. Nothing else.
+
+**LSP + composition over inheritance:** `FeatureFrame`, `PredictionFrame`,
+`TargetFrame`, … are **siblings, not a subtype chain.** Do **not** make one
+inherit another. They share behavior by (a) satisfying the same Protocols and
+(b) composing a `SpatioTemporalIndex` and a small internal validation helper —
+**not** by extending a fat base class. A subtype must be substitutable wherever
+its protocol is expected; that holds for protocol conformance, and it is exactly
+what a `CMDataset`-style inheritance tree gets wrong. The cm/pgm distinction is a
+**value** (`SpatialLevel`) carried by the index, never a class axis.
+
+> Anti-pattern, explicitly banned: a `_BaseFrame` god-class that
+> `FeatureFrame`/`PredictionFrame` extend and that accretes everyone's methods.
+> That recreates `_ViewsDataset` (C-36). Keep the base a **Protocol**; share code
+> by composition.
+>
+> **Unification model — Option C (decided, §13a).** v1 unifies **only** the shared
+> `SpatioTemporalIndex` + `_validation` + protocols + `io/`; the frame classes are
+> relocated as **separate sibling classes**, not merged. This captures the real
+> reused core (the index) at the lowest churn and zero god-class risk. A composed,
+> shared metadata header across frames (Option B) is a later upgrade *only if* a
+> third frame proves the header is genuinely reused. A shared concrete base
+> (Option A) is **rejected in writing**.
+
+---
+
+## 6. Physical layout (the repo must scream "data contracts")
+
+```
+views-frames/
+├── README.md                      # this file (the design bible)
+├── pyproject.toml                 # numpy core; [arrow] optional extra for io/arrow
+├── LICENSE
+├── src/views_frames/              # the pure data contract (numpy only, depends on nothing)
+│   ├── __init__.py                # EXPLICIT re-exports only (no `import *`)
+│   ├── index.py                   # SpatioTemporalIndex value object + alignment
+│   ├── spatial_level.py           # SpatialLevel enum (cm/pgm) — relocated here
+│   ├── protocols.py               # Frame / SpatioTemporalIndexed / Sampled / Persistable
+│   ├── _validation.py             # shared construction-time invariants (private helper)
+│   ├── feature_frame.py           # FeatureFrame              ── one concept per file
+│   ├── prediction_frame.py        # PredictionFrame
+│   ├── target_frame.py            # TargetFrame
+│   ├── conformance/               # the published contract suite consumers re-run (§9)
+│   └── io/                        # serialization adapters — SEPARATE from frames (SRP)
+│       ├── __init__.py
+│       ├── npz.py                 # native save()/load() (.npy + .npz)
+│       └── arrow.py               # flat columnar (.parquet) — the scalable disk format
+├── src/views_frames_summarize/    # sample-axis summarization OVER frames (ADR-017)
+│   ├── __init__.py                #   depends on views_frames + numpy only; never the reverse
+│   ├── collapse.py                # collapse(frame, reducer) — generic point fold
+│   ├── point.py                   # map_estimate (histogram MAP)
+│   ├── interval.py                # hdi, quantiles  → arrays aligned to the frame index
+│   └── aggregate.py               # conservation-correct cross-level aggregation
+└── tests/
+    ├── conformance/               # the published contract suite consumers re-run (see §9)
+    └── unit/
+```
+
+Layout rules (these *are* the screaming-architecture requirements):
+
+- **One main class/concept per file.** Multiple classes in a file is the
+  exception, allowed only for genuinely inseparable units.
+- **Serialization is not the frame's job.** I/O adapters live under `io/`, import
+  the frame, and change for *their own* reasons (a new store format) — not when
+  the frame's schema changes (SRP + CCP). `PredictionFrameConverter`
+  (PF↔list-in-cell DataFrame, a pipeline-core boundary format) **stays in
+  pipeline-core**; it is an adapter, not a frame concern.
+- **No dumping grounds.** A file accumulating loose helpers/types/constants/
+  classes means a boundary is wrong — split it. (`handlers.py`/`file.py`-style
+  13-class files are the failure mode we are escaping.)
+- **Explicit `__init__.py` re-exports** (named, not `import *`) so the public API
+  is statically analyzable.
+- A new developer should infer every responsibility from the file tree without
+  reading bodies.
+
+---
+
+## 7. On-disk / serialization contract (where "doesn't scale" is actually decided)
+
+The scaling failure in the platform today is the **list-in-cell `object`-dtype
+DataFrame** (a cell holds a Python list of S samples) — measured ~33× blow-up
+(C-40/C-66), and ~50–160× per-row over dense float32 in the #181 report-stage
+investigation (C-186; `perspectives/from_views-pipeline-core_perspective.md`).
+`views-frames` standardizes two scalable formats and **bans list-in-cell**:
+
+- **Native (`io/npz.py`):** `values.npy` (contiguous float32) + `identifiers.npz`.
+  Supports `mmap` load so peak RAM = working set, not full array. (This is the
+  existing `PredictionFrame.save/load`; keep it.)
+- **Interchange (`io/arrow.py`):** **flat columnar** parquet — one row per
+  `(time, unit[, sample])`, scalar cells only, zero-copy Arrow write. This is the
+  scalable replacement for the list-in-cell format and is what crosses to the
+  forecasts store / delivery. (Mirrors the existing `to_arrow_table()` path.)
+
+The **boundary adapters** that convert a frame to a *pandas/views-forecasts*
+representation (because those external stores mandate pandas) live in the
+**consumer** repo, depend on `views-frames`, and are explicitly out of scope
+here (CRP). `views-frames` makes the array authoritative; pandas becomes a thin
+edge adapter, never the internal type.
+
+---
+
+## 8. Contract evolution & versioning (SemVer for a thing N repos import)
+
+Because everyone depends on this, breakage is expensive — version it as a
+**published contract**, not as app code:
+
+- **MAJOR** (breaking): removing/renaming a field, changing a dtype or axis
+  meaning, adding a **required** identifier, tightening an invariant.
+- **MINOR** (additive, back-compatible): a new frame type, a new **optional**
+  metadata key, a new method, a new `io/` format.
+- **PATCH:** bug/doc fixes with identical contract.
+- Adding a required identifier is the canonical breaking change — prefer optional
+  + a deprecation window. Provide a `from_legacy_*` shim path when a consumer
+  format changes.
+- **SAP in practice:** if this package needs frequent MAJOR bumps, it is not
+  abstract/stable enough — push volatility *out* into consumer adapters.
+
+---
+
+## 9. Testing strategy (closes the cross-repo contract-test gap, C-30)
+
+- **Conformance suite (`tests/conformance/`):** a *published*, importable set of
+  contract tests asserting the invariants of each Protocol (round-trip
+  save/load, identifier completeness, collapse semantics, alignment laws). Every
+  consumer repo runs it in CI against its own adapters. This is the missing
+  cross-repo contract test (C-30) and the safety net that lets the frames evolve
+  without silently breaking N repos.
+- **Property tests** for `SpatioTemporalIndex` alignment (intersection is
+  commutative; align then collapse == collapse then align; etc.).
+- **No mocks needed** — frames are pure value objects over numpy. If a test needs
+  a mock, the thing under test probably doesn't belong in this package.
+
+---
+
+## 10. Migration / adoption plan (Strangler, not big-bang)
+
+1. **Stand up the package** with `SpatioTemporalIndex`, `protocols.py`,
+   `_validation.py`, and `io/npz.py`.
+2. **Relocate `PredictionFrame` here (contract-preserving, but _not_ verbatim).**
+   `PredictionFrame` today **imports pandas** and uses `pd.isna` for its identifier
+   NaN-check (`prediction_frame.py:5,68`); §3.1 forbids pandas in the core, so the
+   move is **not a verbatim copy — its identifier validation is rewritten
+   numpy-only** (the observable contract from §4.1 is preserved; the implementation
+   is not). Re-export from `views-pipeline-core/data/prediction_frame.py` as a thin
+   shim (`from views_frames import PredictionFrame`) so existing imports keep
+   working.
+3. **Unify `FeatureFrame`:** move datafactory's implementation here; datafactory
+   re-exports a shim. The twins now share `SpatioTemporalIndex` + validation.
+4. **Add `TargetFrame`** and migrate the evaluation adapter
+   (`modules/validation/adapter.py`) off pandas actuals — the highest-value early
+   win.
+5. **Relocate `SpatialLevel`** here; replace bare `"cm"`/`"pgm"` strings and
+   `_ViewsDataset._entity_id` reads with `index.level.entity_column`.
+6. **Add `io/arrow.py`**; point savers at the flat columnar format; retire
+   list-in-cell on the internal path (keep a boundary adapter only where an
+   external store mandates pandas).
+7. Consumers drop their direct `_ViewsDataset` private-internal access in favor
+   of the published frame/index protocols.
+
+Each step is independently shippable and back-compatible via shims (REP/CCP: the
+twins now release together; nothing changes that doesn't change together).
+
+---
+
+## 11. Scope boundaries — what does NOT live here
+
+- **Adapters to pandas / views-forecasts / appwrite / parquet-store** → consumer
+  repos (pipeline-core, datafactory). External stores mandate pandas; that is an
+  *edge*, not the core.
+- **`_ViewsDataset` (pandas↔tensor handler, densification)** → stays in
+  pipeline-core; it is heavy, pandas-bound, and a different stability tier.
+- **Reconciliation math, model code, report rendering, wandb, viewser** → their
+  owning repos.
+- **`EvaluationFrame`** → stays in views-evaluation; conform it to our index
+  protocol instead.
+
+If something here starts needing pandas, a `views_*` import, or app logic, it is
+in the wrong package — extract it to a consumer adapter.
+
+---
+
+## 12. Risk-register & decisions this resolves / informs
+
+Resolves or directly addresses (views-pipeline-core register): **C-36**
+(`_ViewsDataset` god class — frames replace its transport role with a published
+interface), **C-40 / C-66** (list-in-cell memory blow-up — flat columnar +
+arrays) and **C-186** (the #181 report-stage OOM — the first observed-in-production
+instance of that blow-up), **C-48** (concrete dependencies → protocols), **C-135** (private-internal
+cross-repo leakage → published interface), **C-164** (unwired `DataFetchStrategy`
+— frames give the strategy a typed payload), **C-165** (stable package, zero
+abstractions — this *is* the abstraction), **C-167** (reconciliation I/O has no
+typed contract → frame I/O contract), **C-184** (cross-repo mutation of
+`reconciled_dataframe` → immutable frames). Keystone for views-reporting **#113**
+(circular dependency) and informs **D-28** (relocate reconciliation) and **D-33**
+(collapse the `CMDataset/PGMDataset` hierarchy into a `SpatialLevel` value).
+
+From the **views-reporting** consumer (its own register; see
+`perspectives/from_views-reporting_perspective.md`) this package *forbids* its
+**C-184** (the `reconciled_dataframe` mutation) and the reporting side of
+**C-135** (private `_entity_id`/`_time_id` reads → published index protocol), and
+*enables* fixing **C-48** (wandb eval scrape → a typed `MetricFrame`) and **C-44**
+(undeclared wandb → isolated to one consumer adapter). It does **not** by itself
+resolve **C-22** (viewser metadata fetch) or **C-27** (wandb runtime dependency) —
+those remain consumer-side acquisition concerns; `views-frames` only gives their
+output a typed home. (Note: reporting's **C-48** is distinct from the
+pipeline-core **C-48** listed above — two registers, same number.)
+
+---
+
+## 13. Design decisions
+
+### 13a. Resolved (ratified 2026-06-21 — these were the blocking pre-code decisions)
+
+1. **Twin-unification model — Option C.** Unify only the shared
+   `SpatioTemporalIndex` + `_validation` + protocols + `io/`; relocate
+   `FeatureFrame`/`PredictionFrame` as **separate sibling classes**. Reject the
+   shared `_BaseFrame` (Option A); defer the composed header (Option B) until a
+   third frame proves it. See §5.
+2. **Sample axis — decided: always an explicit trailing axis (`S ≥ 1`).**
+   `PredictionFrame (N, S)`, `FeatureFrame (N, F, S)`, `TargetFrame (N, 1)`;
+   `is_sample` is `S > 1`; sample-axis reduction lives in
+   `views_frames_summarize` (point 7), not the leaf. One shape contract, no
+   `ndim` branching. See §4.1.
+3. **Metadata / identifier model — typed header + fixed identifiers.** `metadata`
+   is a **typed, optional-extensible header** (not a free-form dict — that re-opens
+   C-48 store-side and cannot be validated), carrying provenance (model, run_type,
+   timestamp, seed) and `feature_names`. Identifiers stay a fixed required
+   `{time, unit}` for v1; any future identifier (`step`, `origin`, `scenario`) is
+   added as **optional only** (MINOR), never required (a required identifier is the
+   §8 MAJOR break). This is the typed home for the C-48 / #178 run-identity cure.
+4. **Cross-level (cm↔pgm) alignment — leaf owns the protocol, consumer injects the
+   mapping.** Same-level alignment lives in the leaf; the cross-level country↔grid
+   join needs a viewser-sourced, time-varying `priogrid→country` **mapping** that
+   is **injected by the consumer and never embedded in the leaf**. The leaf owns
+   only `cross_level_align(index, mapping)`. See §4.3; resolves
+   `critiqus/critique_02.md`.
+5. **`SpatialLevel` lives here, as identifier vocabulary only** — relocated with
+   the C-65 reversed index-tuple and the `priogrid_gid`/`priogrid_id`
+   inconsistency **fixed, not ported** (§4.3). It carries the level labels, never
+   the cross-level mapping or any unit values/ranges.
+6. **`MetricFrame` / `EvaluationFrame` — out of the leaf.** `EvaluationFrame` stays
+   in views-evaluation; `MetricFrame` is keyed `(target, step, unit)` and does not
+   satisfy the §4 frame definition, so it stays **out of (the) leaf** for v1 (it
+   may re-enter only if the index protocol is *deliberately* generalised to a
+   non-spatiotemporal key — a v2 decision). The leaf may define the *key/index
+   protocol* they conform to.
+7. **Sample-axis summarization is a sibling package, not the leaf (ADR-017, v0.2.0).**
+   `collapse`/MAP/HDI/quantiles and conservation-correct cross-level aggregation move
+   to `views_frames_summarize` (numpy-only, depends on `views_frames`, import-DAG
+   enforced). The leaf keeps only the *structural* `sample_count`/`is_sample`. This
+   de-duplicates the HDI/MAP logic faoapi and reporting each re-derive, and keeps the
+   leaf free of volatile statistics. (The older prose in §4.1/§5/§7/§9/§14 that lists
+   `collapse` as a frame method predates this and is superseded by ADR-017.)
+
+### 13b. Still open (lower-stakes, resolve at/around first code)
+
+1. **Separate repo (this) vs. interim `views_pipeline_core/frames/` sub-package.**
+   This scaffold assumes the separate repo (the SDP/SAP/REP end state, and the only
+   thing that de-duplicates datafactory's `FeatureFrame`).
+2. **`TargetFrame` vs `ActualsFrame` naming** (and whether targets/actuals are one
+   type with a role flag).
+3. **Minimum numpy version / typed-array (nptyping vs bare) policy.**
+4. **Conformance-suite packaging** — it must ship as an importable artifact
+   (installable subpackage / pytest plugin) with a governed **conformance-floor**
+   version every consumer runs in CI regardless of its runtime pin (closes C-30
+   without the version-coordination paradox).
+5. **Owner + release cadence** — name the keystone's owner and the process for a
+   MAJOR bump that must land across N repos at once (governance is otherwise the
+   largest unaddressed cost for a leaf this many repos import).
+
+---
+
+## 14. Glossary
+
+- **Frame:** an immutable value object = numeric array (first axis = N rows) +
+  complete spatiotemporal identifiers, optionally with a sample axis S.
+- **Identifier:** a length-N integer array locating each row in space/time
+  (`time`, `unit`).
+- **`SpatioTemporalIndex`:** the `{time, unit, level}` triple + pure-numpy
+  alignment logic; the genuinely reused primitive.
+- **`SpatialLevel`:** cm (country-month) | pgm (PRIO-GRID-month); defines the unit
+  column.
+- **Sample axis (S):** posterior draws / ensemble members; reduced by
+  `views_frames_summarize` (e.g. `collapse(frame, reducer)`), not the leaf.
+- **list-in-cell:** the banned `object`-dtype encoding (a DataFrame cell holding a
+  Python list of samples); the actual non-scaler.
+
+---
+
+*Build against this document. If the code and this README disagree, that is a bug
+in one of them — reconcile before merging.*

views_frames-1.0.0.dist-info/RECORD

@@ -0,0 +1,27 @@
+views_frames/__init__.py,sha256=C4CKDCwVd1veYQvZnhBdNr2D7CIGRKbLP4ysR8-DIjY,1124
+views_frames/_typing.py,sha256=YtOmRz1s8KVbnO_P94lAppKlTJWxAUIRD0t-9ieGc-Q,897
+views_frames/_validation.py,sha256=0KvM8KAZdXwSC3Ew-uuaxSDvJ6q9V26DCkCgEOjJmw8,4388
+views_frames/feature_frame.py,sha256=NzBDXQoppKImhmpssbLjTnPhqAKB3SAT4WakKoKbHw8,6842
+views_frames/index.py,sha256=lAWH3lVG4V4qo8NLHiw8k6fKjc2bqAQEWi0FocAQd18,13465
+views_frames/metadata.py,sha256=mPLQTDTS-xe996DOg9zq-UfOPKxuuZRW-E80_L9rQ9c,1272
+views_frames/prediction_frame.py,sha256=nCY9c564GRjj83XoDoK6WgbsBb5V7kebDeoQs7p_SCY,5214
+views_frames/protocols.py,sha256=bMeEALEbbplJF_8Oy0OSepkxgS7Y8vEqFhMTslKAt40,2331
+views_frames/py.typed,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
+views_frames/spatial_level.py,sha256=FmMDEFVjSfCTBzEiaf0DoOxQYad_xCsQZCsyGIdjGjo,1288
+views_frames/target_frame.py,sha256=DTZ2c1QQhP8RwD7frtWfeOPk5RE7MeTYBdbpv53Jfgc,4964
+views_frames/conformance/__init__.py,sha256=A8xXgoxMuWeRiXVUyFRRWoV6fV1vXgiqiX30QO0e5r8,4777
+views_frames/io/__init__.py,sha256=zrDpTcigF7D09-ZztZFI__eDIzpbTWap-iNI6PKdJoE,445
+views_frames/io/arrow.py,sha256=TR1ONiSrSl377ovfyIv2o7qxjEs_bEdWyQq4BM9abzA,3288
+views_frames/io/npz.py,sha256=LC30HLOvpc7racUdUgvexTECKOWmnw9VjJXd2ATsBOw,2075
+views_frames_summarize/__init__.py,sha256=z-I_hkSEQ3S8nQy8yjY7Pz1Pj3qmrYz8NaSqBBPIYLo,1021
+views_frames_summarize/_common.py,sha256=3ASSbwLQTWcD3eMd2jN353_E4nKqfPurWLVNVSfSVOY,2391
+views_frames_summarize/aggregate.py,sha256=RRSxdARlc6UICGX1t0J735LuWVTkie-f9XsZZRq6fEo,3371
+views_frames_summarize/collapse.py,sha256=KBaRoMnaDDhE4EIgrDNjW3FLGyLNAN7VuLu5e8ozCSw,1437
+views_frames_summarize/conformance.py,sha256=IM97aW5aa9gMT2GDvwXPI3RV5BhGGQHic_WuRAgXKpo,1540
+views_frames_summarize/interval.py,sha256=oBUZ5aHR78gdBB7qyHAzyC7dK2R7ipeyPHqaesFaP74,2502
+views_frames_summarize/point.py,sha256=9HqWI9i4cAgMQqZ6sMe1W4Rtc8i4-09wXTG0kbrTuzE,4800
+views_frames_summarize/py.typed,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
+views_frames-1.0.0.dist-info/METADATA,sha256=s7cxDsDfgxggRN44xMWD0tLocYhYAw-dQoqeAYo9T9s,36946
+views_frames-1.0.0.dist-info/WHEEL,sha256=mffPy8wBnZQn2VnJUU5jE99KsxaSfiyMHV9Yt0aLVxs,87
+views_frames-1.0.0.dist-info/licenses/LICENSE,sha256=Pd39JkiREFciWHbwg50y6drerp2JC7dmpJaVMVPYRdo,1087
+views_frames-1.0.0.dist-info/RECORD,,

views_frames-1.0.0.dist-info/WHEEL

@@ -0,0 +1,4 @@
+Wheel-Version: 1.0
+Generator: hatchling 1.30.1
+Root-Is-Purelib: true
+Tag: py3-none-any