holoscript-trait-inference 0.1.0__py3-none-any.whl

holoscript_trait_inference-0.1.0.dist-info/METADATA

@@ -0,0 +1,193 @@
+Metadata-Version: 2.4
+Name: holoscript-trait-inference
+Version: 0.1.0
+Summary: Paper 19 (ATI) — Automated Trait Inference for HoloScript .hsplus. Phase 3 training pipeline + baselines + eval harness.
+Author: HoloScript Core
+License-Expression: MIT
+Keywords: holoscript,trait-inference,paper-19,ml
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+Requires-Dist: numpy>=1.24
+Requires-Dist: scikit-learn>=1.3
+Requires-Dist: scipy>=1.11
+Requires-Dist: pandas>=2.0
+Provides-Extra: model
+Requires-Dist: torch>=2.1; extra == "model"
+Requires-Dist: transformers>=4.40; extra == "model"
+Requires-Dist: sentence-transformers>=2.6; extra == "model"
+Requires-Dist: outlines>=0.0.40; extra == "model"
+Requires-Dist: accelerate>=0.30; extra == "model"
+Provides-Extra: dev
+Requires-Dist: pytest>=7.4; extra == "dev"
+Requires-Dist: pytest-cov>=4.1; extra == "dev"
+Requires-Dist: ruff>=0.1.0; extra == "dev"
+Requires-Dist: mypy>=1.7; extra == "dev"
+
+# trait-inference — Paper 19 (ATI) Phase 3 Pipeline
+
+Python package implementing the **frozen** Paper 19 (Automated Trait
+Inference) Phase 3 training pipeline + baselines + eval harness, per:
+
+- Spec: `ai-ecosystem/research/paper-19-trait-inference/phase-1-spec.md`
+- Pre-registration: `ai-ecosystem/research/paper-19-trait-inference/preregistration.md`
+- Brain: `ai-ecosystem/compositions/trait-inference-brain.hsplus`
+- GPU-claim ticket: `task_1777072040695_mrr3`
+
+**Status (2026-04-24)**: Phase 1 (CPU pipeline) shipped — dataset
+loader/audit/splits + 3 baselines (keyword + TF-IDF + Brittney-stub) +
+eval metrics with bootstrap CI + CLI runner + Vast.ai launcher.
+**Phase 2 (model module)** — sentence-transformer encoder + constrained-
+decoder LLM, requires `[model]` extra — pending follow-up commit.
+
+---
+
+## Quick start
+
+### 1. Install (CPU baselines + eval only)
+
+```bash
+cd packages/trait-inference
+pip install -e .
+```
+
+### 2. Smoke test (synthetic data, end-to-end)
+
+Validates the pipeline runs without needing real data or GPU. ~2 min.
+
+```bash
+trait-inference smoke --n 200 --bootstrap-b 200
+```
+
+Should emit a JSON measurement bundle to stdout with `"smoke_test": true,
+"passed": true`. Use this to validate a fresh install before committing
+to a Vast.ai run.
+
+### 3. Extract trait label space from HoloScript core
+
+```bash
+trait-inference extract-traits \
+  --constants-dir ../core/src/traits/constants/ \
+  --output trait_inference/data/trait_label_space.json \
+  --verbose
+```
+
+Reads the 113 TS constant files, extracts string-array exports, writes
+a single JSON consumed by the dataset + model modules.
+
+### 4. Audit a real dataset
+
+```bash
+trait-inference dataset audit data/atimark.jsonl --output measurements/audit.json
+```
+
+Returns exit 0 if the dataset passes spec §1.4 acceptance (≥2k pairs,
+≥300 novel combinations, ≥500 each major source, ≥200 negatives, no
+novelty leak); exit 1 with `issues` list otherwise.
+
+### 5. Run baselines
+
+```bash
+trait-inference dataset split data/atimark.jsonl --output-dir splits/ --seed 42
+trait-inference baseline run keyword --train splits/train.jsonl --eval splits/held_out_novel.jsonl --output measurements/keyword.json
+trait-inference baseline run tfidf   --train splits/train.jsonl --eval splits/held_out_novel.jsonl --val splits/val.jsonl --tune-threshold --output measurements/tfidf.json
+trait-inference baseline run brittney --train splits/train.jsonl --eval splits/held_out_novel.jsonl --output measurements/brittney.json
+```
+
+Each emits `f1_macro`, `exact_match`, `bootstrap_ci`, sample predictions.
+
+---
+
+## Vast.ai GPU launch
+
+Orchestration script: `scripts/vast-launch-paper-19.ps1` (PowerShell;
+mirrors the existing `ai-ecosystem/scripts/vast-bench-runner.ps1`
+pattern).
+
+```powershell
+# Cheapest end-to-end pipeline validation (~$0.30, ~5 min)
+.\scripts\vast-launch-paper-19.ps1 -Phase smoke -Label paper19-smoke
+
+# Run all 3 baselines on the real dataset (~$0.30, ~10 min)
+.\scripts\vast-launch-paper-19.ps1 -Phase baseline `
+  -DatasetPath data/atimark.jsonl -Label paper19-baselines
+
+# Full training run (REQUIRES preregistration.md frozen + Phase 2 model module shipped)
+.\scripts\vast-launch-paper-19.ps1 -Phase train -GpuName RTX_4090 `
+  -DatasetPath data/atimark.jsonl -Label paper19-headline-cell-1
+```
+
+Pre-flight: requires `vastai set api-key` configured (see
+`ai-ecosystem/.env` `VAST_API_KEY`); requires `~/.ssh/id_rsa` with the
+matching public key registered on the Vast.ai account; requires
+≥$0.50 credit for `train`.
+
+---
+
+## Cost estimate (per
+
+`ai-ecosystem/research/paper-19-trait-inference/README.md` Phase 2-4 task table + GPU-claim ticket `_mrr3`)
+
+| Job                                           | GPU      |                           Hours |   Cost |
+| --------------------------------------------- | -------- | ------------------------------: | -----: |
+| Smoke test                                    | RTX 4090 |                             0.1 |  $0.03 |
+| Baselines (CPU-bound)                         | RTX 4090 |                             0.2 |  $0.06 |
+| Single training cell                          | RTX 4090 |                              ~6 | ~$1.80 |
+| Full sweep (30 cells × N=5 reseed = 150 runs) | RTX 4090 | ~900 (parallel: 30 GPUs × 30hr) |  ~$240 |
+
+(A100 estimates are roughly 4-8× higher; A100 supply is also tighter.
+4090 is sufficient for ≤1B-param decoder per spec §3.1.)
+
+---
+
+## Per-spec deliverable map
+
+| Spec section                         | Module                                                              | Status                                           |
+| ------------------------------------ | ------------------------------------------------------------------- | ------------------------------------------------ |
+| §1.1 Sourcing 3-source mix           | `dataset.py` Pair + Source                                          | done (loader; data construction is Phase 2 task) |
+| §1.2 Schema                          | `dataset.py` Pair dataclass                                         | done                                             |
+| §1.3 Splits (train/val/indist/novel) | `dataset.py` make_splits                                            | done                                             |
+| §1.4 Audit protocol                  | `dataset.py` audit + AuditReport                                    | done                                             |
+| §2.1 Keyword baseline                | `baselines.py` KeywordBaseline                                      | done                                             |
+| §2.2 TF-IDF + LogReg baseline        | `baselines.py` TfidfLogregBaseline                                  | done                                             |
+| §2.3 Brittney few-shot baseline      | `baselines.py` BrittneyFewShotBaseline                              | stub (real impl needs Brittney API integration)  |
+| §3.1 Constrained-decoder model       | `model/` (Phase 2 commit)                                           | pending                                          |
+| §3.2 Conditioning fields             | `model/` (Phase 2 commit)                                           | pending                                          |
+| §3.3 Hyperparameter sweep            | `model/sweep.py` (Phase 2 commit)                                   | pending                                          |
+| §4.1 Metric definitions              | `metrics.py` f1_macro, f1_micro, exact_match_rate, bootstrap_ci     | done                                             |
+| §4.2 Statistical protocol            | `metrics.py` bootstrap_ci, evaluate_headline                        | done                                             |
+| §4.3 Ablation matrix                 | `eval/ablations.py` (Phase 2 commit)                                | pending                                          |
+| §4.4 Required user study             | (separate UX-research task)                                         | pending                                          |
+| §4.5 Pre-registration freeze         | `ai-ecosystem/research/paper-19-trait-inference/preregistration.md` | FROZEN (do not edit)                             |
+
+---
+
+## Anti-pattern guards (binding — inherited from
+
+`compositions/trait-inference-brain.hsplus`)
+
+- **No train-set evaluation.** Headline metric on novel-combination split only.
+- **No easy-split-only F1.** Reports include both indist (sanity) and novel (headline).
+- **No single-source dataset.** Audit rejects datasets <500 from any of {existing, brittney, community}.
+- **No optional user study.** §4.4 is required not optional (per F.031).
+- **No after-the-fact threshold-shopping.** preregistration.md is frozen before any Phase 3 board task is filed.
+- **No qualitative-only claims.** ML venue requires numbers; pipeline emits structured measurements.
+- **No validity gap as "scoped contribution"** — constrained-decoding architecture (Phase 2 module) bakes ≥90% validity into the decoder, not into a post-filter.
+
+---
+
+## Known limitations / future work
+
+- Brittney few-shot baseline is a stub returning empty predictions; real impl needs HoloScript MCP integration (separate task).
+- Constrained-decoder model module (`model/`) is the Phase 2 deliverable — not in this commit.
+- Training loop + ablation matrix runner pending Phase 2.
+- User study (Phase 4 §4.4) is a separate UX-research deliverable.
+- The PowerShell Vast.ai launcher targets Windows; a bash equivalent for macOS/Linux is a follow-up.
+
+---
+
+## Provenance
+
+- Authored by `trait-inference-brain` (`compositions/trait-inference-brain.hsplus`).
+- GPU-claim ticket: `task_1777072040695_mrr3` (live on team_1775935947314_f0noxi board).
+- Capability-build provenance commit: `fc294af` (lean-theorist-brain — sibling).
+- F.031 pre-emptions baked into spec; constrained decoding ships in Phase 2 model module.

holoscript_trait_inference-0.1.0.dist-info/RECORD

@@ -0,0 +1,16 @@
+trait_inference/__init__.py,sha256=xDTQIAVanZUfWnPGIR75AVMt5wyFIkQKM_nUltM39UE,860
+trait_inference/baselines.py,sha256=5T9gGDd-ZWSvS9dSJenPIeIDN4gGYBXuRKs7-x8SxvU,9430
+trait_inference/cli.py,sha256=qOqA8UKjWFhuuBhoOlxsxatVFgboWVDmLgD3wW_w0EQ,20011
+trait_inference/dataset.py,sha256=yeWf-wRnIX8gicy2Ej_XHwVhXLc-f-cfpd14zYWoePA,12644
+trait_inference/metrics.py,sha256=azwMBnbevZXGV51GIczbCbVWkf9YiOGeg1xoZV0W19U,9934
+trait_inference/eval/__init__.py,sha256=vb3NIkjYFdoeLsozAudUskKTzXL2Gf1ydpb2xebTyUE,659
+trait_inference/eval/ablations.py,sha256=tB7ktuHT6q1cJGvAuo9MsP4HllvwiX6t69mIAES4TcQ,13960
+trait_inference/model/__init__.py,sha256=2NMRfCqfec3prig_URrkuvfADnNu_mtVBBUynBMKcfA,1310
+trait_inference/model/decoder.py,sha256=phpasQIa4Y3ebws3dNWMif4hp94BPADl-R0_lsaIBcM,7840
+trait_inference/model/sweep.py,sha256=OmUA34J3AZaJ8iXUYmXcMx3ye2cLxay3kN7SAfcv6VQ,9016
+trait_inference/model/trainer.py,sha256=Nz9anThl_Uwk2ZOfZZqL41pzl0t_6l_SNpkImyhnLcc,9164
+holoscript_trait_inference-0.1.0.dist-info/METADATA,sha256=dsOQ1ZEL6hj3E6yek57WFXck4Ea50ERtt1xfQkEtabA,10062
+holoscript_trait_inference-0.1.0.dist-info/WHEEL,sha256=K260EYznzXsJYBQGqmI8VTxEdiZYNvDZwW9cBh9-_MA,91
+holoscript_trait_inference-0.1.0.dist-info/entry_points.txt,sha256=IO3AzHJLf9aNRzlM8PObHGJGX61AA2MZU-hYvQZCoVk,61
+holoscript_trait_inference-0.1.0.dist-info/top_level.txt,sha256=qwYoGcHApqn612UU2QXZSfYFdIe0uEgydsZ8LV7gY3M,16
+holoscript_trait_inference-0.1.0.dist-info/RECORD,,

holoscript_trait_inference-0.1.0.dist-info/WHEEL

@@ -0,0 +1,5 @@
+Wheel-Version: 1.0
+Generator: setuptools (83.0.0)
+Root-Is-Purelib: true
+Tag: py3-none-any
+

holoscript_trait_inference-0.1.0.dist-info/entry_points.txt

@@ -0,0 +1,2 @@
+[console_scripts]
+trait-inference = trait_inference.cli:main

holoscript_trait_inference-0.1.0.dist-info/top_level.txt

@@ -0,0 +1 @@
+trait_inference

trait_inference/__init__.py

@@ -0,0 +1,27 @@
+"""trait-inference — Paper 19 (ATI) training pipeline + baselines + eval.
+
+See ai-ecosystem/research/paper-19-trait-inference/{phase-1-spec,
+preregistration}.md for the frozen spec.
+
+Modules:
+  dataset    — schema, JSONL loader, splits, audit (CPU-only)
+  baselines  — keyword + TF-IDF + few-shot baselines (CPU-only)
+  metrics    — F1 macro, exact-match, validity rate, bootstrap CI
+  cli        — argparse runner; entrypoint `trait-inference`
+
+Optional (install [model] extra):
+  model      — sentence-transformer encoder + constrained-decoder LLM
+  trainer    — training loop (PyTorch + transformers)
+"""
+
+__version__ = "0.1.0"
+
+# Re-export commonly used types so callers can `from trait_inference import Pair`.
+from trait_inference.dataset import (  # noqa: F401
+    Pair,
+    Source,
+    Split,
+    audit,
+    load_jsonl,
+    write_jsonl,
+)

trait_inference/baselines.py

@@ -0,0 +1,238 @@
+"""Baselines — Paper 19 ATI.
+
+Per `phase-1-spec.md` §2 — ≥3 baselines required:
+  1. Keyword match (trivial; CPU; ~30-55% F1 expected on novel split)
+  2. TF-IDF + multi-label LogReg (classical; CPU; ~50-65% F1 expected)
+  3. Brittney few-shot (strong; needs Brittney API; ~60-75% F1 expected)
+
+Per `trait-inference-brain` priority 3 + `phase-1-spec.md` §2:
+  > Build baselines BEFORE contribution model. If a baseline already
+  > hits the gate, the contribution is null.
+
+Each baseline implements:
+  fit(train_pairs) -> None        # may be no-op for keyword
+  predict(description) -> list[str]
+  predict_batch(descs) -> list[list[str]]
+
+This module is CPU-only. Brittney baseline imports lazily so missing
+API key doesn't break keyword/tfidf usage.
+"""
+from __future__ import annotations
+
+import re
+from collections.abc import Iterable
+from dataclasses import dataclass, field
+from typing import Protocol
+
+import numpy as np
+from sklearn.feature_extraction.text import TfidfVectorizer
+from sklearn.linear_model import LogisticRegression
+from sklearn.multiclass import OneVsRestClassifier
+from sklearn.preprocessing import MultiLabelBinarizer
+
+from trait_inference.dataset import Pair
+
+
+# ----------------------------------------------------------------------------
+# Baseline protocol
+# ----------------------------------------------------------------------------
+
+class Baseline(Protocol):
+    """Common interface for all baselines."""
+    name: str
+
+    def fit(self, train_pairs: list[Pair]) -> None: ...
+    def predict(self, description: str) -> list[str]: ...
+    def predict_batch(self, descriptions: list[str]) -> list[list[str]]: ...
+
+
+# ----------------------------------------------------------------------------
+# Baseline 1: Keyword match (trivial)
+# ----------------------------------------------------------------------------
+
+@dataclass
+class KeywordBaseline:
+    """Trivial baseline: predict trait T if T's name (or known synonym)
+    appears in the description.
+
+    Per `phase-1-spec.md` §2.1 — expected F1 ≈ 30-55% on novel split.
+    """
+    name: str = "keyword"
+    label_space: tuple[str, ...] = ()
+    synonyms: dict[str, list[str]] = field(default_factory=dict)
+    _patterns: dict[str, re.Pattern[str]] = field(default_factory=dict, init=False)
+
+    def __post_init__(self) -> None:
+        self._compile()
+
+    def _compile(self) -> None:
+        """Build per-trait word-boundary regex from name + synonyms."""
+        self._patterns = {}
+        for trait in self.label_space:
+            # Strip @-prefix and _-separators for matching.
+            normalized = trait.lstrip("@").replace("_", " ")
+            tokens = [normalized] + self.synonyms.get(trait, [])
+            # Word-boundary regex; case-insensitive.
+            pattern = r"\b(" + "|".join(re.escape(t) for t in tokens) + r")\b"
+            self._patterns[trait] = re.compile(pattern, re.IGNORECASE)
+
+    def fit(self, train_pairs: list[Pair]) -> None:
+        # Could optionally derive synonyms from train co-occurrence here.
+        # Phase 1: synonyms come from constructor (caller-supplied).
+        return
+
+    def predict(self, description: str) -> list[str]:
+        hits = [t for t, p in self._patterns.items() if p.search(description)]
+        return sorted(hits)
+
+    def predict_batch(self, descriptions: list[str]) -> list[list[str]]:
+        return [self.predict(d) for d in descriptions]
+
+
+# ----------------------------------------------------------------------------
+# Baseline 2: TF-IDF + multi-label Logistic Regression (classical)
+# ----------------------------------------------------------------------------
+
+@dataclass
+class TfidfLogregBaseline:
+    """TF-IDF vectorizer + one-vs-rest LogReg per trait.
+
+    Per `phase-1-spec.md` §2.2 — expected F1 ≈ 50-65% on novel split.
+    Captures bag-of-words associations beyond lexical match but cannot
+    generalize across novel combinations because training only sees
+    existing combinations.
+    """
+    name: str = "tfidf_logreg"
+    threshold: float = 0.5      # decision threshold for trait inclusion
+    max_features: int = 10000
+    ngram_range: tuple[int, int] = (1, 2)
+    _vectorizer: TfidfVectorizer | None = None
+    _classifier: OneVsRestClassifier | None = None
+    _binarizer: MultiLabelBinarizer | None = None
+    _label_space: list[str] = field(default_factory=list)
+
+    def fit(self, train_pairs: list[Pair]) -> None:
+        if not train_pairs:
+            raise ValueError("cannot fit on empty training set")
+
+        descriptions = [p.description for p in train_pairs]
+        trait_sets = [list(p.trait_set) for p in train_pairs]
+
+        # Establish label space from training data
+        self._binarizer = MultiLabelBinarizer()
+        Y = self._binarizer.fit_transform(trait_sets)
+        self._label_space = list(self._binarizer.classes_)
+
+        self._vectorizer = TfidfVectorizer(
+            max_features=self.max_features,
+            ngram_range=self.ngram_range,
+            sublinear_tf=True,
+            stop_words="english",
+        )
+        X = self._vectorizer.fit_transform(descriptions)
+
+        # Multi-label via one-vs-rest. LogReg's predict_proba enables
+        # threshold tuning on the validation split.
+        self._classifier = OneVsRestClassifier(
+            LogisticRegression(max_iter=500, C=1.0, solver="liblinear"),
+            n_jobs=-1,
+        )
+        self._classifier.fit(X, Y)
+
+    def _ensure_fit(self) -> None:
+        if self._vectorizer is None or self._classifier is None or self._binarizer is None:
+            raise RuntimeError("call fit() before predict()")
+
+    def predict(self, description: str) -> list[str]:
+        return self.predict_batch([description])[0]
+
+    def predict_batch(self, descriptions: list[str]) -> list[list[str]]:
+        self._ensure_fit()
+        assert self._vectorizer is not None and self._classifier is not None
+        assert self._binarizer is not None
+        X = self._vectorizer.transform(descriptions)
+        # predict_proba returns shape (n_samples, n_labels)
+        proba = self._classifier.predict_proba(X)
+        out: list[list[str]] = []
+        for row in proba:
+            chosen = [
+                self._label_space[i]
+                for i, p in enumerate(row)
+                if p >= self.threshold
+            ]
+            out.append(sorted(chosen))
+        return out
+
+    def tune_threshold(
+        self,
+        val_pairs: list[Pair],
+        candidates: Iterable[float] = (0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8),
+    ) -> float:
+        """Find the threshold that maximizes F1 macro on the validation
+        split. Updates self.threshold and returns it."""
+        from trait_inference.metrics import f1_macro
+
+        descriptions = [p.description for p in val_pairs]
+        gold = [list(p.trait_set) for p in val_pairs]
+
+        best_thr, best_f1 = self.threshold, -1.0
+        for thr in candidates:
+            self.threshold = thr
+            preds = self.predict_batch(descriptions)
+            score = f1_macro(gold, preds, label_space=self._label_space)
+            if score > best_f1:
+                best_thr, best_f1 = thr, score
+        self.threshold = best_thr
+        return best_thr
+
+
+# ----------------------------------------------------------------------------
+# Baseline 3: Brittney few-shot (strong; lazy import — needs API access)
+# ----------------------------------------------------------------------------
+
+@dataclass
+class BrittneyFewShotBaseline:
+    """In-context-learn from K=8 stratified examples + Brittney prompt.
+
+    Per `phase-1-spec.md` §2.3 — expected F1 ≈ 60-75% on novel split.
+    Strongest baseline — represents "what you get for free from the
+    existing system."
+
+    Phase 3 implementation deferred — depends on Brittney API surface
+    decision (HoloScript MCP `brittney_*` tools). This stub captures the
+    interface so the eval harness can be tested end-to-end with mock
+    behavior; the real implementation lands when training Phase 3 starts.
+    """
+    name: str = "brittney_fewshot"
+    k_shots: int = 8
+    _examples: list[Pair] = field(default_factory=list)
+
+    def fit(self, train_pairs: list[Pair]) -> None:
+        # Stratified sampling: pick K examples that maximize trait diversity
+        # in the prompt context. Greedy: each next example adds the most
+        # not-yet-seen traits.
+        if not train_pairs:
+            self._examples = []
+            return
+        seen: set[str] = set()
+        chosen: list[Pair] = []
+        remaining = list(train_pairs)
+        while len(chosen) < self.k_shots and remaining:
+            best_idx, best_new = 0, -1
+            for i, p in enumerate(remaining):
+                new_traits = len(set(p.trait_set) - seen)
+                if new_traits > best_new:
+                    best_idx, best_new = i, new_traits
+            picked = remaining.pop(best_idx)
+            chosen.append(picked)
+            seen.update(picked.trait_set)
+        self._examples = chosen
+
+    def predict(self, description: str) -> list[str]:
+        # Phase 1 stub: return empty (will be replaced by Brittney API call
+        # in Phase 3). Empty prediction is the most honest non-call default
+        # — the eval harness will record this baseline as needing impl.
+        return []
+
+    def predict_batch(self, descriptions: list[str]) -> list[list[str]]:
+        return [self.predict(d) for d in descriptions]