@machinespirits/eval 0.2.1 → 0.3.0

package/docs/research/slides-header.tex

@@ -0,0 +1,188 @@
+% Custom Beamer styling for "Geist in the Machine" presentation
+% Uses Metropolis theme with a refined academic color palette
+
+% ─── Color Palette ───────────────────────────────────────────
+% Deep slate blue primary, warm amber accent, clean backgrounds
+\definecolor{GeistPrimary}{HTML}{1B2838}      % Dark slate (titles, bars)
+\definecolor{GeistAccent}{HTML}{D4872C}        % Warm amber (highlights, alerts)
+\definecolor{GeistLight}{HTML}{F5F2EB}         % Warm off-white (backgrounds)
+\definecolor{GeistMid}{HTML}{5C6B7A}           % Medium slate (subtitles)
+\definecolor{GeistText}{HTML}{2D3436}          % Near-black (body text)
+\definecolor{GeistTableHead}{HTML}{E8E2D6}     % Warm beige (table headers)
+\definecolor{GeistTableAlt}{HTML}{F9F7F3}      % Light warm (table alt rows)
+\definecolor{GeistPositive}{HTML}{27AE60}      % Green (positive results)
+\definecolor{GeistNegative}{HTML}{C0392B}      % Red (negative results)
+\definecolor{GeistCode}{HTML}{F0ECE3}          % Code background
+
+% ─── Metropolis Configuration ────────────────────────────────
+\metroset{
+  progressbar=frametitle,
+  sectionpage=none,
+  numbering=fraction,
+  block=fill,
+}
+
+% Apply custom colors to metropolis elements
+\setbeamercolor{normal text}{fg=GeistText, bg=white}
+\setbeamercolor{background canvas}{bg=white}
+\setbeamercolor{alerted text}{fg=GeistAccent}
+\setbeamercolor{frametitle}{fg=white, bg=GeistPrimary}
+\setbeamercolor{title separator}{fg=GeistAccent}
+\setbeamercolor{progress bar}{fg=GeistAccent, bg=GeistPrimary!20}
+\setbeamercolor{progress bar in head/foot}{fg=GeistAccent, bg=GeistPrimary!20}
+\setbeamercolor{progress bar in section page}{fg=GeistAccent, bg=GeistPrimary!20}
+\setbeamercolor{title}{fg=white}
+\setbeamercolor{subtitle}{fg=GeistAccent!80!white}
+\setbeamercolor{author}{fg=GeistLight!90!white}
+\setbeamercolor{date}{fg=GeistLight!70!white}
+\setbeamercolor{institute}{fg=GeistLight!80!white}
+\setbeamercolor{section title}{fg=white}
+\setbeamercolor{block title}{fg=white, bg=GeistPrimary}
+\setbeamercolor{block body}{bg=GeistLight}
+\setbeamercolor{block title alerted}{fg=white, bg=GeistAccent}
+\setbeamercolor{block body alerted}{bg=GeistAccent!10}
+\setbeamercolor{block title example}{fg=white, bg=GeistPositive!80!black}
+\setbeamercolor{block body example}{bg=GeistPositive!5}
+\setbeamercolor{itemize item}{fg=GeistAccent}
+\setbeamercolor{itemize subitem}{fg=GeistMid}
+\setbeamercolor{enumerate item}{fg=GeistAccent}
+\setbeamercolor{description item}{fg=GeistPrimary}
+\setbeamercolor{footnote}{fg=GeistMid}
+\setbeamercolor{page number in head/foot}{fg=GeistMid}
+
+% ─── Typography ──────────────────────────────────────────────
+\usepackage{FiraSans}
+\usepackage{FiraMono}
+
+% Title page font sizes
+\setbeamerfont{title}{size=\Large, series=\bfseries}
+\setbeamerfont{subtitle}{size=\normalsize}
+\setbeamerfont{author}{size=\small}
+\setbeamerfont{date}{size=\small}
+\setbeamerfont{institute}{size=\footnotesize}
+\setbeamerfont{frametitle}{size=\large}
+
+% ─── Title Page ──────────────────────────────────────────────
+\setbeamertemplate{title page}{
+  \begin{minipage}[b][\paperheight]{\textwidth}
+    \vfill
+    \ifx\inserttitlegraphic\@empty\else\usebeamertemplate*{title graphic}\fi
+
+    {\usebeamerfont{title}\usebeamercolor[fg]{title}\inserttitle\par}
+
+    \vspace{0.6em}
+    {\usebeamercolor{title separator}\textcolor{fg}{\rule{0.35\textwidth}{1.2pt}}\par}
+
+    \vspace{0.6em}
+    \ifx\insertsubtitle\@empty\else
+      {\usebeamerfont{subtitle}\usebeamercolor[fg]{subtitle}\insertsubtitle\par}
+      \vspace{0.4em}
+    \fi
+
+    \ifx\insertauthor\@empty\else
+      {\usebeamerfont{author}\usebeamercolor[fg]{author}\insertauthor\par}
+      \vspace{0.2em}
+    \fi
+
+    \ifx\insertinstitute\@empty\else
+      {\usebeamerfont{institute}\usebeamercolor[fg]{institute}\insertinstitute\par}
+      \vspace{0.2em}
+    \fi
+
+    \ifx\insertdate\@empty\else
+      {\usebeamerfont{date}\usebeamercolor[fg]{date}\insertdate\par}
+    \fi
+
+    \vfill
+    \vspace{1em}
+  \end{minipage}
+}
+
+% Dark background for title page
+\makeatletter
+\def\ps@titlepage{%
+  \setbeamertemplate{background}{%
+    \begin{tikzpicture}
+      \useasboundingbox (0,0) rectangle (\the\paperwidth,\the\paperheight);
+      \fill[GeistPrimary] (0,0) rectangle (\the\paperwidth,\the\paperheight);
+    \end{tikzpicture}
+  }
+}
+\addtobeamertemplate{title page}{\thispagestyle{titlepage}}{}
+\makeatother
+
+% ─── Table Styling ───────────────────────────────────────────
+\usepackage{booktabs}
+\usepackage{colortbl}
+\usepackage{array}
+
+% Redefine \toprule/\bottomrule colors
+\renewcommand{\toprule}{\arrayrulecolor{GeistPrimary}\specialrule{0.8pt}{0pt}{4pt}}
+\renewcommand{\midrule}{\arrayrulecolor{GeistPrimary!30}\specialrule{0.4pt}{3pt}{4pt}}
+\renewcommand{\bottomrule}{\arrayrulecolor{GeistPrimary}\specialrule{0.8pt}{4pt}{0pt}}
+
+% Scale tables to fit slides
+\usepackage{adjustbox}
+\let\oldtabular\tabular
+\let\endoldtabular\endtabular
+\renewenvironment{tabular}[1]{%
+  \adjustbox{max width=\textwidth}{%
+  \oldtabular{#1}%
+}{%
+  \endoldtabular}%
+}
+
+% Smaller font in tables for readability on slides
+\AtBeginEnvironment{longtable}{\footnotesize}
+
+% ─── Code Block Styling ─────────────────────────────────────
+\usepackage{etoolbox}
+\AtBeginDocument{%
+  \apptocmd{\Shaded}{\scriptsize\setlength{\fboxsep}{6pt}}{}{}%
+}
+
+% ─── Spacing & Layout ───────────────────────────────────────
+\setlength{\parskip}{0.4em}
+\linespread{1.1}
+
+% Tighter list spacing
+\setbeamertemplate{itemize/enumerate body begin}{\vspace{-0.2em}}
+\setbeamertemplate{itemize/enumerate body end}{\vspace{-0.2em}}
+\setlength{\leftmargini}{1.2em}
+\setlength{\leftmarginii}{1.2em}
+
+% ─── Blockquote Styling ─────────────────────────────────────
+% Style quotes with a left accent bar
+\usepackage{tikz}
+\renewenvironment{quote}{%
+  \begin{tikzpicture}
+  \node[
+    inner sep=8pt,
+    text width=0.88\textwidth,
+    fill=GeistLight,
+    draw=none,
+    font=\small\itshape\color{GeistText},
+    execute at begin node={\hspace{0pt}},
+  ] (quote) \bgroup
+}{%
+  \egroup;
+  \fill[GeistAccent] ([xshift=-4pt]quote.north west) rectangle ([xshift=-1pt]quote.south west);
+  \end{tikzpicture}
+}
+
+% ─── Standout colors (used only for Thank You slide) ─────────
+% Note: pandoc+metropolis standout frames leak color changes to
+% subsequent frames. Use sparingly — only as the final slide.
+
+% ─── Unicode Support ─────────────────────────────────────────
+\usepackage{unicode-math}
+\setmathfont{latinmodern-math.otf}
+\usepackage{newunicodechar}
+\newunicodechar{≈}{$\approx$}
+\newunicodechar{≥}{$\geq$}
+\newunicodechar{𝜒}{$\chi$}
+\tracinglostchars=0
+
+% ─── Misc ────────────────────────────────────────────────────
+\setbeamertemplate{caption}[numbered]
+\setbeamertemplate{navigation symbols}{}

package/docs/research/slides-pptx.md

@@ -0,0 +1,363 @@
+---
+title: "*Geist* in the Machine"
+subtitle: "Mutual Recognition and Multiagent Architecture for Dialectical AI Tutoring"
+author: "Liam Magee"
+date: "February 2026"
+bibliography: references.bib
+csl: apa.csl
+---
+
+# Motivation
+
+## The Problem
+
+Current AI tutoring treats learners as **knowledge deficits** to be filled.
+
+- Learner says something interesting → tutor redirects to curriculum
+- Learner struggles → tutor simplifies or restates
+- Learner resists → tutor notes "engagement metrics" and moves on
+
+**The learner is never encountered as a subject.**
+
+This maps onto Hegel's master--slave dialectic: the master (tutor) consumes the slave's (learner's) labor without genuine encounter.
+
+---
+
+## Hegel's Alternative: Mutual Recognition
+
+**Recognition** (*Anerkennung*): each party acknowledges the other as an autonomous consciousness whose understanding has intrinsic validity.
+
+:::::::::::::: {.columns}
+::: {.column width="50%"}
+
+**What it is**
+
+- A **relational stance**
+- How the tutor constitutes the learner
+- Achievable without consciousness
+
+:::
+::: {.column width="50%"}
+
+**What it is not**
+
+- Not agreement --- can disagree while recognizing
+- Not affirmation --- "good job!" is not recognition
+- Not a consciousness requirement
+
+:::
+::::::::::::::
+
+---
+
+# Architecture
+
+## The Drama Machine
+
+:::::::::::::: {.columns}
+::: {.column width="50%"}
+
+**Ego** (Response Generator)
+
+- Generates pedagogical suggestions
+- Has **final authority** over output
+- Can override or incorporate Superego feedback
+
+:::
+::: {.column width="50%"}
+
+**Superego** (Internal Critic)
+
+- Evaluates Ego's draft
+- Checks pedagogical quality
+- Structured critique: approve / revise / reject
+
+:::
+::::::::::::::
+
+**Recognition prompts** add Hegelian theory to both Ego and Superego:
+
+- *"Acknowledge the learner as an autonomous subject..."*
+- *"Evaluate whether the response treats the learner's understanding as having intrinsic validity..."*
+
+---
+
+## Phase 2: Advanced Mechanisms
+
+Nine architectural mechanisms tested beyond base Ego/Superego:
+
+| Mechanism | What it does |
+|:----------|:-------------|
+| Self-reflection | Ego reviews own prior performance |
+| Bidirectional profiling | Theory of Mind models of each party |
+| Intersubjective recognition | Explicit other-awareness prompts |
+| Combined (all three) | Full mechanism stack |
+| Cross-turn superego memory | Superego retains conversation context |
+| Prompt rewriting | Dynamic prompt evolution mid-dialogue |
+| Quantitative disposition | Numeric stance tracking |
+| Prompt erosion | Gradual prompt degradation test |
+
+---
+
+# Method
+
+## Evaluation Design
+
+**37 evaluations**, N=3,383 primary scored responses
+
+- **2x2x2 factorial** (N=350): Recognition x Architecture x Learner type
+- **Memory isolation** (N=120): Disentangle recognition from episodic memory
+- **Multi-model probe** (N=655): 5 ego models, architecture held constant
+- **Dynamic learner tests** (N=660): Mechanisms with feedback-capable learners
+- **Cross-judge replication** (N=977): GPT-5.2 independent validation
+
+**14-dimension rubric** scored by Claude Opus 4.6
+
+---
+
+# Results
+
+## Finding 1: Memory Isolation (The Definitive Finding)
+
+2x2 design (N=120, 30/cell) disentangles recognition from episodic memory:
+
+|  | No Memory | Memory |
+|:--|:-----------:|:--------:|
+| **No Recognition** | 75.4 | 80.2 |
+| **Recognition** | **90.6** | **91.2** |
+
+- **Recognition**: **+15.2 pts**, d=1.71, p<.001
+- **Memory**: +4.8 pts, d=0.46, n.s.
+- **Interaction**: -4.2 pts (ceiling effect, not synergy)
+
+Recognition alone accounts for nearly the entire improvement.
+
+---
+
+## Finding 2: Full Factorial (2x2x2)
+
+N=350, Kimi K2.5 ego, Opus 4.6 judge:
+
+| Cell | Recog | Arch | Learner | M (SD) |
+|:------:|:-------:|:------:|:---------:|:--------:|
+| 1 | - | Single | Single | 73.4 (16.2) |
+| 2 | - | Multi | Single | 69.9 (23.3) |
+| 3 | - | Single | Multi | 75.5 (15.2) |
+| 4 | - | Multi | Multi | 75.2 (18.1) |
+| 5 | + | Single | Single | **90.2** (7.1) |
+| 6 | + | Multi | Single | **83.9** (18.1) |
+| 7 | + | Single | Multi | **90.1** (7.1) |
+| 8 | + | Multi | Multi | **87.3** (10.3) |
+
+**Recognition**: **+14.4 pts**, F(1,342)=110.04, p<.001, d=1.11
+
+---
+
+## Finding 3: Architecture is Additive
+
+Multi-model probe (N=655, 5 ego models):
+
+| Model | Base | +Arch | +Recog | +Both | AxB |
+|:-------|:------:|:-------:|:--------:|:-------:|:-----:|
+| Kimi K2.5 | 73.4 | 75.5 | **90.2** | 90.1 | +0.5 |
+| Haiku | 78.2 | 81.9 | **93.3** | 93.5 | -3.7 |
+| DeepSeek-R1 | 71.1 | 71.3 | **88.9** | 83.2 | -5.7 |
+| GLM-4.7 | 63.9 | 62.2 | **73.5** | 74.9 | +3.1 |
+| Nemotron | 62.3 | 62.6 | **78.2** | 72.5 | -5.7 |
+
+- AxB interaction: -5.7 to +3.1 (mean -1.8) --- **no synergy**
+- Recognition range: +9.6 to +17.8 across all models
+
+---
+
+## Finding 4: Domain Generalizability
+
+Recognition effect across 6 tutorial domains (N=60):
+
+| Domain | Base | Recog | Delta |
+|:--------|:------:|:-------:|:---:|
+| Climate science | 72.0 | 93.8 | **+21.8** |
+| Ethics | 72.3 | 89.3 | **+17.0** |
+| Mathematics | 73.0 | 89.2 | **+16.2** |
+| Philosophy | 75.2 | 89.7 | **+14.5** |
+| Machine learning | 78.0 | 91.5 | **+13.5** |
+| Poetry | 86.0 | 92.5 | +6.5 |
+
+Strong for conceptual domains (+14 to +22 pts). Weakest for poetry (+6.5) --- high baseline leaves less room for improvement.
+
+---
+
+## Finding 5: Scripted vs. Dynamic Learners
+
+:::::::::::::: {.columns}
+::: {.column width="50%"}
+
+**Scripted learners**
+
+- Pre-written responses
+- 9 mechanisms cluster within 2.4 pts
+- No differentiation --- noise floor
+
+:::
+::: {.column width="50%"}
+
+**Dynamic learners**
+
+- LLM-generated, ego/superego
+- Mechanisms spread 5+ pts
+- Recognition doubles: +7.6 → **+14.8**
+
+:::
+::::::::::::::
+
+**Lesson**: Mechanism effects require genuine feedback loops to manifest.
+
+---
+
+## Finding 6: Dynamic Learner Mechanisms
+
+Complete 2x4 matrix (N=480, Haiku ego, dynamic learner):
+
+| Mechanism | Base | Recog | Delta |
+|:-----------|:------:|:-------:|:---:|
+| Self-reflection | 72.3 | 85.6 | +13.3 |
+| Bidirectional profiling | 74.6 | **88.8** | +14.2 |
+| Intersubjective | 67.7 | 82.8 | +15.1 |
+| Combined | 73.7 | 87.8 | +14.1 |
+
+- Variance collapses with added mechanisms (SD: 22.5 → 11.8)
+- Recognition delta stable (+13.3 to +15.1) regardless of mechanism
+- Profiling = highest ceiling; intersubjective = lowest floor
+
+---
+
+## Finding 7: Cognitive Prosthesis Fails
+
+Can a strong Superego (Kimi K2.5) compensate for a weak Ego (Nemotron)?
+
+**No.** Full mechanism stack scores **49.5** --- that's **-15 pts below** Nemotron simple base (64.2)
+
+- Same mechanisms boost Haiku by +20 pts but hurt Nemotron by -15 pts
+- Static dimensions fine (spec accuracy 4.0); dynamic dimensions fail (adaptation 1.8)
+- Parse failures: Kimi returns malformed JSON 16--45% of turns
+
+**Minimum ego capability threshold**: The mechanisms amplify what the Ego can already do --- they cannot substitute for missing capability.
+
+---
+
+## Finding 8: Cross-Judge Robustness
+
+GPT-5.2 independently rejudged N=977 paired responses:
+
+| Finding | Claude | GPT-5.2 | Replicates? |
+|:---------|:--------:|:---------:|:-------------:|
+| Recognition (memory) | d=1.71 | d=1.54 | Yes |
+| Memory effect | d=0.46 | d=0.49 | Yes (small) |
+| Architecture effect | +2.6 | -0.2 | Yes (null) |
+| Mechanism clustering | 2.8 pt | 4.4 pt | Yes (null) |
+
+- Inter-judge r = 0.44--0.64 (all p<.001)
+- GPT-5.2 finds 37--59% of Claude's effect magnitudes
+- Always same direction --- **no sign reversals**
+
+---
+
+# Qualitative Evidence
+
+## What Recognition Looks Like
+
+**Base tutor** to a struggling learner:
+
+> "You left off at the neural networks section. Complete this lecture to maintain your learning streak."
+
+**Recognition tutor** to the same learner:
+
+> "This is your third session --- you've persisted through quiz-479-3 three times, which signals you're wrestling with how recognition operates in the dialectic..."
+
+Three systematic changes:
+
+1. The ego **listens to its internal critic** (superego feedback incorporated)
+2. The tutor **builds on learner contributions** (not redirecting to curriculum)
+3. **Mid-conversation strategy shifts** occur (30% of recognition dialogues vs 0% base)
+
+---
+
+## Dialectical Impasse: The Strongest Test
+
+Three 5-turn scenarios with escalating resistance (N=24):
+
+- **Epistemic resistance** (Popperian critique): Recognition **+43 pts**
+- **Productive deadlock** (incompatible frameworks): Recognition **+29 pts**
+- **Affective shutdown** (emotional retreat): Recognition -1.1 (null)
+
+Resolution strategy coding (chi-square=24.00, p<.001, V=1.000):
+
+- **Base**: 12/12 withdraw from encounter entirely
+- **Recognition**: 10/12 scaffolded reframing (*Aufhebung*), 1 mutual recognition, 1 domination
+
+The null on affective shutdown sharpens the claim: recognition's contribution is **epistemological**, not primarily affective.
+
+---
+
+## The Learner Superego Paradox
+
+Multi-agent learner architecture **hurts** learner quality (d=1.43, F=68.28, p<.001):
+
+- Designed to improve through internal self-critique
+- Actually over-edits --- polishes away messy, authentic engagement
+- Recognition partially rescues multi-agent learner (d=0.79, p=.004)
+
+**Hegelian interpretation**: External recognition from an Other is structurally more effective than internal self-critique. You cannot bootstrap genuine dialogue from a monologue.
+
+---
+
+# Implications
+
+## Practical Recommendations
+
+1. **Add recognition prompts** --- immediate +14 pt improvement, no architecture changes needed
+2. **Architecture is optional** --- modest additive benefit (+2 pts), not required
+3. **Use dynamic learners** for testing --- scripted learners mask mechanism effects
+4. **Theory of Mind profiling** --- best mechanism for ceiling performance
+5. **Token budgets can be cut 4--16x** with no quality loss
+6. **Minimum ego capability matters** --- mechanisms amplify, don't substitute
+
+---
+
+## Limitations
+
+1. **Simulated learners, not humans** --- all "learners" are LLM agents
+2. **LLM-as-judge** --- Claude Opus evaluates (mitigated by GPT-5.2 cross-judge)
+3. **Single content domain** --- primarily philosophy of education
+4. **No longitudinal data** --- snapshots, not learning trajectories
+5. **Prompt-level intervention** --- recognition embedded in prompts, not weights
+6. **Small N per cell** --- 30 observations per condition in key experiments
+
+---
+
+## Conclusion
+
+**Recognition theory** produces robust, replicable improvements in AI tutoring quality:
+
+- d=1.11 to d=1.71 depending on experiment
+- Replicates across 5 models, 6 domains, 2 judges
+- Survives all controls: memory isolation, prompt elaboration, token budget
+
+**Multi-agent architecture** contributes additively but modestly.
+
+**The key insight**: Philosophical theories of intersubjectivity can serve as productive design heuristics for AI systems. Recognition is better understood as an achievable relational stance than a requirement for machine consciousness.
+
+---
+
+## Thank You
+
+**Paper**: *Geist* in the Machine (v2.3.14)
+
+**37 evaluations** | **N=3,383 scored** | **5 ego models** | **2 judges**
+
+Liam Magee
+
+Education Policy, Organization and Leadership
+
+University of Illinois Urbana-Champaign
+