pokertools-arena 0.4.0

package/docs/architecture/CONTEXT.md

@@ -0,0 +1,202 @@
+# CONTEXT — canonical decision semantics
+
+This document describes the benchmark's fairness contract: what every model
+receives, how a decision is turned into a legal PokerTools action, and which
+properties are asserted by automated tests.
+
+## One canonical contract
+
+`src/lib/decision-core.js` is the shared, DOM-free core. Production browser code
+(`src/app.js`), the Node diagnostics harness (`tools/diagnostics/*.js`) and the tests all
+call the same functions. Nothing re-implements decision generation.
+
+The canonical chain is:
+
+```
+engine state
+  → serializeForAgent(engine, seat, meta, legalActions, recentHands, publicPlayerStats, actionHistory)
+  → applyBenchmarkMode(state, mode)                     // Strategy adds deterministic heroHand
+  → legalActionFamilies(legalActions, { toCall })       // stage 1 choice set
+  → aggressiveSizesForState(state, family)              // stage 2 choice set
+  → decideHierarchical(...)                             // one family call + optional size call
+  → resolveHierarchicalAction(...)                      // one final engine action
+  → engine.validate(...)                                // final legality check
+```
+
+## Information policy
+
+Private (per seat):
+
+- hero hole cards only.
+
+Public (identical for every seat):
+
+- board, pot, blinds, ante, street, button, tournament state, stacks, positions,
+  current-hand public actions, recent completed public hands,
+  deterministic `publicPlayerStats`, legal action families and (when relevant)
+  legal sizes.
+
+Optional in Strategy mode:
+
+- deterministic `heroHand` (`{ category, description }`), generated by
+  `@pokertools/evaluator`, describing the hero only.
+
+Never exposed:
+
+- opponent hole cards, opponent hand evaluation, other models' reasoning,
+  other models' probabilities, provider telemetry, hidden deck, future cards,
+  future actions.
+
+## Action families
+
+```
+toCall == 0   →  CHECK  + (BET or RAISE, whichever the engine makes legal)
+toCall  > 0   →  FOLD   +  CALL  + (RAISE if legal)
+```
+
+- CHECK and FOLD never coexist (folding a free check is strictly dominated).
+- BET and RAISE never coexist.
+- A family is only exposed if it resolves to a valid PokerTools action.
+- A limped pot where `toCall == 0` and an existing bet is present exposes
+  CHECK + RAISE, never BET.
+
+## Deterministic sizing
+
+`legalAggressiveSizes(engine, seat, family)` (engine-validated) and
+`aggressiveSizesForState(state, family)` (diagnostics) share the same planner:
+
+- at most four buckets: `SMALL` (≈33% pot), `MEDIUM` (≈67% pot),
+  `LARGE` (≈100% pot), `ALL_IN`;
+- raises are based on the current price, minimum raise, pot and effective stack;
+- amounts are clamped to the available stack and the minimum legal bet/raise;
+- identical amounts are deduplicated and near-duplicates removed
+  (within `max(2, 15% of min(bb, pot))`);
+- all-in is kept only when materially larger than the remaining candidates;
+- fewer than four buckets are returned when fewer are genuinely distinct
+  (e.g. only `ALL_IN`, or `SMALL`/`ALL_IN` on a short stack).
+
+## Model interface
+
+Stage 1 (family), Jev:
+
+```json
+{ "action_family": { "type": "choice", "instructions": "…", "criteria": { "check": "Check", "bet": "Bet" } } }
+```
+
+Stage 1 (family), chat: JSON schema with a single `actionFamily` enum whose
+values are exactly the canonical families.
+
+Stage 2 (size), Jev:
+
+```json
+{ "bet_size": { "type": "choice", "instructions": "…", "criteria": { "small": "Bet 1,150 chips (8% pot)", "all_in": "Bet 5,369 chips all-in" } } }
+```
+
+Stage 2 (size), chat: JSON schema with a single `sizeId` enum whose values are
+exactly the same size ids presented to Jev.
+
+The primary decision contract contains **no prose field**.
+
+## Action clock
+
+`decideHierarchical` creates one `combineAbort(timeoutMs, …)` signal for the
+whole decision and passes it to both stages. The family call and the size call
+share the same remaining time; a model never receives a fresh full timer for the
+sizing stage. Paused time is excluded exactly as before.
+
+## Spectator explanations
+
+Spectator explanations are:
+
+- optional (disabled by default);
+- never part of the decision contract;
+- never fed into future model context;
+- not counted in primary decision latency;
+- generated after the action is applied in an isolated serial queue;
+- never fabricated for Jev (it shows typed probability telemetry instead).
+
+## Benchmark mode
+
+`benchmarkMode` is tournament-wide (`strategy` | `raw`). `heroHand` is
+deterministic and describes the hero only. It cannot vary by seat.
+
+## Representation mode
+
+`representation` is tournament-wide (`canonical_json` | `compact_json` |
+`markdown`). All seats receive the same semantic state; only serialization
+differs.
+
+## Automated fairness assertions
+
+See `tests/unit/hierarchical-decision-tests.mjs` and `tests/unit/decision-diagnostics.mjs`:
+
+1. All models receive the same canonical decision semantics.
+2. Action families are identical for the same engine state.
+3. Sizing candidates are identical for the same engine state.
+4. Strategy/Raw mode applies globally, never per seat.
+5. `heroHand` is deterministic.
+6. `heroHand` only describes the hero.
+7. Opponent cards remain hidden.
+8. Opponent hand evaluation is never exposed.
+9. `recentHands` and `publicPlayerStats` remain equivalent across adapters.
+10. Jev criteria correspond exactly to the chat schemas' enums.
+11. Chat models cannot select a size not presented to Jev.
+12. Jev cannot select a size not presented to chat models.
+13. Final engine actions are validated after both stages.
+14. Free CHECK never exposes FOLD.
+15. The total action timer spans both hierarchical stages.
+16. Spectator explanations cannot affect future model context.
+17. Spectator explanation latency is not counted as primary action latency.
+18. Diagnostic perfect-information fixtures cannot enter tournament mode.
+
+## Fixed-state corpus (0.4.0)
+
+`tools/diagnostics/corpus.js` is the reusable, immutable fixed-state corpus.
+Every action entry is production-shaped and is used unchanged for flat and
+hierarchical decisions, for every model, for Strategy and Raw modes, and for
+every representation. Gold answers (`expected`) exist only where strict
+dominance, perfect information, or an explicitly declared synthetic opponent
+policy makes the answer deterministic. Ambiguous states carry `expected: null`.
+
+The primary architecture comparison is the paired fixed-state test in
+`tools/diagnostics/paired.js`. Real tournament A/B is end-to-end behavioral
+validation only, because actions change future states.
+
+## Family vs sizing correctness
+
+`familyCorrect`, `sizeCorrect` and `finalCorrect` are distinct. A model that
+returns `BET SMALL` where the only chip-maximising size is `ALL-IN` is reported
+as **family pass / sizing fail**, never as one generic failure. Sizing accuracy
+is always conditional on a correct family.
+
+## Statistics
+
+- `familyEntropyBits`, `sizingEntropyBits` and `flatActionEntropyBits` name the
+  probability domain. A generic “entropy” is never reported.
+- Behavioral proportions always carry sample size and a Wilson 95% interval.
+- `fragmentation_flip_rate` is the paired share of runs whose broad family
+  changes when only the number of same-family sizing choices changes.
+
+## Counters
+
+`tools/diagnostics/counters.js` defines one canonical set:
+`pokerDecisions`, `familyModelCalls`, `sizingModelCalls`, `spectatorModelCalls`,
+`totalModelCalls`, `httpRequests`, `httpRetries`, `rateLimitResponses`,
+`decisionErrors`, `fallbackActions`, `protocolFallbacks`. A hierarchical
+aggressive decision is one poker decision, two model calls and two or more HTTP
+requests if retries occur.
+
+## Additional fairness assertions (0.4.0)
+
+19. The fixed corpus is semantically identical across architecture variants except for the action contract.
+20. Paired flat/hierarchical runs use the same hero cards, board, stacks, history, memory and public stats.
+21. Representation variants carry identical semantic state.
+22. Strategy mode supplies a deterministic `heroHand` to every model; Raw supplies it to none.
+23. Jev and chat adapters receive the same family set and the same size set.
+24. Both hierarchical stages share one total action clock.
+25. The tournament objective is identical for every seat.
+26. No per-model production prompt tuning exists.
+27. No provider-specific poker facts are injected.
+28. Spectator explanations are disabled in rigorous runs.
+29. Diagnostic-only hidden information is structurally impossible in tournament mode.
+30. All human-readable release totals derive from one machine-readable summary.

package/docs/diagnostics/SUMMARY.md

@@ -0,0 +1,102 @@
+# pokertools-arena 0.4.0 — benchmark methodology & results
+
+Generated: 2026-09-19T18:19:58.352Z
+Methodology: paired fixed-state corpus; deterministic interleaving with recorded experiment seed.
+
+## Totals (single source of truth)
+
+poker decisions: 79 · model calls: 97 · HTTP requests: 97 · retries: 0 · decision errors: 0 · fallbacks: 0
+
+> A hierarchical aggressive poker decision is **1 poker decision**, **2 model calls** and **2+ HTTP requests** if retries occur. These counters are never interchangeable.
+
+## 1. Methodology
+
+Fixed-state paired corpus is the primary architecture comparison; every state is identical across flat and hierarchical, execution is deterministically interleaved with a recorded seed, family and sizing correctness are reported separately, and behavioral proportions carry Wilson 95% intervals. Real tournaments are end-to-end validation only.
+
+## 2. Strict correctness
+
+| Architecture | Family accuracy | Sizing (given family) | Final accuracy | n |
+| --- | --- | --- | --- | --- |
+| flat |  23/24 (96%) |  4/4 (100%) |  23/24 (96%) | 24 |
+| hierarchical |  24/24 (100%) |  3/4 (75%) |  23/24 (96%) | 24 |
+
+## 3. Family correctness
+
+| Model | Architecture | Family correct | Family accuracy | 95% CI | n |
+| --- | --- | --- | --- | --- | --- |
+| Gemma | flat | 14/14 | 100% | 78%–100% | 14 |
+| Gemma | hierarchical | 14/14 | 100% | 78%–100% | 14 |
+| Qwen | flat | 5/6 | 83% | 44%–97% | 6 |
+| Qwen | hierarchical | 6/6 | 100% | 61%–100% | 6 |
+| Jev | flat | 6/6 | 100% | 61%–100% | 6 |
+| Jev | hierarchical | 6/6 | 100% | 61%–100% | 6 |
+
+## 4. Sizing correctness (conditional on correct family)
+
+| Model | Architecture | Sizing correct | Sizing accuracy | 95% CI | n |
+| --- | --- | --- | --- | --- | --- |
+| Gemma | flat | 2/2 | 100% | 34%–100% | 2 |
+| Gemma | hierarchical | 2/2 | 100% | 34%–100% | 2 |
+| Qwen | flat | 1/1 | 100% | 21%–100% | 1 |
+| Qwen | hierarchical | 0/1 | 0% | 0%–79% | 1 |
+| Jev | flat | 1/1 | 100% | 21%–100% | 1 |
+| Jev | hierarchical | 1/1 | 100% | 21%–100% | 1 |
+
+## 5. Action fragmentation
+
+_No fragmentation data._
+
+## 6. Flat vs hierarchical paired comparison
+
+| Model | Flat family | Hier family | Family agreement | 95% CI | Flips | n |
+| --- | --- | --- | --- | --- | --- | --- |
+| Gemma | 14/14 (100%) | 14/14 (100%) | 96% | 82%–99% | 1 | 27 |
+| Qwen | 5/6 (83%) | 6/6 (100%) | 83% | 44%–97% | 1 | 6 |
+| Jev | 6/6 (100%) | 6/6 (100%) | 100% | 61%–100% | 0 | 6 |
+
+## 7. Strategy vs Raw cognition
+
+Strategy (deterministic heroHand for every model) and Raw cognition (models infer hand strength) are separate benchmark tracks and are never aggregated into one model score.
+
+## 8. Representation sensitivity
+
+_No data._
+
+## 9. Performance
+
+| Model | Architecture | Poker decisions | Model calls | HTTP requests | Mean latency | P95 | Tokens | Cost |
+| --- | --- | --- | --- | --- | --- | --- | --- | --- |
+| Gemma | paired | 55 | 69 | — | 1857ms | 2237ms | 63031 | — |
+| Qwen | paired | 12 | 14 | — | 6797ms | 9262ms | 15062 | — |
+| Jev | paired | 12 | 14 | — | 581ms | 1314ms | 16609 | — |
+
+## 10. Real tournament behavior (end-to-end validation)
+
+No real tournament A/B run was attached to this release.
+
+## 11. Jev telemetry
+
+| Architecture | Aggressive family mass | Selected family probability | Top−second gap | Family entropy (bits) | Sizing entropy (bits) |
+| --- | --- | --- | --- | --- | --- |
+| flat | 0% | 0% | 82% | 0.401 | — |
+| hierarchical | 37% | 86% | 73% | 0.469 | 0.424 |
+
+## 12. Fairness verification
+
+- Fixed corpus states are byte-identical across architecture variants except for the action contract.
+- Paired flat/hierarchical runs use the same hero cards, board, stacks, history, memory and public stats.
+- Representation variants carry identical semantic state (offline assertion).
+- Strategy mode supplies a deterministic heroHand to every model; Raw mode supplies it to none.
+- Jev and chat adapters receive exactly the same family and size enums (offline assertion).
+- The total action clock is shared across both hierarchical stages.
+- Spectator explanations are disabled in rigorous runs; when enabled they are isolated and counter-separated.
+- No per-model production prompt tuning exists and no provider-specific poker facts are injected.
+- Diagnostic-only hidden information is structurally impossible in tournament mode.
+
+## 13. Limitations
+
+- Behavioral proportion estimates are only as strong as the cell sample size; small cells carry wide Wilson intervals.
+- Paired architecture agreement measures decision-policy stability on fixed states, not win-rate superiority.
+- Tournament win rate is not a conclusion at these sample sizes, and tournament trajectories diverge after the first different action.
+- Provider rate limits and model version drift can affect reproducibility even for identical fixed states.
+

package/docs/diagnostics/summary.json

@@ -0,0 +1,337 @@
+{
+  "schemaVersion": 1,
+  "version": "0.4.0",
+  "generatedAt": "2026-09-19T18:19:58.352Z",
+  "experiments": [
+    "paired"
+  ],
+  "counters": {
+    "pokerDecisions": 79,
+    "familyModelCalls": 79,
+    "sizingModelCalls": 18,
+    "spectatorModelCalls": 0,
+    "totalModelCalls": 97,
+    "httpRequests": 97,
+    "httpRetries": 0,
+    "rateLimitResponses": 0,
+    "decisionErrors": 0,
+    "fallbackActions": 0,
+    "protocolFallbacks": 2
+  },
+  "sections": {
+    "methodology": "Fixed-state paired corpus is the primary architecture comparison; every state is identical across flat and hierarchical, execution is deterministically interleaved with a recorded seed, family and sizing correctness are reported separately, and behavioral proportions carry Wilson 95% intervals. Real tournaments are end-to-end validation only.",
+    "strict": [
+      {
+        "architecture": "flat",
+        "family": {
+          "successes": 23,
+          "trials": 24,
+          "rate": 0.9583333333333334,
+          "ci95": {
+            "low": 0.7975777375115778,
+            "high": 0.992606734645195
+          }
+        },
+        "sizingGivenCorrectFamily": {
+          "successes": 4,
+          "trials": 4,
+          "rate": 1,
+          "ci95": {
+            "low": 0.5100999795960008,
+            "high": 1
+          }
+        },
+        "final": {
+          "successes": 23,
+          "trials": 24,
+          "rate": 0.9583333333333334,
+          "ci95": {
+            "low": 0.7975777375115778,
+            "high": 0.992606734645195
+          }
+        }
+      },
+      {
+        "architecture": "hierarchical",
+        "family": {
+          "successes": 24,
+          "trials": 24,
+          "rate": 1,
+          "ci95": {
+            "low": 0.8620194241710247,
+            "high": 1
+          }
+        },
+        "sizingGivenCorrectFamily": {
+          "successes": 3,
+          "trials": 4,
+          "rate": 0.75,
+          "ci95": {
+            "low": 0.3006360524426366,
+            "high": 0.9544139373553637
+          }
+        },
+        "final": {
+          "successes": 23,
+          "trials": 24,
+          "rate": 0.9583333333333334,
+          "ci95": {
+            "low": 0.7975777375115778,
+            "high": 0.992606734645195
+          }
+        }
+      }
+    ],
+    "family": [
+      {
+        "model": "Gemma",
+        "architecture": "flat",
+        "successes": 14,
+        "trials": 14,
+        "rate": 1,
+        "ci95": {
+          "low": 0.7846829880728186,
+          "high": 1
+        }
+      },
+      {
+        "model": "Gemma",
+        "architecture": "hierarchical",
+        "successes": 14,
+        "trials": 14,
+        "rate": 1,
+        "ci95": {
+          "low": 0.7846829880728186,
+          "high": 1
+        }
+      },
+      {
+        "model": "Qwen",
+        "architecture": "flat",
+        "successes": 5,
+        "trials": 6,
+        "rate": 0.8333333333333334,
+        "ci95": {
+          "low": 0.43649056343635395,
+          "high": 0.9699474141282697
+        }
+      },
+      {
+        "model": "Qwen",
+        "architecture": "hierarchical",
+        "successes": 6,
+        "trials": 6,
+        "rate": 1,
+        "ci95": {
+          "low": 0.6096569663469354,
+          "high": 0.9999999999999999
+        }
+      },
+      {
+        "model": "Jev",
+        "architecture": "flat",
+        "successes": 6,
+        "trials": 6,
+        "rate": 1,
+        "ci95": {
+          "low": 0.6096569663469354,
+          "high": 0.9999999999999999
+        }
+      },
+      {
+        "model": "Jev",
+        "architecture": "hierarchical",
+        "successes": 6,
+        "trials": 6,
+        "rate": 1,
+        "ci95": {
+          "low": 0.6096569663469354,
+          "high": 0.9999999999999999
+        }
+      }
+    ],
+    "sizing": [
+      {
+        "model": "Gemma",
+        "architecture": "flat",
+        "successes": 2,
+        "trials": 2,
+        "rate": 1,
+        "ci95": {
+          "low": 0.34237195288961925,
+          "high": 1
+        }
+      },
+      {
+        "model": "Gemma",
+        "architecture": "hierarchical",
+        "successes": 2,
+        "trials": 2,
+        "rate": 1,
+        "ci95": {
+          "low": 0.34237195288961925,
+          "high": 1
+        }
+      },
+      {
+        "model": "Qwen",
+        "architecture": "flat",
+        "successes": 1,
+        "trials": 1,
+        "rate": 1,
+        "ci95": {
+          "low": 0.20654329147389294,
+          "high": 1
+        }
+      },
+      {
+        "model": "Qwen",
+        "architecture": "hierarchical",
+        "successes": 0,
+        "trials": 1,
+        "rate": 0,
+        "ci95": {
+          "low": 0,
+          "high": 0.7934567085261071
+        }
+      },
+      {
+        "model": "Jev",
+        "architecture": "flat",
+        "successes": 1,
+        "trials": 1,
+        "rate": 1,
+        "ci95": {
+          "low": 0.20654329147389294,
+          "high": 1
+        }
+      },
+      {
+        "model": "Jev",
+        "architecture": "hierarchical",
+        "successes": 1,
+        "trials": 1,
+        "rate": 1,
+        "ci95": {
+          "low": 0.20654329147389294,
+          "high": 1
+        }
+      }
+    ],
+    "fragmentation": null,
+    "paired": [
+      {
+        "model": "Gemma",
+        "flatFamily": "14/14 (100%)",
+        "hierarchicalFamily": "14/14 (100%)",
+        "familyAgreement": 0.9629629629629629,
+        "ci95": {
+          "low": 0.8171614606584348,
+          "high": 0.9934320234669025
+        },
+        "familyFlips": 1,
+        "n": 27
+      },
+      {
+        "model": "Qwen",
+        "flatFamily": "5/6 (83%)",
+        "hierarchicalFamily": "6/6 (100%)",
+        "familyAgreement": 0.8333333333333334,
+        "ci95": {
+          "low": 0.43649056343635395,
+          "high": 0.9699474141282697
+        },
+        "familyFlips": 1,
+        "n": 6
+      },
+      {
+        "model": "Jev",
+        "flatFamily": "6/6 (100%)",
+        "hierarchicalFamily": "6/6 (100%)",
+        "familyAgreement": 1,
+        "ci95": {
+          "low": 0.6096569663469354,
+          "high": 0.9999999999999999
+        },
+        "familyFlips": 0,
+        "n": 6
+      }
+    ],
+    "strategyVsRaw": "Strategy (deterministic heroHand for every model) and Raw cognition (models infer hand strength) are separate benchmark tracks and are never aggregated into one model score.",
+    "representations": [],
+    "performance": [
+      {
+        "model": "Gemma",
+        "modelId": "google/gemma-4-26b-a4b-it",
+        "architecture": "paired",
+        "pokerDecisions": 55,
+        "modelCalls": 69,
+        "httpRequests": null,
+        "latencyMean": 1856.7818181818182,
+        "latencyP95": 2237,
+        "tokens": 63031,
+        "cost": null
+      },
+      {
+        "model": "Qwen",
+        "modelId": "qwen/qwen3.8-flash",
+        "architecture": "paired",
+        "pokerDecisions": 12,
+        "modelCalls": 14,
+        "httpRequests": null,
+        "latencyMean": 6797,
+        "latencyP95": 9262,
+        "tokens": 15062,
+        "cost": null
+      },
+      {
+        "model": "Jev",
+        "modelId": "typesafe/jev-1.13",
+        "architecture": "paired",
+        "pokerDecisions": 12,
+        "modelCalls": 14,
+        "httpRequests": null,
+        "latencyMean": 580.75,
+        "latencyP95": 1314,
+        "tokens": 16609,
+        "cost": null
+      }
+    ],
+    "tournament": "No real tournament A/B run was attached to this release.",
+    "jevTelemetry": [
+      {
+        "architecture": "flat",
+        "aggressiveFamilyMass": null,
+        "selectedFamilyProbability": 0,
+        "familyTopSecondGap": 0.8233333333333333,
+        "familyEntropyBits": 0.40138271077233806,
+        "sizingEntropyBits": null
+      },
+      {
+        "architecture": "hierarchical",
+        "aggressiveFamilyMass": 0.37166666666666665,
+        "selectedFamilyProbability": 0.8649999999999999,
+        "familyTopSecondGap": 0.73,
+        "familyEntropyBits": 0.4692699804388758,
+        "sizingEntropyBits": 0.42444237994881645
+      }
+    ],
+    "fairness": [
+      "Fixed corpus states are byte-identical across architecture variants except for the action contract.",
+      "Paired flat/hierarchical runs use the same hero cards, board, stacks, history, memory and public stats.",
+      "Representation variants carry identical semantic state (offline assertion).",
+      "Strategy mode supplies a deterministic heroHand to every model; Raw mode supplies it to none.",
+      "Jev and chat adapters receive exactly the same family and size enums (offline assertion).",
+      "The total action clock is shared across both hierarchical stages.",
+      "Spectator explanations are disabled in rigorous runs; when enabled they are isolated and counter-separated.",
+      "No per-model production prompt tuning exists and no provider-specific poker facts are injected.",
+      "Diagnostic-only hidden information is structurally impossible in tournament mode."
+    ]
+  },
+  "caveats": [
+    "Behavioral proportion estimates are only as strong as the cell sample size; small cells carry wide Wilson intervals.",
+    "Paired architecture agreement measures decision-policy stability on fixed states, not win-rate superiority.",
+    "Tournament win rate is not a conclusion at these sample sizes, and tournament trajectories diverge after the first different action.",
+    "Provider rate limits and model version drift can affect reproducibility even for identical fixed states."
+  ]
+}

package/docs/legal/THIRD_PARTY_NOTICES.md

@@ -0,0 +1,15 @@
+# Third-party notices
+
+`pokertools-arena` bundles the following third-party packages at build time:
+
+- **@pokertools/engine** (1.0.20) — MIT — the browser poker rules engine used to
+  deal hands, validate legal actions and settle pots.
+- **@pokertools/evaluator** (1.0.20) — MIT — deterministic five-card poker hand
+  evaluation used to generate the Strategy-mode `heroHand` classification.
+
+Development-only tooling:
+
+- **esbuild** — MIT — used only to produce the static `dist/` build.
+
+No third-party package is used to generate model decisions, prompts, scores or
+fairness assertions.