pokertools-arena 0.4.0

package/.env.example

@@ -0,0 +1,8 @@
+# Optional local/CDP real-model bootstrap. Never commit real keys.
+OPENAI_BASE_URL=https://openrouter.ai/api/v1
+OPENAI_API_KEY="sk-or-v1-..."
+OPENAI_PLAYER1=google/gemma-4-26b-a4b-it
+OPENAI_PLAYER2=qwen/qwen3.8-flash
+OPENAI_PLAYER3=typesafe/jev-1.13
+# OPENAI_PLAYER4=...
+OPENAI_STARTING_STACK=10000

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Art
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -0,0 +1,229 @@
+# pokertools-arena
+
+**A browser-first AI poker benchmark.** Seat Jev and OpenAI-compatible models at the same no-limit Texas Hold'em table, watch every card and decision as a spectator, and let the tournament run autonomously until one model wins.
+
+[![CI](https://github.com/pokertools-arena/pokertools-arena.github.io/actions/workflows/ci.yml/badge.svg)](https://github.com/pokertools-arena/pokertools-arena.github.io/actions/workflows/ci.yml)
+[![Deploy](https://github.com/pokertools-arena/pokertools-arena.github.io/actions/workflows/pages.yml/badge.svg)](https://github.com/pokertools-arena/pokertools-arena.github.io/actions/workflows/pages.yml)
+[![npm](https://img.shields.io/npm/v/pokertools-arena.svg)](https://www.npmjs.com/package/pokertools-arena)
+[![license: MIT](https://img.shields.io/badge/license-MIT-6e9875.svg)](./LICENSE)
+[![node: >=24](https://img.shields.io/badge/node-%3E%3D24-6e9875.svg)](./package.json)
+
+[![pokertools-arena social preview](src/assets/og-image.png)](https://pokertools-arena.github.io/)
+
+| | |
+| --- | --- |
+| **Live app** | <https://pokertools-arena.github.io/> |
+| **Source** | <https://github.com/pokertools-arena/pokertools-arena.github.io> |
+| **Package** | `npx pokertools-arena` |
+| **Runtime** | Node.js ≥ 24 for tooling; any modern browser for the app |
+
+> **Current release: 0.4.0.** This is the methodology and reproducibility release: a paired fixed-state corpus, separated family/sizing correctness, fragmentation-invariance metrics with confidence intervals, standardized counters, and a reorganized repository with one source of truth per report.
+
+---
+
+## Contents
+
+- [Why pokertools-arena](#why-pokertools-arena)
+- [Features](#features)
+- [Quick start](#quick-start)
+- [Configure the table](#configure-the-table)
+- [`.env` bootstrap](#env-bootstrap)
+- [Benchmark modes](#benchmark-modes)
+- [Fairness](#fairness)
+- [Methodology](#methodology)
+- [Testing](#testing)
+- [Repository layout](#repository-layout)
+- [Documentation](#documentation)
+- [Privacy and security](#privacy-and-security)
+- [License](#license)
+
+---
+
+## Why pokertools-arena
+
+Text-model benchmarks are usually static question sets. Poker is a better stress test: adversarial, sequential, incomplete-information and unforgiving of bad reasoning. `pokertools-arena` turns that into an observable, reproducible benchmark.
+
+- **Same information for every seat.** Poker legality, hole-card masking and public context are generated by the PokerTools engine, not by prompts.
+- **Any OpenAI-compatible endpoint.** Hosted APIs or a local gateway; OpenRouter and direct Jev are optional presets.
+- **Nothing to deploy.** The benchmark is fully client-side. There is no application server, database or backend API.
+- **Everything is inspectable.** Watch live, replay any decision, export logs and replay PNGs.
+
+Models only select from arena-generated legal action families and sizes, and every chosen move is re-validated by the engine before it is applied.
+
+## Features
+
+| Area | What you get |
+| --- | --- |
+| **Table** | 2–10 seats, no-limit Hold'em tournaments, rising blinds, antes, time banks, elimination and podium flow |
+| **Connections** | Generic OpenAI-compatible base URL, OpenRouter preset, TypeSafe System One preset |
+| **Protocols** | Tool call, strict JSON Schema, prompt JSON, Jev Decisions, native Jev |
+| **Fairness** | One canonical `DecisionState` per decision, fail-closed hole-card masking, deterministic legal actions |
+| **Hierarchy** | Two-stage family → size decisions shared by every model and adapter |
+| **Observability** | Live decision panel, typed telemetry, latency/error counters, JSONL export |
+| **Replay** | Per-decision snapshots with hero cards, board, stacks, public history, legal menu |
+| **Sharing** | Local 1080×1350 PNG replay cards, table-only WebM recording, native share sheet |
+| **Benchmarks** | In-app sanity suite plus offline and real-API diagnostics with a paired fixed-state corpus |
+
+## Quick start
+
+Prerequisites: **Node.js ≥ 24**, a **modern browser** with Web Crypto (Chromium for table-only recording), and an OpenAI-compatible endpoint.
+
+### Option A — npm launcher (recommended)
+
+```bash
+npx pokertools-arena
+```
+
+The CLI starts a tiny local static server and opens the browser. It exists only to avoid `file://` restrictions. Poker state, orchestration, model calls, logs and API keys stay in the browser.
+
+| Flag | Default | Purpose |
+| --- | --- | --- |
+| `--port <n>` | `4173` | Port to serve on (auto-increments if busy). |
+| `--host <addr>` | `127.0.0.1` | Interface to bind. |
+| `--chrome` | off | Open Google Chrome specifically. |
+| `--no-open` | off | Do not launch a browser. |
+| `--no-env` | off | Ignore `.env` bootstrap. |
+| `--help`, `-h` | — | Print usage. |
+
+### Option B — single HTML file
+
+```bash
+npm install
+npm run build
+```
+
+Then open `dist/pokertools-arena.html`. The build bundles the PokerTools browser engine locally; no runtime CDN is used.
+
+### Option C — GitHub Pages
+
+The canonical deployment is <https://pokertools-arena.github.io/>. Enable **Settings → Pages → Source: GitHub Actions** once; `.github/workflows/pages.yml` builds and deploys `dist/` on every push to `main`.
+
+## Configure the table
+
+The table **is** the player editor. A fresh launch shows empty seats around the felt.
+
+1. Click a seat to add or edit a model.
+2. Choose a connection, enter a model ID and pick a protocol.
+3. Configure **Settings → Connections** (base URLs and keys) and **Settings → Tournament** (stack, blinds, clock, benchmark mode, architecture, representation).
+4. Press **Start**. Only configured seats enter the tournament, and seat editing locks while it runs.
+
+## `.env` bootstrap
+
+The launcher reads `.env` from the current working directory (falling back to the package root) and injects only supported fields into page memory. Disable with `--no-env`. A real user `.env` is never moved, printed or archived.
+
+```dotenv
+OPENAI_BASE_URL=https://openrouter.ai/api/v1
+OPENAI_API_KEY="sk-or-v1-..."
+OPENAI_PLAYER1=google/gemma-4-26b-a4b-it
+OPENAI_PLAYER2=qwen/qwen3.8-flash
+OPENAI_PLAYER3=typesafe/jev-1.13
+OPENAI_STARTING_STACK=10000
+```
+
+| Variable | Notes |
+| --- | --- |
+| `OPENAI_BASE_URL` | Defaults to `https://api.openai.com/v1`; OpenRouter URLs are auto-detected. |
+| `OPENAI_API_KEY` | Optional for local/unauthenticated endpoints. |
+| `OPENAI_PLAYER1` … `OPENAI_PLAYER10` | Each value seats one model; Jev models auto-route to Decisions on OpenRouter. |
+| `OPENAI_STARTING_STACK` | Chip stack per seat (minimum 100, default 10000). |
+
+See [`.env.example`](./.env.example).
+
+## Benchmark modes
+
+Two tournament-wide settings keep every model on equal footing:
+
+- **Strategy** (default): every seat receives a deterministic `heroHand` classification from `@pokertools/evaluator`.
+- **Raw cognition**: no `heroHand`; each model infers hand strength from raw cards.
+
+They are **separate benchmark tracks**. Public reports never aggregate them into a single model score.
+
+**Decision architecture** is also tournament-wide. `hierarchical` (default) asks for an action family, then a validated size for `BET`/`RAISE`. `flat` is retained only as a diagnostic baseline. **Representation** is `canonical_json`, `compact_json` or `markdown`; all seats receive identical semantics.
+
+## Fairness
+
+Every seat receives the same canonical information policy: its own hole cards, the public board/pot/stacks/positions/current bets, the same deterministic legal actions, current-hand public history, last eight public hands, and identical deterministic public opponent statistics. Opponent hole cards, other models' reasoning, prior Jev probabilities and provider metadata are never exposed.
+
+Player masking is **fail-closed**. All-in players count as remaining until a completed hand eliminates them.
+
+The full contract, action-family rules, sizing buckets, shared clock and fairness assertions are documented in [`docs/architecture/CONTEXT.md`](./docs/architecture/CONTEXT.md).
+
+## Methodology
+
+The benchmark methodology is built around a **paired fixed-state corpus**:
+
+- the exact same immutable state is used for flat and hierarchical, for every model, mode and representation;
+- execution is deterministically interleaved with a recorded `experimentSeed`;
+- **family accuracy** and **sizing accuracy (conditional on a correct family)** are reported separately from **final-action accuracy**;
+- behavioral proportions carry sample size and Wilson 95% confidence intervals;
+- **`fragmentation_flip_rate`** measures how often the broad family changes when only the number of same-family size choices changes;
+- counters are standardized (`pokerDecisions`, `familyModelCalls`, `sizingModelCalls`, `totalModelCalls`, `httpRequests`, `httpRetries`, …) and one machine-readable summary is the single source of truth for every report.
+
+> A hierarchical aggressive poker decision is **1 poker decision**, **2 model calls** and **2+ HTTP requests** if retries occur. Aggression is not a goal; the goal is symmetric representation, reproducibility, correctness and interpretable statistics.
+
+Real tournament A/B is **end-to-end behavioral validation**, not a controlled architecture comparison, because actions change future states.
+
+## Testing
+
+```bash
+npm test                    # build + release checks + offline unit + integration
+npm run test:unit           # corpus, hierarchy/fairness, methodology, sizing, paired, archive, report
+npm run test:integration    # @pokertools/engine/browser + launcher .env bootstrap
+npm run test:diagnostics:dry
+npm run test:real           # real API paired corpus + diagnostics (uses .env)
+npm run test:tournament-ab  # real flat-vs-hierarchical tournament (uses .env)
+npm run report:diagnostics  # build logs/release-<version>/summary.json + summary.md
+npm run release:archive     # source archive without secrets/logs/dist
+```
+
+`npm test` is fully offline apart from dependency installation. Real-API diagnostics are manual or secret-gated in CI. See [`tests/README.md`](./tests/README.md).
+
+## Repository layout
+
+```text
+pokertools-arena/
+├── README.md · LICENSE · package.json · build.mjs
+├── bin/pokertools-arena.mjs            local static launcher + .env bootstrap
+├── src/                                browser application source
+│   ├── index.html · app.js · styles.css
+│   ├── lib/decision-core.js            canonical DOM-free decision core
+│   ├── benchmark/scenarios.js          in-app sanity scenarios
+│   ├── env/arena-env.js                .env placeholder injected by the launcher
+│   ├── shims/crypto.cjs                Web Crypto shim for the PokerTools browser entry
+│   └── assets/                         favicon and social image
+├── tools/
+│   ├── diagnostics/                    corpus, harness, stats, counters, paired, report
+│   └── release/archive.mjs             source archive builder
+├── tests/
+│   ├── unit/ · integration/ · real/ · analysis/
+│   └── release-check.mjs
+├── docs/
+│   ├── architecture/CONTEXT.md
+│   ├── releases/RELEASE_NOTES.md
+│   ├── diagnostics/SUMMARY.md + summary.json
+│   ├── verification/0.3.x.md
+│   └── legal/THIRD_PARTY_NOTICES.md
+└── .github/workflows/                  CI, GitHub Pages, npm publish, manual real diagnostics
+```
+
+Generated and gitignored: `dist/`, `pokertools-arena.html`, `logs/`, `node_modules/`.
+
+## Documentation
+
+- Architecture and invariants → [`docs/architecture/CONTEXT.md`](./docs/architecture/CONTEXT.md)
+- Chronological release history → [`docs/releases/RELEASE_NOTES.md`](./docs/releases/RELEASE_NOTES.md)
+- Benchmark methodology and results → [`docs/diagnostics/SUMMARY.md`](./docs/diagnostics/SUMMARY.md)
+- Machine-readable release summary → [`docs/diagnostics/summary.json`](./docs/diagnostics/summary.json)
+- Test suites → [`tests/README.md`](./tests/README.md)
+- Third-party notices → [`docs/legal/THIRD_PARTY_NOTICES.md`](./docs/legal/THIRD_PARTY_NOTICES.md)
+
+## Privacy and security
+
+- API keys live only in current page memory and are excluded from saved setup state, tournament snapshots, exported logs and release archives.
+- The launcher serves the `.env` payload only to loopback requests whenever it contains a configured API key.
+- Models never receive hidden information and never construct engine commands.
+- Replay images and recordings are produced locally; nothing is uploaded.
+
+## License
+
+MIT — see [`LICENSE`](./LICENSE).

package/bin/pokertools-arena.mjs

@@ -0,0 +1,197 @@
+#!/usr/bin/env node
+import { createServer } from 'node:http';
+import { readFile, stat } from 'node:fs/promises';
+import { extname, dirname, join, normalize, resolve } from 'node:path';
+import { fileURLToPath } from 'node:url';
+import { spawn } from 'node:child_process';
+
+const packageRoot = dirname(dirname(fileURLToPath(import.meta.url)));
+const root = join(packageRoot, 'dist');
+const args = process.argv.slice(2);
+const valueAfter = (flag, fallback) => {
+  const index = args.indexOf(flag);
+  return index >= 0 && args[index + 1] ? args[index + 1] : fallback;
+};
+const host = valueAfter('--host', '127.0.0.1');
+const requestedPortRaw = valueAfter('--port', '4173');
+const requestedPort = requestedPortRaw === '0' ? 0 : (Number(requestedPortRaw) || 4173);
+const shouldOpen = !args.includes('--no-open');
+const help = args.includes('--help') || args.includes('-h');
+const useEnv = !args.includes('--no-env');
+const useChrome = args.includes('--chrome');
+const envOverrides = {
+  OPENAI_STARTING_STACK: valueAfter('--starting-stack', null),
+  OPENAI_MAX_DECISIONS: valueAfter('--max-decisions', null),
+  OPENAI_AUTOSTART: args.includes('--autostart') ? '1' : null,
+};
+
+function parseDotEnv(text) {
+  const out = {};
+  for (const rawLine of String(text || '').split(/\r?\n/)) {
+    let line = rawLine.trim();
+    if (!line || line.startsWith('#')) continue;
+    if (line.startsWith('export ')) line = line.slice(7).trim();
+    const eq = line.indexOf('=');
+    if (eq <= 0) continue;
+    const key = line.slice(0, eq).trim();
+    if (!/^[A-Za-z_][A-Za-z0-9_]*$/.test(key)) continue;
+    let value = line.slice(eq + 1).trim();
+    if ((value.startsWith('"') && value.endsWith('"')) || (value.startsWith("'") && value.endsWith("'"))) {
+      const quote = value[0];
+      value = value.slice(1, -1);
+      if (quote === '"') value = value.replace(/\n/g, '\n').replace(/\r/g, '\r').replace(/\t/g, '\t').replace(/\\"/g, '"').replace(/\\\\/g, '\\');
+    } else {
+      value = value.replace(/\s+#.*$/, '').trim();
+    }
+    out[key] = value;
+  }
+  return out;
+}
+
+async function loadArenaEnv() {
+  if (!useEnv) return { source: null, values: {} };
+  const candidates = [resolve(process.cwd(), '.env'), join(packageRoot, '.env')];
+  for (const candidate of [...new Set(candidates)]) {
+    try {
+      const values = parseDotEnv(await readFile(candidate, 'utf8'));
+      return { source: candidate, values };
+    } catch {}
+  }
+  return { source: null, values: {} };
+}
+
+function envBootstrap(values, overrides = {}) {
+  const baseUrl = String(values.OPENAI_BASE_URL || process.env.OPENAI_BASE_URL || '').trim();
+  const apiKey = String(values.OPENAI_API_KEY || process.env.OPENAI_API_KEY || '').trim();
+  const models = [];
+  for (let i = 1; i <= 10; i++) {
+    const model = String(values[`OPENAI_PLAYER${i}`] || process.env[`OPENAI_PLAYER${i}`] || '').trim();
+    if (model) models.push({ index: i, model });
+  }
+  if (!baseUrl && !apiKey && !models.length) return null;
+  const resolvedBase = baseUrl || 'https://api.openai.com/v1';
+  const isOpenRouter = /(^|\.)openrouter\.ai(?=\/|$)/i.test(new URL(resolvedBase).hostname);
+  const connectionId = 'env-connection';
+  const truthy = value => /^(1|true|yes|on)$/i.test(String(value ?? '').trim());
+  const flagOrEnv = flag => {
+    if (overrides[flag] !== undefined && overrides[flag] !== null && overrides[flag] !== '') return overrides[flag];
+    return values[flag] || process.env[flag];
+  };
+  const autostart = truthy(flagOrEnv('OPENAI_AUTOSTART'));
+  const maxDecisions = Math.max(0, Math.round(Number(flagOrEnv('OPENAI_MAX_DECISIONS') || 0) || 0));
+  const startingStack = Math.max(100, Math.round(Number(flagOrEnv('OPENAI_STARTING_STACK') || 10_000) || 10_000));
+  return {
+    source: '.env',
+    settings: {
+      // Default chip stack for every model seat. Override with --starting-stack
+      // or OPENAI_STARTING_STACK.
+      startingStack,
+      // Opt-in launcher demo/benchmark controls. Autostart is off unless the
+      // launcher explicitly enables it; maxDecisions hard-stops the tournament.
+      autostart,
+      maxDecisions,
+    },
+    connections: [{
+      id: connectionId,
+      name: isOpenRouter ? 'OpenRouter' : 'OpenAI-compatible',
+      kind: isOpenRouter ? 'openrouter' : 'openai',
+      baseUrl: resolvedBase,
+      apiKey,
+      headers: '',
+    }],
+    players: models.map(({ index, model }, order) => ({
+      lobbySeat: Math.max(0, index - 1),
+      name: `Player ${index}`,
+      connectionId,
+      model,
+      protocol: isOpenRouter && /(^|\/)typesafe\/jev-/i.test(model.replace(/^~/, '')) ? 'jev_decisions' : 'tool',
+      provider: '',
+    })),
+  };
+}
+
+const loadedEnv = await loadArenaEnv();
+const arenaEnv = envBootstrap(loadedEnv.values, envOverrides);
+
+if (help) {
+  console.log(`pokertools-arena\n\nUsage:\n  npx pokertools-arena [--port 4173] [--host 127.0.0.1] [--chrome] [--no-open] [--no-env] [--autostart] [--starting-stack N] [--max-decisions N]\n\nThe launcher serves the prebuilt static benchmark locally and opens it in your browser so you can watch the tournament. --chrome opens Google Chrome specifically. By default it also reads .env from the current directory and preloads OPENAI_BASE_URL / OPENAI_API_KEY / OPENAI_PLAYER1..10 into page memory. --autostart starts the tournament once seats are loaded, and --max-decisions hard-stops it after N poker decisions. --starting-stack and --max-decisions override .env values. Use --no-env to disable the .env bootstrap.`);
+  process.exit(0);
+}
+
+const mime = {
+  '.html': 'text/html; charset=utf-8', '.js': 'text/javascript; charset=utf-8', '.css': 'text/css; charset=utf-8',
+  '.json': 'application/json; charset=utf-8', '.svg': 'image/svg+xml', '.png': 'image/png', '.ico': 'image/x-icon'
+};
+
+function isLoopbackRequest(req) {
+  const address = String(req.socket?.remoteAddress || '');
+  return address === '127.0.0.1' || address === '::1' || address === '::ffff:127.0.0.1';
+}
+
+function safePath(urlPath) {
+  const clean = decodeURIComponent((urlPath || '/').split('?')[0]);
+  const target = clean === '/' ? '/index.html' : clean;
+  const resolved = normalize(join(root, target));
+  return resolved.startsWith(root) ? resolved : null;
+}
+
+const server = createServer(async (req, res) => {
+  try {
+    const urlPath = (req.url || '/').split('?')[0];
+    if (urlPath === '/arena-env.js') {
+      const exposedEnv = arenaEnv && (isLoopbackRequest(req) || !arenaEnv.connections?.some(c => c.apiKey)) ? arenaEnv : null;
+      const payload = `window.__POKERTOOLS_ENV__ = ${JSON.stringify(exposedEnv)};\n`;
+      res.writeHead(200, { 'Content-Type': 'text/javascript; charset=utf-8', 'Cache-Control': 'no-store' });
+      res.end(payload);
+      return;
+    }
+    const file = safePath(req.url);
+    if (!file) throw new Error('Invalid path');
+    const info = await stat(file);
+    if (!info.isFile()) throw new Error('Not a file');
+    const body = await readFile(file);
+    res.writeHead(200, {
+      'Content-Type': mime[extname(file)] || 'application/octet-stream',
+      'Cache-Control': 'no-store',
+      'Cross-Origin-Opener-Policy': 'same-origin-allow-popups'
+    });
+    res.end(body);
+  } catch {
+    res.writeHead(404, { 'Content-Type': 'text/plain; charset=utf-8' });
+    res.end('Not found');
+  }
+});
+
+function openBrowser(url) {
+  const platform = process.platform;
+  let command, commandArgs;
+  if (useChrome) {
+    if (platform === 'darwin') { command = 'open'; commandArgs = ['-a', 'Google Chrome', url]; }
+    else if (platform === 'win32') { command = 'cmd'; commandArgs = ['/c', 'start', '', 'chrome', url]; }
+    else { command = 'google-chrome'; commandArgs = [url]; }
+  } else if (platform === 'darwin') { command = 'open'; commandArgs = [url]; }
+  else if (platform === 'win32') { command = 'cmd'; commandArgs = ['/c', 'start', '', url]; }
+  else { command = 'xdg-open'; commandArgs = [url]; }
+  try { spawn(command, commandArgs, { detached: true, stdio: 'ignore' }).unref(); } catch {}
+}
+
+function listen(port) {
+  server.once('error', error => {
+    if (error.code === 'EADDRINUSE' && port < requestedPort + 20) return listen(port + 1);
+    console.error(`pokertools-arena: ${error.message}`);
+    process.exit(1);
+  });
+  server.listen(port, host, () => {
+    const address = server.address();
+    const actualPort = typeof address === 'object' && address ? address.port : port;
+    const url = `http://${host === '0.0.0.0' ? '127.0.0.1' : host}:${actualPort}/`;
+    console.log(`pokertools-arena running at ${url}`);
+    if (arenaEnv) console.log(`Loaded .env bootstrap${loadedEnv.source ? ` from ${loadedEnv.source}` : ''}: ${arenaEnv.players.length} player(s), ${arenaEnv.connections[0].baseUrl}`);
+    else if (useEnv) console.log('No supported .env bootstrap values found; using browser-saved/manual setup.');
+    else console.log('.env bootstrap disabled by --no-env.');
+    console.log('Press Ctrl+C to stop the local launcher.');
+    if (shouldOpen) openBrowser(url);
+  });
+}
+
+listen(requestedPort);