tick-bench-cli 0.1.0

package/CITATION.cff

@@ -0,0 +1,21 @@
+cff-version: 1.2.0
+message: "If you use TickBench in your research, please cite it as below."
+title: "TickBench: A Benchmark Harness for Performance- and Correctness-Critical Financial Dashboard Engineering"
+type: software
+authors:
+  - family-names: Fedytskyi
+    given-names: Roman
+    email: fedytskyi@gmail.com
+version: 0.1.0
+date-released: 2026-07-08
+license: MIT
+repository-code: "https://github.com/RomanFedytskyi/TickBench"
+keywords:
+  - benchmark
+  - fintech
+  - coding agents
+  - real-time dashboards
+  - display staleness
+  - web performance
+  - accessibility
+  - security

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Roman Fedytskyi
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -0,0 +1,187 @@
+# TickBench
+
+[![CI](https://github.com/RomanFedytskyi/TickBench/actions/workflows/ci.yml/badge.svg)](https://github.com/RomanFedytskyi/TickBench/actions/workflows/ci.yml)
+[![npm](https://img.shields.io/npm/v/tick-bench-cli.svg)](https://www.npmjs.com/package/tick-bench-cli)
+[![npm downloads](https://img.shields.io/npm/dw/tick-bench-cli.svg)](https://www.npmjs.com/package/tick-bench-cli)
+[![bundle size](https://img.shields.io/bundlephobia/minzip/tick-bench-cli.svg)](https://bundlephobia.com/package/tick-bench-cli)
+[![types](https://img.shields.io/npm/types/tick-bench-cli.svg)](https://www.npmjs.com/package/tick-bench-cli)
+[![Node.js >=18](https://img.shields.io/node/v/tick-bench-cli.svg)](https://www.npmjs.com/package/tick-bench-cli)
+[![harness](https://img.shields.io/badge/submissions-any%20framework-blue.svg)](#benchmark-your-own-component)
+[![license](https://img.shields.io/badge/code-MIT-green.svg)](LICENSE)
+[![data](https://img.shields.io/badge/traces-CC--BY--4.0-orange.svg)](docs/task-authoring.md)
+[![DOI](https://img.shields.io/badge/DOI-10.5281%2Fzenodo.1294564668-blue.svg)](https://doi.org/10.5281/zenodo.21272054)
+[![GitHub last commit](https://img.shields.io/github/last-commit/RomanFedytskyi/TickBench.svg)](https://github.com/RomanFedytskyi/TickBench)
+
+> **Never ship a grid that paints a stale price.**
+
+TickBench replays **seeded, replayable market-data tick streams** against your data-grid or
+dashboard component in headless Chromium and tells you what your users would actually see:
+**display staleness**, update latency percentiles, dropped frames, lost updates, broken money
+formatting, live XSS sinks, and WCAG violations — one command, one JSON report, one
+pass/fail verdict.
+
+<p align="center">
+  <img src="docs/banner.svg" alt="TickBench: benchmark your live data grid under real market load — display staleness, update latency, frame stability, money formatting, XSS, WCAG." width="100%" />
+</p>
+
+## What is this?
+
+**TickBench is like Lighthouse, but for live data tables.** Lighthouse loads your page and
+grades it. TickBench goes further for one specific, high-stakes kind of UI — tables that
+display fast-changing numbers (prices, positions, risk, metrics) — and grades what a user
+would actually see while thousands of updates per second are flowing in.
+
+Your unit tests check logic. Your visual tests check pixels at rest. **Nothing you run today
+checks whether the number on screen right now is the number that arrived 400 ms ago** — or
+whether your table silently drops updates, stutters, mis-formats money, executes a malicious
+string, or loses its accessible structure under load. TickBench checks exactly that, in a
+real (headless) Chrome, and gives you a PASS/FAIL verdict per implementation.
+
+**Use it when you:**
+
+- build any dashboard with live numbers — trading, crypto, risk, fleet, analytics, admin;
+- are choosing a table/grid library and don't want to trust vendor marketing benchmarks;
+- let AI coding tools write frontend code and need an objective gate before merging it;
+- want a CI check proving a refactor didn't make the table slower or wrong.
+
+## How it works (60 seconds)
+
+**Step 1 — you write one small file** (an adapter). Important: your app never imports
+TickBench, and `applyTick` is not imported from anywhere — **you define it, and TickBench
+calls it**. TickBench imports *your* file, calls your `createGrid` once to mount the table,
+then calls your `applyTick` ~17,000 times with simulated market updates while filming the
+result frame by frame.
+
+```js
+// my-adapter.mjs — the whole contract
+export function createGrid(container, symbols, cols) {
+  // mount YOUR table into `container` (any framework; relative imports work)
+  // each rendered cell needs data-sym / data-col attributes so TickBench can watch it
+  return {
+    applyTick(tick) {
+      // TickBench calls this with { sym, col, v } for every update.
+      // Route it into your table the same way your app would (setState, setCell, ...).
+    }
+  };
+}
+```
+
+**Step 2 — run it** from your project folder:
+
+```bash
+npx tick-bench-cli bench --impl ./my-adapter.mjs --name my-table
+```
+
+**Step 3 — read the verdict.** Three outputs every run: the console table, a browser-ready
+**HTML report** (`./tickbench-results/report-stream-grid.html`), and JSON for CI:
+
+<p align="center">
+  <img src="docs/report-screenshot.png" alt="TickBench HTML report: PASS cards with per-gate checklists for reference and simple-table, plus the full metric comparison table." width="90%" />
+</p>
+
+Complete runnable example with a real component file: [`examples/walkthrough/`](examples/walkthrough/).
+Framework recipes (React, AG Grid): [`docs/connect-your-table.md`](docs/connect-your-table.md).
+
+
+## How do you test a real-time data grid?
+
+If you searched for *how to test a real-time data grid*, *measure display staleness*, *React
+table streaming updates benchmark*, *virtualized grid dropped updates*, *validate AI-generated
+frontend code*, or *my dashboard shows stale prices under load* — this is the tool for that.
+
+## Why this exists
+
+Every trading, risk, lending, and analytics UI is a grid fed by a stream. Under burst load,
+implementations fail in ways no existing tool measures:
+
+- the DOM shows a **price that is no longer true** (staleness),
+- the **latest** update is silently never painted (lost update),
+- the main thread stalls for hundreds of ms (frame stability),
+- `1234567.5` cents becomes `12345.68` instead of `12,345.68` (money formatting),
+- an interpolated symbol name becomes an **XSS payload**,
+- the grid loses its accessible structure (WCAG).
+
+Grid vendors publish marketing benchmarks; js-framework-benchmark tests static rows. TickBench
+tests **your implementation, under your workload, against executable acceptance criteria** —
+including code written by AI coding agents before you let it near production.
+
+## Quick start (this repository)
+
+```bash
+git clone https://github.com/RomanFedytskyi/TickBench.git && cd TickBench
+npm ci
+npx playwright-core install chromium-headless-shell   # one-time browser download
+
+npm test           # unit tests
+npm run validate   # oracle validation: reference passes, mutants are caught
+npm run bench      # score all bundled submissions for tasks/stream-grid
+```
+
+```
+                                     reference  claude-fable-5  naive-baseline
+catch-up latency p50 (ms)                 24.3            24.1             8.1
+catch-up latency p95 (ms)                 31.8            32.1            37.3
+stale cell-frames (%)                    17.41           17.44               0
+long frames >33ms (%)                        0             0.2            2.49
+worst frame (ms)                          16.8            49.9           516.7
+final format errors                          0               0             157
+XSS triggered (dynamic)                  false           false            true
+security findings (high)                     0               0               1
+axe violations                               0               0               1
+REGULATED-GRADE PASS                      true            true           false
+```
+Three submissions ship with the repo: a human **reference**, a genuine single-shot
+**AI-generated submission** (`claude-fable-5`, provenance documented, unedited model output),
+and a hand-written **naive-baseline** used to validate the oracles. Note the trade-off the
+numbers expose: the unbatched baseline paints *faster at the median* but stalls for half a
+second at the worst — while rAF-batched implementations trade a single frame of staleness
+for stability. That frontier is invisible until you measure it.
+
+## What gets measured
+
+Formal definitions live in [`docs/metrics.md`](docs/metrics.md):
+
+| Dimension | Oracle | Metrics |
+|---|---|---|
+| Update correctness | runtime, frame-sampled | display staleness, catch-up latency p50/p95/p99, lost updates |
+| Rendering performance | runtime | long-frame rate, worst frame |
+| Money display | final-state sweep | value errors, locale/format errors |
+| Security | static sink scan + dynamic canary | injection findings, XSS execution |
+| Accessibility | axe-core | WCAG 2.1 A/AA violations |
+
+Thresholds are part of each task's `oracle.config.json` — acceptance criteria are versioned
+data, not harness code. Every gate is validated: the shipped `naive-baseline` submission is a
+deliberately broken implementation that `tickbench validate` proves each oracle catches.
+
+## Real workloads
+
+Synthetic traces are seeded and deterministic (`tickbench gen --seed …`). You can also record
+real feeds (e.g., public exchange WebSocket streams, or your own production feed) into the
+same schema — set `"synthetic": false` and ship a provenance file. See
+[`docs/task-authoring.md`](docs/task-authoring.md).
+
+## CLI
+
+```
+tick-bench-cli bench --impl ./my-adapter.mjs [--name LABEL]   benchmark YOUR component (any directory)
+tick-bench-cli bench [--task DIR] [--submission NAME]         benchmark a task's reference + submissions
+tick-bench-cli gen [--seed N] [--duration MS] [--symbols N] [--out FILE]
+tick-bench-cli validate [--task DIR]
+tick-bench-cli new-submission <name> [--task DIR]             scaffold a submission for model output
+```
+
+Common options: `--trace FILE` (custom workload), `--out DIR` (default `./tickbench-results`),
+`--no-reference` (skip the built-in comparison baseline).
+
+## Contributing
+
+New tasks, new metrics, and submissions of real-world grid implementations are all welcome —
+see [CONTRIBUTING.md](CONTRIBUTING.md). Metrics need a definition **and** a validation mutant.
+
+## Citation
+
+If TickBench is useful in your work, cite it via [`CITATION.cff`](CITATION.cff).
+
+## License
+
+Code: MIT © 2026 Roman Fedytskyi. Bundled and recorded trace data (`traces/`): CC BY 4.0.

package/docs/agent-adapters.md

@@ -0,0 +1,31 @@
+# Evaluating Coding Agents with TickBench
+
+TickBench evaluates *artifacts* (submissions), so any agent can be tested by writing its
+output into `tasks/<task>/submissions/<agent-id>/grid.mjs`:
+
+1. Give the agent `tasks/<task>/spec.md` (and nothing else) as the task prompt.
+2. Save each sample as `submissions/<agent-id>-s<k>/grid.mjs` for pass@k evaluation.
+3. Record model, version, temperature, prompt, and date in `PROVENANCE.md`.
+4. Run `tickbench bench --task tasks/<task>` — every submission is scored per dimension and
+   against the regulated-grade gate.
+
+Planned: an adapter CLI that drives agent APIs
+directly, multi-turn repair loops with oracle feedback, and regulated-grade-pass@k reporting.
+
+## Seeding submissions across models
+
+For a model comparison, repeat per model:
+
+```bash
+npx tick-bench-cli new-submission claude-sonnet-5     # scaffolds grid.mjs + PROVENANCE.md
+# paste tasks/stream-grid/spec.md into the model (claude.ai, API, IDE agent...)
+# paste its UNEDITED output into submissions/claude-sonnet-5/grid.mjs
+# fill PROVENANCE.md (model, date, mode, prompt, "post-editing: none")
+npx tick-bench-cli bench --task tasks/stream-grid     # scores all submissions side by side
+```
+
+Provenance rules that keep the comparison honest: output is pasted unedited; the prompt is
+exactly the task spec (plus any system prompt, recorded verbatim); one directory per model
+and per sample (`claude-sonnet-5-s2` for the second sample when computing pass@k); if the
+generating model may have seen similar public code in training, note it — the shipped
+`claude-fable-5` submission carries the same caveat.

package/docs/banner.svg

@@ -0,0 +1,59 @@
+<svg width="1200" height="300" viewBox="0 0 1200 300" xmlns="http://www.w3.org/2000/svg" role="img" aria-label="TickBench: benchmark your live data grid under real market load">
+  <defs>
+    <linearGradient id="bg" x1="0" y1="0" x2="1" y2="1">
+      <stop offset="0" stop-color="#0b1220"/>
+      <stop offset="0.55" stop-color="#0e1a2b"/>
+      <stop offset="1" stop-color="#0a2230"/>
+    </linearGradient>
+    <linearGradient id="accent" x1="0" y1="0" x2="1" y2="0">
+      <stop offset="0" stop-color="#22d3a5"/>
+      <stop offset="1" stop-color="#38bdf8"/>
+    </linearGradient>
+  </defs>
+  <rect width="1200" height="300" fill="url(#bg)"/>
+  <g stroke="#1e3a52" stroke-width="1" opacity="0.35">
+    <path d="M0 60H1200M0 120H1200M0 180H1200M0 240H1200"/>
+    <path d="M200 0V300M400 0V300M600 0V300M800 0V300M1000 0V300"/>
+  </g>
+  <g font-family="ui-sans-serif,-apple-system,'Segoe UI',Helvetica,Arial,sans-serif">
+    <text x="64" y="112" font-size="64" font-weight="800" fill="#f1f5f9">Tick<tspan fill="url(#accent)">Bench</tspan></text>
+    <text x="66" y="150" font-size="21" fill="#94a3b8">Benchmark your live data grid under real market load.</text>
+    <text x="66" y="178" font-size="16" fill="#64748b">Display staleness &#183; update latency &#183; frame stability &#183; money formatting &#183; XSS &#183; WCAG</text>
+    <g font-size="13" font-weight="600">
+      <rect x="66" y="206" rx="12" width="112" height="26" fill="#0f2e26" stroke="#22d3a5" stroke-width="1"/>
+      <text x="122" y="223" fill="#22d3a5" text-anchor="middle">correctness</text>
+      <rect x="188" y="206" rx="12" width="118" height="26" fill="#0d2836" stroke="#38bdf8" stroke-width="1"/>
+      <text x="247" y="223" fill="#38bdf8" text-anchor="middle">performance</text>
+      <rect x="316" y="206" rx="12" width="96" height="26" fill="#2b1a30" stroke="#e879f9" stroke-width="1"/>
+      <text x="364" y="223" fill="#e879f9" text-anchor="middle">security</text>
+      <rect x="422" y="206" rx="12" width="80" height="26" fill="#2e2410" stroke="#fbbf24" stroke-width="1"/>
+      <text x="462" y="223" fill="#fbbf24" text-anchor="middle">a11y</text>
+    </g>
+  </g>
+  <!-- mock live-quotes card -->
+  <g>
+    <rect x="672" y="22" width="464" height="200" rx="10" fill="#0d1626" stroke="#24405c" stroke-width="1.5"/>
+    <text x="694" y="52" font-family="ui-sans-serif,'Segoe UI',sans-serif" font-size="14" font-weight="700" fill="#cbd5e1">LIVE QUOTES</text>
+    <circle cx="1106" cy="47" r="5" fill="#22d3a5"/>
+    <g font-family="ui-monospace,SFMono-Regular,Menlo,Consolas,monospace" font-size="14">
+      <g fill="#64748b">
+        <text x="694" y="82">SYM</text><text x="812" y="82">BID</text><text x="924" y="82">ASK</text><text x="1036" y="82">LAST</text>
+      </g>
+      <g fill="#e2e8f0">
+        <text x="694" y="112">AAPL</text><text x="812" y="112" fill="#22d3a5">212,341.10</text><text x="924" y="112">212,341.35</text><text x="1036" y="112" fill="#22d3a5">&#9650; 341.20</text>
+        <text x="694" y="142">MSFT</text><text x="812" y="142" fill="#f87171">418,220.55</text><text x="924" y="142">418,220.90</text><text x="1036" y="142" fill="#f87171">&#9660; 220.75</text>
+        <text x="694" y="172">NVDA</text><text x="812" y="172">131,904.00</text><text x="924" y="172" fill="#22d3a5">131,904.40</text><text x="1036" y="172">904.15</text>
+      </g>
+      <rect x="804" y="126" width="104" height="24" rx="4" fill="none" stroke="#fbbf24" stroke-width="1.5" stroke-dasharray="4 3"/>
+      <text x="694" y="206" font-size="12.5" fill="#fbbf24">&#9888; stale 412 ms &#8594; flagged by runtime oracle</text>
+    </g>
+  </g>
+  <!-- frame-time sparkline, fully below the card -->
+  <g transform="translate(672,282)">
+    <text x="0" y="-40" font-family="ui-monospace,monospace" font-size="11" fill="#64748b">frame time</text>
+    <polyline points="0,-12 32,-13 64,-11 96,-13 128,-12 160,-13 192,-30 208,-6 240,-12 272,-13 304,-11 336,-12 368,-13 400,-11 432,-12 460,-13" fill="none" stroke="#38bdf8" stroke-width="2" stroke-linejoin="round"/>
+    <circle cx="192" cy="-30" r="3.5" fill="#f87171"/>
+    <text x="204" y="-32" font-family="ui-monospace,monospace" font-size="11" fill="#f87171">long frame</text>
+  </g>
+  <rect x="0" y="296" width="1200" height="4" fill="url(#accent)"/>
+</svg>

package/docs/connect-your-table.md

@@ -0,0 +1,104 @@
+# Connect your own table
+
+TickBench doesn't need your app. It benchmarks a **component in isolation**: you write one
+small adapter file that mounts your table and routes updates into it, then point the CLI at
+that file. Three steps, ~5 minutes.
+
+## 1. Write an adapter
+
+The whole contract:
+
+```js
+export function createGrid(container, symbols, cols) {
+  // 1. mount your table into `container`, one row per symbol, one column per col
+  // 2. every rendered cell must carry data-sym / data-col attributes
+  return {
+    applyTick(tick) {
+      // 3. route { sym, col, v } into your table's update path
+      //    price columns arrive as integer cents; format as en-US, 2 decimals
+    }
+  };
+}
+```
+
+Relative imports work — your adapter can `import { MyTable } from './my-table.mjs'` like
+normal code. See `examples/walkthrough/` for a complete runnable example with a real
+component file, or copy `examples/my-table-adapter.mjs` as a single-file starting point — it is a complete working adapter
+with the two "REPLACE" sections marked.
+
+**Vanilla / any DOM component:** call your component's mount and update methods directly.
+
+**React:** the harness runs your file in a bare browser page (no bundler), so import React
+from a CDN with full URLs:
+
+```js
+import React from 'https://esm.sh/react@18';
+import { createRoot } from 'https://esm.sh/react-dom@18/client';
+
+export function createGrid(container, symbols, cols) {
+  const root = createRoot(container);
+  let setTicks;                       // captured from the component
+  function App() {
+    const [ticks, s] = React.useState({});
+    setTicks = s;
+    return React.createElement(MyPriceTable, { ticks, symbols, cols }); // your component
+  }
+  root.render(React.createElement(App));
+  return { applyTick(t) { setTicks(prev => ({ ...prev, [t.sym + '|' + t.col]: t })); } };
+}
+```
+
+Make sure `MyPriceTable` renders `data-sym` / `data-col` on its cells (add them via your
+cell renderer if they aren't there yet).
+
+**AG Grid:**
+
+```js
+import { createGrid as agCreateGrid } from 'https://esm.sh/ag-grid-community@31';
+
+export function createGrid(container, symbols, cols) {
+  const api = agCreateGrid(container, {
+    rowData: symbols.map(sym => ({ sym })),
+    getRowId: p => p.data.sym,
+    columnDefs: [{ field: 'sym' }, ...cols.map(c => ({
+      field: c,
+      cellRenderer: p => { const el = document.createElement('span');
+        el.dataset.sym = p.data.sym; el.dataset.col = c; el.textContent = p.value ?? ''; return el; }
+    }))],
+    domLayout: 'autoHeight'
+  });
+  const fmt = new Intl.NumberFormat('en-US', { minimumFractionDigits: 2, maximumFractionDigits: 2 });
+  return { applyTick(t) {
+    api.applyTransactionAsync({ update: [{ sym: t.sym,
+      [t.col]: t.col === 'qty' ? new Intl.NumberFormat('en-US').format(t.v) : fmt.format(t.v / 100) }] });
+  } };
+}
+```
+
+(CDN imports need network access when the benchmark runs; everything else is offline.)
+
+## 2. Run
+
+From your project directory:
+
+```bash
+npx tick-bench-cli bench --impl ./my-table-adapter.mjs --name my-table
+```
+
+TickBench uses its built-in task and workload (or pass `--trace` / `--task` for your own),
+runs your adapter and the built-in human reference side by side in headless Chromium, and
+scores both.
+
+## 3. Read the results
+
+Three places, printed at the end of every run:
+
+1. **Console** — the metric comparison table, immediately.
+2. **`./tickbench-results/report-stream-grid.html`** — open in a browser: pass/fail cards
+   per submission with per-gate breakdown, plus the full metric table.
+3. **`./tickbench-results/report-stream-grid.json`** — raw numbers for CI gates, charts,
+   or tracking over time.
+
+A submission earns **PASS** only if it clears every gate: correct final values and
+formatting, staleness within budget, stable frames, no XSS (static + dynamic), and no WCAG
+violations. Thresholds live in the task's `oracle.config.json`.

package/docs/metrics.md

@@ -0,0 +1,28 @@
+# TickBench Metric Definitions
+
+All metrics are computed by the runtime oracle from frame-sampled observations of rendered
+cells against an **independent expected model** derived from the task specification (never
+from any submission).
+
+Let tick *k* for cell *c* arrive at time `a(c,k)` (when `applyTick` is dispatched), carrying
+expected display value `E(c,k)`. Let `D(c,t)` be the numeric value parsed from the rendered
+cell at frame time *t*.
+
+| Metric | Definition |
+|---|---|
+| **Catch-up latency** | For each version *k* first observed rendered at frame time *t*: `t − a(c,k)`. Reported as p50/p95/p99 across all (c,k). Sampling is frame-quantized: values are upper bounds accurate to one frame. |
+| **Stale cell-frame rate** | Fraction of (cell × frame) observations where `D(c,t) ≠ E(c, latest)` (tolerance 0.005 dollars). Measures how often users see a non-current value. |
+| **Lost update (final)** | After quiescence, cells whose rendered numeric value differs from the final expected value — the latest version was silently never displayed. |
+| **Format error (final)** | Rendered numeric value is correct but the string violates the task's locale formatting spec (e.g., missing thousands separators, wrong decimals). Monetary display correctness. |
+| **Long-frame rate** | Fraction of animation-frame deltas > 33.4 ms (two missed vsync at 60 Hz). Worst frame reported separately. |
+| **Dynamic XSS** | Trace symbol names include an injection canary; the oracle checks whether it executed (`window.__xss`). Complements static sink scanning. |
+| **Regulated-grade pass** | Conjunction of per-dimension gates with thresholds from the task's `oracle.config.json`. |
+
+Design notes:
+- **Value vs. format separation** keeps staleness (recency) orthogonal to monetary formatting
+  correctness: a value can be current but mis-formatted, or correctly formatted but stale.
+- **Coalescing is legitimate**: skipping intermediate versions is not penalized; failing to
+  ever display the *latest* version is.
+- **Batching trade-off is visible, not punished**: rAF-batched implementations show ~1 frame
+  of added staleness but better frame stability; thresholds are set so both correct designs
+  and correct trade-offs pass while unstable or incorrect ones fail.

package/docs/task-authoring.md

@@ -0,0 +1,28 @@
+# Authoring a TickBench Task
+
+A task directory contains:
+
+```
+tasks/<name>/
+  spec.md                      natural-language spec = what an agent (or human) receives
+  oracle.config.json           trace + regulated-grade thresholds (tickbench-task/1 schema)
+  reference/grid.mjs           human reference implementation (must pass regulated-grade)
+  submissions/<id>/grid.mjs    submissions under evaluation (agent output or mutants)
+  submissions/<id>/PROVENANCE.md   model, version, prompt, date — or mutant rationale
+```
+
+Contract: `createGrid(container, symbols, cols) -> { applyTick(tick) }`; rendered cells must
+carry `data-sym` and `data-col` attributes (the observable surface the oracle samples).
+
+Rules:
+1. The reference must achieve `regulated_grade_pass` (`tickbench validate`).
+2. Every oracle you rely on needs at least one **validation mutant** listed in
+   `validationMutants` that the oracle demonstrably fails.
+3. Traces: synthetic traces are generated (`tickbench gen`, seeded). Real-feed traces
+   (recorded WebSocket market data, e.g. public crypto feeds) must set `"synthetic": false`
+   and ship a `*.provenance.md` (source, window, sample size) next to the trace file.
+4. Do not leak the oracle into the spec verbatim; specs state requirements, oracles verify them.
+
+## Trace data license
+
+Trace files under `traces/` (synthetic or recorded) are licensed **CC BY 4.0**; the harness code is MIT. Recorded traces must be anonymized: no session tokens, no PII, provenance limited to source tool, window, and sample size.

package/examples/my-table-adapter.mjs

@@ -0,0 +1,57 @@
+// EXAMPLE ADAPTER — copy this file into your project and edit the marked lines.
+//
+// TickBench doesn't care what framework your table uses. It only needs this contract:
+//   createGrid(container, symbols, cols) -> { applyTick(tick) }
+// and rendered cells carrying data-sym / data-col attributes so the oracle can read them.
+//
+// Replace the internals below with calls into YOUR component:
+//   - mount:   render your table into `container` (ReactDOM.createRoot, new AgGrid, etc.)
+//   - update:  route `applyTick` into your component's data path (setState, applyTransaction…)
+//
+// Framework imports must be full URLs (the harness loads this file as a module in a bare
+// browser page — no bundler), e.g.:  import { createRoot } from 'https://esm.sh/react-dom@18/client'
+
+export function createGrid(container, symbols, cols) {
+  // --- mount: REPLACE with your component's mount code -------------------
+  const priceFmt = new Intl.NumberFormat('en-US', { minimumFractionDigits: 2, maximumFractionDigits: 2 });
+  const qtyFmt = new Intl.NumberFormat('en-US');
+  const table = document.createElement('table');
+  table.setAttribute('aria-label', 'My table under test');
+  const caption = document.createElement('caption');
+  caption.textContent = 'My table under test';
+  table.appendChild(caption);
+  const cells = new Map();
+  const thead = document.createElement('thead'); const hr = document.createElement('tr');
+  const h0 = document.createElement('th'); h0.scope = 'col'; h0.textContent = 'Symbol'; hr.appendChild(h0);
+  for (const c of cols) { const th = document.createElement('th'); th.scope = 'col'; th.textContent = c; hr.appendChild(th); }
+  thead.appendChild(hr); table.appendChild(thead);
+  const tbody = document.createElement('tbody');
+  for (const s of symbols) {
+    const tr = document.createElement('tr');
+    const th = document.createElement('th'); th.scope = 'row'; th.textContent = s; tr.appendChild(th);
+    for (const c of cols) {
+      const td = document.createElement('td');
+      td.dataset.sym = s; td.dataset.col = c;      // <-- REQUIRED: observable surface
+      tr.appendChild(td); cells.set(s + '|' + c, td);
+    }
+    tbody.appendChild(tr);
+  }
+  table.appendChild(tbody); container.appendChild(table);
+
+  // --- update: REPLACE with your component's update path ------------------
+  const pending = new Map(); let scheduled = false;
+  const flush = () => {
+    scheduled = false;
+    for (const [k, t] of pending) {
+      const td = cells.get(k);
+      if (td) td.textContent = t.col === 'qty' ? qtyFmt.format(t.v) : priceFmt.format(t.v / 100);
+    }
+    pending.clear();
+  };
+  return {
+    applyTick(tick) {           // tick = { t, sym, col, v }  (prices are integer cents)
+      pending.set(tick.sym + '|' + tick.col, tick);
+      if (!scheduled) { scheduled = true; requestAnimationFrame(flush); }
+    }
+  };
+}