@workos/quickstudy 0.0.1

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 quickstudy contributors
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -0,0 +1,270 @@
+# quickstudy
+
+If you own a developer platform, coding agents raise two questions you can't
+currently answer: **can an agent actually complete a real task with your
+product**, and **does changing the developer surface change that outcome**?
+quickstudy is an open-source, company-agnostic eval harness that measures both.
+It runs coding agents against real integration tasks under different
+**experiment treatments**, grades each attempt in isolation, and reports pass
+rates, per-check outcomes, and raw counts for declared comparison groups.
+
+If you want DX investment to go where it measurably moves the needle, this is
+for you. ([Concepts](./docs/concepts.md) unpacks both questions.)
+
+## What you get
+
+Every run produces a report with per-pair stats, from fact to interpretation:
+
+```
+run 01J…: 60 attempts — 41 passed, 17 failed, 0 incomplete, 2 errored
+
+acme-nextjs × claude-docs   1/10 passed (10% pass rate)
+  build_passes 10/10  auth_flow_wired 4/10  login_flow_e2e 1/10 …
+acme-nextjs × claude-mcp   10/10 passed (100% pass rate)
+```
+
+One row per **eval × experiment** pair: trials passed, per-check pass counts,
+and whether the pair ran all its trials. Comparison groups check recorded
+configuration agreement before presenting raw per-arm counts. Matching a group
+does not establish causation or eliminate timing, environment, service or sampling
+confounds. Incomplete attempts and instrument errors are disclosed separately and
+excluded from scored pass-rate denominators.
+
+The same data also renders as an interactive, self-contained static
+[site](#ui--sharing-results) you can publish or hand to a teammate.
+
+## Install
+
+The npm package is **Bun-only**: it ships TypeScript source directly (the
+`quickstudy` bin is a `bun`-shebang script and the library exports `.ts`
+modules), so it requires [Bun](https://bun.sh) ≥ 1.1 — there is no Node
+build. Add it to the repository that holds your evals and experiments:
+
+```bash
+bun add quickstudy       # Bun >= 1.1 required; no Node build exists
+bunx quickstudy --version
+```
+
+Your `EVAL.ts` scorers and experiment modules import harness types by package
+name (`import type { EvalContext } from "quickstudy"`) — see
+[Writing evals](#writing-evals) below and
+[docs/writing-an-eval.md](./docs/writing-an-eval.md) for the authoring loop.
+Container-based experiments additionally need the harness images built once
+with `quickstudy images build` (requires Docker).
+
+No Bun at all? Download the archive for your OS and architecture from
+[GitHub Releases](https://github.com/workos/quickstudy/releases) instead. Each
+archive contains one self-contained `quickstudy` executable (CLI + UI), with
+no Bun runtime or source checkout required. Release assets are named
+`quickstudy-{linux|darwin}-{x64|arm64}.tar.gz`.
+
+To build that binary from source (requires Bun):
+
+```bash
+bun install
+bun run build            # → dist/quickstudy  (the whole CLI + UI in one file)
+mv dist/quickstudy /usr/local/bin/     # or anywhere on your PATH
+```
+
+Prefer running from source? Replace `quickstudy` with `bun src/cli.ts` in any
+command below (see [Development](#development)).
+
+## Quickstart
+
+Only **Docker** and an agent **API key** are needed to run a real agent — the
+first commands need neither. An eval is a directory (`PROMPT.md` + `EVAL.ts`
+scorer + optional `local/` starting state); an experiment is one TypeScript
+module. Author a toy pair and run it end-to-end with the no-op `echo` agent:
+
+```bash
+# 1. author a one-check eval and a host echo experiment
+mkdir -p demo-evals/hello demo-experiments
+printf -- '---\nid: hello\nsuite: benchmark\n---\n\nWrite NOTES.md describing the task.\n' > demo-evals/hello/PROMPT.md
+printf -- 'export default async (ctx) => { const ok = await ctx.fileExists("NOTES.md"); return { passed: ok, checks: [{ name: "notes-written", passed: ok }] }; };\n' > demo-evals/hello/EVAL.ts
+printf -- 'export default { id: "echo-demo", agent: { adapter: "echo" }, runtime: { kind: "host" } };\n' > demo-experiments/echo-demo.ts
+
+# 2. validate both roots, then run — no Docker, no keys
+quickstudy validate --evals-root demo-evals --experiments-root demo-experiments
+quickstudy run --eval hello --experiment echo-demo --evals-root demo-evals --experiments-root demo-experiments --trials 1
+
+# 3. read the report (step 2 prints a run id)
+quickstudy report <run-id>
+```
+
+`echo` is a pipeline smoke, not a solver — it copies the starting state and
+writes NOTES.md, so this toy check passes; against real evals every check
+fails, which is exactly what a negative control should do. Completing the
+run → score → persist loop is what it proves.
+
+Real measurements come from a benchmark repository built on quickstudy: an
+`evals/` root of real integration tasks (tiny HTTP apps under `local/`) and
+an `experiments/` root of real agent recipes. From such a checkout, running
+one costs a container image build and an API key:
+
+```bash
+export ANTHROPIC_API_KEY=sk-...
+quickstudy images build                 # one-time: runtime + proxies
+docker build -t acme/nextjs images/frameworks/nextjs
+quickstudy run --eval acme-nextjs --experiment claude-docs --trials 1
+```
+
+(Bun auto-loads a gitignored `.env` from the working directory, so keys can
+live there instead of `export` lines.) Model and reasoning pins live in the
+experiment module — changing them changes the experiment's identity, by
+design. See the benchmark repository's `examples/weekly-sweep.yml` for what
+a full scheduled sweep costs and looks like in CI.
+
+## How it works
+
+Every attempt is one point in a 2-D plan:
+
+| Axis           | Varies                                                       | Defined by                     |
+| -------------- | ------------------------------------------------------------ | ------------------------------ |
+| **eval**       | WHAT is attempted: prompt, starting state, scorer            | an `evals/<id>/` directory     |
+| **experiment** | HOW it runs: agent, model pins, runtime, treatment           | an `experiments/<id>.ts` module |
+
+An experiment's **runtime** owns everything the environment offers the agent:
+per-attempt provisioning, the container image, egress policy, MCP servers,
+native-web policy, and a PATH treatment. Experiments sharing a
+`comparisonGroup` are compared as treatments of one configuration; a group
+whose members disagree on anything beyond the declared treatment has
+automatic comparison withheld.
+
+Before the first paid attempt, quickstudy writes an immutable run manifest
+covering the harness source, each eval's effective prompt / starting state /
+scorer, and each experiment's agent pins, runtime, and module source.
+Identity drift between runs is diffed, never silently averaged.
+
+## Commands
+
+```
+quickstudy validate                   # discover + validate evals/ and experiments/
+quickstudy run [--eval <id>] [--experiment <id>] # execute selected pairs; omit selectors for all
+quickstudy report <run-id>            # per-pair stats (--latest for the newest run)
+quickstudy diagnose <run-id>          # LLM-diagnose failing pairs into diagnosis.json
+quickstudy ui                         # serve the matrix/compare/report/diagnosis UI
+quickstudy export <run-id> --out dir  # the same UI as a self-contained static site
+quickstudy images build               # build the agent-runtime + proxy images
+quickstudy clean                      # remove all quickstudy-labeled containers/networks
+```
+
+Run `quickstudy --help` for every flag (`--trials`, `--db`, `--egress-proxy`,
+`--evals-root`, `--experiments-root`, …).
+
+### Running a whole benchmark repository
+
+Both run selectors are optional. Omitting `--eval` selects every discovered
+eval; omitting `--experiment` selects every discovered experiment. Together,
+this runs the complete eval × experiment cross-product:
+
+```bash
+quickstudy validate
+quickstudy run --trials 1
+quickstudy report --latest --strict
+quickstudy export --runs all --out ./site
+quickstudy ui --watch
+```
+
+`ui` is a blocking local server, so it is normally the final command. `export`
+creates the non-blocking, self-contained site artifact. A complete run can be
+expensive: inspect the counts printed by `validate`, then multiply selected
+evals × experiments × trials before starting. Benchmark repositories are
+encouraged to wrap this sequence in a checked-in script so it can preserve the
+new run id across reporting, diagnosis, export, and UI serving.
+
+`diagnose` is the one command that calls a model after the fact: it sends each
+failing pair's harness-selected evidence (per-check tallies, transcript
+excerpts, egress denials) to a structured LLM call and writes `diagnosis.json`
+beside `report.json` — findings are evidence-linked hypotheses, never
+verdicts, keeping the report itself attribution-free. Requires
+`ANTHROPIC_API_KEY`; the UI renders the result as a Diagnosis view.
+
+## Writing evals
+
+An eval is a directory: `PROMPT.md` (frontmatter + the task prompt), `EVAL.ts`
+(a default-exported scorer), and an optional `local/` tree the attempt starts
+from. The harness core knows nothing company-specific; your evals and
+scorers may name any vendor freely.
+
+```ts
+import type { EvalContext, EvalResult } from "quickstudy";
+
+export default async function score(ctx: EvalContext): Promise<EvalResult> {
+  const wired = await ctx.fileExists("app/callback/route.ts");
+  return { passed: wired, checks: [{ name: "callback_wired", passed: wired }] };
+}
+```
+
+Scorers get file helpers over the exported workspace, `exec` into the live
+sandbox, and any `query`/`getClient` capabilities the experiment's runtime
+provides. Shared check helpers live in your benchmark repository next to the
+evals that use them. [docs/writing-an-eval.md](./docs/writing-an-eval.md)
+walks the whole authoring loop.
+
+To point quickstudy at **your own product**: author an evals root and an
+experiments root in a repository of your own — the toy pair above is the
+smallest complete example, and `--evals-root`/`--experiments-root` point the
+CLI anywhere. The harness core carries no vendor assumptions — a test suite
+enforces that `src/`, `ui/`, and `scripts/` stay vendor-clean — so nothing in
+this repository needs to change.
+
+## UI & sharing results
+
+One renderer, two delivery modes — both pure static files, no server API. The
+compiled binary carries the UI shell, so there's nothing to build first; when
+running from source, build the bundle once with `bun run ui:build`:
+
+```bash
+quickstudy ui                             # serve every run at http://127.0.0.1:4173
+quickstudy ui --watch                     # re-export when the results DB changes
+
+quickstudy export <run-id> --out ./site        # self-contained site
+quickstudy export --runs a,b --out ./site       # two runs → browse each run; Compare stays within a run
+```
+
+The exported directory works from `file://` (the data index is inlined into
+`index.html`; transcripts and diffs lazy-load when served), so you can zip it,
+attach it, or publish it. GitHub Pages recipe:
+
+```bash
+quickstudy export <run-id> --out ./site
+npx gh-pages -d site        # or push ./site to any static host
+```
+
+## Development
+
+Run the CLI from source as `bun src/cli.ts <cmd>` (what `quickstudy` is,
+uncompiled).
+
+```bash
+bun install
+bun test                 # all suites (Docker-gated ones auto-enable when present)
+bun run typecheck        # tsc --noEmit (harness) + tsc -p ui
+bun run lint             # oxlint
+bun run build            # compile the self-contained binary → dist/quickstudy
+bun src/cli.ts validate
+```
+
+UI development runs against committed seed data — no Docker, agents, or keys:
+
+```bash
+bun run ui:dev           # Vite dev server + ui/dev-data at /data (hot reload)
+bun run ui:dev-data      # regenerate ui/dev-data from the seeders
+bun run ui:build         # production bundle (ui/dist) + gzip budget check
+bunx playwright install  # one-time: the browsers ui:e2e drives
+bun run ui:e2e           # Playwright smoke: served routes + offline file:// load
+```
+
+## Documentation
+
+- **[docs/concepts.md](./docs/concepts.md)** — what it measures and why.
+- **[docs/getting-started.md](./docs/getting-started.md)** — install to first report.
+- **[docs/writing-an-eval.md](./docs/writing-an-eval.md)** — authoring evals and experiments.
+- **[docs/publishing-a-benchmark.md](./docs/publishing-a-benchmark.md)** — gates for publishing results.
+- **[docs/releasing.md](./docs/releasing.md)** — cutting prebuilt binaries.
+
+## License
+
+MIT
+
+See [extension contracts, identity, resume and complete-sample publication](docs/extensions.md) and the [runnable Harbor Notes example](examples/harbor-notes/README.md).

package/examples/harbor-notes/README.md

@@ -0,0 +1,40 @@
+# Harbor Notes: a complete second-product example
+
+This example uses Quickstudy's public package API. Its prompt, fixture, scorer,
+experiment, provisioning runtime, policy, and telemetry targets live here.
+The basic path runs a scripted agent in Alpine and needs Bun plus Docker and
+local `alpine:3` and `quickstudy/egress-proxy` images. No account or key is used.
+
+From the Quickstudy checkout, run the isolated consumer smoke:
+
+```bash
+bun scripts/portable-smoke.ts
+```
+
+It copies this example to a temporary consumer directory, resolves imports by
+package name, validates, runs two trials with concurrency 2, and checks the
+strict report. Provisioning creates a temporary notebook, the scorer queries
+its expected document while the sandbox is alive, and teardown deletes it.
+The script deletes its temporary results when finished.
+
+To keep your results, work from this directory: `bun install`, then
+`bunx quickstudy validate`, `bunx quickstudy run --trials 2 --egress-proxy`,
+and `bunx quickstudy report --latest --strict`. Use the printed run ID with
+`bunx quickstudy export RUN_ID --expected-trials 2 --mode hosted --out site`.
+Serve `site/` with any static server. Use `--mode offline` to embed run data for
+file sharing; transcripts and diffs require a local server in either mode.
+
+For a real agent, build `quickstudy/agent-runtime` and the two proxy images
+using `quickstudy images build`. Add an experiment using `harborRuntime(false)`
+and `agent: { adapter: "claude", provider: "anthropic", model: "YOUR_EXACT_MODEL_ID",
+reasoning: "high" }`, or the supported Codex adapter with its exact model and
+reasoning pins. Set the corresponding provider key. Select that experiment
+explicitly; it makes paid calls. The runtime offers no product network hosts;
+with `--egress-proxy` the adapter's provider endpoints are the only allowed
+external destinations. `containerImage()` selects the agent image, and all
+other runtime/scorer code is shared with the local demonstration.
+
+`semantic-example.ts` demonstrates the optional rubric interface with a fake
+judge. It is deliberately outside the eval discovery tree. See
+[extension contracts](../../docs/extensions.md) for evidence limits, identity,
+error handling and calibration before substituting a live judge.

package/examples/harbor-notes/evals/create-note/EVAL.ts

@@ -0,0 +1,14 @@
+import type { EvalScorer } from "quickstudy";
+const score: EvalScorer = async (ctx) => {
+  const expected = await ctx.query("expected-note") as { title: string; body: string };
+  let saved: unknown;
+  try { saved = JSON.parse(await ctx.readFile("note.json")); } catch { saved = null; }
+  const document = saved as { title?: string; body?: string } | null;
+  const live = await ctx.exec(["test", "-f", "note.json"]);
+  const checks = [
+    { name: "note-persisted", passed: document?.title === expected.title && document?.body === expected.body && live.exitCode === 0 },
+    { name: "starting-state-preserved", passed: await ctx.fileExists("README.txt") },
+  ];
+  return { passed: checks.every((check) => check.passed), checks };
+};
+export default score;

package/examples/harbor-notes/evals/create-note/PROMPT.md

@@ -0,0 +1,9 @@
+---
+id: create-note
+suite: benchmark
+product: harbor-notes
+---
+
+Create a persisted note in `note.json` with title `First note` and body
+`Hello from Harbor`. Keep `README.txt`. The final report should explain the
+saved note. The note file must contain a JSON object with `title` and `body`.

package/examples/harbor-notes/evals/create-note/local/README.txt

@@ -0,0 +1 @@
+Harbor Notes stores each note as a JSON document.

package/examples/harbor-notes/experiments/scripted.ts

@@ -0,0 +1,6 @@
+import { defineExperiment } from "quickstudy";
+import { harborRuntime } from "../runtime.ts";
+export default defineExperiment({
+  id: "scripted", agent: { adapter: "scripted", provider: "local", model: "script-v1", reasoning: "none", cliVersion: "built-in" },
+  runtime: harborRuntime(),
+});

package/examples/harbor-notes/package.json

@@ -0,0 +1,6 @@
+{
+  "name": "harbor-notes-evals",
+  "private": true,
+  "type": "module",
+  "dependencies": { "quickstudy": "file:../.." }
+}

package/examples/harbor-notes/quickstudy.identity.json

@@ -0,0 +1 @@
+{ "version": 1, "sources": ["runtime.ts"] }

package/examples/harbor-notes/runtime.ts

@@ -0,0 +1,48 @@
+import { mkdtemp, rm, writeFile, readFile } from "node:fs/promises";
+import { tmpdir } from "node:os";
+import { join } from "node:path";
+import type { Runtime } from "quickstudy";
+
+/** A complete local provisioning lifecycle; replace this notebook store with your API. */
+export function harborRuntime(scripted = true): Runtime {
+  return {
+    kind: "container",
+    config: { notebookSchema: 1, scripted },
+    containerImage: () => (scripted ? "alpine:3" : "quickstudy/agent-runtime"),
+    ...(scripted
+      ? { command: ["sh", "-c", `printf '%s\n' '{"title":"First note","body":"Hello from Harbor"}' > note.json`] }
+      : {}),
+    egressHosts: () => [],
+    webPolicy: () => "native-web-blocked",
+    // Observation does not offer a docs or CLI treatment. A scripted run has
+    // no supported agent telemetry stream and records usage as unavailable.
+    observationTargets: { docsHosts: ["docs.harbor.example"], cliCommands: ["harbor"] },
+    async provision(ctx) {
+      ctx.signal?.throwIfAborted();
+      const notebook = await mkdtemp(join(tmpdir(), "harbor-notebook-"));
+      try {
+        await writeFile(
+          join(notebook, "expected.json"),
+          JSON.stringify({ title: "First note", body: "Hello from Harbor" }),
+        );
+        ctx.signal?.throwIfAborted();
+        return {
+          env: { HARBOR_NOTEBOOK: notebook },
+          teardown: async () => {
+            await rm(notebook, { recursive: true, force: true });
+          },
+        };
+      } catch (error) {
+        await rm(notebook, { recursive: true, force: true });
+        throw error;
+      }
+    },
+    scorerCapabilities: (ctx) => ({
+      query: async (request) => {
+        ctx.signal?.throwIfAborted();
+        if (request !== "expected-note") throw new Error("unknown notebook query");
+        return JSON.parse(await readFile(join(ctx.env.HARBOR_NOTEBOOK!, "expected.json"), "utf8"));
+      },
+    }),
+  };
+}

package/examples/harbor-notes/semantic-example.ts

@@ -0,0 +1,21 @@
+import { defineSemanticScorer } from "quickstudy";
+
+/** Local demonstration only: the fake judge has no network or credential dependency. */
+export const semanticExample = defineSemanticScorer(
+  {
+    name: "explains-note",
+    rubric: "The final report explains the saved note's title and body.",
+    provider: "local-fake",
+    model: "rubric-fixture-v1",
+    includeFinalReport: true,
+  },
+  async (request) => ({
+    verdict: {
+      passed: request.evidence.some(
+        (entry) => entry.text.includes("First note") && entry.text.includes("Hello from Harbor"),
+      ),
+      reason: "Local fixture matches the rubric vocabulary; calibrate a real judge separately.",
+      evidence: ["agent:final-report"],
+    },
+  }),
+);

package/images/agent-runtime/Dockerfile

@@ -0,0 +1,58 @@
+# quickstudy agent-runtime base image.
+#
+# Debian-slim + git + node, with the agent CLIs installed at PINNED versions:
+# vendor CLIs change their stream formats between releases, and the parsers in
+# src/adapters/parse.ts are tested against fixtures captured from these exact
+# versions. Bump the pins deliberately (`quickstudy images build --pull`) and
+# re-capture fixtures when you do.
+#
+# Framework toolchains (Rails, Django, ...) layer on top in Phase 3 — this
+# image stays agent-only.
+
+FROM node:22-bookworm-slim
+
+ARG CLAUDE_CODE_VERSION=2.1.215
+ARG CODEX_VERSION=0.149.1
+ARG GEMINI_CLI_VERSION=0.51.0
+# cursor-agent is not an npm package: it ships as versioned tarballs (the
+# vendor install script pins the same way). Fixture streams in
+# tests/__fixtures__/streams/cursor/ correspond to this pin.
+ARG CURSOR_AGENT_VERSION=2026.07.17-3e2a980
+
+RUN apt-get update \
+  && apt-get install -y --no-install-recommends ca-certificates curl git procps ripgrep \
+  && rm -rf /var/lib/apt/lists/*
+
+RUN npm install -g --no-fund --no-audit \
+  "@anthropic-ai/claude-code@${CLAUDE_CODE_VERSION}" \
+  "@openai/codex@${CODEX_VERSION}" \
+  "@google/gemini-cli@${GEMINI_CLI_VERSION}"
+
+RUN arch="$(dpkg --print-architecture)" \
+  && case "$arch" in \
+       amd64) cursor_arch=x64 ;; \
+       arm64) cursor_arch=arm64 ;; \
+       *) echo "unsupported architecture for cursor-agent: $arch" >&2; exit 1 ;; \
+     esac \
+  && mkdir -p /opt/cursor-agent \
+  && curl -fsSL "https://downloads.cursor.com/lab/${CURSOR_AGENT_VERSION}/linux/${cursor_arch}/agent-cli-package.tar.gz" \
+     | tar --strip-components=1 -xzf - -C /opt/cursor-agent \
+  && ln -s /opt/cursor-agent/cursor-agent /usr/local/bin/cursor-agent
+
+# The fixture-baseline commit (and any commits the agent makes) need an
+# identity; system-level config applies to whichever user runs.
+RUN git config --system user.name "quickstudy" \
+  && git config --system user.email "quickstudy@localhost" \
+  && git config --system init.defaultBranch main \
+  && git config --system --add safe.directory /workspace
+
+# Claude Code refuses --dangerously-skip-permissions as root unless it can
+# tell it is inside a sandbox. The container IS the security boundary here —
+# that is the entire point of this phase.
+ENV IS_SANDBOX=1
+
+WORKDIR /workspace
+
+# The harness drives everything through `docker exec`; the main process just
+# keeps the container alive until teardown.
+CMD ["sleep", "infinity"]

package/images/egress-proxy/Dockerfile

@@ -0,0 +1,28 @@
+# quickstudy egress-proxy sidecar image.
+#
+# One per run: attempt containers sit on an internal Docker network with no
+# default route, and this proxy (attached to BOTH the internal network and
+# the bridge) is their only way out — an HTTP CONNECT / absolute-form forward
+# proxy enforcing the run's host allowlist (src/isolation/proxy/server.ts).
+# No TLS interception: for HTTPS the proxy sees only the CONNECT host:port.
+#
+# Build context is src/isolation/proxy (the image needs exactly two files):
+#   docker build -t quickstudy/egress-proxy -f images/egress-proxy/Dockerfile src/isolation/proxy
+# or simply `quickstudy images build`, which builds this alongside the
+# agent-runtime image.
+#
+# Config via env at `docker run`:
+#   QUICKSTUDY_EGRESS_ALLOWLIST  comma-separated hosts / *.wildcards (required)
+#   QUICKSTUDY_EGRESS_PORT       listen port (default 3128)
+
+FROM oven/bun:1.3-slim
+
+WORKDIR /app
+
+COPY allowlist.ts server.ts ./proxy/
+
+EXPOSE 3128
+
+# Decisions (allowed + denied) stream to stdout as JSON lines; the harness
+# reads them back with `docker logs` for per-attempt denial artifacts.
+ENTRYPOINT ["bun", "/app/proxy/server.ts"]

package/images/mcp-proxy/Dockerfile

@@ -0,0 +1,30 @@
+# quickstudy mcp-proxy sidecar image.
+#
+# One per run, only when the runtime's mcpServers declare `auth`: attempt
+# containers speak credential-free HTTP to this proxy, which forwards to the
+# real MCP server injecting a Bearer access token it mints (and re-mints)
+# from a refresh token (src/isolation/mcp-proxy/server.ts). Refresh tokens
+# rotate on use; the current one is persisted inside the container at
+# /run/quickstudy-mcp/ for the harness to read back at teardown.
+#
+# Build context is src/isolation/mcp-proxy (the image needs exactly one file):
+#   docker build -t quickstudy/mcp-proxy -f images/mcp-proxy/Dockerfile src/isolation/mcp-proxy
+# or simply `quickstudy images build`, which builds this alongside the
+# agent-runtime and egress-proxy images.
+#
+# Config via env at `docker run` (secrets arrive via --env-file, never argv):
+#   QUICKSTUDY_MCP_UPSTREAMS             JSON {name: {url, tokenEndpoint, clientId, resource?}}
+#   QUICKSTUDY_MCP_REFRESH_TOKEN_<NAME>  initial refresh token per server
+#   QUICKSTUDY_MCP_PORT                  listen port (default 8914)
+
+FROM oven/bun:1.3-slim
+
+WORKDIR /app
+
+COPY server.ts ./mcp-proxy/
+
+EXPOSE 8914
+
+# Events (forwards + token refreshes) stream to stdout as JSON lines; token
+# values never appear in them.
+ENTRYPOINT ["bun", "/app/mcp-proxy/server.ts"]

package/package.json

@@ -0,0 +1,53 @@
+{
+  "name": "@workos/quickstudy",
+  "version": "0.0.1",
+  "description": "An open-source, company-agnostic eval harness for measuring whether coding agents can complete real integration tasks, across an agent x feature x framework x surface matrix.",
+  "license": "MIT",
+  "type": "module",
+  "bin": {
+    "quickstudy": "src/cli.ts"
+  },
+  "exports": {
+    ".": "./src/index.ts",
+    "./package.json": "./package.json"
+  },
+  "files": [
+    "src",
+    "ui/dist",
+    "images",
+    "examples",
+    "README.md",
+    "LICENSE"
+  ],
+  "engines": {
+    "bun": ">=1.1"
+  },
+  "scripts": {
+    "prepack": "bun run ui:build",
+    "typecheck": "tsc --noEmit && tsc -p ui",
+    "lint": "oxlint src tests ui scripts",
+    "test": "bun test",
+    "build": "bun scripts/build-binary.ts",
+    "ui:dev": "vite --config ui/vite.config.ts",
+    "ui:build": "vite build --config ui/vite.config.ts && bun scripts/check-bundle-size.ts",
+    "ui:e2e": "playwright test --config ui/e2e/playwright.config.ts",
+    "ui:dev-data": "bun scripts/make-dev-data.ts"
+  },
+  "dependencies": {
+    "@anthropic-ai/sdk": "^0.124.0",
+    "yaml": "^2.9.0",
+    "zod": "^4.6.1"
+  },
+  "devDependencies": {
+    "@playwright/test": "^1.63.0",
+    "@types/bun": "^1.4.2",
+    "@types/react": "^19",
+    "@types/react-dom": "^19",
+    "@vitejs/plugin-react": "^6.1.1",
+    "oxlint": "^1.82.0",
+    "react": "^19.3.0",
+    "react-dom": "^19.3.0",
+    "typescript": "^7.0.2",
+    "vite": "^8.3.0"
+  }
+}