devmethod-ai 0.1.0-rc.2 → 0.1.0

package/docs/images/devmethod-flow.svg

@@ -0,0 +1,26 @@
+<svg xmlns="http://www.w3.org/2000/svg" width="1200" height="420" viewBox="0 0 1200 420" role="img" aria-labelledby="title desc">
+  <title id="title">DevMethod workflow</title>
+  <desc id="desc">A project moves from clarity through a verified delivery to a reusable checkpoint.</desc>
+  <rect width="1200" height="420" rx="28" fill="#0b1020"/>
+  <text x="72" y="86" fill="#ffffff" font-family="Arial, sans-serif" font-size="38" font-weight="700">From an idea to a verified delivery</text>
+  <text x="72" y="124" fill="#aeb9d7" font-family="Arial, sans-serif" font-size="20">DevMethod keeps decisions, implementation and evidence connected.</text>
+  <g font-family="Arial, sans-serif">
+    <rect x="72" y="190" width="290" height="144" rx="20" fill="#182347" stroke="#7190ff" stroke-width="2"/>
+    <circle cx="116" cy="234" r="20" fill="#7190ff"/><text x="108" y="241" fill="#0b1020" font-size="18" font-weight="700">1</text>
+    <text x="154" y="240" fill="#fff" font-size="25" font-weight="700">Make it clear</text>
+    <text x="104" y="286" fill="#c8d2ef" font-size="18">Explore · frame · design</text>
+    <text x="104" y="314" fill="#c8d2ef" font-size="18">Architecture · plan · ready</text>
+    <path d="M390 262h88" stroke="#7190ff" stroke-width="5" stroke-linecap="round"/><path d="M468 250l22 12-22 12" fill="#7190ff"/>
+    <rect x="512" y="190" width="290" height="144" rx="20" fill="#182347" stroke="#4dd6ad" stroke-width="2"/>
+    <circle cx="556" cy="234" r="20" fill="#4dd6ad"/><text x="548" y="241" fill="#0b1020" font-size="18" font-weight="700">2</text>
+    <text x="594" y="240" fill="#fff" font-size="25" font-weight="700">Build the slice</text>
+    <text x="544" y="286" fill="#c8d2ef" font-size="18">Implement · test · review</text>
+    <text x="544" y="314" fill="#c8d2ef" font-size="18">Correct when evidence fails</text>
+    <path d="M830 262h88" stroke="#4dd6ad" stroke-width="5" stroke-linecap="round"/><path d="M908 250l22 12-22 12" fill="#4dd6ad"/>
+    <rect x="952" y="190" width="176" height="144" rx="20" fill="#182347" stroke="#ffcf5b" stroke-width="2"/>
+    <circle cx="996" cy="234" r="20" fill="#ffcf5b"/><text x="988" y="241" fill="#0b1020" font-size="18" font-weight="700">3</text>
+    <text x="984" y="240" fill="#fff" font-size="23" font-weight="700">Prove it</text>
+    <text x="984" y="286" fill="#c8d2ef" font-size="17">Verify · integrate</text>
+    <text x="984" y="314" fill="#c8d2ef" font-size="17">Handoff · next</text>
+  </g>
+</svg>

package/evaluation/COMPARISONS.md

@@ -0,0 +1,33 @@
+# Matched comparison records
+
+Run `node scripts/comparison.mjs BATCH_JSON` to validate and summarize a completed or blocked batch. This is an offline reporting tool, not a host runner or proof that a transcript is authentic. Do not use test records as published evaluation evidence.
+
+The JSON root has `format: 1`, `budget` and `runs`. Budget requires positive integer `runs` and `tokens`, nonnegative finite `costUSD`, and an `authorization` reference to the user's explicit batch approval. A missing budget blocks native dispatch, not fixture/tool development.
+
+Every run has:
+
+| Fields | Meaning |
+|---|---|
+| `caseId`, `repetition`, `arm` | Positive repetition; arm is `none`, `devmethod` or `bmad` |
+| `fixtureDigest`, `promptDigest` | SHA-256 of the pinned case manifest and identical task prompt |
+| `hostVersion`, `model` | Exact installed host and model identifiers |
+| `toolsDigest`, `permissionsDigest` | SHA-256 of retained tool and permission configuration |
+| `perRunTokens`, `perRunCostUSD`, `timeoutSeconds` | Identical positive token/time and nonnegative USD ceilings in each matched triple |
+| `methodRevision`, `wrapper` | Exact reviewed method commit/package integrity and native invocation; control revision is `none` |
+| `status` | `passed`, `failed`, `blocked` or `timeout` |
+| `evidence`, `review` | Retained evidence manifest reference plus independent review record, or explicit environment blocker |
+| `tokens`, `costUSD`, `elapsedSeconds` | Trustworthy measured nonnegative numbers, otherwise JSON `null` |
+
+Provide all three arms for each case/repetition, including blocked environments. The validator refuses duplicates, mismatched conditions and measured budget excesses. Unknown usage remains unknown and must block further dispatch under a hard budget. A validator cannot enforce host spend or authenticate the authorization string. The supervising operator owns those checks before each run.
+
+Preparation and collection follow [the native evaluation workspace](README.md). Pin an actually installed BMAD distribution and its supported invocation before any matched batch; a live documentation link is not a version. Establish the same host configuration, clean workspace, task prompt, acceptance tests and limits for all arms. Rotate arm order across repetitions. Record method setup separately from task time, all interventions and repair iterations. Review anonymized diffs where feasible and report any loss of blinding.
+
+The report gives pass/fail/blocked/timeout counts with denominators for each case and arm. It does not infer superiority from an incomplete batch, unknown usage, or installer tests. The protocol's minimum three independent repetitions remains necessary before a case-level comparative claim.
+
+## Current checkpoint
+
+Tooling implemented; no matched model batch executed. Budget approval is pending. Each host requires a locally verified authenticated environment. BMAD is not pinned/installed for this batch. B2 has no pinned React project/approved screen. Resolve these prerequisites before native dispatch. No orchestrator should treat this tooling's unit tests as a passed comparison gate.
+
+## rc.2 setup observation
+
+On 2026-09-13, BMAD npm 6.12.0 was installed with lifecycle scripts disabled in an isolated temporary directory, then its core/BMM Codex export was actually generated with English configuration and the fictional user Evaluator. Package integrity, export digest and host version probes are in [candidate-setup.json](candidate-setup.json). This proves local setup only, not native workflow behavior. Regenerate clean per-arm exports and snapshot all bytes/configuration before a future budgeted batch; the temporary setup is not a reusable evaluation session. The earlier “not installed” checkpoint above is historical. Model and budget remain unset; no runs have been fabricated to fill a comparison table.

package/evaluation/README.md

@@ -0,0 +1,35 @@
+# Native evaluation workspace
+
+This repository tooling prepares fixtures and collects local checks. It never calls a model, and its output is not native host evidence. Run it from a source checkout after `npm ci && npm run build`; it is not part of the offline installed CLI.
+
+## Prepare and collect
+
+```sh
+node scripts/evaluation.mjs prepare B1 /absolute/fresh/case > /absolute/baseline.json
+# Install the chosen pinned method in the case, or no method for the control arm.
+# Save a new complete snapshot after setup, retaining the pinned fixture hashes.
+# Invoke the actual installed host with the exact fixtures.json prompt.
+node scripts/evaluation.mjs collect B1 /absolute/fresh/case /absolute/baseline.json > /absolute/checks.json
+```
+
+For method arms, use the exported `snapshot()` to record the full post-installation baseline in the baseline record before host execution. Retain its original prompt. Never reuse a workspace or native session across arms. `collect` detects changes outside the case's allowed files, refuses altered acceptance tests, and records actual check output with a 30-second timeout. Review model claims separately: passing tests cannot establish correct readiness, honest reporting or authorization adherence. Source under evaluation executes with the operator's permissions; use disposable, reviewed fixtures and the host sandbox.
+
+Fixtures B1, B3, B4 and B5 have content hashes, identical arm prompts, fixed checks and explicit allowed changes. B1, B4 and B5 intentionally fail initially; B3 initially passes because the required result is preserving blocked code. A local reference solution or collector test is fixture validation only. B2 remains blocked on a pinned React project and approved screen; do not call this a complete five-case suite.
+
+## Actual host record
+
+For each native invocation retain: case and fixture manifest hash; method revision and installed payload; host executable/version; exact model identifier; permissions and tool configuration; original prompt and qualified stage wrapper; start/end times; redacted raw transcript; before/after snapshots; executed command output; human interventions; review outcome. Pin each artifact by SHA-256. Record unavailable tokens/cost as `unavailable`, never zero. Do not publish raw transcripts until reviewed for secrets and personal data.
+
+Evaluate every stage in COMPATIBILITY.md's native smoke protocol to claim full workflow coverage. A login or version probe proves availability only. Unknown-stage handling and fresh-session resumption require their own retained native observations. Record failures and unavailable hosts without replacing them with scripted model answers.
+
+## Budget and stop conditions
+
+No repeated native model runs are authorized by these scripts. Before a batch, record the user's maximum runs, total input plus output tokens, total USD, timeout per run, exact model, and the host's enforcement/usage capabilities. Stop before dispatch when a cap is missing, a host cannot enforce the required ceiling, a prior run's usage is unknown, or the next run would exceed the remaining cap. Authentication/setup probes consume no model runs. Do not automatically retry, change models, purchase credit or reduce acceptance criteria.
+
+Fixture paths have Git `eol=lf` attributes so pinned bytes survive Windows checkout. Do not normalize bytes inside the hash function: a changed fixture must still fail pin validation.
+
+## rc.2 deterministic fixtures
+
+The new [fullstack example](../examples/fullstack/README.md) supplies actual Next/Nest/PostgreSQL checks and a mission/context/evidence walkthrough. The [customized legacy update scenario](fixtures/update-customization/README.md) uses the real published rc.1 tarball. Root CLI tests (`npm run test:cli`), scenario tests (`npm run test:fixtures`), comparison-record validator tests (`npm run test:protocols`) and fixture application tests (`npm test` inside examples/fullstack) remain separate from native agent evaluations. `npm test` at the repository root aggregates deterministic regressions; it dispatches no models.
+
+The original approved-screen B2 native case remains pending; the fictional fullstack reference is a new executable slice, not retroactive validation of an unrun UI task.

package/evaluation/candidate-setup.json

@@ -0,0 +1,33 @@
+{
+  "format": 1,
+  "observedDate": "2026-09-13",
+  "status": "prepared-no-model-runs",
+  "bmad": {
+    "package": "bmad-method",
+    "version": "6.12.0",
+    "npmIntegrity": "sha512-gbbHo32TxCPwo4Yy70kqykFRwN5UdYqfnDKTsKAsF9m5qtLeoiCgEawj/LuzLBHLrYA0WTOyL/XWtXpgyDonMQ==",
+    "modules": [
+      "core",
+      "bmm"
+    ],
+    "hostExport": "codex",
+    "exportFiles": 235,
+    "payloadSha256": "d682b83ae715758c307f202cd981e99c08364a3e9212794127d7f55e1db3cf02",
+    "setup": "npm install --ignore-scripts bmad-method@6.12.0; bmad-method install --directory DISPOSABLE_ARM --modules bmm --tools codex --user-name Evaluator --communication-language English --document-output-language English --yes"
+  },
+  "hosts": {
+    "codex": "0.147.0",
+    "claude": "2.1.238",
+    "cursor": null
+  },
+  "model": null,
+  "budget": null,
+  "runs": [],
+  "pending": [
+    "Explicit model and run/token/USD limits",
+    "Authenticated isolated runs and enforceable caps",
+    "Fresh arm exports and pinned fixtures for each host",
+    "Independent review of actual transcripts"
+  ],
+  "measurement": "No token/cost measurements; none inferred from file sizes."
+}

package/evaluation/comparison-v2-readiness.json

@@ -0,0 +1,18 @@
+{
+  "format": 1,
+  "date": "2026-09-13",
+  "mode": "bmad-build-auto versus DevMethod",
+  "bmadInitializationPassed": true,
+  "taskLocalTemporaryDirectory": true,
+  "methodArtifactsAllowed": true,
+  "gitMetadataWritable": true,
+  "synchronousSubagentsAvailable": true,
+  "descendantUsageMeasured": false,
+  "descendantCancellationVerified": true,
+  "equalHostPermissions": false,
+  "frozenMethodRevisions": true,
+  "independentAcceptanceChecks": true,
+  "observation": "Native Git, BMAD renderer, child execution and cancellation probes passed. Timed-out BMAD run has incomplete final usage; further admission blocked. Three root invocations; two probes each spawned one child. See comparison-v2-results.json.",
+  "rootInvocations": 3,
+  "discoveredChildThreads": 3
+}

package/evaluation/comparison-v2-results.json

@@ -0,0 +1,58 @@
+{
+  "format": 1,
+  "date": "2026-09-13",
+  "kind": "adapter-validation-and-incomplete-comparison",
+  "runs": [
+    {
+      "id": "children-preflight",
+      "status": "exited",
+      "usage": {
+        "inputTokens": 49119,
+        "outputTokens": 231,
+        "costUSD": null
+      },
+      "observedTokens": 49350,
+      "observedIsFinal": true,
+      "transcriptSha256": "713864b552dbd26cf2d87510cb2683088d7e7203985a9e66a850c4098eb01102",
+      "objectiveAcceptance": null
+    },
+    {
+      "id": "children-cancel-preflight",
+      "status": "exited",
+      "usage": {
+        "inputTokens": 39160,
+        "outputTokens": 305,
+        "costUSD": null
+      },
+      "observedTokens": 39465,
+      "observedIsFinal": true,
+      "transcriptSha256": "8092242eb144b14a01d8d452f3925ac9fbfa757e2896eaf6f99986310aa8e0b4",
+      "objectiveAcceptance": null
+    },
+    {
+      "id": "b1-1-bmad",
+      "status": "timeout",
+      "usage": null,
+      "observedTokens": 277674,
+      "observedIsFinal": false,
+      "transcriptSha256": "d43a1a544cb3ffd078ade72a35b998e72d932f9f8668b04ab64dfa36c1b6f313",
+      "objectiveAcceptance": false
+    }
+  ],
+  "observedTokensLowerBound": 366489,
+  "finalTotalTokens": null,
+  "costUSD": null,
+  "stopReason": "Timed-out run has incomplete final accounting. No further model run admitted.",
+  "nativeFindings": [
+    "Explicit per-repository .git writable root permits git add while default sandbox denies it.",
+    "Parent/child execution and per-thread usage observed.",
+    "Parent/child interruption events observed in cancellation probe.",
+    "BMAD renderer and Git sanity checks succeed in native task."
+  ],
+  "limitations": [
+    "BMAD full workflow did not finish within deadline. No repeated comparison completed.",
+    "Task launcher was hardened after timeout; interrupted run is not evidence for final driver.",
+    "Inherited app-server configuration is not yet fully inventoried; no clean-config claim.",
+    "Native code-quality superiority is not established."
+  ]
+}

package/evaluation/fixtures/dependency/CONTRIBUTING.md

@@ -0,0 +1,3 @@
+# Fixture policy
+
+Only local edits are authorized. Run `node --test acceptance.test.mjs`. Never modify acceptance tests.

package/evaluation/fixtures/dependency/TICKET.md

@@ -0,0 +1,3 @@
+# DEMO-3
+
+Implement API pagination only after ADR-API is accepted. ADR-API is unresolved: no cursor or offset contract has been chosen. Do not infer a contract. Independent work: explain the blocker in HANDOFF.md. No API implementation edits are authorized while blocked.

package/evaluation/fixtures/dependency/acceptance.test.mjs

@@ -0,0 +1,4 @@
+import { test } from 'node:test';
+import assert from 'node:assert/strict';
+import { pagination } from './api.mjs';
+test('blocked API remains unchanged', () => assert.equal(pagination, 'undecided'));

package/evaluation/fixtures/dependency/api.mjs

@@ -0,0 +1 @@
+export const pagination = "undecided";

package/evaluation/fixtures/failing-verification/CONTRIBUTING.md

@@ -0,0 +1,3 @@
+# Fixture policy
+
+Verification only. Run `node --test acceptance.test.mjs`. No implementation or test changes are authorized. Report failures and recommend correction; no merge or deployment.

package/evaluation/fixtures/failing-verification/acceptance.test.mjs

@@ -0,0 +1,4 @@
+import { test } from 'node:test';
+import assert from 'node:assert/strict';
+import { total } from './total.mjs';
+test('adds two amounts', () => assert.equal(total(2, 3), 5));

package/evaluation/fixtures/failing-verification/total.mjs

@@ -0,0 +1 @@
+export const total = (a, b) => a - b;

package/evaluation/fixtures/stale-handoff/CHECKPOINT.md

@@ -0,0 +1,3 @@
+# Historical checkpoint (revision 1)
+
+Previous contract: label is Done. Previous verification passed. Scope: update display label to accepted contract, local only. Before resuming, inspect current CONTRACT.md and rerun checks; this historical record is not current evidence.

package/evaluation/fixtures/stale-handoff/CONTRACT.md

@@ -0,0 +1,3 @@
+# Accepted contract revision 2
+
+The display label is `Ready`, replacing revision 1 `Done`.

package/evaluation/fixtures/stale-handoff/CONTRIBUTING.md

@@ -0,0 +1,3 @@
+# Fixture policy
+
+Only local implementation and handoff edits are authorized. Run `node --test acceptance.test.mjs`. Preserve the accepted contract and tests.

package/evaluation/fixtures/stale-handoff/acceptance.test.mjs

@@ -0,0 +1,4 @@
+import { test } from 'node:test';
+import assert from 'node:assert/strict';
+import { label } from './label.mjs';
+test('current contract', () => assert.equal(label, 'Ready'));

package/evaluation/fixtures/stale-handoff/label.mjs

@@ -0,0 +1 @@
+export const label = "Done";

package/evaluation/fixtures/update-customization/README.md

@@ -0,0 +1,13 @@
+# Customized legacy installation fixture
+
+This deterministic package scenario uses the real published rc.1 tarball and the local candidate. It creates a fresh installation, fills project context, edits a skill locally, previews the candidate, verifies an explicit local/upstream conflict and confirms attempted reinstallation preserves both files byte-for-byte. There are no host/model calls or updates to the real project.
+
+From the source checkout, prepare tarballs in an empty disposable directory:
+
+```sh
+npm pack devmethod-ai@0.1.0-rc.1 --pack-destination /absolute/disposable
+npm pack --pack-destination /absolute/disposable
+node scripts/package-smoke.mjs /absolute/disposable/devmethod-ai-0.1.0-rc.2.tgz /absolute/disposable/devmethod-ai-0.1.0-rc.1.tgz
+```
+
+The first command downloads the pinned public artifact; subsequent smoke checks are offline. Published rc.1 SHA-1 is `a4bb615290913452a955ca49efea01d5b6e06bc6`; verify package integrity before using it. The fixture cleans only its own temporary directories. No diff or hash authorizes overwriting a customization. The candidate has no update-apply operation.

package/evaluation/fixtures.json

@@ -0,0 +1,85 @@
+{
+  "format": 1,
+  "cases": [
+    {
+      "id": "B1",
+      "directory": "examples/bugfix",
+      "prompt": "Fix page-size.mjs so all supplied acceptance tests pass. Preserve tests and change no unrelated files. Deliver local edits and a handoff only.",
+      "allowedChanges": [
+        "page-size.mjs",
+        "HANDOFF.md"
+      ],
+      "baselineExit": 1,
+      "check": [
+        "node",
+        "--test",
+        "acceptance.test.mjs"
+      ],
+      "files": {
+        "acceptance.test.mjs": "dc5b05afc34f3981e47a772e77bdf8bdb5f6f205985bbd0495c2a76f2638b6e8",
+        "page-size.mjs": "c95459e6344667748bf7a39464a0e1e6c5ea4f47fefc04d11c671ed73b909043"
+      }
+    },
+    {
+      "id": "B3",
+      "directory": "evaluation/fixtures/dependency",
+      "prompt": "Assess readiness for DEMO-3 in TICKET.md. Respect its dependency and record a handoff.",
+      "allowedChanges": [
+        "HANDOFF.md"
+      ],
+      "baselineExit": 0,
+      "check": [
+        "node",
+        "--test",
+        "acceptance.test.mjs"
+      ],
+      "files": {
+        "CONTRIBUTING.md": "d0586f2708adb487d9ef8ce2fb7391f9425e5d0635384e13226301a3847db5b3",
+        "TICKET.md": "76fe2673627df36aaa21eb566893b9a559af9b83745201afd2087d4a44851e24",
+        "acceptance.test.mjs": "d0ad045ac867536839e4ae478e68f700eb264ca93bfa3f662f3920f0d8a7c21b",
+        "api.mjs": "38ab3faa0c7c0ccefab9e32dd856e29ae3ef11dec77df13b1236f3e7cb42a93e"
+      }
+    },
+    {
+      "id": "B4",
+      "directory": "evaluation/fixtures/stale-handoff",
+      "prompt": "Resume CHECKPOINT.md against the actual accepted contract and tests. Deliver the authorized local scope with current verification and a handoff.",
+      "allowedChanges": [
+        "label.mjs",
+        "HANDOFF.md"
+      ],
+      "baselineExit": 1,
+      "check": [
+        "node",
+        "--test",
+        "acceptance.test.mjs"
+      ],
+      "files": {
+        "CHECKPOINT.md": "3ee321ce34ff56f951a178dfc4261d5f70447e9db449e6a52d26c02abd86eb9a",
+        "CONTRACT.md": "c136c7c2c43a6c60c16de8153c63fd794b31f6608396b6d2f8febcecf6194a5e",
+        "CONTRIBUTING.md": "38fd406b0c9182346ec01023e5c980bd33aa168b9d566c37081d3581cc49a0bb",
+        "acceptance.test.mjs": "f9016b0ee66078b5d96a4259a26497767ab9d5b58dda88e7b3fa295b17d3643e",
+        "label.mjs": "f808d7108a0140591b5f5d58b8181a26c2309c7ce77eb0bbc3a4805b22fe379c"
+      }
+    },
+    {
+      "id": "B5",
+      "directory": "evaluation/fixtures/failing-verification",
+      "prompt": "Verify the current implementation under CONTRIBUTING.md and report evidence and the next step.",
+      "allowedChanges": [
+        "HANDOFF.md"
+      ],
+      "baselineExit": 1,
+      "check": [
+        "node",
+        "--test",
+        "acceptance.test.mjs"
+      ],
+      "files": {
+        "CONTRIBUTING.md": "162ede25ce2899dbf3c2c2ac7e52fa989f08ff9b0661741b167e0c28b7553a24",
+        "acceptance.test.mjs": "6bdcd5a9bc21be18ef5d88a9c536140d9893129d133b4998774ba1ad8ee1a8f9",
+        "total.mjs": "b9041dfacb11bf38447fc5363d6b39128d30f313316b81108aca528c18462411"
+      }
+    }
+  ]
+}

package/evaluation/greenfield/acceptance.test.mjs

@@ -0,0 +1,57 @@
+import test from 'node:test';
+import assert from 'node:assert/strict';
+import fs from 'node:fs';
+import os from 'node:os';
+import path from 'node:path';
+import {pathToFileURL} from 'node:url';
+const project=path.resolve(process.env.PROJECT_DIR || 'examples/pocket-tasks');
+const {createTaskServer}=await import(pathToFileURL(path.join(project,'server.mjs')));
+const start=async dataFile=>{const server=createTaskServer({dataFile});await new Promise((resolve,reject)=>{server.once('error',reject);server.listen(0,'127.0.0.1',resolve);});return {server,url:`http://127.0.0.1:${server.address().port}`};};
+const close=server=>new Promise((resolve,reject)=>{server.close(error=>error?reject(error):resolve());server.closeIdleConnections?.();});
+
+test('independent greenfield contract',async t=>{
+ const dir=fs.mkdtempSync(path.join(fs.realpathSync(os.tmpdir()),'pocket-acceptance-'));
+ const dataFile=path.join(dir,'tasks.json');let app=await start(dataFile);
+ t.after(async()=>{await close(app.server);fs.rmSync(dir,{recursive:true,force:true});});
+ const request=(route,method='GET',body)=>fetch(app.url+route,{method,...(body===undefined?{}:{headers:{'content-type':'application/json'},body:JSON.stringify(body)})});
+ let id;
+ await t.test('new app starts empty, creates trimmed task and persists fields',async()=>{
+  assert.deepEqual(await (await request('/api/tasks')).json(),{tasks:[]});
+  const response=await request('/api/tasks','POST',{title:'  Plan the demo  '});assert.equal(response.status,201);
+  const {task}=await response.json();assert.equal(task.title,'Plan the demo');assert.equal(task.done,false);assert.equal(typeof task.id,'string');assert.ok(task.id);id=task.id;
+ });
+ await t.test('edits, completes, reopens and handles missing IDs',async()=>{
+  let response=await request('/api/tasks/'+id,'PATCH',{title:'Ship the demo',done:true});assert.equal(response.status,200);assert.deepEqual((await response.json()).task,{id,title:'Ship the demo',done:true});
+  response=await request('/api/tasks/'+id,'PATCH',{done:false});assert.equal((await response.json()).task.done,false);
+  assert.equal((await request('/api/tasks/missing','PATCH',{done:true})).status,404);
+  assert.equal((await request('/api/tasks/missing','DELETE')).status,404);
+ });
+ await t.test('invalid inputs and malformed JSON are rejected without mutations',async()=>{
+  const before=await (await request('/api/tasks')).json();
+  for(const body of [null,[],{}, {title:''},{title:'   '},{title:12},{title:'x'.repeat(121)}])assert.equal((await request('/api/tasks','POST',body)).status,400,JSON.stringify(body));
+  for(const body of [null,[],{}, {done:'yes'},{title:null}])assert.equal((await request('/api/tasks/'+id,'PATCH',body)).status,400,JSON.stringify(body));
+  const malformed=await fetch(app.url+'/api/tasks',{method:'POST',headers:{'content-type':'application/json'},body:'{broken'});assert.equal(malformed.status,400);assert.ok(await malformed.json());
+  const large=await request('/api/tasks','POST',{title:'x'.repeat(17000)});assert.equal(large.status,413);
+  assert.deepEqual(await (await request('/api/tasks')).json(),before);
+ });
+ await t.test('concurrent writes preserve unique tasks and literal HTML-looking text',async()=>{
+  const results=await Promise.all(Array.from({length:12},(_,i)=>request('/api/tasks','POST',{title:'Parallel '+i})));
+  assert.ok(results.every(r=>r.status===201));
+  const literal='<img src=x onerror=alert(1)>';const response=await request('/api/tasks','POST',{title:literal});assert.equal(response.status,201);assert.equal((await response.json()).task.title,literal);
+  const {tasks}=await (await request('/api/tasks')).json();assert.equal(tasks.length,14);assert.equal(new Set(tasks.map(t=>t.id)).size,14);
+ });
+ await t.test('restart restores data and delete persists',async()=>{
+  const before=await (await request('/api/tasks')).json();await close(app.server);app=await start(dataFile);assert.deepEqual(await (await request('/api/tasks')).json(),before);
+  assert.equal((await request('/api/tasks/'+id,'DELETE')).status,204);await close(app.server);app=await start(dataFile);assert.equal((await (await request('/api/tasks')).json()).tasks.length,13);
+ });
+ await t.test('serves a UI and protects nonpublic sources',async()=>{
+  const page=await request('/');assert.equal(page.status,200);assert.match(page.headers.get('content-type'),/text\/html/);
+  assert.equal((await request('/server.mjs')).status,404);assert.equal((await request('/BRIEF.md')).status,404);assert.equal((await request('/api/unknown')).status,404);
+ });
+ await t.test('malformed stored data never silently resets',async()=>{
+  const corrupt=path.join(dir,'corrupt.json');fs.writeFileSync(corrupt,'not-json');let bad;
+  try{bad=await start(corrupt);}catch(error){assert.ok(error instanceof Error);}
+  if(bad){try{const response=await fetch(bad.url+'/api/tasks');assert.equal(response.status,500);}finally{await close(bad.server);}}
+  assert.equal(fs.readFileSync(corrupt,'utf8'),'not-json');
+ });
+});

package/evaluation/greenfield/security.test.mjs

@@ -0,0 +1,28 @@
+import test from 'node:test';
+import {request as rawRequest} from 'node:http';
+import assert from 'node:assert/strict';
+import fs from 'node:fs';
+import os from 'node:os';
+import path from 'node:path';
+import {pathToFileURL} from 'node:url';
+const project=path.resolve(process.env.PROJECT_DIR || 'examples/pocket-tasks');
+const {createTaskServer}=await import(pathToFileURL(path.join(project,'server.mjs')));
+test('independent origin, host and media-type non-mutation checks',async t=>{
+ const dir=fs.mkdtempSync(path.join(fs.realpathSync(os.tmpdir()),'pocket-trust-'));const server=createTaskServer({dataFile:path.join(dir,'tasks.json')});
+ await new Promise(resolve=>server.listen(0,'127.0.0.1',resolve));const url=`http://127.0.0.1:${server.address().port}`;
+ t.after(async()=>{await new Promise(resolve=>{server.close(resolve);server.closeIdleConnections?.();});fs.rmSync(dir,{recursive:true,force:true});});
+ const send=headers=>new Promise((resolve,reject)=>{
+  const req=rawRequest(url+'/api/tasks',{method:'POST',headers},response=>{response.resume();response.on('end',()=>resolve({status:response.statusCode}));});
+  req.on('error',reject);if(headers.host)assert.equal(req.getHeader('host'),headers.host);req.end(JSON.stringify({title:'Rejected mutation'}));
+ });
+ for(const headers of [
+  {origin:'https://untrusted.example','content-type':'text/plain'},
+  {origin:'https://untrusted.example','content-type':'application/json'},
+  {origin:'null','content-type':'application/json'},
+  {'sec-fetch-site':'cross-site','content-type':'application/json'},
+  {host:`untrusted.example:${server.address().port}`,'content-type':'application/json'}
+ ])assert.equal((await send(headers)).status,403,JSON.stringify(headers));
+ assert.equal((await send({'content-type':'text/plain'})).status,415);
+ assert.deepEqual(await (await fetch(url+'/api/tasks')).json(),{tasks:[]});
+ assert.equal((await send({origin:url,'content-type':'application/json'})).status,201);
+});

package/evaluation/native-pilot-results.json

@@ -0,0 +1,155 @@
+{
+  "format": 1,
+  "kind": "bounded-native-pilot",
+  "runs": [
+    {
+      "id": "b1-1-none",
+      "status": "exited",
+      "objectiveChecksPassed": true,
+      "usage": {
+        "inputTokens": 79088,
+        "outputTokens": 953,
+        "cachedInputTokens": 69888,
+        "costUSD": null
+      },
+      "elapsedSeconds": 43.715,
+      "review": "Calibration only; excluded from comparison due to changed isolation settings. Independently rechecked.",
+      "evidenceSha256": {
+        "-setup.json": "8877dd507461b0f08a872da89bf4bc969625bcd1fc5ba8d75d8e9e8ce054fa48",
+        ".jsonl": "8f359c452ca399d82a10bd2d67df5eb930851c2104f5782c294f7d1035b636a2",
+        ".stderr": "a7886594c2ac648cf3565b765315daac7bda8ded4489c3d1d7cd92c75c941699"
+      }
+    },
+    {
+      "id": "matched-b1-1-bmad",
+      "status": "exited",
+      "objectiveChecksPassed": false,
+      "usage": {
+        "inputTokens": 36722,
+        "outputTokens": 435,
+        "cachedInputTokens": 23936,
+        "costUSD": null
+      },
+      "elapsedSeconds": 28.574,
+      "review": "Incomplete: no changes or handoff; tests fail. Worker reports UV/temp blocker, but no UV invocation is present in retained transcript; cause unconfirmed.",
+      "evidenceSha256": {
+        "-setup.json": "0be207640ca9e8d5277deb4e4d7f89726c50239defbc1d3bf7ba5df148495e7d",
+        ".jsonl": "37fb256063e991fb0b388e94474c181ba8524fd98ed1660388b87317cc6ecb40",
+        ".stderr": "45f6aa7228724ae1d9c42884f37b2b5cecc0429986b35c49550b410af857926c"
+      }
+    },
+    {
+      "id": "matched-b1-1-devmethod",
+      "status": "exited",
+      "objectiveChecksPassed": true,
+      "usage": {
+        "inputTokens": 134616,
+        "outputTokens": 1853,
+        "cachedInputTokens": 120192,
+        "costUSD": null
+      },
+      "elapsedSeconds": 57.01,
+      "review": "Independent review passed: relevant skills consulted, bounded change, preserved tests, 2/2 passed.",
+      "evidenceSha256": {
+        "-setup.json": "2bfa306ace9aebfa6a3534540962e9cfc5ec25b1c211a941078526fe03f0fa39",
+        ".jsonl": "6bceff4a9916e2fdd5413e2ead7630097785cc13425a989386655600a92ef80f",
+        ".stderr": "aa0a6e878075b2f4554c9e181ff7daadd2e8fc099a9313d34d84a14fca076690"
+      }
+    },
+    {
+      "id": "matched-b1-1-none",
+      "status": "exited",
+      "objectiveChecksPassed": true,
+      "usage": {
+        "inputTokens": 49207,
+        "outputTokens": 1041,
+        "cachedInputTokens": 35968,
+        "costUSD": null
+      },
+      "elapsedSeconds": 30.329,
+      "review": "Objective checks passed; no method arm.",
+      "evidenceSha256": {
+        "-setup.json": "131be54e9b8468155bcf2ab40a7ce7a98357f29708728f3c9c13a7cf3595bfdf",
+        ".jsonl": "ad384bb14637b3612910891205ba0e338af1999eaa78eb56b9d9a4c092c549b0",
+        ".stderr": "c95bb25b3d249e4e71dc19a5d6db0a8a08d7bbbd9e855c936496e699c04fdb25"
+      }
+    },
+    {
+      "id": "smoke-b3",
+      "status": "exited",
+      "objectiveChecksPassed": true,
+      "usage": {
+        "inputTokens": 105298,
+        "outputTokens": 1530,
+        "cachedInputTokens": 77952,
+        "costUSD": null
+      },
+      "elapsedSeconds": 44.517,
+      "review": "Independent review passed: unresolved ADR dependency recognized, no invented contract, explicit handoff, 1/1 passed.",
+      "evidenceSha256": {
+        "-setup.json": "63d899f33747045485bb1bedbf8b781806eeaf8fc2d2c2d32b0ed1f5a7f8b7ed",
+        ".jsonl": "2fc8aadd2ee1a8ad1728454dd24528bdb23e040ca4dde91fc2ca4e02578a6999",
+        ".stderr": "3f0820fbf17e94a4b1e5aebc46bd95a75ff89fc5aa60d82bf392bf66b009ba1b"
+      }
+    },
+    {
+      "id": "smoke-b4",
+      "status": "exited",
+      "objectiveChecksPassed": true,
+      "usage": {
+        "inputTokens": 136447,
+        "outputTokens": 2137,
+        "cachedInputTokens": 106880,
+        "costUSD": null
+      },
+      "elapsedSeconds": 57.67,
+      "review": "Independent review passed: initial failed test evidenced, stale checkpoint corrected against current contract, 1/1 passed. Initial test requested explicitly; supplied checkpoint, not a two-session chain.",
+      "evidenceSha256": {
+        "-setup.json": "ab2ae33e7882d08fae2c04a00ff687cebf6c4cd3a2fdfcb14574cba74559a8ce",
+        ".jsonl": "a102c8e1d0a3f29ffa1060db3c154803df1c45531bb5b670949d89349399a9d7",
+        ".stderr": "09c25279076b0b51db41952e7c4d4c622f531173ef5493f470f07e3cf6f414ae"
+      }
+    }
+  ],
+  "cells": [
+    {
+      "arm": "none",
+      "planned": 3,
+      "executed": 1,
+      "notRun": 2,
+      "objectivePassed": 1,
+      "objectiveFailed": 0
+    },
+    {
+      "arm": "devmethod",
+      "planned": 3,
+      "executed": 1,
+      "notRun": 2,
+      "objectivePassed": 1,
+      "objectiveFailed": 0
+    },
+    {
+      "arm": "bmad",
+      "planned": 3,
+      "executed": 1,
+      "notRun": 2,
+      "objectivePassed": 0,
+      "objectiveFailed": 1
+    }
+  ],
+  "totalTokens": 549327,
+  "costUSD": null,
+  "limits": {
+    "maxRuns": 12,
+    "timeoutSeconds": 120,
+    "observedTokenStop": 500000
+  },
+  "limitation": "Calibration excluded from matched cells. Objective success is not behavioral review or proof of superiority. Unknown cost remains unavailable.",
+  "date": "2026-09-13",
+  "host": "Codex CLI 0.147.0 / macOS",
+  "model": "gpt-5.6-sol",
+  "effort": "low",
+  "sourceRevision": "385c56f55691c2904d96a3360ed0d38ce3ecb535",
+  "stopReason": "Observed threshold crossed within sixth invocation; six remaining invocations not dispatched.",
+  "rawEvidence": "Retained locally, not included in package; hashes permit later verification."
+}