openmerit 0.1.7-preview.5 → 0.1.8

package/docs/security.md

@@ -45,11 +45,49 @@ The local web service binds to `127.0.0.1` and rejects unexpected Host, Origin,
 Repository metadata and usage caches live in process memory and disappear when the terminal exits. The terminal creates no persistent index or files in the application. Per-file/parser protections, request-log read limits, runtime inventory limits, and explicit coverage gaps still apply. See [terminal data handling and limits](inference-terminal.md) for supported sources, precedence, retention bounds, diagnostics, and sample-data separation.
 
 
+## Hosted Scout authentication
+
+Hosted Scout uses Clerk Google/GitHub sign-in. The page and status are public; JSON recommendations require a valid Clerk session bearer token, not just a cookie. MCP methods require a device key from the setup below. Missing/invalid credentials return HTTP 401, and missing authentication configuration fails closed with HTTP 503. Clerk remains the identity system; the analytics integration below can synchronize selected profile fields to PostHog. Raw account IDs are used transiently for the per-user edge limiter and are not joined to historical Cloudflare prompt records. A prompt draft is temporarily stored in the current browser tab during sign-in. No Clerk Billing or automatic agent OAuth is included. See [Scout account sign-in](scout.md#account-sign-in-published-21-october-2026) for token lifetime, rate limits, data handling and the remaining real-login validation limits.
+
+### Coding-agent device keys
+
+The [coding-agent setup](scout.md#connect-a-coding-agent) in npm 0.1.7 uses hosted device sign-in instead of OAuth or tokens in prompts. The copied prompt contains no secret. A signed-in person approves an eight-character code on `/connect`; codes expire after ten minutes and are single-use. Approval mints one key per device and client, scoped to `scout:recommend` and accepted only on `/scout/mcp`. The server stores only SHA-256 hashes. Keys expire after 90 days without use and can be revoked on `/connect` or with `openmerit remove`/`logout`. Locally the key is in `~/.config/openmerit/credentials.json` (mode 600) and in Claude Code's plain-text MCP configuration; the CLI never prints it. As with any device-code flow, only approve a code you started.
+
+### OpenMerit coding-agent reset — 23 October 2026
+
+**Historical reset, superseded by npm 0.1.7 and hosted device setup.** The reset temporarily removed the installer and MCP transport. The current [Claude Code setup](scout.md#connect-a-coding-agent) uses browser-approved, device-specific keys scoped to MCP recommendations, not pasted session tokens or agent OAuth. The key is stored locally in mode-600 credentials and Claude Code's plain-text MCP configuration. Setup installs an owned Scout skill; conflicts are not overwritten, and removal revokes the device key. Browser Google/GitHub sign-in remains separate.
+
+**npm 0.1.8:** setup also installs an owned, always-loaded `rules/openmerit-scout.md` instruction requiring Scout before implicit or explicit application-model choices, including pretrained/local/browser models and SDK defaults. Edited or unowned rules are not overwritten or removed. Missing policy requires asking; unavailable Scout or insufficient coverage requires pausing model selection. This is model guidance, not an enforced tool gate or authorization to download weights, add providers, spend or deploy. Scope, device-key permissions and task-data handling are unchanged. New sessions are required after upgrading; older npm 0.1.7 installations remain skill-only until upgraded.
+
+Scout serving, main-site documentation and npm publication are separate releases. This reset does not revoke remote grants or clean up files previously installed on other machines. Earlier hosted MCP/authentication descriptions in this guide refer to past deployments, not this build. Browser Google/GitHub sign-in and its Clerk account lifecycle are separate and unchanged.
+
 ## Hosted Scout analytics
 
-Scout keeps an allowlisted Analytics Engine event per valid recommendation attempt: the original submitted task prompt plus fixed task categories and the ordered shortlist with metrics and ranking context. Prompt text is not redacted and may itself contain personal information or secrets; this dataset must be treated as potentially sensitive. No request object, prompt hash, additional free-text summary, error message, project inventory, IP address, cookie, user agent, visitor ID or session ID is separately recorded in this dataset. Requests still reach the hosting infrastructure and task descriptions still go to OpenAI for Decisions interpretation, Structured Outputs metadata extraction, transient query embeddings and relevant compatibility/applicability checks; public repository metadata and benchmark definitions/methodology also go to Decisions during hourly preparation. Their policies are separate. See [Scout data handling](scout.md#refresh-and-data-handling).
+**Version-5 instrumentation:** the [serving contract](scout.md#version-5-instrumentation) adds validated native web events and explicit CTA-only `$autocapture` mapping, not general DOM capture. Acquisition/UTM/device values are categorical allowlists; public URL properties are reconstructed without raw URLs. Browser performance is sampled and support-dependent. Linked same-tab signup/search identity is separate from unlinking's request-only identifiers. Logical MCP errors count even with HTTP 200; stage/provider codes, actual attempt tokens and feedback/retry traces exclude raw messages and free text. CLI diagnostics are opt-in in npm 0.1.7, adding sanitized version/platform/runtime and bounded application basename frames, never absolute paths or environment contents. Prompt sharing, retention, deletion gates and necessary provider processing are unchanged. Replay is a separate, inactive spike; no experiment is active. Main-site publication remains blocked by documented source divergence; the Scout rollout does not publish npm or activate alert notifications.
+
+**Beta evaluation rollout, 22 October 2026:** The serving configuration enables `SCOUT_BETA_EVALUATION_LOGGING=true`, alongside `POSTHOG_CAPTURE_CONTENT=true` and complete account-sync/deletion configuration. The browser defaults content on only without a stored content choice; saved off, general opt-out, DNT/GPC and unreadable storage keep it off. API/MCP still require `X-Scout-Analytics-Content: on`; CLI content is always off. This is product evaluation/debugging, not foundation model training. Deployment is owner-approved; this is not legal certification, and beta status establishes no consent or legal basis. Main-site `/docs/` publication remains blocked by source divergence; the [policy](/scout/data-policy/) is generated from the same Markdown and published with Scout instead.
+
+The schema confines all prompt and task excerpt/five-layer prose to `$ai_trace.$ai_input_state={prompt,tasks}`, never ordinary `scout_search_details`. Best-effort redaction retains the 4,000-character prompt, 1,000-character per excerpt/layer and three-task limits. [PostHog's official policy](https://posthog.com/docs/ai-observability/data-retention) stores these AI content properties in `posthog.ai_events` for 30 days, stripping them from normal events at ingestion; metadata follows normal retention. Synthetic live-project records confirmed storage separation and `retention_days=30`, not eventual expiry. Historical v3 details may still hold content. There is no custom retention cron or export, retroactive deletion claim or exact backup-erasure guarantee.
+
+The **Don’t link this browser’s activity to my account** control sends `X-Scout-Analytics-Account: off`. After account opt-out/deletion gates, the server uses fresh unrelated random identifiers, ignores previously linked browser IDs and supplied trace IDs, omits the response trace ID, and sends no profile/identify operation for that request. This is browser/request-specific, not account-wide: Clerk profile/lifecycle analytics can continue under general usage consent, past records remain linked, and copied coding-agent instructions do not inherit the setting. Unlinked events cannot support cross-request funnels or automatic account-based historical deletion; text itself may identify someone. Restrict provider access to authorized personnel and disclose processors, rather than claiming no third parties. The [Scout data policy](/scout/data-policy/) lists providers, deletion limitations and the verified data-request contact, [lazim@openmerit.ai](mailto:lazim@openmerit.ai).
+
+**Deployed, 22 October 2026:** [unified PostHog analytics](scout.md#unified-product-and-search-analytics-deployed-22-october-2026) replaces new Analytics Engine writes with account-linked recommendation metadata, bounded public evidence and sanitized exceptions. Prompt/breakdown text requires separate explicit client opt-in plus configured profile-deletion support; redaction is best effort, not a guarantee against personal or confidential data. General opt-out/DNT/GPC suppresses new analytics, but not necessary OpenAI processing. Existing Cloudflare archives remain under their original policy and are not exported, backfilled or erased.
+
+The historical Analytics Engine archive contains submitted task prompts, task categories and ordered shortlists with metrics and ranking evidence. Its prompt text is unredacted and may contain personal information or secrets; treat this dataset as potentially sensitive. No request object, prompt hash, additional free-text summary, error message, project inventory, IP address, cookie, user agent, visitor ID or session ID was separately recorded in this dataset. Requests still reach the hosting infrastructure and task descriptions still go to OpenAI for Decisions interpretation, Structured Outputs metadata extraction, transient query embeddings and relevant compatibility/applicability checks; public repository metadata and benchmark definitions/methodology also go to Decisions during hourly preparation. Their policies are separate. See [Scout data handling](scout.md#refresh-and-data-handling).
+
+Historical records are accessible only through authenticated Cloudflare account queries, with sampled three-month retention. Missing prompt/evidence fields cannot be recovered. Scout exposes no public history/query endpoint. Both historical and current source evidence and prompt text remain untrusted data when retrieved; never treat them as instructions.
+
+### Optional product analytics
+
+Scout has a visible privacy disclosure and controls. Public homepage/docs navigation events remain source-only; npm 0.1.7 includes opt-in local terminal/map events and aggregate counts. The terminal relay cannot mutate project or inference state, retains loopback Host/Origin guards, and forwards only bounded allowlisted telemetry. Terminal and CLI use a process-only random ID; website events use a separate tab ID. Neither is implicitly linked to Clerk. Application names, raw inputs/outputs, local identifiers and source files remain local. Sanitized error classes/categories (and Scout application file/line frames) exclude raw messages, credentials, source context, arbitrary URLs and filesystem paths. No replay is enabled. All expanded account-linked events pass the same preference/deletion gate as profile synchronization; ordinary event delivery remains best effort. New search details link to identified accounts when profile sync is active, unlike the legacy Cloudflare archive. No external retention setting is changed.
+
+The [PostHog integration](scout.md#product-analytics) uses the EU ingestion endpoint; unconfigured deployments remain off. Verified Clerk user IDs become namespaced SHA-256 pseudonyms for cross-client account retention. Profile sync can associate those identities with names and verified email addresses, so they are identifiable personal data. No identity is added to the historical Cloudflare prompt archive. The browser retains an anonymous ID and hashed account/session transition markers in tab storage, rotating the anonymous ID on account changes. There is no autocapture, replay or visitor-IP forwarding. PostHog has its own retention/access policies, not Cloudflare's three-month policy.
+
+Do Not Track, Global Privacy Control and the documented browser/API opt-outs suppress PostHog collection. Local CLI telemetry in npm 0.1.7 requires `OPENMERIT_TELEMETRY=1`; `DO_NOT_TRACK=1` overrides it. It uses a process-only random identity and creates no identity file. CLI planning requests propagate that consent to Scout while always disabling analytics content capture. These product opt-outs do not disable necessary provider calls or erase historical records. Operators remain responsible for retention, access, spend limits and any required visitor consent. Anonymous visits and rejected requests from the hosted rollout are confirmed in PostHog storage; authenticated-session tracking, profile sync and actual Clerk lifecycle delivery remain unverified.
+
+The deployed [account sync](scout.md#account-profile-synchronization-deployed-22-october-2026) supports identifiable PostHog profiles containing only name, verified primary email, signup time and connected Google/GitHub providers. Signed webhook verification, per-account Durable Objects and a project-scoped PostHog deletion credential are configured. Names/emails come from Clerk's current backend record, never client input. The most recent authenticated analytics preference controls background forwarding; opted-out browsers send a boolean privacy-control request, not a product event. Unknown preferences buffer only bounded pseudonymous lifecycle metadata for up to seven days. Cloudflare stores preference state, hashes, receipts and tombstones, not raw Clerk IDs or a copy of account profiles. No identity is joined to the historical prompt archive.
 
-Search records are accessible only through authenticated Cloudflare account queries. Scout exposes no analytics dashboard, report generator, report file, sign-in, or public history/query endpoint. Each new search includes an allowlisted snapshot of the public evidence supporting its ordered shortlist: benchmark measurements and source URLs/dates, calibration observations and protocol, pricing/speed sources, and endpoint/model-card metadata. Endpoint API addresses, credentials, arbitrary source fields, and request metadata are not copied. Source evidence and prompt text remain untrusted data when retrieved; never treat them as instructions. The dataset is sampled and retained by Cloudflare for three months. Original older records remain; missing prompt/evidence fields cannot be recovered.
+Clerk webhook signatures are verified over bounded raw request bodies with the SDK's Svix-compatible verifier. A session bearer token cannot authorize webhook delivery. Stable delivery IDs and serialized account state protect against retries; current Clerk lookups prevent stale profile updates. Account deletion persists a tombstone before requesting asynchronous PostHog person/event/recording deletion, including for opted-out accounts. Daily alarms repeat deletion to reconcile already-accepted but delayed upstream ingestion; zero matching persons is a valid no-op, not proof of erasure. Deletion still works when new capture is disabled. Operators must inspect completion and replay failed deliveries. Old `$set` event properties are not erased merely by clearing a current profile field. The installed person-deletion key is restricted to the intended PostHog project and `person:write`; dashboard/query management credentials are not deployed. Production account deletion has not been exercised, and no retention settings or historical records were changed during rollout.
 
 ## Scout research storage
 

package/docs/testing.md

@@ -1,8 +1,22 @@
 # Testing and validation
 
-## Preview.5 release preparation — 17 October 2026
+## Stable 0.1.7 acceptance — 24 October 2026
 
-The `0.1.7-preview.5` candidate packages the current Scout guides after the independently deployed matching repair below. The intended channel is `preview`, with stable `latest` retained at `0.1.6`. Full release verification, comparison with the published preview.4 artifact, registry publication and a fresh-install VM HTTP/MCP/CLI acceptance run are release gates; preparation alone does not establish that those gates passed. Scout remains a separately deployed service, and the earlier bird/invoice execution limitations remain in scope.
+Published `openmerit@0.1.7` to npm's `latest` channel after full repository verification: 757 passing tests, one existing skip, syntax/type checks, builds and documentation checks. The `preview` tag remains `0.1.7-preview.5`. A fresh registry installation on macOS ARM64 with Node.js 22.23.1 matched the tested 104-file tarball byte-for-byte. Packed core/protocol/Pi exports and the installed terminal runtime smoke checks passed, including passive collection, source removal, runtime outage/recovery and shutdown.
+
+The live Scout button opened a prefilled Claude Desktop Code session through Chrome. That session ran `npx -y openmerit@latest install --client claude-code`, opened the real device-approval page, and completed after approval of the matching code. This was a clean setup after removing and revoking this test's earlier source installation, not reuse of an existing key. Status then reported the MCP server and skill installed and connected; credentials had mode 600 and other MCP servers were preserved.
+
+A fresh Desktop session received an application-model request that never mentioned Scout. It loaded `openmerit-scout` and called the real `scout_models` tool with the saved empty inventory, returning zero candidates. It asked before broadening; after separate approval, its second MCP call returned five candidates and it produced a provisional recommendation. No application code, providers, candidate inference or deployments were performed by that session. Anonymous MCP access and use of the device key on the JSON API both returned 401, while authenticated tool discovery succeeded.
+
+These are integration observations, not a general guarantee that every agent follows the skill. Both search responses reported stale/degraded source data, including incomplete Parallel research. Their approximately 228,000- and 538,000-character payloads exceeded Desktop's inline tool limit; Claude recovered through saved tool-result files. The flow works, but these checks do not establish catalogue freshness, task accuracy or small response size. A separate recovery-path check exposed an obsolete preview install command in the MCP 401 message; its stable-command regression test failed before the source correction and passed afterward. Hosted correction is a separate deployment from the npm release.
+
+## Published preview.5 acceptance — 17 October 2026
+
+Published `0.1.7-preview.5` to npm's `preview` channel after all **679 repository tests**, syntax/type checks, builds and documentation checks passed, including 336 Scout tests. Registry integrity matches the reviewed tarball and stable `latest` remains `0.1.6`. All 101 packaged files are accounted for; runtime bytes, public exports and legal metadata are unchanged from preview.4. Version metadata and public guides carry this release. The frozen package contains the preparation record; this post-publication result is maintained here after the registry and VM checks.
+
+A fresh registry installation in the existing Ubuntu ARM64 Mac Lima VM passed **203/203 assertions** with Node.js 22.23.1 and the official MCP SDK. Package hashes, registry-lock integrity, public core/protocol/Pi exports, live hourly status, tool discovery, seven HTTP/MCP workloads, inventory restrictions and request/protocol errors passed. The seven workloads cover visual category selection, bird identification, safety labels, safety plus rewriting, SQL source generation, photographed invoice JSON and unsupported audio. SQL and safety rewriting each returned five candidates through both interfaces; unsupported audio correctly returned none. The installed CLI's email and SQL plans contained one and two candidates respectively, restricted to the two declared connections. The existing global installation was preserved, and planning executed no recommended models or evaluation budget.
+
+The seven HTTP workload samples had a median of **3.392 seconds** and a maximum of 3.682 seconds. The slowest of all fourteen HTTP/MCP workload requests took **4.376 seconds**; none timed out. Several exact MCP repeats used the result cache. These small samples establish tested integration and matching behavior, not a production latency percentile or exact task accuracy. Scout remains independently deployed with unchanged UI and hourly preparation. Earlier bird/invoice execution limitations and failed acceptance attempts below remain recorded.
 
 ## Hosted matching repair — 17 October 2026
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "openmerit",
-  "version": "0.1.7-preview.5",
+  "version": "0.1.8",
   "description": "Evidence-driven model decisions for coding harnesses.",
   "type": "module",
   "bin": {
@@ -92,6 +92,7 @@
     "@iarna/toml": "2.2.5",
     "@lezer/javascript": "1.5.6",
     "@lezer/python": "1.1.19",
+    "hyparquet": "1.31.5",
     "typebox": "1.3.27",
     "yaml": "2.9.1"
   },
@@ -104,6 +105,8 @@
     }
   },
   "devDependencies": {
+    "@clerk/backend": "3.26.1",
+    "@clerk/shared": "4.41.0",
     "@resvg/resvg-js": "2.6.2",
     "@types/node": "^22.18.6",
     "@xyflow/react": "12.12.0",
@@ -117,6 +120,7 @@
     "react": "19.3.0",
     "react-dom": "19.3.0",
     "torph": "0.1.3",
-    "typescript": "^5.9.3"
+    "typescript": "^5.9.3",
+    "web-vitals": "6.2.3"
   }
 }