openmerit 0.1.7-preview.4 → 0.1.7

package/docs/security.md

@@ -45,11 +45,47 @@ The local web service binds to `127.0.0.1` and rejects unexpected Host, Origin,
 Repository metadata and usage caches live in process memory and disappear when the terminal exits. The terminal creates no persistent index or files in the application. Per-file/parser protections, request-log read limits, runtime inventory limits, and explicit coverage gaps still apply. See [terminal data handling and limits](inference-terminal.md) for supported sources, precedence, retention bounds, diagnostics, and sample-data separation.
 
 
+## Hosted Scout authentication
+
+Hosted Scout uses Clerk Google/GitHub sign-in. The page and status are public; JSON recommendations require a valid Clerk session bearer token, not just a cookie. MCP methods require a device key from the setup below. Missing/invalid credentials return HTTP 401, and missing authentication configuration fails closed with HTTP 503. Clerk remains the identity system; the analytics integration below can synchronize selected profile fields to PostHog. Raw account IDs are used transiently for the per-user edge limiter and are not joined to historical Cloudflare prompt records. A prompt draft is temporarily stored in the current browser tab during sign-in. No Clerk Billing or automatic agent OAuth is included. See [Scout account sign-in](scout.md#account-sign-in-published-21-october-2026) for token lifetime, rate limits, data handling and the remaining real-login validation limits.
+
+### Coding-agent device keys
+
+The [coding-agent setup](scout.md#connect-a-coding-agent) in npm 0.1.7 uses hosted device sign-in instead of OAuth or tokens in prompts. The copied prompt contains no secret. A signed-in person approves an eight-character code on `/connect`; codes expire after ten minutes and are single-use. Approval mints one key per device and client, scoped to `scout:recommend` and accepted only on `/scout/mcp`. The server stores only SHA-256 hashes. Keys expire after 90 days without use and can be revoked on `/connect` or with `openmerit remove`/`logout`. Locally the key is in `~/.config/openmerit/credentials.json` (mode 600) and in Claude Code's plain-text MCP configuration; the CLI never prints it. As with any device-code flow, only approve a code you started.
+
+### OpenMerit coding-agent reset — 23 October 2026
+
+**Historical reset, superseded by npm 0.1.7 and hosted device setup.** The reset temporarily removed the installer and MCP transport. The current [Claude Code setup](scout.md#connect-a-coding-agent) uses browser-approved, device-specific keys scoped to MCP recommendations, not pasted session tokens or agent OAuth. The key is stored locally in mode-600 credentials and Claude Code's plain-text MCP configuration. Setup installs an owned Scout skill; conflicts are not overwritten, and removal revokes the device key. Browser Google/GitHub sign-in remains separate.
+
+Scout serving, main-site documentation and npm publication are separate releases. This reset does not revoke remote grants or clean up files previously installed on other machines. Earlier hosted MCP/authentication descriptions in this guide refer to past deployments, not this build. Browser Google/GitHub sign-in and its Clerk account lifecycle are separate and unchanged.
+
 ## Hosted Scout analytics
 
-Scout keeps an allowlisted Analytics Engine event per valid recommendation attempt: the original submitted task prompt plus fixed task categories and the ordered shortlist with metrics and ranking context. Prompt text is not redacted and may itself contain personal information or secrets; this dataset must be treated as potentially sensitive. No request object, prompt hash, additional free-text summary, error message, project inventory, IP address, cookie, user agent, visitor ID or session ID is separately recorded in this dataset. Requests still reach the hosting infrastructure and task descriptions still go to OpenAI for Decisions interpretation, Structured Outputs metadata extraction, transient query embeddings and relevant compatibility/applicability checks; public repository metadata and benchmark definitions/methodology also go to Decisions during hourly preparation. Their policies are separate. See [Scout data handling](scout.md#refresh-and-data-handling).
+**Version-5 instrumentation:** the [serving contract](scout.md#version-5-instrumentation) adds validated native web events and explicit CTA-only `$autocapture` mapping, not general DOM capture. Acquisition/UTM/device values are categorical allowlists; public URL properties are reconstructed without raw URLs. Browser performance is sampled and support-dependent. Linked same-tab signup/search identity is separate from unlinking's request-only identifiers. Logical MCP errors count even with HTTP 200; stage/provider codes, actual attempt tokens and feedback/retry traces exclude raw messages and free text. CLI diagnostics are opt-in in npm 0.1.7, adding sanitized version/platform/runtime and bounded application basename frames, never absolute paths or environment contents. Prompt sharing, retention, deletion gates and necessary provider processing are unchanged. Replay is a separate, inactive spike; no experiment is active. Main-site publication remains blocked by documented source divergence; the Scout rollout does not publish npm or activate alert notifications.
+
+**Beta evaluation rollout, 22 October 2026:** The serving configuration enables `SCOUT_BETA_EVALUATION_LOGGING=true`, alongside `POSTHOG_CAPTURE_CONTENT=true` and complete account-sync/deletion configuration. The browser defaults content on only without a stored content choice; saved off, general opt-out, DNT/GPC and unreadable storage keep it off. API/MCP still require `X-Scout-Analytics-Content: on`; CLI content is always off. This is product evaluation/debugging, not foundation model training. Deployment is owner-approved; this is not legal certification, and beta status establishes no consent or legal basis. Main-site `/docs/` publication remains blocked by source divergence; the [policy](/scout/data-policy/) is generated from the same Markdown and published with Scout instead.
+
+The schema confines all prompt and task excerpt/five-layer prose to `$ai_trace.$ai_input_state={prompt,tasks}`, never ordinary `scout_search_details`. Best-effort redaction retains the 4,000-character prompt, 1,000-character per excerpt/layer and three-task limits. [PostHog's official policy](https://posthog.com/docs/ai-observability/data-retention) stores these AI content properties in `posthog.ai_events` for 30 days, stripping them from normal events at ingestion; metadata follows normal retention. Synthetic live-project records confirmed storage separation and `retention_days=30`, not eventual expiry. Historical v3 details may still hold content. There is no custom retention cron or export, retroactive deletion claim or exact backup-erasure guarantee.
+
+The **Don’t link this browser’s activity to my account** control sends `X-Scout-Analytics-Account: off`. After account opt-out/deletion gates, the server uses fresh unrelated random identifiers, ignores previously linked browser IDs and supplied trace IDs, omits the response trace ID, and sends no profile/identify operation for that request. This is browser/request-specific, not account-wide: Clerk profile/lifecycle analytics can continue under general usage consent, past records remain linked, and copied coding-agent instructions do not inherit the setting. Unlinked events cannot support cross-request funnels or automatic account-based historical deletion; text itself may identify someone. Restrict provider access to authorized personnel and disclose processors, rather than claiming no third parties. The [Scout data policy](/scout/data-policy/) lists providers, deletion limitations and the verified data-request contact, [lazim@openmerit.ai](mailto:lazim@openmerit.ai).
+
+**Deployed, 22 October 2026:** [unified PostHog analytics](scout.md#unified-product-and-search-analytics-deployed-22-october-2026) replaces new Analytics Engine writes with account-linked recommendation metadata, bounded public evidence and sanitized exceptions. Prompt/breakdown text requires separate explicit client opt-in plus configured profile-deletion support; redaction is best effort, not a guarantee against personal or confidential data. General opt-out/DNT/GPC suppresses new analytics, but not necessary OpenAI processing. Existing Cloudflare archives remain under their original policy and are not exported, backfilled or erased.
+
+The historical Analytics Engine archive contains submitted task prompts, task categories and ordered shortlists with metrics and ranking evidence. Its prompt text is unredacted and may contain personal information or secrets; treat this dataset as potentially sensitive. No request object, prompt hash, additional free-text summary, error message, project inventory, IP address, cookie, user agent, visitor ID or session ID was separately recorded in this dataset. Requests still reach the hosting infrastructure and task descriptions still go to OpenAI for Decisions interpretation, Structured Outputs metadata extraction, transient query embeddings and relevant compatibility/applicability checks; public repository metadata and benchmark definitions/methodology also go to Decisions during hourly preparation. Their policies are separate. See [Scout data handling](scout.md#refresh-and-data-handling).
+
+Historical records are accessible only through authenticated Cloudflare account queries, with sampled three-month retention. Missing prompt/evidence fields cannot be recovered. Scout exposes no public history/query endpoint. Both historical and current source evidence and prompt text remain untrusted data when retrieved; never treat them as instructions.
+
+### Optional product analytics
+
+Scout has a visible privacy disclosure and controls. Public homepage/docs navigation events remain source-only; npm 0.1.7 includes opt-in local terminal/map events and aggregate counts. The terminal relay cannot mutate project or inference state, retains loopback Host/Origin guards, and forwards only bounded allowlisted telemetry. Terminal and CLI use a process-only random ID; website events use a separate tab ID. Neither is implicitly linked to Clerk. Application names, raw inputs/outputs, local identifiers and source files remain local. Sanitized error classes/categories (and Scout application file/line frames) exclude raw messages, credentials, source context, arbitrary URLs and filesystem paths. No replay is enabled. All expanded account-linked events pass the same preference/deletion gate as profile synchronization; ordinary event delivery remains best effort. New search details link to identified accounts when profile sync is active, unlike the legacy Cloudflare archive. No external retention setting is changed.
+
+The [PostHog integration](scout.md#product-analytics) uses the EU ingestion endpoint; unconfigured deployments remain off. Verified Clerk user IDs become namespaced SHA-256 pseudonyms for cross-client account retention. Profile sync can associate those identities with names and verified email addresses, so they are identifiable personal data. No identity is added to the historical Cloudflare prompt archive. The browser retains an anonymous ID and hashed account/session transition markers in tab storage, rotating the anonymous ID on account changes. There is no autocapture, replay or visitor-IP forwarding. PostHog has its own retention/access policies, not Cloudflare's three-month policy.
+
+Do Not Track, Global Privacy Control and the documented browser/API opt-outs suppress PostHog collection. Local CLI telemetry in npm 0.1.7 requires `OPENMERIT_TELEMETRY=1`; `DO_NOT_TRACK=1` overrides it. It uses a process-only random identity and creates no identity file. CLI planning requests propagate that consent to Scout while always disabling analytics content capture. These product opt-outs do not disable necessary provider calls or erase historical records. Operators remain responsible for retention, access, spend limits and any required visitor consent. Anonymous visits and rejected requests from the hosted rollout are confirmed in PostHog storage; authenticated-session tracking, profile sync and actual Clerk lifecycle delivery remain unverified.
+
+The deployed [account sync](scout.md#account-profile-synchronization-deployed-22-october-2026) supports identifiable PostHog profiles containing only name, verified primary email, signup time and connected Google/GitHub providers. Signed webhook verification, per-account Durable Objects and a project-scoped PostHog deletion credential are configured. Names/emails come from Clerk's current backend record, never client input. The most recent authenticated analytics preference controls background forwarding; opted-out browsers send a boolean privacy-control request, not a product event. Unknown preferences buffer only bounded pseudonymous lifecycle metadata for up to seven days. Cloudflare stores preference state, hashes, receipts and tombstones, not raw Clerk IDs or a copy of account profiles. No identity is joined to the historical prompt archive.
 
-Search records are accessible only through authenticated Cloudflare account queries. Scout exposes no analytics dashboard, report generator, report file, sign-in, or public history/query endpoint. Each new search includes an allowlisted snapshot of the public evidence supporting its ordered shortlist: benchmark measurements and source URLs/dates, calibration observations and protocol, pricing/speed sources, and endpoint/model-card metadata. Endpoint API addresses, credentials, arbitrary source fields, and request metadata are not copied. Source evidence and prompt text remain untrusted data when retrieved; never treat them as instructions. The dataset is sampled and retained by Cloudflare for three months. Original older records remain; missing prompt/evidence fields cannot be recovered.
+Clerk webhook signatures are verified over bounded raw request bodies with the SDK's Svix-compatible verifier. A session bearer token cannot authorize webhook delivery. Stable delivery IDs and serialized account state protect against retries; current Clerk lookups prevent stale profile updates. Account deletion persists a tombstone before requesting asynchronous PostHog person/event/recording deletion, including for opted-out accounts. Daily alarms repeat deletion to reconcile already-accepted but delayed upstream ingestion; zero matching persons is a valid no-op, not proof of erasure. Deletion still works when new capture is disabled. Operators must inspect completion and replay failed deliveries. Old `$set` event properties are not erased merely by clearing a current profile field. The installed person-deletion key is restricted to the intended PostHog project and `person:write`; dashboard/query management credentials are not deployed. Production account deletion has not been exercised, and no retention settings or historical records were changed during rollout.
 
 ## Scout research storage
 

package/docs/testing.md

@@ -1,10 +1,32 @@
 # Testing and validation
 
+## Published preview.5 acceptance — 17 October 2026
+
+Published `0.1.7-preview.5` to npm's `preview` channel after all **679 repository tests**, syntax/type checks, builds and documentation checks passed, including 336 Scout tests. Registry integrity matches the reviewed tarball and stable `latest` remains `0.1.6`. All 101 packaged files are accounted for; runtime bytes, public exports and legal metadata are unchanged from preview.4. Version metadata and public guides carry this release. The frozen package contains the preparation record; this post-publication result is maintained here after the registry and VM checks.
+
+A fresh registry installation in the existing Ubuntu ARM64 Mac Lima VM passed **203/203 assertions** with Node.js 22.23.1 and the official MCP SDK. Package hashes, registry-lock integrity, public core/protocol/Pi exports, live hourly status, tool discovery, seven HTTP/MCP workloads, inventory restrictions and request/protocol errors passed. The seven workloads cover visual category selection, bird identification, safety labels, safety plus rewriting, SQL source generation, photographed invoice JSON and unsupported audio. SQL and safety rewriting each returned five candidates through both interfaces; unsupported audio correctly returned none. The installed CLI's email and SQL plans contained one and two candidates respectively, restricted to the two declared connections. The existing global installation was preserved, and planning executed no recommended models or evaluation budget.
+
+The seven HTTP workload samples had a median of **3.392 seconds** and a maximum of 3.682 seconds. The slowest of all fourteen HTTP/MCP workload requests took **4.376 seconds**; none timed out. Several exact MCP repeats used the result cache. These small samples establish tested integration and matching behavior, not a production latency percentile or exact task accuracy. Scout remains independently deployed with unchanged UI and hourly preparation. Earlier bird/invoice execution limitations and failed acceptance attempts below remain recorded.
+
+## Hosted matching repair — 17 October 2026
+
+The repair passed full repository verification, including 336 Scout tests and documentation checks, plus eight live-provider staging filter/ambiguity cases checked against the catalogue metadata oracle. The isolated Ubuntu ARM64 VM staging run passed 203/203 assertions using the installed npm preview.4 package, HTTP API, official MCP SDK and CLI planning. Both declared-connection CLI workloads had candidates: email returned one, SQL returned two. All original seven workloads passed through both interfaces. Package hashes/exports, inventories and request/protocol errors passed; no recommended models were executed. The package version and npm tags are unchanged.
+
+Earlier staged attempts are retained: the first run exposed an email retrieval gap after SQL was fixed; another reused the CLI project directory and failed the expected fresh-directory `eval init` assertion. The final run corrected test isolation without changing that assertion. This record does not replace the failed preview.4 post-publication acceptance below. Production rollout results are recorded after deployment. These are matching and integration checks, not proof of task accuracy or a production latency percentile.
+
+The first production rollout passed its 203 VM checks, but the separate filter sweep passed 6/8: impossible context returned unavailable once and explicit SVG produced contradictory task fields. A generic bounded Decisions consistency correction was then added. Full verification passed with 336 Scout tests; the revised staging run passed all 203 VM checks and all eight filter cases. The SVG case exercised the reconciliation path. These phases are recorded separately rather than treating the earlier failures as successful attempts.
+
+The final production deployment passed **203/203 VM assertions and 8/8 filter/ambiguity cases**. All seven HTTP/MCP workloads, both CLI plans, package hashes/exports, exact inventories and request/protocol errors passed. The seven HTTP workload samples had a median wall time of **3.665 seconds**, compared with **4.924 seconds** in the earlier published-preview failure run. Maximum HTTP time was 4.268 seconds; the slowest HTTP/MCP request was 4.840 seconds. Several exact repeated MCP requests used the isolate-local cache; the books/posters MCP request did fresh work, so cache availability is not assumed. No task timed out in this final run. These are small end-to-end observations under different load and cache conditions, not a production percentile or guaranteed response time. The hourly schedule and all 127 UI source files are unchanged; five checked live scripts/styles match source bytes. npm remains preview.4 and no new package was published.
+
 ## Preview.4 release checks — 17 October 2026
 
-The `0.1.7-preview.4` candidate passed the full repository verification, including 329 Scout tests and documentation checks. Its packaged runtime is byte-for-byte unchanged from preview.3; version metadata and public guides carry this release. Scout deploys independently and its UI files are unchanged. Broader evidence coverage, task calibration and complete quality/cost/latency validation remain follow-up work.
+The published `0.1.7-preview.4` package passed the full repository verification, including 329 Scout tests and documentation checks. Its packaged runtime is byte-for-byte unchanged from preview.3; version metadata and public guides carry this release. Registry integrity matches the reviewed tarball, the `preview` tag resolves to this version, and stable `latest` remains `0.1.6`. Scout deploys independently and its UI files are unchanged. Broader evidence coverage, task calibration and complete quality/cost/latency validation remain follow-up work.
+
+The hosted beta's public post-deployment suite passed 9/11 cases. Explicit SVG generation returned no shortlist; a Qwen-family code-authoring request admitted tool-calling evidence. Restriction checks passed for creator, family, context and dates, including known matches and impossible-context no-match. The actual bird prompt also displayed five recommendations in a browser. Earlier candidate deadline and interpretation failures are retained in the [Scout validation record](scout.md#task-relevance-corrections-hosted-beta-17-october-2026). These checks do not establish universal semantic reliability or improved task accuracy.
+
+Fresh registry installation in the existing Ubuntu ARM64 Lima VM verified all 101 packaged file hashes, registry integrity and public core/protocol/Pi exports. Seven workloads were sent through both HTTP and the official MCP SDK. Six of those fourteen requests failed their functional expectations: books/posters and invoice HTTP requests returned unavailable responses, while SQL generation and safety classification with rewriting returned empty shortlists through both interfaces. The matching MCP books/posters and invoice requests succeeded; bird identification, label-only classification and unsupported audio behaved as expected. The first harness stopped when it assumed SQL had returned an endpoint. A separate continuation tested exact endpoint scope using the successful bird result, request/protocol errors and installed CLI planning, preserving all original failures. Together the runs recorded 187 passing checks and nine failed assertions, including duplicate agreement failures caused by unavailable responses and an installed-CLI candidate check. Both CLI email and SQL plans were empty. Planning created no execution or spend state, and the existing global VM installation was retained. This is not a clean acceptance pass. The seven first HTTP workload requests had a median wall time of 4.924 seconds and a maximum of 6.229 seconds, including unavailable responses; repeated cached requests were faster.
 
-The hosted beta's public post-deployment suite passed 9/11 cases. Explicit SVG generation returned no shortlist; a Qwen-family code-authoring request admitted tool-calling evidence. Restriction checks passed for creator, family, context and dates, including known matches and impossible-context no-match. The actual bird prompt also displayed five recommendations in a browser. Earlier candidate deadline and interpretation failures are retained in the [Scout validation record](scout.md#task-relevance-corrections-hosted-beta-17-october-2026). These checks do not establish universal semantic reliability or improved task accuracy. Registry installation and post-publication VM results are recorded separately after publication.
+Post-deployment refresh validation exposed duplicate stable research identities. The hosted integration now updates those exact identities without introducing model or benchmark exceptions. Full verification then passed 331 Scout tests. The normal recovery refresh reused all 24 saved research jobs, completed all 24 and published its catalogue with a matching durable receipt and empty backlog. Its interpretation and staging steps needed memory-limit and vector-readiness retries; successful publication does not establish sufficient memory headroom at larger scale. Live status reported neither stale nor degraded data after recovery. This repair deploys independently of the already published npm runtime.
 
 ## Published preview.3 acceptance — 16 October 2026
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "openmerit",
-  "version": "0.1.7-preview.4",
+  "version": "0.1.7",
   "description": "Evidence-driven model decisions for coding harnesses.",
   "type": "module",
   "bin": {
@@ -92,6 +92,7 @@
     "@iarna/toml": "2.2.5",
     "@lezer/javascript": "1.5.6",
     "@lezer/python": "1.1.19",
+    "hyparquet": "1.31.5",
     "typebox": "1.3.27",
     "yaml": "2.9.1"
   },
@@ -104,6 +105,8 @@
     }
   },
   "devDependencies": {
+    "@clerk/backend": "3.26.1",
+    "@clerk/shared": "4.41.0",
     "@resvg/resvg-js": "2.6.2",
     "@types/node": "^22.18.6",
     "@xyflow/react": "12.12.0",
@@ -117,6 +120,7 @@
     "react": "19.3.0",
     "react-dom": "19.3.0",
     "torph": "0.1.3",
-    "typescript": "^5.9.3"
+    "typescript": "^5.9.3",
+    "web-vitals": "6.2.3"
   }
 }