@stsepelin/checktrail 0.1.0-alpha.1

package/docs/CLIENTS.md

@@ -0,0 +1,109 @@
+# Application client compatibility
+
+The recorded snapshots below predate the Checktrail rename and retain their
+original package identities. A release candidate needs its own client checks;
+see [RENAMING.md](RENAMING.md).
+
+The local package was installed offline into a fresh temporary consumer and tested
+with Claude Code 2.1.263 and Codex CLI 0.154.0 on macOS arm64, Node 26.8.1. These
+are specific client surfaces and versions; they do not establish compatibility
+with every editor or with model-driven tool selection.
+
+| Client surface                  | Observed protocol | Verified behavior                                                                                                                                                                                               |
+| ------------------------------- | ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| Claude Code `mcp list`          | `2025-11-25`      | Successful connection and discovery of every advertised tool                                                                                                                                                    |
+| Codex app-server direct MCP API | `2025-06-18`      | Tool discovery, inspection, planning, HTTPS guidance results, default execution denial, rejection of tool-supplied trust, trusted native passing/failing/skipped-only runs, and exact retained report retrieval |
+
+Both clients use the SDK's existing legacy negotiation path. The separate SDK
+integration suite exercises `2026-07-28`; these application observations do not
+establish modern protocol or standard Tasks support. See
+[MCP compatibility](MCP-COMPATIBILITY.md).
+
+The initial Claude Code check exposed an ignored `starts_with` format in the
+guidance-reference URL output schema. That observation is retained in the
+[pre-fix snapshot](measurements/client-claude-before-schema-fix-darwin-arm64-node26.json).
+Guidance and exported Vue/Nuxt path constraints now use standard patterns, with
+URI format metadata for guidance links. The current health check requires empty
+stderr, rather than accepting that warning. It still does not call a tool or
+establish full client-side validation of tool results.
+
+Published schemas are also compiled with strict Ajv 2020-12 and standard formats.
+Runtime and JSON Schema regressions cover valid HTTPS links, wrong schemes/prefixes,
+length boundaries, route prefixes, authority-style probe paths, and Nuxt fragments
+and newlines. The latter probe restrictions previously existed only in runtime
+refinements; the exported schemas now express them too. This corrects the schema
+representation without permitting values the runtime previously rejected. Zod's
+[JSON Schema metadata](https://zod.dev/json-schema#metadata) carries the standard
+URI format; [JSON Schema validation](https://json-schema.org/draft/2020-12/json-schema-validation)
+defines the pattern and format vocabulary. Runtime-only refinements elsewhere are
+not thereby proven equivalent to every exported schema.
+
+Codex tests call its real app-server MCP API directly, without starting a model
+turn. A synthetic Node assertion passes, is changed to fail, and is replaced by a
+skipped test. The helper checks exact executed-test counters, result categories,
+unchanged source during each run, and equality of retained reports. Summary
+responses must not contain the temporary project path. This does not test an
+editor UI, model reasoning, every tool, or cancellation of active native work.
+
+## Reproduce
+
+The client executables must already be installed. The helper does not install or
+update clients, authenticate, invoke a model, register a persistent user server,
+or publish a package. Run from the source checkout after building:
+
+```sh
+node scripts/verify-mcp-clients.mjs claude
+node scripts/verify-mcp-clients.mjs codex
+```
+
+Each run packs the current checkout, installs the tarball with npm offline and
+lifecycle scripts disabled, creates an original synthetic fixture, and removes
+its temporary installation afterward. A transparent stdio observer records only
+protocol versions, methods, advertised tool names and process lifecycle metadata;
+it forwards protocol bytes unchanged. Tool calls go through the actual client.
+No custom MCP compatibility implementation is introduced.
+
+Claude Code uses a fresh `CLAUDE_CONFIG_DIR`, bare mode, user settings from that
+directory, and disabled nonessential traffic/marketplace auto-installation.
+Its own `mcp add-json` writes only that temporary configuration. See the official
+[configuration directory](https://code.claude.com/docs/en/claude-directory) and
+[environment settings](https://code.claude.com/docs/en/env-vars) references.
+
+Codex receives process-local configuration overrides. Existing named MCP servers
+are individually disabled, and a preflight requires only the two synthetic
+servers to be enabled. Apps, plugins and hooks are disabled for this invocation;
+its ephemeral thread uses temporary state/log directories. Existing configuration
+files and home-directory variables are not rewritten. Client policy constraints
+can still prevent this profile from running; such failures are not passes.
+
+The configured Codex provider points to a temporary loopback endpoint. It answers
+only the observed model-catalog GET request; any inference request fails the
+measurement. No model turn is submitted and no provider credentials are inherited
+from the invoking environment. This is a direct client-integration check, not a
+network-sandbox or client-telemetry attestation. The API schema was generated by
+the installed CLI; the official [app-server reference](https://developers.openai.com/codex/app-server)
+and [configuration reference](https://developers.openai.com/codex/config-reference)
+describe those interfaces.
+
+The observer checks that its own process and each observed server process have
+stopped before deleting the temporary directory. Client shutdown can terminate an
+observer before it records the server's exit event; recorded exit counts are
+therefore separate from the process-liveness check. Neither establishes cleanup
+of an active validation's descendants. That behavior has separate engine/SDK
+lifecycle tests.
+
+## Recorded evidence
+
+- [Claude Code observation](measurements/client-claude-darwin-arm64-node26.json)
+- [Codex app-server observation](measurements/client-codex-darwin-arm64-node26.json)
+
+Snapshots include the client/runtime versions, package and installed runtime
+hashes, harness hash, observed protocols, outcomes and available exit records.
+They omit source, raw client logs, configured personal server names and absolute
+paths. The package digest identifies the measured tarball, not every later
+checkout: changing documentation or adding these snapshots changes package bytes.
+The runtime digest identifies the installed `dist/src` tree independently.
+
+These local checks satisfy only the named client profiles. Claude Desktop,
+Cursor, VS Code integrations, other client versions and target operating systems
+remain unverified. No public package installation or hosted CI run is implied.

package/docs/CONTRACTS.md

@@ -0,0 +1,73 @@
+# Captured producer/consumer contracts
+
+Contract validation checks actual JSON samples against a consumer's schema using
+Ajv 8.20.0 and ajv-formats 3.0.1. It does not infer general schema compatibility or
+run a service. Each contract names both sides, their recorded source fingerprints,
+the consumer schema and named captured samples. The artifact format is versioned
+by `contract-bundle.schema.json`.
+
+```sh
+checktrail check-contracts --root /path/to/project --input contract.json
+```
+
+The library exports async `validateContracts(bundle, {timeoutMs?, signal?})`.
+MCP `contract_validation` accepts a local relative `input` and optional `timeoutMs`.
+No project execution permission is needed to read and validate artifacts. MCP
+allows one contract worker at a time, remains responsive to other tools, and
+supports cancellation and connection/signal shutdown. Root, output mode and
+execution permission remain operator settings.
+
+The supported schema dialect is JSON Schema 2020-12. A root `type` is required;
+empty or catch-all schemas cannot establish a meaningful contract here. Strict
+schema/type/required checking and format assertions are enabled. Unknown keywords,
+unknown formats and unsupported dialects are incomplete evidence. Local references
+within the document are supported; external references, asynchronous schemas and
+`$data` and OpenAPI-only `nullable`/`discriminator` extensions are rejected. Literal `$ref` names inside `const` or `enum`
+values are data. No schema loader or network fetch is configured.
+
+Validation does not coerce types, insert defaults or remove fields. For example,
+`"2"` fails an integer requirement instead of being changed to `2`. Required fields,
+numeric limits, extra properties, formats and nested array elements are checked
+according to the schema. A valid payload says nothing about unexpressed business
+rules. Missing samples and incomplete captures cannot pass.
+
+Each contract is passed, failed or incomplete. Known payload failures take
+precedence over other contracts' incompleteness; all counts remain visible.
+`passed + failed + unverified = contracts`, and `accepted + rejected = samples`
+for evaluated samples. Detailed results include contract/sample names and schema
+keyword/instance/schema paths, without echoing payload values. Summaries omit those
+identifiers. Exit codes are 0 passed, 1 failed, and 2 incomplete or malformed input.
+
+Schema compilation and matching run in a worker with a default 10-second budget
+(CLI default 30 seconds), configurable up to 30 seconds, a 64 MiB old-generation
+heap limit, a 16 MiB young-generation limit and a 4 MiB stack limit. Timeout,
+cancellation or worker failure returns incomplete evidence. Worker limits are not
+a sandbox or a promise about total process memory. Inputs are limited to 8 MiB,
+depth 32, 100,000 JSON values, 100 contracts and 1,000 samples per contract. Duplicate
+contract IDs, duplicate sample names, missing payloads and non-JSON values are
+rejected. The worker never loads producer or consumer application code.
+
+Results carry `imported-contract-samples` provenance. A capture timestamp and source
+fingerprints are declared evidence, not proof of origin, freshness, transport,
+authorization, or complete behavior coverage. Use native isolated integration
+tests to capture the producer output and consumer schema when those claims matter.
+
+The synthetic `examples/contracts/` pair can be exercised from this checkout:
+
+```sh
+node examples/contracts/capture.mjs > .checktrail/contracts.json
+node dist/src/cli.js check-contracts --root . --input .checktrail/contracts.json
+node examples/contracts/capture.mjs --broken > .checktrail/contracts.json
+node dist/src/cli.js check-contracts --root . --input .checktrail/contracts.json
+```
+
+Create `.checktrail/` first if it does not exist. The first capture serializes
+numeric quantities; the second simulates a legacy string quantity and fails the
+consumer schema. This example fingerprints its producer module and consumer schema
+only. Tests cover native producer serialization, valid/invalid boundary payloads,
+non-mutating validation, local references, formats, unsupported definitions, input
+limits, pathological regex deadlines, cancellation and CLI/MCP privacy.
+
+References: [JSON Schema 2020-12](https://json-schema.org/draft/2020-12),
+[Ajv strict mode](https://ajv.js.org/strict-mode),
+[Ajv validation options](https://ajv.js.org/options).

package/docs/DEPENDENCIES.md

@@ -0,0 +1,42 @@
+# Dependency provenance and notices
+
+Run `node scripts/audit-dependencies.mjs` from the checkout after installing the
+locked dependencies. It makes no network calls or project changes and emits a
+JSON inventory of production npm packages, their exact versions, declared licenses,
+registry tarball locations, lockfile integrity declarations and notice-file hashes.
+The installed production dependency tree must reconcile with the lock. Missing
+notices produce incomplete evidence; mismatches and stale fallback entries fail.
+
+This audits installed metadata and notice presence. It does not recompute registry
+tarball integrity over installed code, establish legal compliance, cover development
+tools or establish native/container supply-chain provenance. The package uses npm
+dependencies without bundling their code; installed packages retain their own
+notices. Audit and retain those notices if distribution arrangements change.
+
+## Supplementary upstream notice
+
+`@nodable/entities@3.0.0` declares MIT but its installed distribution omits a
+license file. The supplementary notice is preserved verbatim at
+[`licenses/nodable-entities-3.0.0.txt`](licenses/nodable-entities-3.0.0.txt).
+
+The npm registry metadata for that exact version identifies Git commit
+`d2070d76a8ba07e6c7fa142caeb51ffd756e47eb`. The notice was retrieved from that
+commit's [root LICENSE](https://raw.githubusercontent.com/nodable/val-parsers/d2070d76a8ba07e6c7fa142caeb51ffd756e47eb/LICENSE).
+Every distributed source/declaration file and the README was compared byte-for-byte
+with the commit's `Entity/` directory and matched.
+
+The upstream manifest says **2.2.0**, while the installed/registry manifest says
+**3.0.0**. That discrepancy is preserved rather than treated as a matching release
+manifest. The fallback records both versions, the exact installed file inventory
+and SHA-256 digests. The offline audit rejects changed source, a changed notice,
+an unused fallback or a newly bundled notice that makes the fallback stale.
+
+`scripts/license-sources.json` records this preparation evidence. The supplementary
+notice and this explanation ship inside the package's existing `docs/` allowlist.
+No dependency version or package-manager signature setting was changed.
+
+`node scripts/smoke-package.mjs` also compares two packs of the same checkout byte
+for byte, checks the file allowlist and installs the tarball offline into a fresh
+consumer. This demonstrates repeated packaging of one built checkout, not an
+independent cross-platform rebuild or reproducible native toolchain. The dependency
+audit and package smoke passed in the hosted matrix at `52ba415`; see `NATIVE-CI.md`.

package/docs/DJANGO.md

@@ -0,0 +1,70 @@
+# Django URL inventory profile
+
+Select `python.django-routes` and provide project-local
+`checktrail.django.json`:
+
+```json
+{
+  "schemaVersion": 1,
+  "settings": "settings",
+  "assembly": "catalog-api",
+  "environment": "isolated-test"
+}
+```
+
+The verified runtime is Django 6.1.1 on Python 3.12.13 in an isolated Linux
+container. Other Django versions are unavailable until verified. Settings must
+resolve to one inventoried local Python module or package. Planning does not
+import settings, URLconfs or applications. Unselected framework profiles are not
+loaded by other Python checks.
+
+Trusted execution fixes `DJANGO_SETTINGS_MODULE` to the selected module, invokes
+`django.setup()`, then recursively enumerates the actual root URL resolver. This
+includes registrations made by application `ready()` hooks and nested included
+URLconfs. No HTTP server, request, migration or package installation is started
+by the adapter. Application setup can access services or files with the process
+user's privileges; the declared environment label is not a sandbox. Supply isolated
+settings and prepare any required services separately.
+
+The supported profile handles native `URLPattern` and `URLResolver` with
+`RoutePattern` and `RegexPattern`, plus Django's built-in integer, string, UUID,
+slug and path converters. Custom route classes/converters, translated route
+objects and locale-prefix patterns are incomplete evidence. Unsupported callable
+identities, non-JSON default arguments, setup/resolver errors and empty URL tables
+also cannot pass. Traversal is limited to depth 32 and 20,000 visited nodes, within
+the shared execution time/output limits.
+
+Each detailed check result includes a `runtime` route inventory compatible with
+the runtime comparison tool. Entries retain ordered native pattern chains,
+pattern class, regex text, regex flags and endpoint matching mode. The last field
+matters because Django can use full matching or searching for identical regex
+text. Namespace order, route name, handler identity and a hash of inherited/default
+arguments are also captured. Defaults are hashed rather than emitted directly;
+this is not a guarantee of secrecy for low-entropy values.
+
+Exact duplicate pattern chains fail. Prefix/suffix names are compared exactly.
+Equivalent URL matchers expressed through different nesting or regex syntax are
+not normalized, and arbitrary pattern overlap is not analyzed. Request methods,
+authorization, middleware effects, callback behavior and per-request URLconf
+overrides are outside this projection. Regex matching is not executed against
+sample request paths by this check.
+
+The capture records native visited/unsupported counts, and the parser verifies
+collection accounting, profile/source identity and canonical pattern signatures.
+Known duplicates fail even if a different node is unsupported; incomplete
+collection metadata remains available. Summary reports omit runtime entries and
+application logs. The shared engine verifies source and policy after execution.
+Check scope names the settings entry file, not every imported source file.
+
+Native regression fixtures cover nested URLconfs, namespace capture, duplicated
+resolver branches, fixed counterparts, regex endpoint-mode near misses, setup-time
+registrations, empty tables, custom converters/routes and resolver failures. Tests
+also verify non-executing planning, protected settings environment, malformed
+evidence and skipped unused framework profiles. Reproduce prepared Linux tests
+with `node scripts/verify-framework-container.mjs`; preparation dependencies are
+pinned in `scripts/framework-tools.requirements.txt`. Hosted CI remains unexecuted.
+
+References: [Django URL dispatcher](https://docs.djangoproject.com/en/6.0/topics/http/urls/),
+[Django setup and settings](https://docs.djangoproject.com/en/4.2/topics/settings/).
+Collector behavior is verified against installed Django 6.1.1 source and fixtures;
+these references describe the underlying public concepts.

package/docs/DOTNET.md

@@ -0,0 +1,114 @@
+# C# compilation
+
+`dotnet.csharp` compiles all inventoried `.cs` files in a project as one explicitly
+configured assembly. It requires one project-root `.csproj` marker, a prepared
+.NET SDK 10.0.401, Roslyn 5.9.0 from that SDK, and the .NET 10.0.12 runtime and
+reference pack. This profile has native evidence on arm64 Linux.
+
+The project-root `checktrail.dotnet.json` declares the complete supported
+configuration. For example:
+
+```json
+{
+  "schemaVersion": 1,
+  "targetFramework": "net10.0",
+  "assemblyName": "Example.Catalog",
+  "languageVersion": "14",
+  "outputKind": "library",
+  "nullable": "enable",
+  "warningsAsErrors": true,
+  "allowUnsafe": false,
+  "checkedArithmetic": true,
+  "implicitUsings": false,
+  "defines": ["NET", "NET10_0", "NETCOREAPP"],
+  "references": []
+}
+```
+
+The adapter does not infer these settings from project XML, environment variables,
+`.editorconfig`, response files or build targets. C# language versions 12, 13 and
+14 are accepted; preview versions are not. Nullable context accepts `enable`,
+`disable`, `warnings` or `annotations`. The assembly name matters for accessibility,
+including friend assemblies. Conditional symbols are exact explicit identifiers;
+the adapter adds none automatically and does not check every preprocessor branch.
+
+When requested, `implicitUsings` adds the fixed global imports for `System`,
+`System.Collections.Generic`, `System.IO`, `System.Linq`, `System.Net.Http`,
+`System.Threading` and `System.Threading.Tasks`. This owned syntax tree is counted
+separately from project inputs. Source directives retain native compiler semantics.
+The compiler enables its warning waves; `warningsAsErrors` controls promotion.
+Diagnostics suppressed by the compiler cannot establish complete findings and
+make an otherwise successful check inconclusive.
+
+## Dependencies and execution
+
+Each optional reference has the shape `{ "path": "…dll", "sha256": "…" }`.
+Paths resolve from the project, must remain inside the operator root and cannot
+traverse symbolic links. Prepared DLLs can live in the excluded `.checktrail/`
+directory. Exact bytes are checked during planning, before native compilation
+and afterwards. Limits are 128 references, 32 MiB per DLL and 128 MiB total.
+Duplicates, unpinned changes, non-managed files, standalone modules and multi-file
+assemblies are rejected. Assembly-file metadata cannot bring in unpinned sidecars.
+
+The runner locates the exact SDK and invokes its `csc.dll` directly to prepare an
+owned compiler helper in a fresh temporary directory. It uses existing local
+reference assemblies. It does not run MSBuild, NuGet restore, workloads, project
+targets, source generators, analyzers, application initializers or tests. Project
+`global.json` does not select a different SDK. No project assembly is loaded for
+execution; reference assemblies are read as metadata. SDK compiler libraries are
+trusted tooling and are loaded by the owned helper.
+
+Native processes receive the snapshotted PATH and fixed .NET settings, including
+disabled telemetry/diagnostics and invariant globalization. Other inherited or
+operator-supplied environment values, including startup hooks and profiler
+configuration, are not forwarded. Operator trust is still required: native
+compilers and metadata readers have the process user's privileges, and this is
+not a sandbox.
+
+Project output is emitted into a bounded memory stream and discarded. Owned
+helper artifacts are temporary; no `bin`, `obj` or application DLL is written
+inside the project. Output is limited to 32 MiB, diagnostics to 2,000 entries,
+native stdout/stderr to 1 MiB per invocation and 4 MiB total, and the prepared
+argument to 100 KiB. Native managed heaps have a 256 MiB bound, which does not
+bound total process memory. The engine's normal time and process-group limits
+also apply.
+
+## Evidence and limitations
+
+Every inventoried source must have a native syntax tree covering its complete
+text and a completed semantic-model diagnostic query. Input paths are reconciled
+exactly, including empty files and partial classes. The profile requires a
+successful native emit with nonempty output, matching declared options and
+reference counts. Compiler, runtime and SDK identities are retained, together
+with a digest of the reference-pack DLLs and compiler assemblies checked before
+and after the run. This is not a digest of the entire installed SDK or runtime.
+
+The helper uses Roslyn's
+[semantic model](https://learn.microsoft.com/en-us/dotnet/csharp/roslyn-sdk/get-started/semantic-analysis)
+and compilation emit diagnostics. Errors retain their `CS…` codes and original
+source locations. Failed compilation has incomplete findings. Missing, malformed,
+inconsistent or suppressed evidence cannot pass. Native compiler diagnostics are
+not analyzer, framework, authorization or business-logic coverage.
+
+The result means that the inventoried files compile under the explicit profile.
+It does not prove equivalence to an SDK project build. Generated source under
+`obj`/`build` is excluded and never silently included. Prepare required generated
+source as inventoried files or a pinned assembly. F#, Visual Basic, scripts,
+Razor/XAML, source generation, multi-project builds, other target frameworks,
+MSBuild properties, package restore, format checks and test runners require
+separate profiles. Solution files, including `.slnx`, are detected; select actual
+C# project roots for compilation.
+
+## Reproduce
+
+```sh
+npm run build
+docker build --file scripts/dotnet-tools.Dockerfile --tag checktrail-dotnet-test:10.0.401 scripts
+node scripts/verify-dotnet-container.mjs
+```
+
+The helper checks the pinned native SDK/compiler, runs original synthetic cases
+with networking disabled, and tests a fresh offline package installation through
+the library, CLI and MCP. The host npm cache needs the production dependencies.
+The hosted amd64 job passed at `52ba415`; see `NATIVE-CI.md`. Windows process-tree handling and
+native macOS .NET execution remain unverified.

package/docs/ENVIRONMENTS.md

@@ -0,0 +1,52 @@
+# Project environments
+
+A project may declare required environment variable names in `checktrail.json`:
+
+```json
+{
+  "schemaVersion": 1,
+  "projects": [
+    {
+      "path": ".",
+      "checks": ["javascript.node-test"],
+      "environment": ["APP_ENV", "DATABASE_URL"]
+    }
+  ]
+}
+```
+
+The declaration does not grant access. The CLI operator must name each permitted
+variable, whose value is read from the CLI's environment:
+
+```sh
+checktrail plan --root /path/to/project --allow-env APP_ENV --allow-env DATABASE_URL
+checktrail run --root /path/to/project --trust-project --allow-env APP_ENV --allow-env DATABASE_URL
+checktrail serve --root /path/to/project --allow-execution --allow-env APP_ENV --allow-env DATABASE_URL
+```
+
+The library accepts an explicit `environment` record in `createPlan` and
+`validate` options. MCP uses values captured at server startup; tool-call
+arguments cannot add names, supply values or change permissions. Reading a
+`.env` file is not part of this feature. Tools may independently read their own
+configuration because executed project code is trusted, not sandboxed.
+
+Only names requested by each configured project are added to that project's
+validation processes and version probes. Other supplied values are not forwarded.
+The existing minimal process environment remains present. Missing permission or
+an unset required value makes that project's checks unavailable before execution.
+A named but unset CLI permission is a startup error; explicitly empty values are
+allowed. Names use uppercase ASCII letters, digits and underscores, start with a
+letter or underscore, and are bounded along with values and the number of entries.
+Duplicate names and conflicts with fixed adapter settings, such as Go's offline
+proxy setting, are rejected.
+
+Plans contain required names. Detailed results contain supplied names and a
+SHA-256 fingerprint of their sorted values. Values are not copied into command
+metadata; summary output omits names and fingerprints. Fingerprints are identity
+signals, not secret encryption, and invoked tools can print values into detailed
+logs. Tool identity reuse is scoped to the environment fingerprint. Operator
+values are copied at the start of a library invocation and at MCP startup.
+
+This supplies process configuration; it does not start databases, load credentials
+from external services, validate service availability, or promise isolated tests.
+Those prerequisites still need explicit preparation by the consumer.

package/docs/ESLINT.md

@@ -0,0 +1,68 @@
+# ESLint validation
+
+Select `javascript.eslint` in `checktrail.json`:
+
+```json
+{
+  "schemaVersion": 1,
+  "projects": [{ "path": ".", "checks": ["javascript.eslint"] }]
+}
+```
+
+The project must have exactly one `eslint.config.js`, `eslint.config.mjs` or
+`eslint.config.cjs`. The adapter selects that file explicitly; it does not search
+outside the project for configuration or apply nested configuration overrides.
+ESLint must already be installed in the project's `node_modules` or an ancestor's
+`node_modules` within the operator-configured root. Missing or ambiguous
+prerequisites produce an unavailable check. Planning only inspects paths.
+
+## Scope and execution
+
+Candidates are inventoried `.js`, `.mjs`, `.cjs`, `.jsx`, `.ts`, `.mts`, `.cts`,
+`.tsx` and `.vue` files owned by the detected project. Dependency/build exclusions
+and nested project boundaries apply before planning. Configuration files with
+these extensions are candidates too. Every candidate needs matching configuration
+and at least one enabled rule; otherwise the check is incomplete.
+
+After operator trust is granted, a child process loads the consumer's installed
+ESLint API and configuration. It uses `calculateConfigForFile` and `lintText` for
+each explicit file, preserving filenames containing spaces or glob characters.
+It requests no fixes and does not use the file-cache path, preserving an existing
+`.eslintcache`. Project configuration, plugins and processors remain executable
+trusted code with the process user's privileges.
+
+Rule settings and inline suppressions follow the selected configuration. This is
+evidence that configured checks ran, not a judgment that the chosen rules are
+sufficient. Suppression reconciliation and policy packs are separate future work.
+The file must have enabled rules even if its contents happen to be empty.
+
+## Evidence
+
+The child emits versioned JSON with the ESLint version, per-file configuration
+and enabled-rule evidence, native diagnostics and counters. Parsing requires
+exact planned paths, one result per configured file, consistent severity counts
+and complete output. Ignored/unconfigured files, no enabled rules, malformed
+output, unexpected stderr, cancellation or output truncation cannot yield pass.
+Both errors and warnings fail a fully accounted check. Configuration/runtime
+exceptions are execution errors, not code findings.
+
+CLI and MCP use the same engine and interpretation. Summary output omits paths,
+rule messages and source excerpts. Detailed mode includes the structured evidence
+in the process output. Native lint diagnostics may contain source identifiers.
+
+## Verified scope
+
+Integration tests exercise ESLint 10.10.0 on synthetic JavaScript: valid code,
+syntax errors, rule errors/warnings, ignored files, unmatched TypeScript files,
+disabled rules, invalid configuration, root-hoisted tooling, unusual filenames,
+existing caches and CLI/MCP parity. Other versions, legacy `.eslintrc` files,
+TypeScript configuration loaders and parser/processor combinations for TypeScript
+or Vue are not verified by this adapter's integration suite. Those file types
+are candidates so that lack of matching configuration remains visible.
+
+Reference: [ESLint Node.js API](https://eslint.org/docs/latest/integrate/nodejs-api).
+
+A separate [external integration evaluation](EXTERNAL-EVALUATION.md) checks
+upstream-authored synthetic cases for three core rules against a frozen verifier
+and the native CLI. This adds diagnostic-preservation evidence for those profiles;
+it does not establish a general false-positive rate or parser/plugin compatibility.

package/docs/EVALUATION.md

@@ -0,0 +1,124 @@
+# Detection and completeness evaluation
+
+The recorded results use the archived `synthetic-evaluation-v1` corpus under
+`measurements/evaluation-corpus-repo-verifier.json`. The current runnable corpus
+is v2 with Checktrail configuration filenames; these historical measurements
+do not validate v2. See [RENAMING.md](RENAMING.md).
+
+The development evaluation compares Checktrail with direct native commands on
+the same original synthetic Node, TypeScript and GitHub Actions inputs. The
+versioned corpus is separate from the adapter regression tests and was authored
+after those profiles were implemented. It is a small, paired convenience sample,
+not independent held-out evidence or a comparison with a private review workflow.
+
+```sh
+npm run build
+node scripts/measure-evaluation.mjs > evaluation.json
+```
+
+Prepare TypeScript from the locked development dependencies and the official
+actionlint 1.7.12 release on `PATH` first. The harness installs nothing. Its only
+inputs are the checked-in development corpus and prepared tools; it accepts no
+custom executable commands. Fixtures are created in temporary directories and
+removed after each case. The corpus remains unchanged during measurement.
+
+## Labels and observations
+
+`scripts/evaluation-corpus.json` records an ID, family, original source files,
+label and expected diagnostic signals for every case:
+
+- `defect`: a deliberately broken assertion, type contract or workflow construct.
+- `valid`: a corresponding valid example or valid boundary condition.
+- `insufficient-evidence`: empty/skipped tests, a source outside compiler scope or
+  a local action whose metadata has not been prepared.
+- `unsupported-profile`: a valid YAML alias outside this adapter's declared profile.
+
+A failed command counts as a detected defect only when its expected diagnostic
+signal is present. An unrelated runtime failure is not a detected assertion
+defect. Incomplete observations remain in the relevant denominators and have
+their own matrix column. Valid inputs that produce incomplete results are shown
+separately from both passes and false-positive failures. Unknown/duplicate/missing
+cases and malformed measurement processes fail the harness instead of shrinking
+the sample. Source changes also invalidate measurement.
+
+The report preserves per-case classifications, diagnostic identifiers, tool
+versions, source fingerprints, native output digests and raw timings. It also
+records hashes of the corpus, harness, runtime artifacts, compiler distribution
+and package lock. These identities are checked before and after the run. Raw
+source, logs, local paths and command environments are not stored in the report.
+
+## Native baseline and interpretation
+
+The baseline is the exit status from these fixed native operations:
+
+| Family         | Native operation                                                                                | Verifier difference                                                                           |
+| -------------- | ----------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------- |
+| Node           | `node --test --test-reporter=tap value.test.js`                                                 | Structured event accounting requires executed, non-skipped tests                              |
+| TypeScript     | `tsc --project tsconfig.json --noEmit --incremental false --pretty false`                       | Forces `noCheck=false` and accounts for all inventoried source through the compiler file list |
+| GitHub Actions | `actionlint -format '{{json .}}' -shellcheck= -pyflakes= -no-color .github/workflows/check.yml` | Explicit configuration, local input preparation and native per-file completion requirements   |
+
+This compares operational contracts, not competing parsing algorithms. In
+particular, the additional TypeScript defect is caught because the verifier
+overrides the fixture's `noCheck` setting. The native compiler honors that setting;
+this is not a compiler defect. Likewise actionlint's acceptance of a YAML alias
+is correct native behavior, while this adapter reports its unsupported profile.
+Neither difference establishes generally better detection quality.
+
+## Recorded observations
+
+The [macOS/Node 26 report](measurements/evaluation-darwin-arm64-node26.json) and
+[Linux/Node 22 report](measurements/evaluation-linux-arm64-node22.json) agree on
+classifications and expected diagnostic signals. In this fixed corpus, both
+systems detect the ordinary broken/fixed pairs without failing their valid
+counterparts. The verifier additionally identifies the disabled TypeScript
+analysis and keeps the insufficient-evidence/profile cases from passing.
+
+The Linux measurement ran with networking disabled and the repository mounted
+read-only in the explicitly prepared `checktrail-actionlint-test:1.7.12` image.
+The local immutable image reference was
+`sha256:60fe972d5efcb28a85a9557afe5e8428949dfe675b57b52bf33e9b3e560354dd`.
+Fixture directories were writable temporary storage inside the container.
+This is local container evidence; hosted CI and Windows were not measured.
+
+Every case has one observation per system. Native/engine order alternates between
+cases; each engine observation starts a fresh Node worker. `wallMs` includes
+process startup, engine import and child execution as applicable. `engineMs`
+excludes worker startup/import. Fixture preparation, compiler copying and final
+cleanup are outside the measured operation. Filesystem caches and scheduling are
+uncontrolled. These timings describe this run, not a statistically established
+overhead ratio or an isolated comparison between operating systems.
+
+The paired, deliberately selected sample cannot justify population confidence
+intervals or production false-positive estimates. Rates are exact fractions over
+the stated corpus only; raw counts and all incomplete cases are retained. No
+model was called, so model-token usage and provider cost are zero. Machine/runtime
+cost, human review effort and the cost of creating this project are not measured.
+
+## External integration sample
+
+A separate [external ESLint evaluation](EXTERNAL-EVALUATION.md) now uses upstream-
+authored synthetic cases against a verifier frozen before inspecting them. It
+compares diagnostic preservation with the native tool on macOS and Linux. This
+does not turn the development corpus above into held-out evidence or establish
+general review quality.
+
+The [external Ruff cohort](EXTERNAL-RUFF-EVALUATION.md) adds Python diagnostic
+preservation evidence on both platforms. Its mixed files do not supply a separate
+clean-case denominator or establish native-rule effectiveness.
+
+## Remaining evidence
+
+Independent end-to-end review cases beyond that integration sample, additional
+rule/language families, larger workloads,
+native tool configurations beyond these fixed baselines, and comparison with a
+previous review process remain work. The project makes no equivalent-or-better
+review-quality claim. A private baseline can be evaluated privately against the
+same labels, with only aggregate results considered for public documentation.
+
+The tests for the harness reject lost cases, duplicate IDs, invalid observations,
+incorrect failure signals and removal of incomplete cases from denominators.
+Recorded matrices are independently recomputed from their raw observations.
+
+The [prior-workflow protocol](PRIOR-WORKFLOW-EVALUATION.md) specifies a provisional
+public baseline, matched comparison arms, declaration requirements and independent
+adjudication. No prior-workflow comparison has been run.