openmerit 0.1.12 → 0.3.0

package/docs/metrics-and-evidence.md

@@ -1,48 +0,0 @@
-# Metrics and evidence
-
-## Task-specific profiles
-
-The full catalogue is available to every product, but the harness infers a task-specific proposal from the application's outputs, tools, and likely failure modes. Before collection, the user sees and confirms the required metrics, direction, sampling design, minimum sample count, aggregation, tolerance, optional hard constraints, and estimated cost.
-
-## Initial metric catalogue
-
-Quality and behavior: task quality, task success, consistency, instruction following, tool-use performance, structured-output reliability, hallucination rate, reasoning efficiency, recovery ability, context handling, retrieval-use quality, long-horizon performance, and preference fit.
-
-Cost and resources: input tokens, output tokens, total task cost, cost per successful task, and agent steps.
-
-Latency and operations: time to first useful output, end-to-end latency, throughput, retry rate, and human-intervention rate.
-
-## Metric states
-
-- `measured` — numeric value and evidence exist.
-- `not_applicable` — the metric does not apply.
-- `not_configured` — collection or evaluation is missing.
-- `insufficient_evidence` — relevant, but without enough representative samples.
-
-Missing measurements are never converted to zero.
-
-## Single and repeated runs
-
-A single execution can report its cost, latency, token use, steps, success, and evaluator score. Distribution and reliability claims require repeated runs: rates, consistency, variance, averages, percentiles, throughput, prompt sensitivity, and regression stability.
-
-Two forms of repetition may be needed: repeat the same input to measure stochastic variance, and run representative task instances to measure general task performance.
-
-Each objective records one sampling strategy: `single_run`, `repeated_same_input`, `representative_task_set`, `repeated_representative_task_set`, or `load_test`. It also records representative-case count, repetitions per case, aggregation, and rationale. Every measured metric record repeats that aggregation explicitly (`single_value`, `sum`, `mean`, `rate`, `distribution`, or `percentile`). Rate, mean, distribution, percentile, and catalogue-defined reliability metrics are rejected when their plan contains fewer than two runs or uses `single_run`; `total_task_cost` sample windows must use `sum` because that value is used for budget accounting.
-
-The task profile sets the minimum sample count for each required metric; there is no universal number of runs. `minimumSamples` is evidence quantity. An optional `constraint` is a real performance threshold in the metric's unit; OpenMerit never interprets a three-sample requirement as a metric value of at least three. Ratio thresholds must remain between zero and one. A completed-task signal can advance an assessment cadence, but it does not by itself supply a quality score. Quality requires a task-specific evaluator, known outcome, or other justified measurement. Cost, token use, and latency must come from the application call or its observability layer. The baseline and controlled challenger runs supply the measured evidence used for comparison. Catalogue price and model reputation help the harness choose candidates to test; they are not substitutes for these measurements. Setup records the exact incumbent, preliminary leads are stored separately from metric records, and discovery freezes the accepted candidate set. Challenger trials preserve one experiment manifest containing the application revision, recorded shuffle seed, comparable cases, exact model identities, and per-run evidence. Accepted assessments are carried into frontier calculation rather than reconstructed from chat history.
-
-Each `metric_window_available` signal replaces the stored aggregate window for that metric. Send a cumulative, application-run window with its sample count, explicit aggregation, and evidence reference; OpenMerit does not add separate or overlapping window counts. Challenger metric windows must also fall inside their assessment execution window. Core marks the baseline ready only when instrumentation is ready and every required metric has a finite measured value, the confirmed minimum sample count, and an evidence reference. The harness remains responsible for measuring the real application call and authenticating those references; a reported window is not independently re-run or verified by core.
-
-## Evidence references
-
-Results point to durable evidence rather than embedding arbitrary traces. Sources include harness traces, evaluation artifacts, observability records, external sources, and user feedback. References may contain a URI, media type, and digest. Successful intents require evidence; frontier certification also requires calculation evidence and exact assessment IDs.
-
-## Readiness contract
-
-Setup must return an explicit observability coverage report containing every required metric, which metrics are covered, which remain missing, and when the check occurred. OpenMerit derives readiness from those sets; a harness cannot label coverage `ready` while a required metric is absent.
-
-If setup reports a required gap, OpenMerit issues `instrument_observability`. A successful instrumentation result is accepted only when every required metric is reported as covered. OpenMerit explicitly asks Pi to exercise each collector and grader through the application path, inspect output values, and retain artifacts. OpenMerit validates the coverage report but does not independently run the collector or grader, so `ready` is still a harness claim rather than independently verified end-to-end proof. Request a reported production metric window before relying on it for model decisions.
-
-Product instrumentation and OpenMerit auditing are separate. The confirmed application LLM target emits measurements such as quality, success, tokens, cost, latency, retries, tool calls, and intervention. Every metric record carries that target ID. OpenMerit records the signals, policy decision, intent lifecycle, evidence identifiers, and readiness or frontier decision that followed.
-
-Pi's own assistant messages are orchestration data, not application evidence. Application provider cost, input and output tokens, and end-to-end task latency must come from the confirmed application call or its observability layer. Tool-use performance requires harness-trace evidence; code coverage alone is not evidence of tool-use quality. Quality and task-specific behavior can still come from verified project evaluators.

package/docs/operations.md

@@ -1,40 +0,0 @@
-# Project state and operations
-
-**Scope — unreleased Pi replacement:** the evaluation lifecycle files and `/openmerit` commands below describe the core coordinator and historical Pi 0.87 adapter. The Pi 1.5.0 Scout adapter does not create or advance that lifecycle, manage scheduling, or expose those commands. See [current Pi paths and targeted shell commands](pi-extension.md). Terminal files remain current; standalone evaluation files are documented in [agent evaluations](agent-evaluations.md). Preserve older state and separately remove any legacy background job when migrating.
-
-## State location
-
-Operational data lives under `.openmerit/` in the target product repository. It should be ignored by version control because it can contain local lifecycle state and evidence locations.
-
-- `config.json` — task profiles and policy.
-- `state.json` — current cycle, active or pending intent, last result, explicit observability coverage, processed signal IDs, task counters, latest aggregate baseline metric windows with evidence references, catalogue fingerprint, and assessment/swap checkpoints.
-- `events.jsonl` — append-only audit events.
-- `audit-export-errors.jsonl` — failures from optional external event exporters; these failures never replace or erase the canonical local event.
-- `evidence/*.json` — evidence-reference manifests.
-- `scheduler-status.json` — last headless wakeup time and exit code from the macOS background runner, when a wakeup actually ran.
-- `terminal.json` — optional source configuration and declared inventory for the Inference Terminal.
-- `inference.jsonl` — optional request-metadata export supplied by your existing tools for the terminal. The terminal reads this file; it does not instrument your application or create the feed.
-
-Snapshot writes are atomic. Files use owner-only permissions where POSIX modes are supported.
-
-The [Inference Terminal](inference-terminal.md) runs independently of the orchestration lifecycle. Its collector caches are memory-only and stop when the command exits. Permissions and retention for imported source files remain the responsibility of the tool that writes them.
-
-## Diagnostics
-
-Start with `/openmerit status`, which reports profile, metric and evaluation readiness, active work, next assessment, lifecycle, and pause state. `/openmerit doctor` confirms adapter loading without printing secrets.
-
-Pause is stored in `.openmerit/state.json`, blocks automatic due decisions, and unloads an OpenMerit-managed macOS LaunchAgent for that project. Resume restores scheduling only if the confirmed policy still permits persistent execution and infrastructure changes. Incoming signals can still be recorded while paused, and an already-active intent is not cancelled. `/openmerit doctor` shows the LaunchAgent and status-file paths. A LaunchAgent runs only while the Mac user is logged in; a headless run requires Pi credentials available without a shell-only environment variable.
-
-Inspect `scheduler-status.json` after a headless wakeup. A successful baseline-only check records a decision without starting Pi. A nonzero `exitCode` or `activeIntentId` means model-driven work did not complete; open the project in interactive Pi, inspect `/openmerit status` and `.openmerit/events.jsonl`, then use `/openmerit cancel` only if you intend to abandon that work. Model-driven runs have a 15-minute limit. The runner does not silently mark incomplete work successful.
-
-`/openmerit logs` prints the exact local paths to the audit log, exporter failures, evidence manifests, and current state. During an end-to-end run, `tail -f .openmerit/events.jsonl | jq .` shows signals and decisions as they occur.
-
-## Audit and recovery
-
-OpenMerit records initialization, intent requests and completions, automation signals and duplicates, due checks, task checkpoints, catalogue changes, frontier validation, confirmation boundaries, and lifecycle advancement.
-
-Automation-signal records include safe application measurement summaries: application model ID and application-reported token/cost/latency fields for completed tasks; or metric ID, state, scope, direction, method, value, unit, sample count, interval/window, and evidence IDs for metric windows. Pi's own model usage is not recorded as product-task evidence. Intent completions record their kind, status, next stage, harness, summary, and evidence IDs/sources. Secret-shaped keys and common API-key patterns are redacted before local persistence or export.
-
-The local JSONL file is the canonical source of truth. `ProjectStore` can also receive optional event exporters for OpenTelemetry or vendor-specific adapters. Export is best effort: a failed exporter is written to `audit-export-errors.jsonl` and does not interrupt lifecycle state or canonical logging.
-
-If Pi stops during an intent, durable state retains it. Inspect `state.json` and `events.jsonl` before manual changes. Deleting `.openmerit/` discards the audit trail, budget, profile, and evidence references.

package/docs/pareto-spec.md

@@ -1,76 +0,0 @@
-# OpenMerit Pareto conformance specification
-
-Status: initial normative specification
-
-OpenMerit owns this definition. A connected harness performs the calculation, returns a `FrontierSnapshot`, and supplies calculation evidence. OpenMerit verifies the returned result against these rules.
-
-## Comparable assessment set
-
-Every assessment in one frontier calculation must share:
-
-- task-profile ID and revision;
-- product revision;
-- harness identity and materially equivalent tool configuration;
-- objective definitions, units, and directions;
-- a representative evaluation workload and compatible evidence window.
-
-The harness must not silently compare measurements from incompatible conditions.
-
-## Metric readiness
-
-A required objective is ready only when:
-
-- a metric record exists;
-- its state is `measured`;
-- it has a numeric value;
-- its sample count meets the objective's minimum;
-- its unit and direction are compatible across candidates.
-
-Missing or immature evidence is `unresolved`. It is never converted to zero and never treated as success.
-
-## Feasibility
-
-Hard constraints are evaluated conservatively against uncertainty intervals. If an interval is absent, the point value is both bounds.
-
-- `at_least`: eligible only when the lower bound meets the threshold; ineligible when the upper bound is below it; otherwise unresolved.
-- `at_most`: eligible only when the upper bound meets the threshold; ineligible when the lower bound exceeds it; otherwise unresolved.
-
-Only eligible candidates may be certified as members of the frontier.
-
-## Dominance
-
-For candidate A to dominate candidate B, A must be proven no worse than B on every objective and proven materially better on at least one objective.
-
-For a maximize objective with tolerance `t`:
-
-- A is proven no worse when `A.lower + t >= B.upper`.
-- A is proven materially better when `A.lower > B.upper + t`.
-
-For a minimize objective with tolerance `t`:
-
-- A is proven no worse when `A.upper <= B.lower + t`.
-- A is proven materially better when `A.upper + t < B.lower`.
-
-If complete evidence proves A worse on any objective, A does not dominate B. If the evidence cannot establish no-worse or worse because intervals overlap, the comparison is unresolved.
-
-## Frontier
-
-An eligible candidate is on the certified frontier when:
-
-- no eligible candidate is proven to dominate it; and
-- none of its required pairwise comparisons is unresolved.
-
-Candidates with incomplete evidence, unresolved constraints, or unresolved comparisons are reported separately. A dominated, ineligible, or unresolved candidate must not appear on the certified frontier.
-
-## Required result evidence
-
-Every frontier result must include:
-
-- the assessment IDs used;
-- all directed pairwise comparisons between eligible candidates;
-- reasons for every comparison;
-- separate frontier, dominated, ineligible, and unresolved sets;
-- at least one durable calculation-evidence reference.
-
-The calculation evidence should identify the harness execution or artifact that produced the result.
-

package/docs/ux-reference.md

@@ -1,32 +0,0 @@
-# Pi UX behavioral reference
-
-This document is a **historical evaluation-adapter reference**, not the current Pi interface specification. The unreleased Pi 1.5.0 native Scout replacement does not carry forward the `/openmerit` commands, lifecycle, pause controls, or scheduling described here. See [the current Pi guide](pi-extension.md). The list below preserves earlier UX ideas for custom evaluation harnesses only.
-
-## Behaviors worth preserving
-
-- A single `/openmerit` command is the primary in-harness entry point.
-- Running the command without arguments gives a concise operational summary.
-- Long-running work produces short progress notifications instead of appearing silent.
-- Users can pause and resume proactive work without uninstalling the extension.
-- Users can request work explicitly even when proactive behavior is paused.
-- Diagnostics are available from the same command surface.
-- Skipped work explains why it was skipped and how the user can proceed.
-- Failures are visible, actionable, and do not masquerade as successful evidence.
-- The extension behaves sensibly in TUI, RPC, JSON, and print modes; prompts are not attempted when no interactive UI exists.
-
-## Staged behavior
-
-The command shell can carry a proposal through an authorized model change, but only after baseline evidence, budgeted challenger trials, a harness-calculated and OpenMerit-verified frontier, policy approval, and post-swap verification.
-
-## Current direction
-
-The command surface centers on evidence readiness. Its status view should answer:
-
-1. What task profile is active?
-2. Which required metrics are observable?
-3. Which evaluations exist and have been verified?
-4. What intelligent work is Pi currently performing for OpenMerit?
-5. What evidence is still missing or insufficient?
-6. When is the next assessment due?
-
-The approved command surface is documented in [Pi extension](pi-extension.md).