openmerit 0.1.4 → 0.1.6-preview.0

package/instructions/OPENMERIT.md

@@ -1,63 +0,0 @@
-# OpenMerit — Model Merit Harness
-
-You are running under a main agent harness that is observed by **OpenMerit**, an
-external merit harness. OpenMerit's job is to make sure you are always running on
-the best model for the task at hand, at the best price, with a vetted fallback.
-
-## What OpenMerit does
-
-1. **Observes** this harness's session traces (models used, tokens, cost,
-   latency, errors) without intercepting or slowing down your work.
-2. **Evaluates** candidate models for each completed task sequentially through
-   pi and scores their outputs with a judge model. Candidate tools are disabled
-   by default.
-3. **Maintains a pareto frontier** per task (quality vs. cost vs. latency) and
-   an aggregate frontier across all of your tasks.
-4. **Watches for new model releases** (provider catalogs + public benchmarks)
-   and queues promising releases for trial against your existing frontier.
-5. **Recommends or applies model swaps**, with reasoning, and keeps your
-   fallback model up to date.
-
-## What you should do
-
-- **Treat model changes as explicit routing decisions.** If you notice your
-  model identity change, it was approved by the user or passed their opt-in
-  auto-apply policy. Continue the task; your instructions and context are
-  unchanged.
-- **Respect the fallback.** If your current model errors, rate-limits, or is
-  retired, OpenMerit can offer the current fallback or apply it under an
-  opt-in automatic policy. Treat an approved fallback as normal operation.
-- **State your task clearly in your first message** of a session when possible.
-  OpenMerit keys its pareto frontier to task signatures; clear task statements
-  produce better model choices for you.
-- **Do not edit OpenMerit state files** (`~/.openmerit/`). If something looks
-  wrong, tell the user.
-
-## What you can ask the user for
-
-- `/openmerit` — show current model, fallback, frontier position, and pending
-  recommendations.
-- `/openmerit apply` / `/openmerit dismiss` — act on a pending recommendation
-  when automatic application is declined by the policy gate.
-- `/openmerit pause` / `/openmerit resume` — stop or restart automatic task
-  comparisons without changing the saved policy.
-- `/openmerit compare` — explicitly compare the latest completed task, even
-  while automatic comparisons are paused.
-- `/openmerit doctor` — inspect sanitized installation, route, pricing, state,
-  and provider-extension diagnostics inside Pi.
-- Policy changes (auto-apply thresholds, budgets, provider allow-lists) are
-  made by the user in `~/.openmerit/policy.json`, not by you.
-
-## Guarantees
-
-- Per-task comparisons send the task text, uploaded files or images (when
-  present), and candidate answers through the model routes configured in Pi
-  for model runs and judging. Candidate Pi runs default to no tools; any tool access is an
-  explicit user opt-in. Use non-sensitive examples while
-  evaluating this alpha.
-- The shipped policy is supervised. Swaps happen automatically only after the
-  user opts in and the configured quality/cost guardrails pass; otherwise they
-  remain recommendations for a human to approve.
-- Isolated subprocesses disable extension discovery. Provider-registration
-  extensions run only when the user lists their absolute file paths in
-  `policy.json`; OpenMerit refuses to load itself recursively.

package/instructions/openmerit.policy.json

@@ -1,37 +0,0 @@
-{
-  "version": 2,
-  "mode": "recommend",
-  "auto_apply": {
-    "enabled": false,
-    "min_score_gain": 0.1,
-    "max_price_ratio": 1.5,
-    "require_frontier": true
-  },
-  "budgets": {
-    "max_usd_per_trial": 0.25,
-    "max_trials_per_day": 20,
-    "max_usd_per_day": 5.0
-  },
-  "providers": {
-    "allow": ["*"],
-    "deny": []
-  },
-  "watch": {
-    "catalog_interval_min": 360,
-    "traces_interval_sec": 5,
-    "trial_interval_min": 30,
-    "models_per_task": 3
-  },
-  "fallback": {
-    "auto_update": true,
-    "min_score": 0.6,
-    "apply_on_error": true
-  },
-  "judge_model": null,
-  "strategist_model": null,
-  "max_usd_per_m": 20.0,
-  "route_overrides": {},
-  "pi": {
-    "provider_extensions": []
-  }
-}

package/rules.md

@@ -1,43 +0,0 @@
-# OpenMerit product and architecture rules
-
-- **Optimize for useful model choices.** OpenMerit exists to find better task-specific tradeoffs between quality, cost, and latency—not to become a general-purpose agent or telemetry platform.
-
-- **Act as a control plane, not a gateway.** Normal traffic stays between the harness and provider; OpenMerit observes evidence, runs explicit trials, and returns recommendations without intercepting ordinary work.
-
-- **Make Pi plug-and-play first.** Pi and Pi-based harnesses are the first supported user experience because that is where early users already work; broader harness support can follow without weakening this path.
-
-- **Keep the core harness-neutral.** Pi is the first `HarnessAdapter`, not a permanent assumption in scoring, recommendations, policy, or stored data, so other harnesses can be added without rewriting the merit loop.
-
-- **Keep model providers interchangeable.** OpenRouter is a first-class provider and discovery source, not the foundation of the domain model; provider-specific behavior belongs behind a `ModelProviderAdapter`.
-
-- **Require end-to-end provider neutrality.** Discovery, authentication, execution, scoring, and swapping must preserve the selected route; a feature is not provider-neutral if only its API client is abstracted.
-
-- **Separate model identity from execution route.** Keep logical model IDs in stable `vendor/model` form, while recording the actual provider or route separately, so the same model can be compared through OpenRouter, a native provider, or a local provider.
-
-- **Respect the harness's eligible model pool.** Prefer models already available and configured in the active harness; external catalogs may enrich or expand discovery but must not silently override harness scope or credentials.
-
-- **Require explicit metadata instead of price guesses.** Missing price, context, output, or modality data for a custom route may be supplied by an exact-route override; never assume a local route is free or copy metadata across providers.
-
-- **Treat public benchmarks as priors, not proof.** Benchmarks help shortlist candidates, but merit comes from trials on the user's actual task.
-
-- **Keep four integration boundaries distinct.** Model execution (`ModelProviderAdapter`), agent execution (`HarnessAdapter`), incoming traces (`ObservationSource`), and outgoing telemetry (`EventSink`) solve different problems and must not be coupled.
-
-- **Use provider-neutral core records.** Normalize integrations into stable concepts such as `TaskObservation`, `CandidateRun`, `TrialScore`, `Frontier`, `Recommendation`, and `MeritEvent`, so integrations do not leak their schemas into the decision engine.
-
-- **Optimize per task before aggregating per agent.** Agent-level conclusions must be built from measured task evidence rather than assumed from global model rankings.
-
-- **Version persisted events and evolve them additively.** Existing 0.1.x state must remain readable, and append-only recommendation history must stay intact; migrations should normalize old records rather than invalidate them.
-
-- **Keep policy as the sole auto-swap authority.** Trials and strategists may recommend changes, but only the policy gate may approve automatic application, and every recommendation must retain its gate reasons.
-
-- **Default to safe, explicit trials.** Candidate tool access stays off unless deliberately allowed, trials remain isolated, and temporary workspaces are cleaned up because trying a model must not expose or damage a user's project by surprise.
-
-- **Allowlist subprocess extensions.** Keep Pi extension discovery disabled in isolated trials and load only explicitly named provider-registration files; never recursively load OpenMerit or inherit unrelated harness extensions.
-
-- **Keep the shadow track isolated, not necessarily concurrent.** Candidate trials may run sequentially to respect cost, rate, and safety limits while remaining separate from the user's live task.
-
-- **Treat local JSONL as the default integration, not a lock-in.** Local traces and events should work without an external service; systems such as Langfuse can later plug in as observation sources or event sinks.
-
-- **Keep integrations optional and the core lightweight.** New providers, harnesses, and observability services should not impose credentials, network calls, or heavy dependencies on users who do not enable them.
-
-- **Keep 0.1.x focused.** The near-term bar is reliable, provider-neutral use for Pi tinkerers and solo hackers; additional harnesses and hosted observability integrations belong in later releases unless required to prove the boundaries work.