@moda-ai/cli 1.55.45 → 1.55.47

package/skills/moda-cli/SKILL.md

@@ -1,7 +1,7 @@
 ---
 name: moda-cli
-version: 3.0.10
-description: Investigate a production AI agent with the Moda CLI (`moda`) - find where something happened in real traces (traces search, then traces get), check agent health, rank what to fix (investigate, problems), and dig into tool failures, user frustration, and hallucinations. Also prompt registry sync/promote, replays, custom signals, and false-positive marking. Use when the user mentions Moda, production traces or conversations of their agent, agent failures, or asks what to fix.
+version: 3.0.11
+description: Investigate a production AI agent with the Moda CLI (`moda`) - find where something happened in real traces (traces search, then traces get), check agent health, rank what to fix (investigate, problems), and dig into tool failures, user frustration, and hallucinations. Also prompt registry sync/promote, harness registry push, custom signals, and false-positive marking. Use when the user mentions Moda, production traces or conversations of their agent, agent failures, or asks what to fix.
 ---
 
 # Moda CLI
@@ -81,7 +81,7 @@ Details: `references/output-protocol.md`.
 | A synthesized answer | `moda ask "<question>"` only when the user asks: each call is a paid agent run (up to about 200k tokens) | Exit 3 means degraded |
 | A detection is wrong | `moda review <kind> <target> --reason="..."` | `references/signals-and-false-positives.md` |
 | Moda data looks wrong or missing | `moda feedback "<what and why>" --conversation-id=<id>` | |
-| Prompts, replays, registry push | Only on explicit request | `references/prompts.md`, `references/replays-and-registry.md` |
+| Prompts, registry push | Only on explicit request | `references/prompts.md`, `references/registry.md` |
 
 Time windows differ by command: `--days-back=N` (1-90) on overview,
 investigate, problems, and detection commands (`detections list
@@ -97,10 +97,7 @@ names the closest values. `moda <noun> <verb> --help` shows which.
 2. `moda problems get <problem_id> --evidence` and open two or three anchors with
    `moda traces get <id> --msg-index=<n>` before deciding on a cause.
 3. Make the change in code; cite the problem id and trace ids in the commit or PR.
-4. Test a prompt change on the Problem's failing traces (a paid replay, so ask
-   first; prompt changes only today):
-   `moda prompts ab --problem=<problem_id> --baseline=<prompt_key> --candidate=<edited file>`.
-5. After it ships, and only if the user agrees:
+4. After it ships, and only if the user agrees:
    `moda problems feedback <problem_id> --action=mark_fixed`.
 More: `references/production-intelligence.md`.
 
@@ -109,9 +106,9 @@ More: `references/production-intelligence.md`.
 - Read evidence before acting on a summary; quote trace ids and message
   indexes when you report.
 - Never run writes (`problems feedback`, `review`,
-  `signals create|update|...`, `prompts sync|promote|ab`, `registry push`)
+  `signals create|update|...`, `prompts sync|promote`, `registry push`)
   unless the user asked. Preview with `--dry-run` (every write except
-  `prompts promote|ab|propose` takes it): it validates and applies nothing.
+  `prompts promote` takes it): it validates and applies nothing.
   Most writes print the exact request and a `next_commands` entry that runs
   it for real (see `references/output-protocol.md`).
 - An empty result in a window means "nothing in that window". Widen the window
@@ -156,6 +153,6 @@ Read only what the task needs:
 - `references/detections.md`: frustrations, emotions, tool failures, anchors and windows
 - `references/production-intelligence.md`: overview, investigate, ask, Problems and the fix loop
 - `references/signals-and-false-positives.md`: custom signals and false-positive marks
-- `references/prompts.md`, `references/replays-and-registry.md`: prompt registry, replays, registry push
+- `references/prompts.md`, `references/registry.md`: prompt registry, registry push
 - `references/dashboard-parity.md`, `references/recipes.md`, `references/feedback.md`
 - `references/troubleshooting.md`, `references/commands.md` (command index; `--help` is authoritative)

package/skills/moda-cli/references/commands.md

@@ -62,8 +62,7 @@ Loaded on demand from the moda-cli skill. This table can lag the CLI: `moda <com
 | `--cohort=<ref>` | Scope `problems list`, `traces list`, `detections list` (emotion/frustration/hallucination per trace, laziness, custom_signal hits), `detections tools` and `clusters` to one cohort (`builtin:<key>` or a saved id from `moda cohorts list`). The summary names the cohort; a warning means the server did not confirm the scope |
 | `moda problems reopen <pid>` | Reopen a Problem marked fixed |
 | `moda signals <create\|update\|activate\|pause\|delete\|examples\|refine\|revise\|proposals\|accept\|reject\|search\|preview>` | Manage custom signals (writes take `--dry-run`; LLM-backed ones are rate-limited per key) |
-| `moda prompts list\|show\|proposals\|usage\|replay-runs\|decide` | Read the prompt registry; merge/reject proposals |
-| `moda replays sets\|runs\|run\|playouts\|playout` | Replay results: run history, per-seed verdicts, playout transcripts with tool provenance (read-only) |
+| `moda prompts list\|show\|proposals\|usage\|decide` | Read the prompt registry; merge/reject proposals |
 | `moda skills list\|show\|policy` / `moda harness list\|show\|versions\|diff` | Skill library and synced harnesses |
 
 `moda traces search` flags: `--mode=keyword|semantic|hybrid` (default `hybrid`),

package/skills/moda-cli/references/dashboard-parity.md

@@ -5,7 +5,7 @@ Loaded on demand from the moda-cli skill.
 ### 10. Everything the dashboard does, from the CLI
 
 Every command below needs only an API key, and the tenant comes from the key.
-Writes listed here accept `--dry-run`: it validates the call, prints the exact request (method, endpoint, body) and sends nothing, and its `next_commands` entry is the command that sends it for real (`mutability: write`, `requires_approval: true`). That covers `problems feedback|reopen|link`, `review`, `feedback`, `cohorts create|update` and the `signals` writes. `prompts promote|ab|propose` refuse `--dry-run` before sending anything, so run them only when you mean it.
+Writes listed here accept `--dry-run`: it validates the call, prints the exact request (method, endpoint, body) and sends nothing, and its `next_commands` entry is the command that sends it for real (`mutability: write`, `requires_approval: true`). That covers `problems feedback|reopen|link`, `review`, `feedback`, `cohorts create|update` and the `signals` writes. `prompts promote` refuses `--dry-run` before sending anything, so run it only when you mean it.
 
 | Dashboard | CLI |
 |---|---|
@@ -29,8 +29,7 @@ Writes listed here accept `--dry-run`: it validates the call, prints the exact r
 | Custom signals: edit, lifecycle | `moda signals update <id> …`, `moda signals activate\|pause\|delete <id>` (delete: drafts only; archiving is dashboard-only) |
 | Custom signals: review & tune | `moda signals examples <id> add\|update\|remove --conversation-id=…`, `refine <id> --request="…"`, `revise <id> --conversation-ids=…`, `proposals <id>`, `accept\|reject <id> <proposal_id>` |
 | Custom signals: find matches | `moda signals search <id>`, `search-status <id> <search_id>`, `search-stop <id> <search_id>`, `preview --file=detector.json` |
-| Replays: run detail, playout transcripts | `moda replays runs <set_id>`, `run <set_id> <run_id>`, `playouts <set_id> <run_id> [--case --arm --seed --transcript]`, `playout <set_id> <run_id> <playout_id>` |
-| Prompts registry | `moda prompts list [--label --search]`, `show <key>`, `proposals <key>`, `usage <key>`, `replay-runs <key>`, `decide <key> <proposal_id> --action=merge\|reject\|reopen` |
+| Prompts registry | `moda prompts list [--label --search]`, `show <key>`, `proposals <key>`, `usage <key>`, `decide <key> <proposal_id> --action=merge\|reject\|reopen` |
 | Skills library | `moda skills list`, `show <key>`, `policy <key> --live\|--local` |
 | Harness pages | `moda harness list`, `show <id>`, `versions <id>`, `diff <id> --from=… --to=…` |
 

package/skills/moda-cli/references/output-protocol.md

@@ -220,7 +220,7 @@ Treat an answer as degraded if **any** of these hold:
   instead of ignoring it. Free text joins every word: `moda traces search
   refund not processed` searches "refund not processed" (same for `ask` and
   `feedback`).
-- **Writes take `--dry-run`** (`prompts promote|ab|propose` excepted).
+- **Writes take `--dry-run`** (`prompts promote` excepted).
   - `problems feedback|reopen|link`, `review`, `feedback`, `signals` writes
     and `skills policy` validate fully, send nothing, and return
     `data.planned` (method, endpoint, body) plus a `next_commands` entry

package/skills/moda-cli/references/production-intelligence.md

@@ -141,9 +141,5 @@ Rules that keep the fix grounded:
   not on the Problem's title alone.
 - Cite the Moda problem id and the trace ids you relied on in the PR
   description, so a reviewer can open the same evidence.
-- To test a prompt change against real traffic before shipping it, A/B it with
-  `moda prompts ab --baseline=<path> --candidate=<path> --problem=<problem_id>`
-  (replays the Problem's failing traces from their evidence anchors) or
-  `--traces=<ids>` (see section 7).
 - Mark the Problem fixed only after the change ships. If the issue is detected
   again, the Problem comes back; `moda problems reopen` reopens it by hand.

package/skills/moda-cli/references/prompts.md

@@ -61,33 +61,7 @@ sources:                      # prompts defined in code or config, mapped in pla
 To attribute runtime calls, stamp the lock entry `{ key, promptId, versionId }`
 on telemetry. With Cloudflare Think / AI SDK 7, pass
 `prompt: lock.prompts['<key>']` to `createThinkTelemetry` (`moda-ai/think`) or
-`createAI7Telemetry` (`moda-ai/ai7`). A `moda fix` that lands `UNROUTED` with
-"top share 0%" means the evidence carries no prompt version: wire this up,
-then check `moda prompts status`.
-
-`prompts ab` and `prompts propose --gate` run paid replays. Before sending
-anything they compute planned playouts (cases × seeds × 2 arms) and print it
-(`plannedPlayouts`). Above 200, or when an existing set's case count can't be
-read, they fail (exit 1) with the number and the exact command to re-run with `--yes`.
-Only add `--yes` when the user has agreed to that spend. Limits: `--cases`
-default 5, max 100; `--seeds` default 3, max 10; `--traces` at most 100
-(`--traces=conv_id@412` replays that trace from msg_index 412, with the
-earlier turns as history, and writes its case from the messages around 412);
-`--problem=<problem_id>` picks a Problem's affected traces, each anchored at
-its earliest evidence msg_index, up to
-`--max-cases` (default 10, max 100), and prints them (`problemCases`);
-`--model` / `--assistant-model` must be one of `openai/gpt-5.6-luna`,
-`openai/gpt-4o-mini`, `openai/gpt-4o`, `anthropic/claude-sonnet-4-5`,
-`anthropic/claude-haiku-4-5`, `google/gemini-2.5-flash`,
-`google/gemini-2.5-pro` unless you pass `--allow-any-model`.
-
-`prompts ab --traces=<ids>` builds one case per trace from Moda's distilled
-scenario and success criteria for that trace. A trace Moda cannot distill
-falls back to its opening user message (for `id@N`, the user message leading
-into N) with three generic criteria (which long
-traces pass easily); the run warns and lists it under `caseSources.fallback`.
-`prompts ab` never makes its replay set the tenant's primary set unless you
-pass `--promote`.
+`createAI7Telemetry` (`moda-ai/ai7`).
 
 Runtime code should render synced prompts through the SDK:
 

package/skills/moda-cli/references/{replays-and-registry.md → registry.md}

@@ -1,53 +1,15 @@
-# Replays and the harness registry
+# Harness registry
 
 Loaded on demand from the moda-cli skill.
 
-### 7a. Check a replay verdict (`moda replays`)
-
-Use after `prompts ab`, `prompts propose --gate`, or a dashboard replay when
-you need to see *why* an arm passed or failed. Read-only; no replay spend.
-
-```bash
-moda replays sets --json                              # set ids
-moda replays runs <set_id> --limit=5 --json           # newest runs, every status
-moda replays run <set_id> <run_id> --json             # per-case + per-seed verdicts
-moda replays playouts <set_id> <run_id> --json        # one row per case x arm x seed
-moda replays playouts <set_id> <run_id> --case=<case_id> --arm=proposed --seed=1 --transcript --json
-moda replays playout <set_id> <run_id> <playout_id> --json
-```
-
-`run` returns `results[]` (one per case × arm × seed: `pass`, `abstained`,
-`coverageReason`, `score`, `rationale`, `criteria[]`) next to the per-case
-majority in `cases[]`. `arm=prod` is the baseline and `arm=proposed` the
-candidate. An `abstained: true` row is not a verdict: the pair hit a replay
-coverage hole and is excluded from pass counts.
-
-A playout's `transcript.events[]` holds the simulated user and agent messages
-and every tool call with its `input`, the `output` the replay served, and
-`provenance.source`: `replay_runtime` (compiled mock over the case's recorded
-source state), `agent_tool_simulator` (LLM simulator grounded in recorded
-receipts; `simulator.model`), `counterfactual_reconstruction`, or
-`generic_handler`. `provenance.errorSubtype` such as `missing_source_state`
-marks an unservable call. `judge` is the judgement stored with the playout and
-`verdict` the stored eval row (null when `seed` is null, which happens on
-older runs whose graphs cannot be matched to a seed).
-
-Limits: `--transcript` pages at most 10 playouts (`--limit` default 50, max
-200 without it); text and tool JSON are clipped at `--max-chars` (default
-4000, max 32000) and `transcript.truncated.fields` counts the clipped fields.
-Page with `--cursor=<nextCursor>`. Playouts exist only for runs whose
-playouts were captured; `playoutCount: 0` on `run` means there is no
-transcript to read.
-
-### 7b. Register the harness for replays (`moda registry`)
+### 7a. Register the harness (`moda registry`)
 
 Use when the user wants their prompts, skills, tools, or agent wiring in Moda
-so replays rebuild their harness faithfully, without the GitHub App or `moda
-harness analyze`. **You** are the analyst: read the codebase you already have,
-write one manifest, and push exactly what it lists. Nothing else in the repo is
-uploaded. For a full guided pass (inventory, prompt switching, context,
-attribution, and verification), run `moda registry prompt` and follow the
-brief it prints.
+without the GitHub App or `moda harness analyze`. **You** are the analyst: read
+the codebase you already have, write one manifest, and push exactly what it
+lists. Nothing else in the repo is uploaded. For a full guided pass (inventory,
+prompt switching, context, attribution, and verification), run `moda registry
+prompt` and follow the brief it prints.
 
 ```bash
 moda registry prompt                                # the step-by-step brief for this job
@@ -113,8 +75,7 @@ tools still works; every section optional, ≤500 entries in prompts, skills, an
 - **Skills:** `key` (letters, digits, `_`, `-`) plus `skillMd` or `file` (full
   SKILL.md). Pushed skills are *local* by default. Set `"live": true`, or pass
   `--live`, to include them in the served skill surface. Leaving `live` out
-  never demotes a skill that is already live. Replays load the skills an agent
-  lists in `agents[].skills`, whether or not they are live.
+  never demotes a skill that is already live.
 - **Tools:** an OpenAI function `schema` (a wrapped
   `{"type":"function","function":{...}}` is accepted), plus optional
   `isActive`. Copy names and parameter schemas exactly as the agent registers
@@ -130,16 +91,11 @@ tools still works; every section optional, ≤500 entries in prompts, skills, an
   `promptKeys`).
 - **Routing (prompt switching):** `strategy` is `single`, `handoff`, `router`,
   `rules`, or `trace_attribute`; `default` is an agent key; put code-level
-  rules in `notes`. Replay runs the agent named on the replayed turn's trace
-  (agent name, then prompt key), else `default`, else the `entry` agent. When
-  the replayed model calls a handoff tool, replay switches to the target
-  agent's prompt, tools, model, and params.
+  rules in `notes`.
 - **Context:** what the runtime injects. `source` is `static` (with `value`),
-  `clock` (`format`: date, datetime, or iso; replay uses the source
-  conversation's time), or `trace`, `memory`, `retrieval`, `tool`, or
-  `unknown`. Those last ones are reported as unresolved in replay fidelity,
-  never invented. `target` is `prompt_variable` (needs `variable`) or
-  `system_append`.
+  `clock` (`format`: date, datetime, or iso), or `trace`, `memory`,
+  `retrieval`, `tool`, or `unknown`. `target` is `prompt_variable` (needs
+  `variable`) or `system_append`.
 - **Runtime:** `framework`, `language`, `maxToolRoundsPerTurn`,
   `parallelToolCalls`, `history` (`strategy`, `maxMessages`, `maxTokens`),
   and `skillLoading` (`tool`, `system_prompt`, or `none`).
@@ -148,9 +104,9 @@ tools still works; every section optional, ≤500 entries in prompts, skills, an
 - `file` paths are relative to the directory you run the command from and must
   stay inside it (symlinks included).
 
-**Traces must name the agent.** For per-turn prompt switching, each LLM call
-should carry `gen_ai.agent.name` (or `moda.agent_name`; with the Vercel AI SDK
-use `experimental_telemetry.metadata["moda.agent_name"]`; `agent_name` on
+**Traces must name the agent.** Each LLM call should carry `gen_ai.agent.name`
+(or `moda.agent_name`; with the Vercel AI SDK use
+`experimental_telemetry.metadata["moda.agent_name"]`; `agent_name` on
 `/v1/ingest`), set to an agent `key` or one of its `match.agentNames`. It
 should also carry `moda.prompt_key` / `moda.prompt_version_id`. Only add these
 to application code when the user asks for it or approves the diff.

package/dist/cli-4pc0a18x.js

@@ -1,12 +0,0 @@
-import {
-  callControlAPI
-} from "./cli-dmm9jm9m.js";
-
-// src/replay-runs.ts
-var DEFAULT_POLL_INTERVAL_MS = 15000;
-var DEFAULT_WAIT_TIMEOUT_MS = 2 * 60 * 60 * 1000;
-async function fetchLatestRun(tenantId, replaySetId, runId, profileOptions) {
-  return callControlAPI(`/tenants/${encodeURIComponent(tenantId)}/replay-sets/${encodeURIComponent(replaySetId)}/runs/latest?runId=${encodeURIComponent(runId)}`, { method: "GET" }, profileOptions, { timeoutMs: 60000, retries: 2 });
-}
-
-export { DEFAULT_POLL_INTERVAL_MS, DEFAULT_WAIT_TIMEOUT_MS, fetchLatestRun };