@moda-ai/cli 1.55.19 → 1.55.21

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -0,0 +1,173 @@
1
+ # Replays and the harness registry
2
+
3
+ Loaded on demand from the moda-cli skill.
4
+
5
+ ### 7a. Check a replay verdict (`moda replays`)
6
+
7
+ Use after `prompts ab`, `prompts propose --gate`, or a dashboard replay when
8
+ you need to see *why* an arm passed or failed. Read-only; no replay spend.
9
+
10
+ ```bash
11
+ moda replays sets --json # set ids
12
+ moda replays runs <set_id> --limit=5 --json # newest runs, every status
13
+ moda replays run <set_id> <run_id> --json # per-case + per-seed verdicts
14
+ moda replays playouts <set_id> <run_id> --json # one row per case x arm x seed
15
+ moda replays playouts <set_id> <run_id> --case=<case_id> --arm=proposed --seed=1 --transcript --json
16
+ moda replays playout <set_id> <run_id> <playout_id> --json
17
+ ```
18
+
19
+ `run` returns `results[]` (one per case × arm × seed: `pass`, `abstained`,
20
+ `coverageReason`, `score`, `rationale`, `criteria[]`) next to the per-case
21
+ majority in `cases[]`. `arm=prod` is the baseline and `arm=proposed` the
22
+ candidate. An `abstained: true` row is not a verdict: the pair hit a replay
23
+ coverage hole and is excluded from pass counts.
24
+
25
+ A playout's `transcript.events[]` holds the simulated user and agent messages
26
+ and every tool call with its `input`, the `output` the replay served, and
27
+ `provenance.source`: `replay_runtime` (compiled mock over the case's recorded
28
+ source state), `agent_tool_simulator` (LLM simulator grounded in recorded
29
+ receipts; `simulator.model`), `counterfactual_reconstruction`, or
30
+ `generic_handler`. `provenance.errorSubtype` such as `missing_source_state`
31
+ marks an unservable call. `judge` is the judgement stored with the playout and
32
+ `verdict` the stored eval row (null when `seed` is null, which happens on
33
+ older runs whose graphs cannot be matched to a seed).
34
+
35
+ Limits: `--transcript` pages at most 10 playouts (`--limit` default 50, max
36
+ 200 without it); text and tool JSON are clipped at `--max-chars` (default
37
+ 4000, max 32000) and `transcript.truncated.fields` counts the clipped fields.
38
+ Page with `--cursor=<nextCursor>`. Playouts exist only for runs whose
39
+ playouts were captured; `playoutCount: 0` on `run` means there is no
40
+ transcript to read.
41
+
42
+ ### 7b. Register the harness for replays (`moda registry`)
43
+
44
+ Use when the user wants their prompts, skills, tools, or agent wiring in Moda
45
+ so replays rebuild their harness faithfully, without the GitHub App or `moda
46
+ harness analyze`. **You** are the analyst: read the codebase you already have,
47
+ write one manifest, and push exactly what it lists. Nothing else in the repo is
48
+ uploaded. For a full guided pass (inventory, prompt switching, context,
49
+ attribution, and verification), run `moda registry prompt` and follow the
50
+ brief it prints.
51
+
52
+ ```bash
53
+ moda registry prompt # the step-by-step brief for this job
54
+ moda registry template --out=.moda/registry.json # starter manifest (moda_registry.v2)
55
+ # ...fill in .moda/registry.json from the codebase...
56
+ moda registry validate # local only, no network
57
+ moda registry push --dry-run # server-side preview, writes nothing
58
+ moda registry push # upload
59
+ moda registry status # what is registered (harness version)
60
+ moda registry coverage --days=7 # production traces vs the registry
61
+ moda registry log # history: one commit per push
62
+ moda registry show <commit|working> # a commit's file diffs / changes since the last push
63
+ ```
64
+
65
+ Every successful `push` records a registry **commit**: the whole registry
66
+ (prompts, tools, skills, agents, harness settings) snapshotted as files, with
67
+ your repo's git SHA, branch, and dirty flag. Pass `--message="..."` to label
68
+ it. The dashboard **Registry** page shows these commits git-style: a log,
69
+ per-file diffs, a file browser, and uncommitted changes made outside a push
70
+ (dashboard edits, MCP discovery, `moda prompts sync`).
71
+
72
+ Manifest (`moda_registry.v2`; `moda_registry.v1` with only prompts, skills, and
73
+ tools still works; every section optional, ≤500 entries in prompts, skills, and tools):
74
+
75
+ ```json
76
+ {
77
+ "schema": "moda_registry.v2",
78
+ "prompts": [
79
+ { "key": "support.triage", "file": "prompts/triage.md" },
80
+ { "key": "support.billing", "content": "You handle billing for {{company_name}}..." }
81
+ ],
82
+ "skills": [
83
+ { "key": "refund-policy", "file": ".claude/skills/refund-policy/SKILL.md" }
84
+ ],
85
+ "tools": [
86
+ { "schema": { "name": "lookup_customer", "description": "...",
87
+ "parameters": { "type": "object", "properties": { "email": { "type": "string" } } } } }
88
+ ],
89
+ "runtime": { "framework": "openai-agents", "maxToolRoundsPerTurn": 6, "skillLoading": "tool" },
90
+ "agents": [
91
+ { "key": "triage", "entry": true, "prompt": "support.triage",
92
+ "model": { "id": "openai/gpt-5.1", "temperature": 0.2, "maxTokens": 1024 },
93
+ "tools": ["lookup_customer"],
94
+ "handoffs": [{ "to": "billing", "tool": "transfer_to_billing" }],
95
+ "match": { "agentNames": ["TriageAgent"] } },
96
+ { "key": "billing", "prompt": "support.billing", "skills": ["refund-policy"] }
97
+ ],
98
+ "routing": { "strategy": "handoff", "default": "triage" },
99
+ "context": [
100
+ { "key": "company", "variable": "company_name", "source": "static", "value": "Acme" },
101
+ { "key": "today", "variable": "current_date", "source": "clock", "format": "date" }
102
+ ]
103
+ }
104
+ ```
105
+
106
+ - **Prompts:** `key` plus one of `content`, `file`, `systemPrompt`, or
107
+ `messages`. Optional `name`, `description`, `category`, `variables`,
108
+ `modelConfig`, `toolRefs`, `responseSchema`, `sourcePath`. Use the key the
109
+ runtime passes to `Moda.prompt(key)` when there is one. Copy each prompt's
110
+ text **verbatim**: for templated prompts, keep the template string, not a
111
+ rendered example. Re-pushing the same text is a no-op; changed text adds a
112
+ new version.
113
+ - **Skills:** `key` (letters, digits, `_`, `-`) plus `skillMd` or `file` (full
114
+ SKILL.md). Pushed skills are *local* by default. Set `"live": true`, or pass
115
+ `--live`, to include them in the served skill surface. Leaving `live` out
116
+ never demotes a skill that is already live. Replays load the skills an agent
117
+ lists in `agents[].skills`, whether or not they are live.
118
+ - **Tools:** an OpenAI function `schema` (a wrapped
119
+ `{"type":"function","function":{...}}` is accepted), plus optional
120
+ `isActive`. Copy names and parameter schemas exactly as the agent registers
121
+ them with the model. Tools are upserted by name and never overwrite
122
+ MCP-server-managed tools (those report `skipped`).
123
+ - **Agents:** one per distinct prompt, tool set, or model. Fields: `key`,
124
+ `name`, `description`, `entry`, `prompt` (prompt key), `model` (`id`,
125
+ `provider`, `temperature`, `maxTokens`, `topP`, `reasoningEffort`), `tools`
126
+ (tool names), `skills` (skill keys), `handoffs` (`to`, `tool`,
127
+ `description`), `subagents` (`agent`, `tool`; the tool must also be in
128
+ `tools`), `context` (context keys), `maxToolRoundsPerTurn`,
129
+ `responseSchema`, `match` (`agentNames` as they appear in traces,
130
+ `promptKeys`).
131
+ - **Routing (prompt switching):** `strategy` is `single`, `handoff`, `router`,
132
+ `rules`, or `trace_attribute`; `default` is an agent key; put code-level
133
+ rules in `notes`. Replay runs the agent named on the replayed turn's trace
134
+ (agent name, then prompt key), else `default`, else the `entry` agent. When
135
+ the replayed model calls a handoff tool, replay switches to the target
136
+ agent's prompt, tools, model, and params.
137
+ - **Context:** what the runtime injects. `source` is `static` (with `value`),
138
+ `clock` (`format`: date, datetime, or iso; replay uses the source
139
+ conversation's time), or `trace`, `memory`, `retrieval`, `tool`, or
140
+ `unknown`. Those last ones are reported as unresolved in replay fidelity,
141
+ never invented. `target` is `prompt_variable` (needs `variable`) or
142
+ `system_append`.
143
+ - **Runtime:** `framework`, `language`, `maxToolRoundsPerTurn`,
144
+ `parallelToolCalls`, `history` (`strategy`, `maxMessages`, `maxTokens`),
145
+ and `skillLoading` (`tool`, `system_prompt`, or `none`).
146
+ - **MCP servers:** `name`, `transport`, `url` or `command`, `tools`, and
147
+ `description`, with no credentials. List their tools in `tools[]` too.
148
+ - `file` paths are relative to the directory you run the command from and must
149
+ stay inside it (symlinks included).
150
+
151
+ **Traces must name the agent.** For per-turn prompt switching, each LLM call
152
+ should carry `gen_ai.agent.name` (or `moda.agent_name`; with the Vercel AI SDK
153
+ use `experimental_telemetry.metadata["moda.agent_name"]`; `agent_name` on
154
+ `/v1/ingest`), set to an agent `key` or one of its `match.agentNames`. It
155
+ should also carry `moda.prompt_key` / `moda.prompt_version_id`. Only add these
156
+ to application code when the user asks for it or approves the diff.
157
+ `moda registry coverage` shows how many recent calls resolve to a registered
158
+ agent, which agent names, prompt keys, and tools are unregistered, and which
159
+ agents were never seen.
160
+
161
+ `push` needs an API key. `push --dry-run` without one only validates locally
162
+ (status `validated`). Each section goes to its own endpoint and reports
163
+ `sections[].status` (`synced`, `dry_run`, `validated`, `skipped`, or `error`).
164
+ The harness sections (`runtime`, `agents`, `routing`, `context`, and
165
+ `mcpServers`) are pushed last, as one versioned document. Re-pushing an
166
+ unchanged one reports `unchanged`. If any section fails, `push` exits `1`,
167
+ lists the failures in `errors[]`, and still pushes the other sections. The
168
+ whole manifest is validated before any request, with every error listed at
169
+ once. `warnings[]` names agent references to prompts, tools, or skills that
170
+ the manifest doesn't define; they must already exist in Moda. Unknown fields
171
+ and wrong types are rejected, matching what the server accepts. Skill keys are
172
+ stricter than the server (letters, digits, `_`, `-`) because they double as
173
+ `.claude/skills/<key>/` directory names.
@@ -0,0 +1,180 @@
1
+ # Search, traces, context, and audit
2
+
3
+ Loaded on demand from the moda-cli skill.
4
+
5
+ ### 1. Search messages (primary content-discovery path)
6
+
7
+ ```bash
8
+ moda search "refund flow timeout" # hybrid (default)
9
+ moda search "customer asked for a refund" --mode=semantic
10
+ moda search "SEARCH_NO_RESULTS" --mode=keyword
11
+ moda search "checkout error" --time-range=7d --limit=10
12
+ moda search "stripe.charges.create failed" --user-id=<id>
13
+ moda search "Иван" --conversation-id=<id> --mode=keyword # inside one trace
14
+ ```
15
+
16
+ Searches trace messages at message grain. Pass `--include-tool-io` to
17
+ also search **tool calls and tool results** (tool name, input arguments, and
18
+ output previews), so it finds where an agent *did* something, not just where
19
+ it talked about it — tool-IO search is opt-in, not the default.
20
+
21
+ Flags: `--mode=keyword|semantic|hybrid` (default `hybrid`), `--user-id`,
22
+ `--conversation-id` (search inside one trace; no trace fallback),
23
+ `--time-range` (`all|1h|3d|7d|24h|30d|90d`, default `all`), `--limit` (1–100,
24
+ default 20). The query is a positional arg (max 500 chars).
25
+
26
+ **Index freshness.** Keyword matches (and hybrid's keyword half) fall back to
27
+ the original messages when the search index returns too few literal matches,
28
+ so recent, Cyrillic and partial-word text is still found. `coverage` reports
29
+ which source served the keyword leg (`keyword_source`) and how far the index
30
+ trails ingestion (`index_freshness`); a warning says so when your
31
+ `--time-range` reaches past the newest indexed message.
32
+
33
+ **Fallback.** If message-level search finds nothing but whole traces match
34
+ (same as `moda traces --search`), those traces come back in `results` with
35
+ `match_grain: "trace"`, plus `fallback.source: "traces_search"` and a warning.
36
+ These rows have no score or message index. Open one with
37
+ `moda audit <conversation_id>` or `moda context <conversation_id>` to find the
38
+ matching turn. When more traces exist, `fallback.has_more` is true and
39
+ `next_commands` includes the `moda traces --search ... --offset` command to
40
+ page. With `--mode=keyword`, ranked traces are not substituted. If the fallback
41
+ itself fails, you get a warning, not a silent empty result. Either way, zero
42
+ results does not prove the topic never came up.
43
+
44
+ Choosing a mode:
45
+
46
+ - `hybrid` (default) — best general recall; fuses keyword + semantic. Use it
47
+ unless you have a reason not to.
48
+ - `keyword` — exact substring/identifier matches (error codes, tool names,
49
+ IDs). `score` is the integer match count.
50
+ - `semantic` — meaning-based; finds paraphrases the user never typed
51
+ verbatim. `score` is cosine similarity.
52
+
53
+ Each result row carries `conversation_id`, `message_index`, `role`,
54
+ `timestamp`, `snippet`, `score`, and `cluster_id` / `cluster_name`. The
55
+ response also reports `search_mode` — the mode that **actually ran**; a
56
+ `semantic`/`hybrid` request transparently degrades to `keyword` when no
57
+ embedding provider is reachable, so check it before trusting semantic ranking.
58
+ `score` scales differ per mode (see above) — branch on `search_mode`, never
59
+ compare scores across modes.
60
+
61
+ Hand the anchors straight to `moda context`:
62
+
63
+ ```bash
64
+ # Find, then read the exact turn. --json prints the bare payload; piped output
65
+ # without it is the agent envelope, where results sit under .data.results.
66
+ hit=$(moda search "double charge" --limit=1 --json | jq -c '.results[0] // empty')
67
+ cid=$(echo "$hit" | jq -r '.conversation_id')
68
+ # Trace-level fallback hits (match_grain: "trace") have no message_index.
69
+ idx=$(echo "$hit" | jq -r '.message_index // empty')
70
+ moda context "$cid" ${idx:+--msg-index="$idx"}
71
+ ```
72
+
73
+ ### 3. Browse topic clusters
74
+
75
+ ```bash
76
+ moda clusters # root-level
77
+ moda clusters --parent-id=<node_id> # drill in
78
+ moda cluster-traces <node_id> # traces in cluster
79
+ ```
80
+
81
+ ### 3b. List / filter traces (structured, not semantic)
82
+
83
+ Use `moda traces` to enumerate or filter by structured fields — not to
84
+ search by meaning (use `moda search` for that). The `--search` flag here
85
+ ranks whole traces: hybrid (keyword + semantic) when embeddings exist, else
86
+ keyword. Check `total_kind`: `candidate_pool` means results are related traces
87
+ that may not contain the words and `pagination.total` is not a match count;
88
+ `matches` means a literal keyword match set.
89
+
90
+ ```bash
91
+ moda traces --user-id=<id> --time-range=7d
92
+ moda traces --cluster-id=<node_id> --environment=production
93
+ moda traces --search="timeout" --limit=20 # ranked trace search
94
+ ```
95
+
96
+ Filters: `--search`, `--cluster-id`, `--user-id`, `--time-range`
97
+ (`all|1h|3d|7d|24h|30d|90d`), `--environment`
98
+ (`all|development|staging|production`), `--outcome`, `--limit`, `--offset`.
99
+
100
+ ### 4. Read a single trace
101
+
102
+ ```bash
103
+ moda context <conversation_id> # default window around middle
104
+ moda context <conversation_id> --msg-index=5 # center on message 5
105
+ moda context <conversation_id> --window=3 # 3 messages each side (max 5)
106
+ moda context <conversation_id> --msg-index=40 --window=0 # just message 40 + total_messages
107
+ moda context <conversation_id> --metadata-only # summary + total_messages, no messages
108
+ moda context <conversation_id> --all # whole trace in order, first 500 messages
109
+ moda context <conversation_id> --all --from=500 --max-messages=500 # continue (max 5000)
110
+ ```
111
+
112
+ `--all` returns `has_more`/`next_from`; when it stops at the cap, the warning
113
+ prints the exact `--from` command to continue.
114
+
115
+ **Several traces? Batch them in one call instead of looping `moda context`
116
+ per id.** Pass up to 25 targets as positionals or one per line with
117
+ `--ids-from=-` (stdin) or `--ids-from=<file>`. A target is `<id>` or
118
+ `<id>@<msg_index>` (the anchor; ids may contain `:`), and the shared flags
119
+ (`--window`, `--metadata-only`, `--all`) apply to every target. Lines may also
120
+ be JSON (`{"id":"...","msg_index":N}`), the only spelling for an id that itself
121
+ ends in `@<digits>`.
122
+
123
+ ```bash
124
+ moda context telegram:8565459080@12 linq:c6f88988-aa@3 web:conv-9 --window=2
125
+ printf '%s\n' telegram:8565459080@12 web:conv-9 | moda context --ids-from=- --window=1
126
+ ```
127
+
128
+ A batch returns `data.results: [{ target, ok, data | error }]` in input order,
129
+ plus `succeeded`/`failed` counts. A missing trace is a per-item
130
+ `error.code: "not_found"`, not a failed call; the exit code is 1 only when
131
+ every target failed. When the agent byte budget would overflow, the items that
132
+ fit are returned and `remaining_targets` plus a `next_commands` entry read the
133
+ rest. One positional id keeps the single-trace output above.
134
+
135
+ `moda context` returns messages whole, with their tool calls and results. An
136
+ older Moda server cuts message text, tool inputs, and tool results at 500
137
+ characters; the response warns when that server is in use.
138
+ `--window` is at most 5 messages per side. For the raw,
139
+ complete picture of what was captured — every span, the parent/child hierarchy,
140
+ tool calls, prompt/response bodies, and `gen_ai.*` attributes — use `moda audit`.
141
+
142
+ ### 4b. Audit raw spans (completeness, orphans, duplicates)
143
+
144
+ ```bash
145
+ moda audit <conversation_id|trace_id> # auto-detects OTLP trace_id vs trace ID (conversation_id)
146
+ moda audit <trace_id> --kind=trace # force OTLP trace_id lookup
147
+ moda audit <conversation_id> --include-raw # attach verbatim raw_event bodies
148
+ moda audit <conversation_id> --include-bodies # keep prompt/response/attribute bodies
149
+ moda audit <conversation_id> --limit=1000 # ingest rows to read (default 200, max 1000)
150
+ moda audit <conversation_id> --cursor=<next_cursor> # next page of a capped audit
151
+ moda audit <id1> <id2> <id3> # batch: up to 25 ids, one envelope
152
+ ```
153
+
154
+ Auditing several traces? Batch the ids (positionals or `--ids-from=-|<file>`)
155
+ instead of looping `moda audit`: the result is `data.results: [{ target, ok,
156
+ data | error }]` with the same per-item errors and byte-budget rules as batch
157
+ `moda context`. Ids are taken verbatim (no `@` anchors); `--cursor` pages one
158
+ trace, so follow each item's `next_commands` for it.
159
+
160
+ `truncated` / `has_more` at the top level say whether the audit is complete.
161
+ A full `--limit` page of ingest rows, the server's ~3 MB response budget
162
+ (`summary.truncated_reason: "byte_limit"`) or a span cap sets both, and a
163
+ warning plus `next_commands` give the follow-up — usually the `--cursor=` page
164
+ from `next_cursor`; for very large traces audit one OTLP trace at a time
165
+ (`summary.trace_ids`, `--kind=trace`). Prompt/response bodies are omitted by
166
+ default (`{"omitted": true, "original_bytes": N}`); `--include-bodies` keeps
167
+ them, cutting strings over 8 KB to `{"truncated": true, "original_bytes": N,
168
+ "preview": …}`. Ingest rows whose stored payload is too large carry
169
+ `payload_capped: true`: their spans keep ids, hierarchy, timing and types but
170
+ no bodies. A trace too large to audit at all fails fast with an
171
+ `audit_too_large` error whose `suggested_commands` list per-trace audits.
172
+
173
+ Returns the raw, NON-deduped span set from the Layer-0 store (`events_raw`), so
174
+ nothing is collapsed. Use it to verify capture completeness: the `summary` gives
175
+ `total_spans`, `by_type` (llm/tool/agent/…), plus `orphan_count` (spans whose
176
+ parent is missing) and `duplicate_count` (double-instrumentation — the same
177
+ logical call emitted as >1 span). Each span carries `span_id`, `parent_span_id`,
178
+ `trace_id`, `type` (tool spans included), timing, `prompt`, `response`, and the
179
+ full semconv `attributes`. This is the honest substrate for auditing whether
180
+ Moda captured every LLM/tool call for a trace.
@@ -0,0 +1,79 @@
1
+ # Custom signals and false positives
2
+
3
+ Loaded on demand from the moda-cli skill.
4
+
5
+ ### 9. Custom signals and false positives
6
+
7
+ **Mark anything as a false positive with one command.** `--reason` (why the
8
+ detection is wrong) is optional but recommended; marks are logged for Moda's review.
9
+ A marked item disappears from every list and count (`moda problems`,
10
+ `hallucinations`, `laziness`, `tool-failures`, problem evidence, the dashboard);
11
+ `--undo` brings it back.
12
+
13
+ ```bash
14
+ moda false-positive problem <problem_id> --reason="two unrelated tools" # dismiss: hidden at once
15
+ moda false-positive problem-detection <problem_id> --conversation-id=<id> \
16
+ --attribution-id=<uuid> --reason="refund was approved" # one piece of evidence
17
+ moda false-positive quote <conversation_id> --family=frustration --turn=3 \
18
+ --quote="this is useless lol" --reason="user was joking" # one user quote
19
+ moda false-positive tool <conversation_id> --tool=send_invoice \
20
+ --tool-use-id=<tool_use_id> --reason="retried upstream" # one failing tool call
21
+ moda false-positive laziness <conversation_id> --reason="reload is expected"
22
+ moda false-positive hallucination <conversation_id> --reason="price came from the tool result"
23
+ moda false-positive custom-signal <signal_id> --conversation-id=<id> --reason="refund was approved"
24
+ moda false-positive laziness <conversation_id> --undo # remove a mark (any kind but problem)
25
+ moda false-positives # every current mark (--kind=tool, --all, --limit)
26
+ moda false-positives <conversation_id> # marks + problem reviews on a trace
27
+ ```
28
+
29
+ What each flag does downstream:
30
+
31
+ | Kind | Effect |
32
+ |---|---|
33
+ | `problem` | Dismisses the problem: it leaves `moda problems` and the dashboard at once, and the reason is folded into the next reconcile. Cannot be undone from the CLI |
34
+ | `problem-detection` | Drops that evidence from the problem and teaches the problem reader (same as `detection-review --verdict=false_positive`) |
35
+ | `custom-signal` | Saved as a `does_not_fit` label with the reason, a labelled counterexample for the signal, and the live match is retracted (same as `signal-label`). `--undo` removes the label |
36
+ | `quote`, `tool`, `laziness`, `hallucination` | Audit only: recorded for review, never fed back to the detectors |
37
+
38
+ Find targets with the read commands: `--family`, `--turn` and `--quote` verbatim
39
+ from a `user_quotes` entry in `moda emotions`; a failing call's `tool_use_id` (or `msg_index` when it has none) from
40
+ `moda tool-failure-detail <tool_name>`;
41
+ laziness traces from `moda laziness`; hallucination traces from
42
+ `moda hallucinations`; custom-signal hits from `moda trace-signals` or
43
+ `moda signal <id> --detections`; problem evidence ids from
44
+ `moda problem <id> --evidence`.
45
+
46
+ **Custom signals** are tenant-defined behaviors (pass/fail criteria) judged on
47
+ every trace. Read them, then label hits so each signal has labelled examples:
48
+
49
+ ```bash
50
+ moda signals # every signal: criteria, status, 7-day judged/matched
51
+ moda signal <signal_id> # one signal's summary
52
+ moda signal <signal_id> --detections --days-back=30 # recent hits (limit 1–20, --offset to page)
53
+ moda trace-signals <conversation_id> # signal hits on one trace
54
+ moda signal-label <signal_id> --conversation-id=<id> --label=does_not_fit \
55
+ --note="refund was approved" --msg-index=<anchor_msg_index>
56
+ ```
57
+
58
+ `--label=does_not_fit` marks a hit a **false positive** and retracts the live
59
+ match; `--label=fits` confirms it. Pass the hit's `anchor_msg_index` (or
60
+ `--msg-dedup-token=<anchor_dedup_token>`) so the example points at the right
61
+ message, and always give a `--note` when rejecting. Labels are throttled to 10
62
+ per minute per API key and are never retried automatically.
63
+
64
+ **Problem detections** (the dashboard's flag on problem evidence) are reviewed with
65
+ `detection-review`. A `false_positive` drops the detection out of
66
+ `moda problem <id> --evidence` and is fed back to the problem reader; `clear`
67
+ undoes an earlier review.
68
+
69
+ ```bash
70
+ moda detection-reviews <conversation_id> # problem detections on a trace + review state
71
+ moda detection-review <problem_id> --conversation-id=<id> \
72
+ --attribution-id=<uuid> --verdict=false_positive --note="retry was expected"
73
+ moda detection-review <problem_id> --conversation-id=<id> \
74
+ --attribution-id=<uuid> --verdict=clear # undo
75
+ ```
76
+
77
+ `attribution_id` comes from `moda detection-reviews <conversation_id>` or
78
+ `moda problem <problem_id> --evidence`. Prefer `detection-review` over
79
+ `problem-feedback --action=flag_attribution` for a single wrong detection.
@@ -0,0 +1,65 @@
1
+ # Validation and troubleshooting
2
+
3
+ Loaded on demand from the moda-cli skill.
4
+
5
+ ## Validation — how to know it worked
6
+
7
+ - Every successful command exits 0 and prints valid JSON to stdout.
8
+ - `moda ask` may exit `3` (degraded): a real, lower-trust answer synthesized
9
+ from local evidence — not a failure (`investigate`/`failures` always exit
10
+ `0` on success). Treat exit `1` as the only hard error (see
11
+ [Agent protocol](output-protocol.md#agent-protocol)).
12
+ - Errors exit non-zero. In agent/JSON mode the error envelope is printed to
13
+ stdout (`status: "error"`, `errors[0].code`), even with `--quiet`; human
14
+ mode prints prose to stderr. Check the exit code before claiming success.
15
+ - An unknown trace, problem, or cluster id exits 1 with
16
+ `errors[0].code: "not_found"`; it is not an empty success.
17
+ - `moda world-state` returns success with a warning when state is missing.
18
+ This can mean an unknown trace or a real trace without generated state.
19
+ Use `moda context <id> --metadata-only` to check whether the trace exists.
20
+ - If the JSON has `"conversations": []` or `"frustrations": []`, the call
21
+ worked but the time window is empty. Widen the window once
22
+ (`--days-back` on detection commands, `--time-range` on `traces`,
23
+ `search`, `clusters`) before concluding "no data".
24
+
25
+ ## Troubleshooting
26
+
27
+ **`No Moda API key found` (exit 4, `auth_required`)**
28
+ Have the user export `MODA_API_KEY` (keys: [moda.dev/settings](https://moda.dev/settings)),
29
+ or save one with `moda auth login --api-key`. `moda init` is the interactive
30
+ setup wizard; do not run it unprompted.
31
+
32
+ **`The Moda API rejected the API key (HTTP 401)` (exit 4, `auth_required`)**
33
+ The key is invalid, revoked, or for another environment. Get a new key and
34
+ save it with `moda auth login --api-key`.
35
+
36
+ **`not_found` (exit 1)**
37
+ The trace, problem, or cluster id does not exist in this tenant (a trace with
38
+ no generated world state returns a warning, not `not_found`).
39
+ `errors[0].cause` names the listing command: `moda search`/`moda traces`,
40
+ `moda problems`, `moda clusters`. A tool name with no failures in the window
41
+ returns an empty result with a warning instead.
42
+
43
+ **`Validation error:`**
44
+ A flag value didn't match the schema. Check enums (`time_range` must be
45
+ `1h|24h|3d|7d|30d|90d|all`, not `"7 days"`) and numeric ranges
46
+ (`days_back` is 1–90). `--window` is 1–5 on `frustrations` and
47
+ `tool-failure-detail`; on `context` a larger value is clamped to 5 with a
48
+ warning that gives the `--all` command for a wider slice.
49
+ `moda <command> --help` lists every flag with its range.
50
+
51
+ **Empty results**
52
+ Widen the window: try `--days-back=30` or `--time-range=30d`. Confirm with
53
+ `moda overview` that data exists in the tenant at all.
54
+
55
+ **`moda search` returns nothing, or semantic feels like keyword**
56
+ The default `--time-range` is `all`, so an empty result usually means the
57
+ phrasing, not the window: try a shorter phrase or the exact identifier with
58
+ `--mode=keyword`, and add `--include-tool-io` for strings that only appear in
59
+ tool inputs or outputs. When a `semantic`/`hybrid` request was served as
60
+ `keyword`, the envelope's `warnings` say so and why; retry or rephrase. For meaning-based queries prefer `--mode=semantic`; for
61
+ exact error codes, tool names, or IDs prefer `--mode=keyword`.
62
+
63
+ **`moda: command not found`**
64
+ Either install globally (`npm install -g @moda-ai/cli`) or invoke through
65
+ npx: `npx -p @moda-ai/cli moda <command>`.