@moda-ai/cli 1.55.19 → 1.55.21
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/dist/{cli-c5c6f0yw.js → cli-38p87ykc.js} +1 -1
- package/dist/{cli-5km7j04z.js → cli-4j8nffsm.js} +6 -2
- package/dist/{cli-ssfq84k5.js → cli-sf0g0fjg.js} +133 -18
- package/dist/cli.js +5 -5
- package/dist/{index-6qqmjqb2.js → index-ewqbpnwz.js} +3 -3
- package/dist/{skills-d35nnm66.js → skills-br82cpt7.js} +2 -2
- package/dist/{skills-bundled-mdmhzfsr.js → skills-bundled-r9a2vs07.js} +2 -2
- package/dist/{sync-n2psdffz.js → sync-25vn78je.js} +2 -2
- package/package.json +1 -1
- package/skills/integration/index.json +2 -2
- package/skills/moda-cli/SKILL.md +114 -1434
- package/skills/moda-cli/references/auth.md +176 -0
- package/skills/moda-cli/references/commands.md +74 -0
- package/skills/moda-cli/references/dashboard-parity.md +29 -0
- package/skills/moda-cli/references/detections.md +79 -0
- package/skills/moda-cli/references/feedback.md +45 -0
- package/skills/moda-cli/references/output-protocol.md +260 -0
- package/skills/moda-cli/references/production-intelligence.md +96 -0
- package/skills/moda-cli/references/prompts.md +104 -0
- package/skills/moda-cli/references/recipes.md +74 -0
- package/skills/moda-cli/references/replays-and-registry.md +173 -0
- package/skills/moda-cli/references/search-and-traces.md +180 -0
- package/skills/moda-cli/references/signals-and-false-positives.md +79 -0
- package/skills/moda-cli/references/troubleshooting.md +65 -0
|
@@ -0,0 +1,173 @@
|
|
|
1
|
+
# Replays and the harness registry
|
|
2
|
+
|
|
3
|
+
Loaded on demand from the moda-cli skill.
|
|
4
|
+
|
|
5
|
+
### 7a. Check a replay verdict (`moda replays`)
|
|
6
|
+
|
|
7
|
+
Use after `prompts ab`, `prompts propose --gate`, or a dashboard replay when
|
|
8
|
+
you need to see *why* an arm passed or failed. Read-only; no replay spend.
|
|
9
|
+
|
|
10
|
+
```bash
|
|
11
|
+
moda replays sets --json # set ids
|
|
12
|
+
moda replays runs <set_id> --limit=5 --json # newest runs, every status
|
|
13
|
+
moda replays run <set_id> <run_id> --json # per-case + per-seed verdicts
|
|
14
|
+
moda replays playouts <set_id> <run_id> --json # one row per case x arm x seed
|
|
15
|
+
moda replays playouts <set_id> <run_id> --case=<case_id> --arm=proposed --seed=1 --transcript --json
|
|
16
|
+
moda replays playout <set_id> <run_id> <playout_id> --json
|
|
17
|
+
```
|
|
18
|
+
|
|
19
|
+
`run` returns `results[]` (one per case × arm × seed: `pass`, `abstained`,
|
|
20
|
+
`coverageReason`, `score`, `rationale`, `criteria[]`) next to the per-case
|
|
21
|
+
majority in `cases[]`. `arm=prod` is the baseline and `arm=proposed` the
|
|
22
|
+
candidate. An `abstained: true` row is not a verdict: the pair hit a replay
|
|
23
|
+
coverage hole and is excluded from pass counts.
|
|
24
|
+
|
|
25
|
+
A playout's `transcript.events[]` holds the simulated user and agent messages
|
|
26
|
+
and every tool call with its `input`, the `output` the replay served, and
|
|
27
|
+
`provenance.source`: `replay_runtime` (compiled mock over the case's recorded
|
|
28
|
+
source state), `agent_tool_simulator` (LLM simulator grounded in recorded
|
|
29
|
+
receipts; `simulator.model`), `counterfactual_reconstruction`, or
|
|
30
|
+
`generic_handler`. `provenance.errorSubtype` such as `missing_source_state`
|
|
31
|
+
marks an unservable call. `judge` is the judgement stored with the playout and
|
|
32
|
+
`verdict` the stored eval row (null when `seed` is null, which happens on
|
|
33
|
+
older runs whose graphs cannot be matched to a seed).
|
|
34
|
+
|
|
35
|
+
Limits: `--transcript` pages at most 10 playouts (`--limit` default 50, max
|
|
36
|
+
200 without it); text and tool JSON are clipped at `--max-chars` (default
|
|
37
|
+
4000, max 32000) and `transcript.truncated.fields` counts the clipped fields.
|
|
38
|
+
Page with `--cursor=<nextCursor>`. Playouts exist only for runs whose
|
|
39
|
+
playouts were captured; `playoutCount: 0` on `run` means there is no
|
|
40
|
+
transcript to read.
|
|
41
|
+
|
|
42
|
+
### 7b. Register the harness for replays (`moda registry`)
|
|
43
|
+
|
|
44
|
+
Use when the user wants their prompts, skills, tools, or agent wiring in Moda
|
|
45
|
+
so replays rebuild their harness faithfully, without the GitHub App or `moda
|
|
46
|
+
harness analyze`. **You** are the analyst: read the codebase you already have,
|
|
47
|
+
write one manifest, and push exactly what it lists. Nothing else in the repo is
|
|
48
|
+
uploaded. For a full guided pass (inventory, prompt switching, context,
|
|
49
|
+
attribution, and verification), run `moda registry prompt` and follow the
|
|
50
|
+
brief it prints.
|
|
51
|
+
|
|
52
|
+
```bash
|
|
53
|
+
moda registry prompt # the step-by-step brief for this job
|
|
54
|
+
moda registry template --out=.moda/registry.json # starter manifest (moda_registry.v2)
|
|
55
|
+
# ...fill in .moda/registry.json from the codebase...
|
|
56
|
+
moda registry validate # local only, no network
|
|
57
|
+
moda registry push --dry-run # server-side preview, writes nothing
|
|
58
|
+
moda registry push # upload
|
|
59
|
+
moda registry status # what is registered (harness version)
|
|
60
|
+
moda registry coverage --days=7 # production traces vs the registry
|
|
61
|
+
moda registry log # history: one commit per push
|
|
62
|
+
moda registry show <commit|working> # a commit's file diffs / changes since the last push
|
|
63
|
+
```
|
|
64
|
+
|
|
65
|
+
Every successful `push` records a registry **commit**: the whole registry
|
|
66
|
+
(prompts, tools, skills, agents, harness settings) snapshotted as files, with
|
|
67
|
+
your repo's git SHA, branch, and dirty flag. Pass `--message="..."` to label
|
|
68
|
+
it. The dashboard **Registry** page shows these commits git-style: a log,
|
|
69
|
+
per-file diffs, a file browser, and uncommitted changes made outside a push
|
|
70
|
+
(dashboard edits, MCP discovery, `moda prompts sync`).
|
|
71
|
+
|
|
72
|
+
Manifest (`moda_registry.v2`; `moda_registry.v1` with only prompts, skills, and
|
|
73
|
+
tools still works; every section optional, ≤500 entries in prompts, skills, and tools):
|
|
74
|
+
|
|
75
|
+
```json
|
|
76
|
+
{
|
|
77
|
+
"schema": "moda_registry.v2",
|
|
78
|
+
"prompts": [
|
|
79
|
+
{ "key": "support.triage", "file": "prompts/triage.md" },
|
|
80
|
+
{ "key": "support.billing", "content": "You handle billing for {{company_name}}..." }
|
|
81
|
+
],
|
|
82
|
+
"skills": [
|
|
83
|
+
{ "key": "refund-policy", "file": ".claude/skills/refund-policy/SKILL.md" }
|
|
84
|
+
],
|
|
85
|
+
"tools": [
|
|
86
|
+
{ "schema": { "name": "lookup_customer", "description": "...",
|
|
87
|
+
"parameters": { "type": "object", "properties": { "email": { "type": "string" } } } } }
|
|
88
|
+
],
|
|
89
|
+
"runtime": { "framework": "openai-agents", "maxToolRoundsPerTurn": 6, "skillLoading": "tool" },
|
|
90
|
+
"agents": [
|
|
91
|
+
{ "key": "triage", "entry": true, "prompt": "support.triage",
|
|
92
|
+
"model": { "id": "openai/gpt-5.1", "temperature": 0.2, "maxTokens": 1024 },
|
|
93
|
+
"tools": ["lookup_customer"],
|
|
94
|
+
"handoffs": [{ "to": "billing", "tool": "transfer_to_billing" }],
|
|
95
|
+
"match": { "agentNames": ["TriageAgent"] } },
|
|
96
|
+
{ "key": "billing", "prompt": "support.billing", "skills": ["refund-policy"] }
|
|
97
|
+
],
|
|
98
|
+
"routing": { "strategy": "handoff", "default": "triage" },
|
|
99
|
+
"context": [
|
|
100
|
+
{ "key": "company", "variable": "company_name", "source": "static", "value": "Acme" },
|
|
101
|
+
{ "key": "today", "variable": "current_date", "source": "clock", "format": "date" }
|
|
102
|
+
]
|
|
103
|
+
}
|
|
104
|
+
```
|
|
105
|
+
|
|
106
|
+
- **Prompts:** `key` plus one of `content`, `file`, `systemPrompt`, or
|
|
107
|
+
`messages`. Optional `name`, `description`, `category`, `variables`,
|
|
108
|
+
`modelConfig`, `toolRefs`, `responseSchema`, `sourcePath`. Use the key the
|
|
109
|
+
runtime passes to `Moda.prompt(key)` when there is one. Copy each prompt's
|
|
110
|
+
text **verbatim**: for templated prompts, keep the template string, not a
|
|
111
|
+
rendered example. Re-pushing the same text is a no-op; changed text adds a
|
|
112
|
+
new version.
|
|
113
|
+
- **Skills:** `key` (letters, digits, `_`, `-`) plus `skillMd` or `file` (full
|
|
114
|
+
SKILL.md). Pushed skills are *local* by default. Set `"live": true`, or pass
|
|
115
|
+
`--live`, to include them in the served skill surface. Leaving `live` out
|
|
116
|
+
never demotes a skill that is already live. Replays load the skills an agent
|
|
117
|
+
lists in `agents[].skills`, whether or not they are live.
|
|
118
|
+
- **Tools:** an OpenAI function `schema` (a wrapped
|
|
119
|
+
`{"type":"function","function":{...}}` is accepted), plus optional
|
|
120
|
+
`isActive`. Copy names and parameter schemas exactly as the agent registers
|
|
121
|
+
them with the model. Tools are upserted by name and never overwrite
|
|
122
|
+
MCP-server-managed tools (those report `skipped`).
|
|
123
|
+
- **Agents:** one per distinct prompt, tool set, or model. Fields: `key`,
|
|
124
|
+
`name`, `description`, `entry`, `prompt` (prompt key), `model` (`id`,
|
|
125
|
+
`provider`, `temperature`, `maxTokens`, `topP`, `reasoningEffort`), `tools`
|
|
126
|
+
(tool names), `skills` (skill keys), `handoffs` (`to`, `tool`,
|
|
127
|
+
`description`), `subagents` (`agent`, `tool`; the tool must also be in
|
|
128
|
+
`tools`), `context` (context keys), `maxToolRoundsPerTurn`,
|
|
129
|
+
`responseSchema`, `match` (`agentNames` as they appear in traces,
|
|
130
|
+
`promptKeys`).
|
|
131
|
+
- **Routing (prompt switching):** `strategy` is `single`, `handoff`, `router`,
|
|
132
|
+
`rules`, or `trace_attribute`; `default` is an agent key; put code-level
|
|
133
|
+
rules in `notes`. Replay runs the agent named on the replayed turn's trace
|
|
134
|
+
(agent name, then prompt key), else `default`, else the `entry` agent. When
|
|
135
|
+
the replayed model calls a handoff tool, replay switches to the target
|
|
136
|
+
agent's prompt, tools, model, and params.
|
|
137
|
+
- **Context:** what the runtime injects. `source` is `static` (with `value`),
|
|
138
|
+
`clock` (`format`: date, datetime, or iso; replay uses the source
|
|
139
|
+
conversation's time), or `trace`, `memory`, `retrieval`, `tool`, or
|
|
140
|
+
`unknown`. Those last ones are reported as unresolved in replay fidelity,
|
|
141
|
+
never invented. `target` is `prompt_variable` (needs `variable`) or
|
|
142
|
+
`system_append`.
|
|
143
|
+
- **Runtime:** `framework`, `language`, `maxToolRoundsPerTurn`,
|
|
144
|
+
`parallelToolCalls`, `history` (`strategy`, `maxMessages`, `maxTokens`),
|
|
145
|
+
and `skillLoading` (`tool`, `system_prompt`, or `none`).
|
|
146
|
+
- **MCP servers:** `name`, `transport`, `url` or `command`, `tools`, and
|
|
147
|
+
`description`, with no credentials. List their tools in `tools[]` too.
|
|
148
|
+
- `file` paths are relative to the directory you run the command from and must
|
|
149
|
+
stay inside it (symlinks included).
|
|
150
|
+
|
|
151
|
+
**Traces must name the agent.** For per-turn prompt switching, each LLM call
|
|
152
|
+
should carry `gen_ai.agent.name` (or `moda.agent_name`; with the Vercel AI SDK
|
|
153
|
+
use `experimental_telemetry.metadata["moda.agent_name"]`; `agent_name` on
|
|
154
|
+
`/v1/ingest`), set to an agent `key` or one of its `match.agentNames`. It
|
|
155
|
+
should also carry `moda.prompt_key` / `moda.prompt_version_id`. Only add these
|
|
156
|
+
to application code when the user asks for it or approves the diff.
|
|
157
|
+
`moda registry coverage` shows how many recent calls resolve to a registered
|
|
158
|
+
agent, which agent names, prompt keys, and tools are unregistered, and which
|
|
159
|
+
agents were never seen.
|
|
160
|
+
|
|
161
|
+
`push` needs an API key. `push --dry-run` without one only validates locally
|
|
162
|
+
(status `validated`). Each section goes to its own endpoint and reports
|
|
163
|
+
`sections[].status` (`synced`, `dry_run`, `validated`, `skipped`, or `error`).
|
|
164
|
+
The harness sections (`runtime`, `agents`, `routing`, `context`, and
|
|
165
|
+
`mcpServers`) are pushed last, as one versioned document. Re-pushing an
|
|
166
|
+
unchanged one reports `unchanged`. If any section fails, `push` exits `1`,
|
|
167
|
+
lists the failures in `errors[]`, and still pushes the other sections. The
|
|
168
|
+
whole manifest is validated before any request, with every error listed at
|
|
169
|
+
once. `warnings[]` names agent references to prompts, tools, or skills that
|
|
170
|
+
the manifest doesn't define; they must already exist in Moda. Unknown fields
|
|
171
|
+
and wrong types are rejected, matching what the server accepts. Skill keys are
|
|
172
|
+
stricter than the server (letters, digits, `_`, `-`) because they double as
|
|
173
|
+
`.claude/skills/<key>/` directory names.
|
|
@@ -0,0 +1,180 @@
|
|
|
1
|
+
# Search, traces, context, and audit
|
|
2
|
+
|
|
3
|
+
Loaded on demand from the moda-cli skill.
|
|
4
|
+
|
|
5
|
+
### 1. Search messages (primary content-discovery path)
|
|
6
|
+
|
|
7
|
+
```bash
|
|
8
|
+
moda search "refund flow timeout" # hybrid (default)
|
|
9
|
+
moda search "customer asked for a refund" --mode=semantic
|
|
10
|
+
moda search "SEARCH_NO_RESULTS" --mode=keyword
|
|
11
|
+
moda search "checkout error" --time-range=7d --limit=10
|
|
12
|
+
moda search "stripe.charges.create failed" --user-id=<id>
|
|
13
|
+
moda search "Иван" --conversation-id=<id> --mode=keyword # inside one trace
|
|
14
|
+
```
|
|
15
|
+
|
|
16
|
+
Searches trace messages at message grain. Pass `--include-tool-io` to
|
|
17
|
+
also search **tool calls and tool results** (tool name, input arguments, and
|
|
18
|
+
output previews), so it finds where an agent *did* something, not just where
|
|
19
|
+
it talked about it — tool-IO search is opt-in, not the default.
|
|
20
|
+
|
|
21
|
+
Flags: `--mode=keyword|semantic|hybrid` (default `hybrid`), `--user-id`,
|
|
22
|
+
`--conversation-id` (search inside one trace; no trace fallback),
|
|
23
|
+
`--time-range` (`all|1h|3d|7d|24h|30d|90d`, default `all`), `--limit` (1–100,
|
|
24
|
+
default 20). The query is a positional arg (max 500 chars).
|
|
25
|
+
|
|
26
|
+
**Index freshness.** Keyword matches (and hybrid's keyword half) fall back to
|
|
27
|
+
the original messages when the search index returns too few literal matches,
|
|
28
|
+
so recent, Cyrillic and partial-word text is still found. `coverage` reports
|
|
29
|
+
which source served the keyword leg (`keyword_source`) and how far the index
|
|
30
|
+
trails ingestion (`index_freshness`); a warning says so when your
|
|
31
|
+
`--time-range` reaches past the newest indexed message.
|
|
32
|
+
|
|
33
|
+
**Fallback.** If message-level search finds nothing but whole traces match
|
|
34
|
+
(same as `moda traces --search`), those traces come back in `results` with
|
|
35
|
+
`match_grain: "trace"`, plus `fallback.source: "traces_search"` and a warning.
|
|
36
|
+
These rows have no score or message index. Open one with
|
|
37
|
+
`moda audit <conversation_id>` or `moda context <conversation_id>` to find the
|
|
38
|
+
matching turn. When more traces exist, `fallback.has_more` is true and
|
|
39
|
+
`next_commands` includes the `moda traces --search ... --offset` command to
|
|
40
|
+
page. With `--mode=keyword`, ranked traces are not substituted. If the fallback
|
|
41
|
+
itself fails, you get a warning, not a silent empty result. Either way, zero
|
|
42
|
+
results does not prove the topic never came up.
|
|
43
|
+
|
|
44
|
+
Choosing a mode:
|
|
45
|
+
|
|
46
|
+
- `hybrid` (default) — best general recall; fuses keyword + semantic. Use it
|
|
47
|
+
unless you have a reason not to.
|
|
48
|
+
- `keyword` — exact substring/identifier matches (error codes, tool names,
|
|
49
|
+
IDs). `score` is the integer match count.
|
|
50
|
+
- `semantic` — meaning-based; finds paraphrases the user never typed
|
|
51
|
+
verbatim. `score` is cosine similarity.
|
|
52
|
+
|
|
53
|
+
Each result row carries `conversation_id`, `message_index`, `role`,
|
|
54
|
+
`timestamp`, `snippet`, `score`, and `cluster_id` / `cluster_name`. The
|
|
55
|
+
response also reports `search_mode` — the mode that **actually ran**; a
|
|
56
|
+
`semantic`/`hybrid` request transparently degrades to `keyword` when no
|
|
57
|
+
embedding provider is reachable, so check it before trusting semantic ranking.
|
|
58
|
+
`score` scales differ per mode (see above) — branch on `search_mode`, never
|
|
59
|
+
compare scores across modes.
|
|
60
|
+
|
|
61
|
+
Hand the anchors straight to `moda context`:
|
|
62
|
+
|
|
63
|
+
```bash
|
|
64
|
+
# Find, then read the exact turn. --json prints the bare payload; piped output
|
|
65
|
+
# without it is the agent envelope, where results sit under .data.results.
|
|
66
|
+
hit=$(moda search "double charge" --limit=1 --json | jq -c '.results[0] // empty')
|
|
67
|
+
cid=$(echo "$hit" | jq -r '.conversation_id')
|
|
68
|
+
# Trace-level fallback hits (match_grain: "trace") have no message_index.
|
|
69
|
+
idx=$(echo "$hit" | jq -r '.message_index // empty')
|
|
70
|
+
moda context "$cid" ${idx:+--msg-index="$idx"}
|
|
71
|
+
```
|
|
72
|
+
|
|
73
|
+
### 3. Browse topic clusters
|
|
74
|
+
|
|
75
|
+
```bash
|
|
76
|
+
moda clusters # root-level
|
|
77
|
+
moda clusters --parent-id=<node_id> # drill in
|
|
78
|
+
moda cluster-traces <node_id> # traces in cluster
|
|
79
|
+
```
|
|
80
|
+
|
|
81
|
+
### 3b. List / filter traces (structured, not semantic)
|
|
82
|
+
|
|
83
|
+
Use `moda traces` to enumerate or filter by structured fields — not to
|
|
84
|
+
search by meaning (use `moda search` for that). The `--search` flag here
|
|
85
|
+
ranks whole traces: hybrid (keyword + semantic) when embeddings exist, else
|
|
86
|
+
keyword. Check `total_kind`: `candidate_pool` means results are related traces
|
|
87
|
+
that may not contain the words and `pagination.total` is not a match count;
|
|
88
|
+
`matches` means a literal keyword match set.
|
|
89
|
+
|
|
90
|
+
```bash
|
|
91
|
+
moda traces --user-id=<id> --time-range=7d
|
|
92
|
+
moda traces --cluster-id=<node_id> --environment=production
|
|
93
|
+
moda traces --search="timeout" --limit=20 # ranked trace search
|
|
94
|
+
```
|
|
95
|
+
|
|
96
|
+
Filters: `--search`, `--cluster-id`, `--user-id`, `--time-range`
|
|
97
|
+
(`all|1h|3d|7d|24h|30d|90d`), `--environment`
|
|
98
|
+
(`all|development|staging|production`), `--outcome`, `--limit`, `--offset`.
|
|
99
|
+
|
|
100
|
+
### 4. Read a single trace
|
|
101
|
+
|
|
102
|
+
```bash
|
|
103
|
+
moda context <conversation_id> # default window around middle
|
|
104
|
+
moda context <conversation_id> --msg-index=5 # center on message 5
|
|
105
|
+
moda context <conversation_id> --window=3 # 3 messages each side (max 5)
|
|
106
|
+
moda context <conversation_id> --msg-index=40 --window=0 # just message 40 + total_messages
|
|
107
|
+
moda context <conversation_id> --metadata-only # summary + total_messages, no messages
|
|
108
|
+
moda context <conversation_id> --all # whole trace in order, first 500 messages
|
|
109
|
+
moda context <conversation_id> --all --from=500 --max-messages=500 # continue (max 5000)
|
|
110
|
+
```
|
|
111
|
+
|
|
112
|
+
`--all` returns `has_more`/`next_from`; when it stops at the cap, the warning
|
|
113
|
+
prints the exact `--from` command to continue.
|
|
114
|
+
|
|
115
|
+
**Several traces? Batch them in one call instead of looping `moda context`
|
|
116
|
+
per id.** Pass up to 25 targets as positionals or one per line with
|
|
117
|
+
`--ids-from=-` (stdin) or `--ids-from=<file>`. A target is `<id>` or
|
|
118
|
+
`<id>@<msg_index>` (the anchor; ids may contain `:`), and the shared flags
|
|
119
|
+
(`--window`, `--metadata-only`, `--all`) apply to every target. Lines may also
|
|
120
|
+
be JSON (`{"id":"...","msg_index":N}`), the only spelling for an id that itself
|
|
121
|
+
ends in `@<digits>`.
|
|
122
|
+
|
|
123
|
+
```bash
|
|
124
|
+
moda context telegram:8565459080@12 linq:c6f88988-aa@3 web:conv-9 --window=2
|
|
125
|
+
printf '%s\n' telegram:8565459080@12 web:conv-9 | moda context --ids-from=- --window=1
|
|
126
|
+
```
|
|
127
|
+
|
|
128
|
+
A batch returns `data.results: [{ target, ok, data | error }]` in input order,
|
|
129
|
+
plus `succeeded`/`failed` counts. A missing trace is a per-item
|
|
130
|
+
`error.code: "not_found"`, not a failed call; the exit code is 1 only when
|
|
131
|
+
every target failed. When the agent byte budget would overflow, the items that
|
|
132
|
+
fit are returned and `remaining_targets` plus a `next_commands` entry read the
|
|
133
|
+
rest. One positional id keeps the single-trace output above.
|
|
134
|
+
|
|
135
|
+
`moda context` returns messages whole, with their tool calls and results. An
|
|
136
|
+
older Moda server cuts message text, tool inputs, and tool results at 500
|
|
137
|
+
characters; the response warns when that server is in use.
|
|
138
|
+
`--window` is at most 5 messages per side. For the raw,
|
|
139
|
+
complete picture of what was captured — every span, the parent/child hierarchy,
|
|
140
|
+
tool calls, prompt/response bodies, and `gen_ai.*` attributes — use `moda audit`.
|
|
141
|
+
|
|
142
|
+
### 4b. Audit raw spans (completeness, orphans, duplicates)
|
|
143
|
+
|
|
144
|
+
```bash
|
|
145
|
+
moda audit <conversation_id|trace_id> # auto-detects OTLP trace_id vs trace ID (conversation_id)
|
|
146
|
+
moda audit <trace_id> --kind=trace # force OTLP trace_id lookup
|
|
147
|
+
moda audit <conversation_id> --include-raw # attach verbatim raw_event bodies
|
|
148
|
+
moda audit <conversation_id> --include-bodies # keep prompt/response/attribute bodies
|
|
149
|
+
moda audit <conversation_id> --limit=1000 # ingest rows to read (default 200, max 1000)
|
|
150
|
+
moda audit <conversation_id> --cursor=<next_cursor> # next page of a capped audit
|
|
151
|
+
moda audit <id1> <id2> <id3> # batch: up to 25 ids, one envelope
|
|
152
|
+
```
|
|
153
|
+
|
|
154
|
+
Auditing several traces? Batch the ids (positionals or `--ids-from=-|<file>`)
|
|
155
|
+
instead of looping `moda audit`: the result is `data.results: [{ target, ok,
|
|
156
|
+
data | error }]` with the same per-item errors and byte-budget rules as batch
|
|
157
|
+
`moda context`. Ids are taken verbatim (no `@` anchors); `--cursor` pages one
|
|
158
|
+
trace, so follow each item's `next_commands` for it.
|
|
159
|
+
|
|
160
|
+
`truncated` / `has_more` at the top level say whether the audit is complete.
|
|
161
|
+
A full `--limit` page of ingest rows, the server's ~3 MB response budget
|
|
162
|
+
(`summary.truncated_reason: "byte_limit"`) or a span cap sets both, and a
|
|
163
|
+
warning plus `next_commands` give the follow-up — usually the `--cursor=` page
|
|
164
|
+
from `next_cursor`; for very large traces audit one OTLP trace at a time
|
|
165
|
+
(`summary.trace_ids`, `--kind=trace`). Prompt/response bodies are omitted by
|
|
166
|
+
default (`{"omitted": true, "original_bytes": N}`); `--include-bodies` keeps
|
|
167
|
+
them, cutting strings over 8 KB to `{"truncated": true, "original_bytes": N,
|
|
168
|
+
"preview": …}`. Ingest rows whose stored payload is too large carry
|
|
169
|
+
`payload_capped: true`: their spans keep ids, hierarchy, timing and types but
|
|
170
|
+
no bodies. A trace too large to audit at all fails fast with an
|
|
171
|
+
`audit_too_large` error whose `suggested_commands` list per-trace audits.
|
|
172
|
+
|
|
173
|
+
Returns the raw, NON-deduped span set from the Layer-0 store (`events_raw`), so
|
|
174
|
+
nothing is collapsed. Use it to verify capture completeness: the `summary` gives
|
|
175
|
+
`total_spans`, `by_type` (llm/tool/agent/…), plus `orphan_count` (spans whose
|
|
176
|
+
parent is missing) and `duplicate_count` (double-instrumentation — the same
|
|
177
|
+
logical call emitted as >1 span). Each span carries `span_id`, `parent_span_id`,
|
|
178
|
+
`trace_id`, `type` (tool spans included), timing, `prompt`, `response`, and the
|
|
179
|
+
full semconv `attributes`. This is the honest substrate for auditing whether
|
|
180
|
+
Moda captured every LLM/tool call for a trace.
|
|
@@ -0,0 +1,79 @@
|
|
|
1
|
+
# Custom signals and false positives
|
|
2
|
+
|
|
3
|
+
Loaded on demand from the moda-cli skill.
|
|
4
|
+
|
|
5
|
+
### 9. Custom signals and false positives
|
|
6
|
+
|
|
7
|
+
**Mark anything as a false positive with one command.** `--reason` (why the
|
|
8
|
+
detection is wrong) is optional but recommended; marks are logged for Moda's review.
|
|
9
|
+
A marked item disappears from every list and count (`moda problems`,
|
|
10
|
+
`hallucinations`, `laziness`, `tool-failures`, problem evidence, the dashboard);
|
|
11
|
+
`--undo` brings it back.
|
|
12
|
+
|
|
13
|
+
```bash
|
|
14
|
+
moda false-positive problem <problem_id> --reason="two unrelated tools" # dismiss: hidden at once
|
|
15
|
+
moda false-positive problem-detection <problem_id> --conversation-id=<id> \
|
|
16
|
+
--attribution-id=<uuid> --reason="refund was approved" # one piece of evidence
|
|
17
|
+
moda false-positive quote <conversation_id> --family=frustration --turn=3 \
|
|
18
|
+
--quote="this is useless lol" --reason="user was joking" # one user quote
|
|
19
|
+
moda false-positive tool <conversation_id> --tool=send_invoice \
|
|
20
|
+
--tool-use-id=<tool_use_id> --reason="retried upstream" # one failing tool call
|
|
21
|
+
moda false-positive laziness <conversation_id> --reason="reload is expected"
|
|
22
|
+
moda false-positive hallucination <conversation_id> --reason="price came from the tool result"
|
|
23
|
+
moda false-positive custom-signal <signal_id> --conversation-id=<id> --reason="refund was approved"
|
|
24
|
+
moda false-positive laziness <conversation_id> --undo # remove a mark (any kind but problem)
|
|
25
|
+
moda false-positives # every current mark (--kind=tool, --all, --limit)
|
|
26
|
+
moda false-positives <conversation_id> # marks + problem reviews on a trace
|
|
27
|
+
```
|
|
28
|
+
|
|
29
|
+
What each flag does downstream:
|
|
30
|
+
|
|
31
|
+
| Kind | Effect |
|
|
32
|
+
|---|---|
|
|
33
|
+
| `problem` | Dismisses the problem: it leaves `moda problems` and the dashboard at once, and the reason is folded into the next reconcile. Cannot be undone from the CLI |
|
|
34
|
+
| `problem-detection` | Drops that evidence from the problem and teaches the problem reader (same as `detection-review --verdict=false_positive`) |
|
|
35
|
+
| `custom-signal` | Saved as a `does_not_fit` label with the reason, a labelled counterexample for the signal, and the live match is retracted (same as `signal-label`). `--undo` removes the label |
|
|
36
|
+
| `quote`, `tool`, `laziness`, `hallucination` | Audit only: recorded for review, never fed back to the detectors |
|
|
37
|
+
|
|
38
|
+
Find targets with the read commands: `--family`, `--turn` and `--quote` verbatim
|
|
39
|
+
from a `user_quotes` entry in `moda emotions`; a failing call's `tool_use_id` (or `msg_index` when it has none) from
|
|
40
|
+
`moda tool-failure-detail <tool_name>`;
|
|
41
|
+
laziness traces from `moda laziness`; hallucination traces from
|
|
42
|
+
`moda hallucinations`; custom-signal hits from `moda trace-signals` or
|
|
43
|
+
`moda signal <id> --detections`; problem evidence ids from
|
|
44
|
+
`moda problem <id> --evidence`.
|
|
45
|
+
|
|
46
|
+
**Custom signals** are tenant-defined behaviors (pass/fail criteria) judged on
|
|
47
|
+
every trace. Read them, then label hits so each signal has labelled examples:
|
|
48
|
+
|
|
49
|
+
```bash
|
|
50
|
+
moda signals # every signal: criteria, status, 7-day judged/matched
|
|
51
|
+
moda signal <signal_id> # one signal's summary
|
|
52
|
+
moda signal <signal_id> --detections --days-back=30 # recent hits (limit 1–20, --offset to page)
|
|
53
|
+
moda trace-signals <conversation_id> # signal hits on one trace
|
|
54
|
+
moda signal-label <signal_id> --conversation-id=<id> --label=does_not_fit \
|
|
55
|
+
--note="refund was approved" --msg-index=<anchor_msg_index>
|
|
56
|
+
```
|
|
57
|
+
|
|
58
|
+
`--label=does_not_fit` marks a hit a **false positive** and retracts the live
|
|
59
|
+
match; `--label=fits` confirms it. Pass the hit's `anchor_msg_index` (or
|
|
60
|
+
`--msg-dedup-token=<anchor_dedup_token>`) so the example points at the right
|
|
61
|
+
message, and always give a `--note` when rejecting. Labels are throttled to 10
|
|
62
|
+
per minute per API key and are never retried automatically.
|
|
63
|
+
|
|
64
|
+
**Problem detections** (the dashboard's flag on problem evidence) are reviewed with
|
|
65
|
+
`detection-review`. A `false_positive` drops the detection out of
|
|
66
|
+
`moda problem <id> --evidence` and is fed back to the problem reader; `clear`
|
|
67
|
+
undoes an earlier review.
|
|
68
|
+
|
|
69
|
+
```bash
|
|
70
|
+
moda detection-reviews <conversation_id> # problem detections on a trace + review state
|
|
71
|
+
moda detection-review <problem_id> --conversation-id=<id> \
|
|
72
|
+
--attribution-id=<uuid> --verdict=false_positive --note="retry was expected"
|
|
73
|
+
moda detection-review <problem_id> --conversation-id=<id> \
|
|
74
|
+
--attribution-id=<uuid> --verdict=clear # undo
|
|
75
|
+
```
|
|
76
|
+
|
|
77
|
+
`attribution_id` comes from `moda detection-reviews <conversation_id>` or
|
|
78
|
+
`moda problem <problem_id> --evidence`. Prefer `detection-review` over
|
|
79
|
+
`problem-feedback --action=flag_attribution` for a single wrong detection.
|
|
@@ -0,0 +1,65 @@
|
|
|
1
|
+
# Validation and troubleshooting
|
|
2
|
+
|
|
3
|
+
Loaded on demand from the moda-cli skill.
|
|
4
|
+
|
|
5
|
+
## Validation — how to know it worked
|
|
6
|
+
|
|
7
|
+
- Every successful command exits 0 and prints valid JSON to stdout.
|
|
8
|
+
- `moda ask` may exit `3` (degraded): a real, lower-trust answer synthesized
|
|
9
|
+
from local evidence — not a failure (`investigate`/`failures` always exit
|
|
10
|
+
`0` on success). Treat exit `1` as the only hard error (see
|
|
11
|
+
[Agent protocol](output-protocol.md#agent-protocol)).
|
|
12
|
+
- Errors exit non-zero. In agent/JSON mode the error envelope is printed to
|
|
13
|
+
stdout (`status: "error"`, `errors[0].code`), even with `--quiet`; human
|
|
14
|
+
mode prints prose to stderr. Check the exit code before claiming success.
|
|
15
|
+
- An unknown trace, problem, or cluster id exits 1 with
|
|
16
|
+
`errors[0].code: "not_found"`; it is not an empty success.
|
|
17
|
+
- `moda world-state` returns success with a warning when state is missing.
|
|
18
|
+
This can mean an unknown trace or a real trace without generated state.
|
|
19
|
+
Use `moda context <id> --metadata-only` to check whether the trace exists.
|
|
20
|
+
- If the JSON has `"conversations": []` or `"frustrations": []`, the call
|
|
21
|
+
worked but the time window is empty. Widen the window once
|
|
22
|
+
(`--days-back` on detection commands, `--time-range` on `traces`,
|
|
23
|
+
`search`, `clusters`) before concluding "no data".
|
|
24
|
+
|
|
25
|
+
## Troubleshooting
|
|
26
|
+
|
|
27
|
+
**`No Moda API key found` (exit 4, `auth_required`)**
|
|
28
|
+
Have the user export `MODA_API_KEY` (keys: [moda.dev/settings](https://moda.dev/settings)),
|
|
29
|
+
or save one with `moda auth login --api-key`. `moda init` is the interactive
|
|
30
|
+
setup wizard; do not run it unprompted.
|
|
31
|
+
|
|
32
|
+
**`The Moda API rejected the API key (HTTP 401)` (exit 4, `auth_required`)**
|
|
33
|
+
The key is invalid, revoked, or for another environment. Get a new key and
|
|
34
|
+
save it with `moda auth login --api-key`.
|
|
35
|
+
|
|
36
|
+
**`not_found` (exit 1)**
|
|
37
|
+
The trace, problem, or cluster id does not exist in this tenant (a trace with
|
|
38
|
+
no generated world state returns a warning, not `not_found`).
|
|
39
|
+
`errors[0].cause` names the listing command: `moda search`/`moda traces`,
|
|
40
|
+
`moda problems`, `moda clusters`. A tool name with no failures in the window
|
|
41
|
+
returns an empty result with a warning instead.
|
|
42
|
+
|
|
43
|
+
**`Validation error:`**
|
|
44
|
+
A flag value didn't match the schema. Check enums (`time_range` must be
|
|
45
|
+
`1h|24h|3d|7d|30d|90d|all`, not `"7 days"`) and numeric ranges
|
|
46
|
+
(`days_back` is 1–90). `--window` is 1–5 on `frustrations` and
|
|
47
|
+
`tool-failure-detail`; on `context` a larger value is clamped to 5 with a
|
|
48
|
+
warning that gives the `--all` command for a wider slice.
|
|
49
|
+
`moda <command> --help` lists every flag with its range.
|
|
50
|
+
|
|
51
|
+
**Empty results**
|
|
52
|
+
Widen the window: try `--days-back=30` or `--time-range=30d`. Confirm with
|
|
53
|
+
`moda overview` that data exists in the tenant at all.
|
|
54
|
+
|
|
55
|
+
**`moda search` returns nothing, or semantic feels like keyword**
|
|
56
|
+
The default `--time-range` is `all`, so an empty result usually means the
|
|
57
|
+
phrasing, not the window: try a shorter phrase or the exact identifier with
|
|
58
|
+
`--mode=keyword`, and add `--include-tool-io` for strings that only appear in
|
|
59
|
+
tool inputs or outputs. When a `semantic`/`hybrid` request was served as
|
|
60
|
+
`keyword`, the envelope's `warnings` say so and why; retry or rephrase. For meaning-based queries prefer `--mode=semantic`; for
|
|
61
|
+
exact error codes, tool names, or IDs prefer `--mode=keyword`.
|
|
62
|
+
|
|
63
|
+
**`moda: command not found`**
|
|
64
|
+
Either install globally (`npm install -g @moda-ai/cli`) or invoke through
|
|
65
|
+
npx: `npx -p @moda-ai/cli moda <command>`.
|