@moda-ai/cli 1.55.23 → 1.55.25
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/dist/{agent-f8a1kejk.js → agent-t7dxmpdq.js} +8 -8
- package/dist/{cli-9g8vz4sv.js → cli-0wr3483h.js} +1 -1
- package/dist/{cli-4f6jkrnm.js → cli-7xc585hc.js} +5 -5
- package/dist/{cli-wkmdw40s.js → cli-9btx12wp.js} +1 -1
- package/dist/{cli-4yarp5s1.js → cli-9hc5bg2r.js} +1 -1
- package/dist/{cli-cv3jt1ms.js → cli-dn91fb56.js} +10 -10
- package/dist/{cli-t3a6eknn.js → cli-dve50x7p.js} +233 -28
- package/dist/{cli-54zhv6cx.js → cli-m0045npe.js} +17 -14
- package/dist/{cli-s6tw1fje.js → cli-m805msp6.js} +210 -23
- package/dist/{cli-z3g07t65.js → cli-mebf5vtr.js} +5 -5
- package/dist/{cli-2cggp8ww.js → cli-qvg26twp.js} +5 -4
- package/dist/{cli-d2zews6c.js → cli-qycsmxev.js} +3 -3
- package/dist/cli.js +1292 -335
- package/dist/{doctor-snb87jvy.js → doctor-q21mr5ya.js} +8 -8
- package/dist/{fix-fr48gr0w.js → fix-0qdtdz5h.js} +6 -6
- package/dist/{harness-github-actions-cfkx9ty9.js → harness-github-actions-atc5bbkj.js} +5 -5
- package/dist/{harness-m63b6s0k.js → harness-vzy7n7ea.js} +5 -5
- package/dist/{index-nsaph4as.js → index-4b1ybz51.js} +17 -17
- package/dist/{production-intelligence-fwjh572m.js → production-intelligence-c8e0xw3n.js} +11 -11
- package/dist/{profile-commands-548ts402.js → profile-commands-6e2dgv72.js} +3 -3
- package/dist/prompt-source-mcp.js +5 -5
- package/dist/{prompts-p66y7wep.js → prompts-3h5jcjrw.js} +6 -6
- package/dist/{provision-r9fhrvpw.js → provision-q86phh4a.js} +5 -5
- package/dist/{registry-n7dsdxmg.js → registry-s2vdsfw8.js} +3 -3
- package/dist/{skills-c17gg69y.js → skills-vq3njc8a.js} +5 -5
- package/dist/{sync-bne90xvj.js → sync-hq9dpqpr.js} +7 -7
- package/package.json +1 -1
- package/skills/integration/index.json +2 -2
- package/skills/moda-cli/SKILL.md +35 -30
- package/skills/moda-cli/references/commands.md +22 -22
- package/skills/moda-cli/references/dashboard-parity.md +4 -4
- package/skills/moda-cli/references/detections.md +10 -10
- package/skills/moda-cli/references/feedback.md +2 -2
- package/skills/moda-cli/references/output-protocol.md +2 -2
- package/skills/moda-cli/references/production-intelligence.md +13 -12
- package/skills/moda-cli/references/prompts.md +6 -2
- package/skills/moda-cli/references/recipes.md +14 -14
- package/skills/moda-cli/references/search-and-traces.md +32 -32
- package/skills/moda-cli/references/signals-and-false-positives.md +29 -29
- package/skills/moda-cli/references/troubleshooting.md +4 -4
|
@@ -14,34 +14,34 @@ Loaded on demand from the moda-cli skill. This table can lag the CLI: `moda <com
|
|
|
14
14
|
| `moda auth logout` | Revoke the CLI session server-side, then delete it locally |
|
|
15
15
|
| `moda auth status` | Both credentials; exit `0` if either is usable, `4` if neither (expected, not an error) |
|
|
16
16
|
| `moda auth token` | Print the raw bearer token for `$(moda auth token)` composition |
|
|
17
|
-
| `moda search "<query>"` | **Message-grain semantic/keyword/hybrid search (incl. tool calls); primary content-discovery path** |
|
|
17
|
+
| `moda traces search "<query>"` | **Message-grain semantic/keyword/hybrid search (incl. tool calls); primary content-discovery path** |
|
|
18
18
|
| `moda overview` | Dashboard KPIs, top clusters, recent activity |
|
|
19
19
|
| `moda ask "<question>"` | Natural-language production/harness answer with evidence (exit `3` = degraded local fallback) |
|
|
20
20
|
| `moda investigate` | Rank production issues (tool failures, frustrations, top Problems) with evidence + next commands |
|
|
21
21
|
| `moda clusters` | Browse topic cluster hierarchy; `--search="q"` finds clusters by meaning, `--node-id=ID` resolves a deep link |
|
|
22
22
|
| `moda cluster-traces <node_id>` | Traces in a cluster (legacy alias: `moda cluster-conversations`) |
|
|
23
23
|
| `moda traces` | List/filter traces by structured fields (legacy alias: `moda conversations`) |
|
|
24
|
-
| `moda
|
|
25
|
-
| `moda
|
|
26
|
-
| `moda
|
|
27
|
-
| `moda
|
|
28
|
-
| `moda
|
|
29
|
-
| `moda
|
|
30
|
-
| `moda problems` | Rank cross-signal Problems by root cause (what to fix first); takes only `--days-back`/`--limit`. Filter by family with `moda problems
|
|
31
|
-
| `moda
|
|
32
|
-
| `moda
|
|
33
|
-
| `moda
|
|
34
|
-
| `moda
|
|
35
|
-
| `moda laziness` | Laziness detections per trace (`--days-back`, `--limit` 1-50, `--conversation-id`, `--family`) |
|
|
36
|
-
| `moda
|
|
37
|
-
| `moda
|
|
38
|
-
| `moda
|
|
39
|
-
| `moda
|
|
24
|
+
| `moda traces get <conversation_id>` | Windowed message context (max 5 per side); `--msg-index=N` (alias `--message-index`) anchors the window |
|
|
25
|
+
| `moda detections list --family=frustration` | User frustration detections with evidence (the frustration family of `emotions`, legacy field names) |
|
|
26
|
+
| `moda detections list --family=emotion` | Multi-family emotion detections: frustration, disapproval, sadness, confusion, anxiety, trust, positive (`--family=F`, `--user-id`, `--conversation-id`, limit 1–20) |
|
|
27
|
+
| `moda detections list --family=hallucination` | Grounding detections: contradicted/verified outputs, rule breakdown (`--kind=contradicted\|verified`, `--conversation-id=ID`) |
|
|
28
|
+
| `moda detections tools` | Tool failure overview |
|
|
29
|
+
| `moda detections tools <tool_name>` | Per-tool failure breakdown + examples |
|
|
30
|
+
| `moda problems` | Rank cross-signal Problems by root cause (what to fix first); takes only `--days-back`/`--limit`. Filter by family with `moda problems unexplained --family=F` or `moda problems get <problem_id> --traces --family=F` |
|
|
31
|
+
| `moda problems get <problem_id>` | One Problem: dossier, or `--evidence`/`--reports`/`--traces`/`--feedback` pages (`--conversations` is a legacy alias of `--traces`) (`--limit` 1–50, `--cursor` verbatim keyset token) |
|
|
32
|
+
| `moda problems feedback <problem_id>` | Write: `--action=mark_fixed\|dismiss\|flag_attribution\|rename`. `--reason` required for flag_attribution, optional for dismiss; `--new-name` for rename; `--attribution-id` (UUID) required for flag_attribution |
|
|
33
|
+
| `moda review <kind> <target>` | Write: kinds `problem\|problem-detection\|quote\|tool\|laziness\|hallucination\|custom-signal`; `--reason` optional; problem-detection needs `--conversation-id` + `--attribution-id`; quote needs `--family`, `--turn`, `--quote`; tool needs `--tool` plus `--tool-use-id` or `--msg-index`; custom-signal needs `--conversation-id`; `--verdict=clear` removes a mark (not for problem) |
|
|
34
|
+
| `moda reviews list [conversation_id]` | Current false-positive marks: all (`--kind=quote\|tool\|laziness\|hallucination\|custom-signal`, `--all` to include removed, `--limit`) or one trace with its problem detection reviews. Problem dismissals are read with `moda problems get <problem_id> --feedback`, detection reviews with `moda reviews list <conversation_id> --kind=problem-detection` |
|
|
35
|
+
| `moda detections list --family=laziness` | Laziness detections per trace (`--days-back`, `--limit` 1-50, `--conversation-id`, `--family`) |
|
|
36
|
+
| `moda problems linear <problem_id>` | The Problem's linked Linear issue (key, state, URL) or `link: null`; needs Linear connected (Integrations → Linear) |
|
|
37
|
+
| `moda problems link <problem_id>` | Write: file the Problem as a Linear issue (`--team-id`, `--title`, `--priority=0-4`), or link an existing one with `--issue=ENG-123\|URL` (exclusive with the others). Idempotent per Problem |
|
|
38
|
+
| `moda reviews list <conversation_id> --kind=problem-detection` | Problem detections on a trace with review state (`--attribution-id=UUID` narrows to one) |
|
|
39
|
+
| `moda review problem-detection <problem_id>` | Write: `--conversation-id`, `--attribution-id` (UUID), `--verdict=correct\|false_positive\|clear`, optional `--note` |
|
|
40
40
|
| `moda signals` | Custom signals: criteria, status, 7-day judged/matched stats |
|
|
41
41
|
| `moda signal <signal_id>` | One custom signal; `--detections` pages recent hits (`--days-back` 1–90, `--limit` 1–20, `--offset`) |
|
|
42
|
-
| `moda
|
|
43
|
-
| `moda signal
|
|
44
|
-
| `moda
|
|
42
|
+
| `moda detections list --family=custom_signal --conversation-id=<conversation_id>` | Custom signal hits on one trace |
|
|
43
|
+
| `moda review custom-signal <signal_id>` | Write: `--conversation-id`, `--verdict=correct\|false_positive\|clear`, optional `--note`, `--msg-index`/`--msg-dedup-token` |
|
|
44
|
+
| `moda traces state <conversation_id>` | Agent memory: slots/threads/events; `--summary-only`; `--snapshot --msg-index=N` (state at a turn); `--replay --message-count=N` (state over time) |
|
|
45
45
|
| `moda failures` | Tool failures only, ranked (no frustrations or Problems) |
|
|
46
46
|
| `moda tail` | Live tail: one JSON line per new trace/detection (`--signal=traces\|emotions\|all`, legacy alias `--signal=conversations`, `--interval=N`, `--once`, `--max-events=N`). **Emotions caveat:** `/emotions` is ranked by score with no time ordering or cursor, so the tail follows the *highest-scoring* detections rather than everything; each poll emits a `tail_coverage` line with `scanned`/`total`/`coverage_pct`/`complete`. Pass `--full-scan` for a complete window scan (many more requests, capped by the API's offset ceiling of 10000). |
|
|
47
47
|
| `moda feedback "<note>"` | Flag wrong/missing data or CLI quirks to the Moda team |
|
|
@@ -51,13 +51,13 @@ Loaded on demand from the moda-cli skill. This table can lag the CLI: `moda <com
|
|
|
51
51
|
| `moda prompts promote <key>` | Move a remote prompt label |
|
|
52
52
|
| `moda registry template\|validate\|push\|status\|coverage\|log\|show\|prompt` | Register prompts, skills, tools, and agent wiring from one agent-written manifest (`--file`, `--dry-run`, `--live`, `--message`, `--days`); every push is a commit in the Registry history; no repo scan |
|
|
53
53
|
| `moda users` / `moda user <id>` | End users with health/activity (`--page-size=N`, alias `--limit=N`; `--page=N`); one user's profile |
|
|
54
|
-
| `moda
|
|
54
|
+
| `moda problems reopen <pid>` | Reopen a Problem marked fixed |
|
|
55
55
|
| `moda signals <create\|update\|activate\|pause\|delete\|examples\|refine\|revise\|proposals\|accept\|reject\|search\|preview>` | Manage custom signals (writes take `--dry-run`; LLM-backed ones are rate-limited per key) |
|
|
56
56
|
| `moda prompts list\|show\|proposals\|usage\|replay-runs\|decide` | Read the prompt registry; merge/reject proposals |
|
|
57
57
|
| `moda replays sets\|runs\|run\|playouts\|playout` | Replay results: run history, per-seed verdicts, playout transcripts with tool provenance (read-only) |
|
|
58
58
|
| `moda skills list\|show\|policy` / `moda harness list\|show\|versions\|diff` | Skill library and synced harnesses |
|
|
59
59
|
|
|
60
|
-
`moda search` flags: `--mode=keyword|semantic|hybrid` (default `hybrid`),
|
|
60
|
+
`moda traces search` flags: `--mode=keyword|semantic|hybrid` (default `hybrid`),
|
|
61
61
|
`--user-id`, `--time-range=<range>`, `--limit=N` (1–100, default 20).
|
|
62
62
|
|
|
63
63
|
There is no flag set common to every command: each command rejects a flag it
|
|
@@ -5,15 +5,15 @@ Loaded on demand from the moda-cli skill.
|
|
|
5
5
|
### 10. Everything the dashboard does, from the CLI
|
|
6
6
|
|
|
7
7
|
Every command below needs only an API key, and the tenant comes from the key.
|
|
8
|
-
Writes listed here accept `--dry-run` (prints the exact request, sends nothing) except `
|
|
8
|
+
Writes listed here accept `--dry-run` (prints the exact request, sends nothing) except `problems feedback`, `review`, `problems link`, and `prompts promote|ab|propose`. Those refuse `--dry-run` before sending anything, so run them only when you mean it.
|
|
9
9
|
|
|
10
10
|
| Dashboard | CLI |
|
|
11
11
|
|---|---|
|
|
12
12
|
| Traces list: sort, page | `moda traces --sort=recent\|longest\|slowest --offset=N` |
|
|
13
|
-
| Trace detail: whole transcript | `moda
|
|
13
|
+
| Trace detail: whole transcript | `moda traces get <id> --all` (ordered, with tool calls/results) |
|
|
14
14
|
| Users page / user profile | `moda users [--window --cohort --sort --order --health-status]`, `moda user <id>` |
|
|
15
|
-
| Problems: unexplained remainder | `moda problems
|
|
16
|
-
| Reopen a problem marked fixed | `moda
|
|
15
|
+
| Problems: unexplained remainder | `moda problems unexplained [--family=emotion\|tool_failure\|laziness\|… --cursor]` |
|
|
16
|
+
| Reopen a problem marked fixed | `moda problems reopen <problem_id> --reason="…"` (resolved ones return only via re-detection) |
|
|
17
17
|
| Custom signals: create | `moda signals draft --note="…"`, then `moda signals create --name --description --pass-criteria --fail-criteria [--tool-names=a,b]` or `--file=body.json` |
|
|
18
18
|
| Custom signals: edit, lifecycle | `moda signals update <id> …`, `moda signals activate\|pause\|delete <id>` (delete: drafts only; archiving is dashboard-only) |
|
|
19
19
|
| Custom signals: review & tune | `moda signals examples <id> add\|update\|remove --conversation-id=…`, `refine <id> --request="…"`, `revise <id> --conversation-ids=…`, `proposals <id>`, `accept\|reject <id> <proposal_id>` |
|
|
@@ -5,11 +5,11 @@ Loaded on demand from the moda-cli skill.
|
|
|
5
5
|
### 5. Frustration analysis
|
|
6
6
|
|
|
7
7
|
```bash
|
|
8
|
-
moda
|
|
9
|
-
moda
|
|
10
|
-
moda
|
|
11
|
-
moda
|
|
12
|
-
moda
|
|
8
|
+
moda detections list --family=frustration # last 7 days
|
|
9
|
+
moda detections list --family=frustration --days-back=14 --limit=20
|
|
10
|
+
moda detections list --family=frustration --include-window --window=1 --limit=5 # agent output omits windows unless asked
|
|
11
|
+
moda detections list --family=emotion:frustration --user-id=<id> # one end user
|
|
12
|
+
moda detections list --family=emotion --conversation-id=<id> # every family, one trace
|
|
13
13
|
```
|
|
14
14
|
|
|
15
15
|
In agent output, `frustrations` and `emotions` rows leave out the embedded
|
|
@@ -47,10 +47,10 @@ half-width; `--window=1` ⇒ 3 messages total (anchor ± 1), the canonical
|
|
|
47
47
|
### 6. Tool failure investigation
|
|
48
48
|
|
|
49
49
|
```bash
|
|
50
|
-
moda
|
|
51
|
-
moda
|
|
52
|
-
moda
|
|
53
|
-
moda
|
|
50
|
+
moda detections tools # which tools are breaking
|
|
51
|
+
moda detections tools <tool_name> # subtype breakdown + examples
|
|
52
|
+
moda detections tools search --subtype=dependency.timeout --limit=10 # a value from `subtypes`
|
|
53
|
+
moda detections tools search --include-window --window=1 --limit=5
|
|
54
54
|
```
|
|
55
55
|
|
|
56
56
|
Every example row in `tool-failure-detail` output carries a top-level
|
|
@@ -63,7 +63,7 @@ directly instead of matching by `tool_use_id` against the trace.
|
|
|
63
63
|
|
|
64
64
|
By default each row embeds `conversation`: the anchor ± 2 messages (without
|
|
65
65
|
the duplicate `content_blocks`), plus `context_command`, the exact
|
|
66
|
-
`moda
|
|
66
|
+
`moda traces get <conversation_id> --msg-index=<msg_index> --window=2` read for
|
|
67
67
|
the untrimmed window. `--compact` (same as `--include-window=false` or
|
|
68
68
|
`--no-include-window`) drops `conversation` and keeps the anchor and
|
|
69
69
|
`context_command`.
|
|
@@ -39,7 +39,7 @@ moda feedback "trace is a refund request, not this problem" \
|
|
|
39
39
|
`--message-index`); they go into `refs` and let the team reproduce what
|
|
40
40
|
you saw.
|
|
41
41
|
- Feedback does not change the problem itself. To act on a misroute directly,
|
|
42
|
-
use `moda
|
|
43
|
-
--attribution-id=<uuid> --reason=TEXT` or `moda
|
|
42
|
+
use `moda problems feedback <problem_id> --action=flag_attribution
|
|
43
|
+
--attribution-id=<uuid> --reason=TEXT` or `moda review <kind> <target>
|
|
44
44
|
--reason=TEXT`.
|
|
45
45
|
- Never include secrets or full transcripts in the note.
|
|
@@ -116,7 +116,7 @@ Fields an agent should read:
|
|
|
116
116
|
- `summary.text` + `summary.confidence` — the answer and how much to trust it
|
|
117
117
|
(`high | medium | low | unknown`).
|
|
118
118
|
- `evidence_refs[]` — each `{ id, kind, label, path }`. When `path` looks like
|
|
119
|
-
`conversation/<id>`, open it with `moda
|
|
119
|
+
`conversation/<id>`, open it with `moda traces get <id>`.
|
|
120
120
|
- `next_commands[]` — suggested follow-up commands (each carries `mutability`
|
|
121
121
|
and `requires_approval`).
|
|
122
122
|
- `warnings[]`, `errors[]`.
|
|
@@ -179,7 +179,7 @@ Treat an answer as degraded if **any** of these hold:
|
|
|
179
179
|
items are dropped from `data`. When rows are dropped, the list's
|
|
180
180
|
`pagination` says what you got (`returned`, `limit`, `next_offset`; a
|
|
181
181
|
`next_cursor` that would skip rows becomes `null`) and `next_commands` has
|
|
182
|
-
the exact command for the rest (e.g. `moda
|
|
182
|
+
the exact command for the rest (e.g. `moda detections list --family=frustration --offset=3
|
|
183
183
|
--limit=7`). `truncation.data.full_output_path` (also an `artifacts[]` entry of
|
|
184
184
|
kind `full_output`) is that file; read it in line ranges or with `jq`.
|
|
185
185
|
These files are pruned after 24 hours (`MODA_AGENT_OUTPUT_RETENTION_HOURS`).
|
|
@@ -43,7 +43,7 @@ Data API fallback. Abbreviated envelope:
|
|
|
43
43
|
{ "id": "tool:lookupCustomer", "kind": "tool_failure", "label": "lookupCustomer: 12 failure(s)", "path": null }
|
|
44
44
|
],
|
|
45
45
|
"next_commands": [
|
|
46
|
-
{ "command": "moda
|
|
46
|
+
{ "command": "moda detections tools lookupCustomer --days-back=7 --include-window", "purpose": "Inspect tool failure examples and trace anchors.", "mutability": "read", "requires_approval": false }
|
|
47
47
|
],
|
|
48
48
|
"warnings": ["Cloud ask endpoint unavailable; synthesized answer from local Data API evidence."],
|
|
49
49
|
"errors": [],
|
|
@@ -55,7 +55,7 @@ Data API fallback. Abbreviated envelope:
|
|
|
55
55
|
narrows the tool-failure evidence to one tool, and `--trace` (legacy alias
|
|
56
56
|
`--conversation`) scopes only the frustration evidence (overview, tool
|
|
57
57
|
failures and Problems stay tenant-wide). Its findings include the top 1–3
|
|
58
|
-
Problems from `moda problems`, with `moda
|
|
58
|
+
Problems from `moda problems`, with `moda problems get <id>` as the next command.
|
|
59
59
|
`moda failures` is the tool-failures-only view.
|
|
60
60
|
|
|
61
61
|
After reporting the top behavioral failure and the harness layer it routes to
|
|
@@ -73,14 +73,14 @@ Problem, or before hand-editing prompt/skill files in a repo that talks to Moda.
|
|
|
73
73
|
moda investigate # rank tool failures, frustrations,
|
|
74
74
|
# and top Problems with evidence
|
|
75
75
|
moda problems # cross-signal Problems by root cause
|
|
76
|
-
moda
|
|
77
|
-
moda
|
|
78
|
-
moda
|
|
79
|
-
moda
|
|
80
|
-
moda
|
|
81
|
-
moda
|
|
76
|
+
moda problems get <problem_id> # compact dossier: what, where, how often
|
|
77
|
+
moda problems get <problem_id> --full # every dossier block (large)
|
|
78
|
+
moda problems get <problem_id> --evidence # attribution rationales and anchors
|
|
79
|
+
moda problems get <problem_id> --traces # affected traces with anchors
|
|
80
|
+
moda detections tools <tool_name> # failing calls with surrounding turns
|
|
81
|
+
moda traces get <conversation_id> --window=3 # the messages around one turn
|
|
82
82
|
# ...propose and make the change, citing the problem/trace ids in the PR...
|
|
83
|
-
moda
|
|
83
|
+
moda problems feedback <problem_id> --action=mark_fixed # after it ships
|
|
84
84
|
```
|
|
85
85
|
|
|
86
86
|
Rules that keep the fix grounded:
|
|
@@ -90,7 +90,8 @@ Rules that keep the fix grounded:
|
|
|
90
90
|
- Cite the Moda problem id and the trace ids you relied on in the PR
|
|
91
91
|
description, so a reviewer can open the same evidence.
|
|
92
92
|
- To test a prompt change against real traffic before shipping it, A/B it with
|
|
93
|
-
`moda prompts ab --baseline=<path> --candidate=<path> --
|
|
94
|
-
|
|
93
|
+
`moda prompts ab --baseline=<path> --candidate=<path> --problem=<problem_id>`
|
|
94
|
+
(replays the Problem's failing traces from their evidence anchors) or
|
|
95
|
+
`--traces=<ids>` (see section 7).
|
|
95
96
|
- Mark the Problem fixed only after the change ships. If the issue is detected
|
|
96
|
-
again, the Problem comes back; `moda
|
|
97
|
+
again, the Problem comes back; `moda problems reopen` reopens it by hand.
|
|
@@ -72,7 +72,10 @@ read, they fail (exit 1) with the number and the exact command to re-run with `-
|
|
|
72
72
|
Only add `--yes` when the user has agreed to that spend. Limits: `--cases`
|
|
73
73
|
default 5, max 100; `--seeds` default 3, max 10; `--traces` at most 100
|
|
74
74
|
(`--traces=conv_id@412` replays that trace from msg_index 412, with the
|
|
75
|
-
earlier turns as history);
|
|
75
|
+
earlier turns as history, and writes its case from the messages around 412);
|
|
76
|
+
`--problem=<problem_id>` picks a Problem's affected traces, each anchored at
|
|
77
|
+
its earliest evidence msg_index, up to
|
|
78
|
+
`--max-cases` (default 10, max 100), and prints them (`problemCases`);
|
|
76
79
|
`--model` / `--assistant-model` must be one of `openai/gpt-5.6-luna`,
|
|
77
80
|
`openai/gpt-4o-mini`, `openai/gpt-4o`, `anthropic/claude-sonnet-4-5`,
|
|
78
81
|
`anthropic/claude-haiku-4-5`, `google/gemini-2.5-flash`,
|
|
@@ -80,7 +83,8 @@ earlier turns as history);
|
|
|
80
83
|
|
|
81
84
|
`prompts ab --traces=<ids>` builds one case per trace from Moda's distilled
|
|
82
85
|
scenario and success criteria for that trace. A trace Moda cannot distill
|
|
83
|
-
falls back to its opening user message
|
|
86
|
+
falls back to its opening user message (for `id@N`, the user message leading
|
|
87
|
+
into N) with three generic criteria (which long
|
|
84
88
|
traces pass easily); the run warns and lists it under `caseSources.fallback`.
|
|
85
89
|
`prompts ab` never makes its replay set the tenant's primary set unless you
|
|
86
90
|
pass `--promote`.
|
|
@@ -8,43 +8,43 @@ Loaded on demand from the moda-cli skill.
|
|
|
8
8
|
|
|
9
9
|
```bash
|
|
10
10
|
# Semantic: find the moment a user expressed an intent, in their own words
|
|
11
|
-
moda search "user wants to cancel subscription" --mode=semantic --limit=5 --json \
|
|
11
|
+
moda traces search "user wants to cancel subscription" --mode=semantic --limit=5 --json \
|
|
12
12
|
| jq -r '.results[] | "\(.conversation_id) @\(.message_index) \(.snippet)"'
|
|
13
13
|
|
|
14
14
|
# Keyword: locate every tool call/result mentioning an exact identifier
|
|
15
|
-
moda search "stripe.refunds.create" --mode=keyword --limit=10
|
|
15
|
+
moda traces search "stripe.refunds.create" --mode=keyword --limit=10
|
|
16
16
|
|
|
17
17
|
# Search, then open the top message-level hit at the exact turn (skips
|
|
18
18
|
# trace-level fallback hits, which have no message_index)
|
|
19
|
-
moda search "rate limit exceeded" --limit=5 --json \
|
|
19
|
+
moda traces search "rate limit exceeded" --limit=5 --json \
|
|
20
20
|
| jq -r 'first(.results[] | select(.message_index != null)) | "\(.conversation_id) \(.message_index)"' \
|
|
21
|
-
| xargs -r -n2 sh -c 'moda
|
|
21
|
+
| xargs -r -n2 sh -c 'moda traces get "$0" --msg-index="$1"'
|
|
22
22
|
```
|
|
23
23
|
|
|
24
24
|
### Daily health check
|
|
25
25
|
|
|
26
26
|
```bash
|
|
27
27
|
moda overview --days-back=1
|
|
28
|
-
moda
|
|
29
|
-
moda
|
|
28
|
+
moda detections list --family=frustration --days-back=1 --limit=5
|
|
29
|
+
moda detections tools --days-back=1
|
|
30
30
|
```
|
|
31
31
|
|
|
32
32
|
### Debug a frustrated user
|
|
33
33
|
|
|
34
34
|
```bash
|
|
35
|
-
moda
|
|
35
|
+
moda detections list --family=frustration --days-back=7 --limit=10
|
|
36
36
|
# Each row has an anchor: {conversation_id, msg_index}. Open that turn
|
|
37
37
|
# (without --msg-index, context centers on the middle of the trace):
|
|
38
|
-
moda
|
|
38
|
+
moda traces get <anchor.conversation_id> --msg-index=<anchor.msg_index> --window=3
|
|
39
39
|
```
|
|
40
40
|
|
|
41
41
|
### Diagnose a flaky tool
|
|
42
42
|
|
|
43
43
|
```bash
|
|
44
|
-
moda
|
|
45
|
-
moda
|
|
44
|
+
moda detections tools
|
|
45
|
+
moda detections tools <tool_name>
|
|
46
46
|
# Pick an example conversation_id from the output
|
|
47
|
-
moda
|
|
47
|
+
moda traces get <conversation_id>
|
|
48
48
|
```
|
|
49
49
|
|
|
50
50
|
### Explore what users are doing
|
|
@@ -62,13 +62,13 @@ envelope, where the payload sits under `.data`.
|
|
|
62
62
|
|
|
63
63
|
```bash
|
|
64
64
|
# Pull primary causes of recent frustrations
|
|
65
|
-
moda
|
|
65
|
+
moda detections list --family=frustration --days-back=7 --json | jq -r '.frustrations[].primary_cause'
|
|
66
66
|
|
|
67
67
|
# Get trace IDs for a search and fetch context for all of them in one call
|
|
68
68
|
moda traces --search="timeout" --limit=3 --json \
|
|
69
69
|
| jq -r '.conversations[].conversation_id' \
|
|
70
|
-
| moda
|
|
70
|
+
| moda traces get --ids-from=-
|
|
71
71
|
|
|
72
72
|
# Tool failure leaderboard
|
|
73
|
-
moda
|
|
73
|
+
moda detections tools --json | jq '.tools[] | {tool: .tool_name, failures: .failure_count}'
|
|
74
74
|
```
|
|
@@ -5,12 +5,12 @@ Loaded on demand from the moda-cli skill.
|
|
|
5
5
|
### 1. Search messages (primary content-discovery path)
|
|
6
6
|
|
|
7
7
|
```bash
|
|
8
|
-
moda search "refund flow timeout" # hybrid (default)
|
|
9
|
-
moda search "customer asked for a refund" --mode=semantic
|
|
10
|
-
moda search "SEARCH_NO_RESULTS" --mode=keyword
|
|
11
|
-
moda search "checkout error" --time-range=7d --limit=10
|
|
12
|
-
moda search "stripe.charges.create failed" --user-id=<id>
|
|
13
|
-
moda search "Иван" --conversation-id=<id> --mode=keyword # inside one trace
|
|
8
|
+
moda traces search "refund flow timeout" # hybrid (default)
|
|
9
|
+
moda traces search "customer asked for a refund" --mode=semantic
|
|
10
|
+
moda traces search "SEARCH_NO_RESULTS" --mode=keyword
|
|
11
|
+
moda traces search "checkout error" --time-range=7d --limit=10
|
|
12
|
+
moda traces search "stripe.charges.create failed" --user-id=<id>
|
|
13
|
+
moda traces search "Иван" --conversation-id=<id> --mode=keyword # inside one trace
|
|
14
14
|
```
|
|
15
15
|
|
|
16
16
|
Searches trace messages at message grain. Pass `--include-tool-io` to
|
|
@@ -34,7 +34,7 @@ trails ingestion (`index_freshness`); a warning says so when your
|
|
|
34
34
|
(same as `moda traces --search`), those traces come back in `results` with
|
|
35
35
|
`match_grain: "trace"`, plus `fallback.source: "traces_search"` and a warning.
|
|
36
36
|
These rows have no score or message index. Open one with
|
|
37
|
-
`moda
|
|
37
|
+
`moda traces spans <conversation_id>` or `moda traces get <conversation_id>` to find the
|
|
38
38
|
matching turn. When more traces exist, `fallback.has_more` is true and
|
|
39
39
|
`next_commands` includes the `moda traces --search ... --offset` command to
|
|
40
40
|
page. With `--mode=keyword`, ranked traces are not substituted. If the fallback
|
|
@@ -58,16 +58,16 @@ embedding provider is reachable, so check it before trusting semantic ranking.
|
|
|
58
58
|
`score` scales differ per mode (see above) — branch on `search_mode`, never
|
|
59
59
|
compare scores across modes.
|
|
60
60
|
|
|
61
|
-
Hand the anchors straight to `moda
|
|
61
|
+
Hand the anchors straight to `moda traces get`:
|
|
62
62
|
|
|
63
63
|
```bash
|
|
64
64
|
# Find, then read the exact turn. --json prints the bare payload; piped output
|
|
65
65
|
# without it is the agent envelope, where results sit under .data.results.
|
|
66
|
-
hit=$(moda search "double charge" --limit=1 --json | jq -c '.results[0] // empty')
|
|
66
|
+
hit=$(moda traces search "double charge" --limit=1 --json | jq -c '.results[0] // empty')
|
|
67
67
|
cid=$(echo "$hit" | jq -r '.conversation_id')
|
|
68
68
|
# Trace-level fallback hits (match_grain: "trace") have no message_index.
|
|
69
69
|
idx=$(echo "$hit" | jq -r '.message_index // empty')
|
|
70
|
-
moda
|
|
70
|
+
moda traces get "$cid" ${idx:+--msg-index="$idx"}
|
|
71
71
|
```
|
|
72
72
|
|
|
73
73
|
### 3. Browse topic clusters
|
|
@@ -81,7 +81,7 @@ moda cluster-traces <node_id> # traces in cluster
|
|
|
81
81
|
### 3b. List / filter traces (structured, not semantic)
|
|
82
82
|
|
|
83
83
|
Use `moda traces` to enumerate or filter by structured fields — not to
|
|
84
|
-
search by meaning (use `moda search` for that). The `--search` flag here
|
|
84
|
+
search by meaning (use `moda traces search` for that). The `--search` flag here
|
|
85
85
|
ranks whole traces: hybrid (keyword + semantic) when embeddings exist, else
|
|
86
86
|
keyword. Check `total_kind`: `candidate_pool` means results are related traces
|
|
87
87
|
that may not contain the words and `pagination.total` is not a match count;
|
|
@@ -100,19 +100,19 @@ Filters: `--search`, `--cluster-id`, `--user-id`, `--time-range`
|
|
|
100
100
|
### 4. Read a single trace
|
|
101
101
|
|
|
102
102
|
```bash
|
|
103
|
-
moda
|
|
104
|
-
moda
|
|
105
|
-
moda
|
|
106
|
-
moda
|
|
107
|
-
moda
|
|
108
|
-
moda
|
|
109
|
-
moda
|
|
103
|
+
moda traces get <conversation_id> # default window around middle
|
|
104
|
+
moda traces get <conversation_id> --msg-index=5 # center on message 5
|
|
105
|
+
moda traces get <conversation_id> --window=3 # 3 messages each side (max 5)
|
|
106
|
+
moda traces get <conversation_id> --msg-index=40 --window=0 # just message 40 + total_messages
|
|
107
|
+
moda traces get <conversation_id> --metadata-only # summary + total_messages, no messages
|
|
108
|
+
moda traces get <conversation_id> --all # whole trace in order, first 500 messages
|
|
109
|
+
moda traces get <conversation_id> --all --from=500 --max-messages=500 # continue (max 5000)
|
|
110
110
|
```
|
|
111
111
|
|
|
112
112
|
`--all` returns `has_more`/`next_from`; when it stops at the cap, the warning
|
|
113
113
|
prints the exact `--from` command to continue.
|
|
114
114
|
|
|
115
|
-
**Several traces? Batch them in one call instead of looping `moda
|
|
115
|
+
**Several traces? Batch them in one call instead of looping `moda traces get`
|
|
116
116
|
per id.** Pass up to 25 targets as positionals or one per line with
|
|
117
117
|
`--ids-from=-` (stdin) or `--ids-from=<file>`. A target is `<id>` or
|
|
118
118
|
`<id>@<msg_index>` (the anchor; ids may contain `:`), and the shared flags
|
|
@@ -121,8 +121,8 @@ be JSON (`{"id":"...","msg_index":N}`), the only spelling for an id that itself
|
|
|
121
121
|
ends in `@<digits>`.
|
|
122
122
|
|
|
123
123
|
```bash
|
|
124
|
-
moda
|
|
125
|
-
printf '%s\n' telegram:8565459080@12 web:conv-9 | moda
|
|
124
|
+
moda traces get telegram:8565459080@12 linq:c6f88988-aa@3 web:conv-9 --window=2
|
|
125
|
+
printf '%s\n' telegram:8565459080@12 web:conv-9 | moda traces get --ids-from=- --window=1
|
|
126
126
|
```
|
|
127
127
|
|
|
128
128
|
A batch returns `data.results: [{ target, ok, data | error }]` in input order,
|
|
@@ -132,29 +132,29 @@ every target failed. When the agent byte budget would overflow, the items that
|
|
|
132
132
|
fit are returned and `remaining_targets` plus a `next_commands` entry read the
|
|
133
133
|
rest. One positional id keeps the single-trace output above.
|
|
134
134
|
|
|
135
|
-
`moda
|
|
135
|
+
`moda traces get` returns messages whole, with their tool calls and results. An
|
|
136
136
|
older Moda server cuts message text, tool inputs, and tool results at 500
|
|
137
137
|
characters; the response warns when that server is in use.
|
|
138
138
|
`--window` is at most 5 messages per side. For the raw,
|
|
139
139
|
complete picture of what was captured — every span, the parent/child hierarchy,
|
|
140
|
-
tool calls, prompt/response bodies, and `gen_ai.*` attributes — use `moda
|
|
140
|
+
tool calls, prompt/response bodies, and `gen_ai.*` attributes — use `moda traces spans`.
|
|
141
141
|
|
|
142
142
|
### 4b. Audit raw spans (completeness, orphans, duplicates)
|
|
143
143
|
|
|
144
144
|
```bash
|
|
145
|
-
moda
|
|
146
|
-
moda
|
|
147
|
-
moda
|
|
148
|
-
moda
|
|
149
|
-
moda
|
|
150
|
-
moda
|
|
151
|
-
moda
|
|
145
|
+
moda traces spans <conversation_id|trace_id> # auto-detects OTLP trace_id vs trace ID (conversation_id)
|
|
146
|
+
moda traces spans <trace_id> --kind=trace # force OTLP trace_id lookup
|
|
147
|
+
moda traces spans <conversation_id> --include-raw # attach verbatim raw_event bodies
|
|
148
|
+
moda traces spans <conversation_id> --include-bodies # keep prompt/response/attribute bodies
|
|
149
|
+
moda traces spans <conversation_id> --limit=1000 # ingest rows to read (default 200, max 1000)
|
|
150
|
+
moda traces spans <conversation_id> --cursor=<next_cursor> # next page of a capped audit
|
|
151
|
+
moda traces spans <id1> <id2> <id3> # batch: up to 25 ids, one envelope
|
|
152
152
|
```
|
|
153
153
|
|
|
154
154
|
Auditing several traces? Batch the ids (positionals or `--ids-from=-|<file>`)
|
|
155
|
-
instead of looping `moda
|
|
155
|
+
instead of looping `moda traces spans`: the result is `data.results: [{ target, ok,
|
|
156
156
|
data | error }]` with the same per-item errors and byte-budget rules as batch
|
|
157
|
-
`moda
|
|
157
|
+
`moda traces get`. Ids are taken verbatim (no `@` anchors); `--cursor` pages one
|
|
158
158
|
trace, so follow each item's `next_commands` for it.
|
|
159
159
|
|
|
160
160
|
`truncated` / `has_more` at the top level say whether the audit is complete.
|
|
@@ -7,23 +7,23 @@ Loaded on demand from the moda-cli skill.
|
|
|
7
7
|
**Mark anything as a false positive with one command.** `--reason` (why the
|
|
8
8
|
detection is wrong) is optional but recommended; marks are logged for Moda's review.
|
|
9
9
|
A marked item disappears from every list and count (`moda problems`,
|
|
10
|
-
`
|
|
11
|
-
`--
|
|
10
|
+
`detections list`, `detections tools`, problem evidence, the dashboard);
|
|
11
|
+
`--verdict=clear` brings it back.
|
|
12
12
|
|
|
13
13
|
```bash
|
|
14
|
-
moda
|
|
15
|
-
moda
|
|
14
|
+
moda review problem <problem_id> --reason="two unrelated tools" # dismiss: hidden at once
|
|
15
|
+
moda review problem-detection <problem_id> --conversation-id=<id> \
|
|
16
16
|
--attribution-id=<uuid> --reason="refund was approved" # one piece of evidence
|
|
17
|
-
moda
|
|
17
|
+
moda review quote <conversation_id> --family=frustration --turn=3 \
|
|
18
18
|
--quote="this is useless lol" --reason="user was joking" # one user quote
|
|
19
|
-
moda
|
|
19
|
+
moda review tool <conversation_id> --tool=send_invoice \
|
|
20
20
|
--tool-use-id=<tool_use_id> --reason="retried upstream" # one failing tool call
|
|
21
|
-
moda
|
|
22
|
-
moda
|
|
23
|
-
moda
|
|
24
|
-
moda
|
|
25
|
-
moda
|
|
26
|
-
moda
|
|
21
|
+
moda review laziness <conversation_id> --reason="reload is expected"
|
|
22
|
+
moda review hallucination <conversation_id> --reason="price came from the tool result"
|
|
23
|
+
moda review custom-signal <signal_id> --conversation-id=<id> --reason="refund was approved"
|
|
24
|
+
moda review laziness <conversation_id> --verdict=clear # remove a mark (any kind but problem)
|
|
25
|
+
moda reviews list # every current mark (--kind=tool, --all, --limit)
|
|
26
|
+
moda reviews list <conversation_id> # marks + problem reviews on a trace
|
|
27
27
|
```
|
|
28
28
|
|
|
29
29
|
What each flag does downstream:
|
|
@@ -36,12 +36,12 @@ What each flag does downstream:
|
|
|
36
36
|
| `quote`, `tool`, `laziness`, `hallucination` | Audit only: recorded for review, never fed back to the detectors |
|
|
37
37
|
|
|
38
38
|
Find targets with the read commands: `--family`, `--turn` and `--quote` verbatim
|
|
39
|
-
from a `user_quotes` entry in `moda
|
|
40
|
-
`moda
|
|
41
|
-
laziness traces from `moda laziness`; hallucination traces from
|
|
42
|
-
`moda
|
|
43
|
-
`moda
|
|
44
|
-
`moda
|
|
39
|
+
from a `user_quotes` entry in `moda detections list --family=emotion`; a failing call's `tool_use_id` (or `msg_index` when it has none) from
|
|
40
|
+
`moda detections tools <tool_name>`;
|
|
41
|
+
laziness traces from `moda detections list --family=laziness`; hallucination traces from
|
|
42
|
+
`moda detections list --family=hallucination`; custom-signal hits from `moda detections list --family=custom_signal` or
|
|
43
|
+
`moda detections list --family=custom_signal --signal-id=<id>`; problem evidence ids from
|
|
44
|
+
`moda problems get <id> --evidence`.
|
|
45
45
|
|
|
46
46
|
**Custom signals** are tenant-defined behaviors (pass/fail criteria) judged on
|
|
47
47
|
every trace. Read them, then label hits so each signal has labelled examples:
|
|
@@ -49,9 +49,9 @@ every trace. Read them, then label hits so each signal has labelled examples:
|
|
|
49
49
|
```bash
|
|
50
50
|
moda signals # every signal: criteria, status, 7-day judged/matched
|
|
51
51
|
moda signal <signal_id> # one signal's summary
|
|
52
|
-
moda signal
|
|
53
|
-
moda
|
|
54
|
-
moda signal
|
|
52
|
+
moda detections list --family=custom_signal --signal-id=<signal_id> --days-back=30 # recent hits (limit 1–20, --offset to page)
|
|
53
|
+
moda detections list --family=custom_signal --conversation-id=<conversation_id> # signal hits on one trace
|
|
54
|
+
moda review custom-signal <signal_id> --conversation-id=<id> --verdict=false_positive \
|
|
55
55
|
--note="refund was approved" --msg-index=<anchor_msg_index>
|
|
56
56
|
```
|
|
57
57
|
|
|
@@ -62,18 +62,18 @@ message, and always give a `--note` when rejecting. Labels are throttled to 10
|
|
|
62
62
|
per minute per API key and are never retried automatically.
|
|
63
63
|
|
|
64
64
|
**Problem detections** (the dashboard's flag on problem evidence) are reviewed with
|
|
65
|
-
`detection
|
|
66
|
-
`moda
|
|
65
|
+
`moda review problem-detection`. A `false_positive` drops the detection out of
|
|
66
|
+
`moda problems get <id> --evidence` and is fed back to the problem reader; `clear`
|
|
67
67
|
undoes an earlier review.
|
|
68
68
|
|
|
69
69
|
```bash
|
|
70
|
-
moda
|
|
71
|
-
moda
|
|
70
|
+
moda reviews list <conversation_id> --kind=problem-detection # problem detections on a trace + review state
|
|
71
|
+
moda review problem-detection <problem_id> --conversation-id=<id> \
|
|
72
72
|
--attribution-id=<uuid> --verdict=false_positive --note="retry was expected"
|
|
73
|
-
moda
|
|
73
|
+
moda review problem-detection <problem_id> --conversation-id=<id> \
|
|
74
74
|
--attribution-id=<uuid> --verdict=clear # undo
|
|
75
75
|
```
|
|
76
76
|
|
|
77
|
-
`attribution_id` comes from `moda
|
|
78
|
-
`moda
|
|
79
|
-
`
|
|
77
|
+
`attribution_id` comes from `moda reviews list <conversation_id> --kind=problem-detection` or
|
|
78
|
+
`moda problems get <problem_id> --evidence`. Prefer `review problem-detection` over
|
|
79
|
+
`problems feedback --action=flag_attribution` for a single wrong detection.
|
|
@@ -14,9 +14,9 @@ Loaded on demand from the moda-cli skill.
|
|
|
14
14
|
mode prints prose to stderr. Check the exit code before claiming success.
|
|
15
15
|
- An unknown trace, problem, or cluster id exits 1 with
|
|
16
16
|
`errors[0].code: "not_found"`; it is not an empty success.
|
|
17
|
-
- `moda
|
|
17
|
+
- `moda traces state` returns success with a warning when state is missing.
|
|
18
18
|
This can mean an unknown trace or a real trace without generated state.
|
|
19
|
-
Use `moda
|
|
19
|
+
Use `moda traces get <id> --metadata-only` to check whether the trace exists.
|
|
20
20
|
- If the JSON has `"conversations": []` or `"frustrations": []`, the call
|
|
21
21
|
worked but the time window is empty. Widen the window once
|
|
22
22
|
(`--days-back` on detection commands, `--time-range` on `traces`,
|
|
@@ -36,7 +36,7 @@ save it with `moda auth login --api-key`.
|
|
|
36
36
|
**`not_found` (exit 1)**
|
|
37
37
|
The trace, problem, or cluster id does not exist in this tenant (a trace with
|
|
38
38
|
no generated world state returns a warning, not `not_found`).
|
|
39
|
-
`errors[0].cause` names the listing command: `moda search`/`moda traces`,
|
|
39
|
+
`errors[0].cause` names the listing command: `moda traces search`/`moda traces`,
|
|
40
40
|
`moda problems`, `moda clusters`. A tool name with no failures in the window
|
|
41
41
|
returns an empty result with a warning instead.
|
|
42
42
|
|
|
@@ -52,7 +52,7 @@ warning that gives the `--all` command for a wider slice.
|
|
|
52
52
|
Widen the window: try `--days-back=30` or `--time-range=30d`. Confirm with
|
|
53
53
|
`moda overview` that data exists in the tenant at all.
|
|
54
54
|
|
|
55
|
-
**`moda search` returns nothing, or semantic feels like keyword**
|
|
55
|
+
**`moda traces search` returns nothing, or semantic feels like keyword**
|
|
56
56
|
The default `--time-range` is `all`, so an empty result usually means the
|
|
57
57
|
phrasing, not the window: try a shorter phrase or the exact identifier with
|
|
58
58
|
`--mode=keyword`, and add `--include-tool-io` for strings that only appear in
|