@moda-ai/cli 1.55.22 → 1.55.24
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/dist/{agent-mj622k8s.js → agent-b4tdzy9q.js} +8 -9
- package/dist/{cli-9x9adrtj.js → cli-0wr3483h.js} +1 -1
- package/dist/{cli-a5c9te95.js → cli-7xc585hc.js} +5 -5
- package/dist/{cli-rqk6f6n6.js → cli-9btx12wp.js} +24 -5
- package/dist/cli-9hc5bg2r.js +61 -0
- package/dist/{cli-cehskjde.js → cli-dn91fb56.js} +13 -15
- package/dist/{cli-5ke3xe6y.js → cli-dve50x7p.js} +258 -28
- package/dist/{cli-569hj0bj.js → cli-qvg26twp.js} +5 -4
- package/dist/{cli-033n2mpb.js → cli-qycsmxev.js} +6 -9
- package/dist/{cli-0ef72tqx.js → cli-rt26sbz7.js} +14 -14
- package/dist/{cli-pc5cw7mx.js → cli-vg88mg20.js} +5 -5
- package/dist/{cli-wf7gfa1e.js → cli-y2ty9vtv.js} +8 -10
- package/dist/cli.js +1310 -346
- package/dist/{doctor-mx6jf435.js → doctor-3r19ry92.js} +8 -9
- package/dist/{fix-9fmy6p2x.js → fix-exvxzvtz.js} +6 -7
- package/dist/{harness-github-actions-c826bfxs.js → harness-github-actions-atc5bbkj.js} +5 -6
- package/dist/{harness-xvg6ws6x.js → harness-vzy7n7ea.js} +5 -6
- package/dist/{index-1wbjz3jc.js → index-e1pf1qz2.js} +17 -17
- package/dist/{production-intelligence-dbe9dmc5.js → production-intelligence-c8e0xw3n.js} +17 -16
- package/dist/{profile-commands-kmmd9y86.js → profile-commands-6e2dgv72.js} +7 -48
- package/dist/prompt-source-mcp.js +5 -6
- package/dist/{prompts-1tkhrxkq.js → prompts-cvzt994j.js} +6 -7
- package/dist/{provision-wzttap58.js → provision-q86phh4a.js} +5 -4
- package/dist/{registry-atqyb5fa.js → registry-s2vdsfw8.js} +3 -4
- package/dist/{skills-gejdvjak.js → skills-tadh1jss.js} +5 -6
- package/dist/{sync-57ercmh9.js → sync-g8wtecz1.js} +7 -8
- package/package.json +1 -1
- package/skills/integration/index.json +2 -2
- package/skills/moda-cli/SKILL.md +31 -29
- package/skills/moda-cli/references/commands.md +22 -22
- package/skills/moda-cli/references/dashboard-parity.md +4 -4
- package/skills/moda-cli/references/detections.md +10 -10
- package/skills/moda-cli/references/feedback.md +2 -2
- package/skills/moda-cli/references/output-protocol.md +2 -2
- package/skills/moda-cli/references/production-intelligence.md +10 -10
- package/skills/moda-cli/references/recipes.md +14 -14
- package/skills/moda-cli/references/search-and-traces.md +32 -32
- package/skills/moda-cli/references/signals-and-false-positives.md +29 -29
- package/skills/moda-cli/references/troubleshooting.md +4 -4
- package/dist/cli-7ea02wca.js +0 -30
package/skills/moda-cli/SKILL.md
CHANGED
|
@@ -1,16 +1,18 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: moda-cli
|
|
3
3
|
version: 3.0.0
|
|
4
|
-
description: Investigate a production AI agent with the Moda CLI (`moda`) - find where something happened in real traces (search, then
|
|
4
|
+
description: Investigate a production AI agent with the Moda CLI (`moda`) - find where something happened in real traces (traces search, then traces get), check agent health, rank what to fix (investigate, problems), and dig into tool failures, user frustration, and hallucinations. Also prompt registry sync/promote, replays, custom signals, and false-positive marking. Use when the user mentions Moda, production traces or conversations of their agent, agent failures, or asks what to fix.
|
|
5
5
|
---
|
|
6
6
|
|
|
7
7
|
# Moda CLI
|
|
8
8
|
|
|
9
9
|
`moda` queries one Moda tenant's production analytics. A **trace** is one agent
|
|
10
10
|
run: every message, tool call, and step sharing one `conversation_id` (older
|
|
11
|
-
names: "conversation"; `moda conversations` is an alias of `moda traces`).
|
|
12
|
-
Analytics commands are read-only.
|
|
13
|
-
|
|
11
|
+
names: "conversation"; `moda conversations` is an alias of `moda traces list`).
|
|
12
|
+
Analytics commands are read-only. Commands are `moda <noun> <verb>` (`traces`,
|
|
13
|
+
`problems`, `detections`, `review`); older names such as `moda context` still
|
|
14
|
+
work but warn (`meta.deprecated`), so use the forms below. Detailed guides live
|
|
15
|
+
in `references/`; read one only when the task needs it.
|
|
14
16
|
|
|
15
17
|
## Before the first command
|
|
16
18
|
|
|
@@ -23,7 +25,7 @@ one only when the task needs it.
|
|
|
23
25
|
- Not installed: `npx -y -p @moda-ai/cli moda <command>`.
|
|
24
26
|
- If a response carries `meta.update`, the installed CLI is out of date and
|
|
25
27
|
may not match this guide; tell the user how to update.
|
|
26
|
-
- Flags: `moda <
|
|
28
|
+
- Flags: `moda <noun> <verb> --help` lists every flag the command accepts, with
|
|
27
29
|
types, ranges, and defaults. Each command rejects flags it does not take and
|
|
28
30
|
the error lists the valid ones; read that list instead of guessing.
|
|
29
31
|
|
|
@@ -53,74 +55,74 @@ Details: `references/output-protocol.md`.
|
|
|
53
55
|
|
|
54
56
|
| Goal | First command | Then |
|
|
55
57
|
|---|---|---|
|
|
56
|
-
| Where did X happen / traces about Y | `moda search "<words>"` (`--mode=keyword` for ids, error codes, exact strings; `--include-tool-io` for tool inputs/outputs) | `moda
|
|
58
|
+
| Where did X happen / traces about Y | `moda traces search "<words>"` (`--mode=keyword` for ids, error codes, exact strings; `--include-tool-io` for tool inputs/outputs) | `moda traces get <conversation_id> --msg-index=<message_index> --window=3` |
|
|
57
59
|
| Search returned nothing | Read `.warnings` (degraded mode, reason). Try a shorter phrase, `--mode=keyword`, a wider `--time-range` | `moda traces --search="<words>"` |
|
|
58
60
|
| Is the agent healthy? | `moda overview --days-back=7` | `moda investigate` |
|
|
59
|
-
| What to fix first? | `moda investigate` or `moda problems` (free) | `moda
|
|
60
|
-
| Why is tool T failing? | `moda
|
|
61
|
-
| What frustrates users? | `moda
|
|
62
|
-
| Other emotions, laziness, hallucinations | `moda
|
|
63
|
-
| Read one whole trace | `moda
|
|
64
|
-
| Read several traces or turns | `moda
|
|
65
|
-
| One user's experience | `moda users`, `moda user <user_id>` | `moda search "<words>" --user-id=<id>` |
|
|
61
|
+
| What to fix first? | `moda investigate` or `moda problems` (free) | `moda problems get <problem_id>`, then `--evidence` / `--traces` |
|
|
62
|
+
| Why is tool T failing? | `moda detections tools` | `moda detections tools <T>` (use each row's `anchor`) |
|
|
63
|
+
| What frustrates users? | `moda detections list --family=frustration --limit=10` | `moda traces get <anchor.conversation_id> --msg-index=<anchor.msg_index>` |
|
|
64
|
+
| Other emotions, laziness, hallucinations | `moda detections list --family=emotion:<f>`, `moda detections list --family=laziness`, `moda detections list --family=hallucination --kind=contradicted` | context on the row's anchor |
|
|
65
|
+
| Read one whole trace | `moda traces get <id> --all` (pages of up to 500 messages) | `moda traces spans <id>` for raw spans |
|
|
66
|
+
| Read several traces or turns | `moda traces get <id1>@<msg_index> <id2>@<msg_index> ...` (up to 25; or one per line via `--ids-from=-`) | Same for `moda traces spans <id1> <id2> ...` |
|
|
67
|
+
| One user's experience | `moda users`, `moda user <user_id>` | `moda traces search "<words>" --user-id=<id>` |
|
|
66
68
|
| Topic mix | `moda clusters` | `moda cluster-traces <node_id>` |
|
|
67
|
-
| What the agent believed at a turn | `moda
|
|
69
|
+
| What the agent believed at a turn | `moda traces state <id> --snapshot --msg-index=<n>` | |
|
|
68
70
|
| A synthesized answer | `moda ask "<question>"` only when the user asks: each call is a paid agent run (up to about 200k tokens) | Exit 3 means degraded |
|
|
69
|
-
| A detection is wrong | `moda
|
|
71
|
+
| A detection is wrong | `moda review <kind> <target> --reason="..."` | `references/signals-and-false-positives.md` |
|
|
70
72
|
| Moda data looks wrong or missing | `moda feedback "<what and why>" --conversation-id=<id>` | |
|
|
71
73
|
| Prompts, replays, registry push | Only on explicit request | `references/prompts.md`, `references/replays-and-registry.md` |
|
|
72
74
|
|
|
73
75
|
Time windows differ by command: `--days-back=N` (1-90) on overview,
|
|
74
76
|
investigate, problems, and detection commands; `--time-range=1h|24h|3d|7d|30d|90d|all`
|
|
75
|
-
on search, traces, and clusters. `moda <
|
|
77
|
+
on search, traces, and clusters. `moda <noun> <verb> --help` shows which.
|
|
76
78
|
|
|
77
79
|
## Fixing a Problem
|
|
78
80
|
|
|
79
81
|
1. `moda investigate` or `moda problems` to pick one; note its `problem_id`.
|
|
80
|
-
2. `moda
|
|
81
|
-
`moda
|
|
82
|
+
2. `moda problems get <problem_id> --evidence` and open two or three anchors with
|
|
83
|
+
`moda traces get <id> --msg-index=<n>` before deciding on a cause.
|
|
82
84
|
3. Make the change in code; cite the problem id and trace ids in the commit or PR.
|
|
83
85
|
4. After it ships, and only if the user agrees:
|
|
84
|
-
`moda
|
|
86
|
+
`moda problems feedback <problem_id> --action=mark_fixed`.
|
|
85
87
|
More: `references/production-intelligence.md`.
|
|
86
88
|
|
|
87
89
|
## Rules
|
|
88
90
|
|
|
89
91
|
- Read evidence before acting on a summary; quote trace ids and message
|
|
90
92
|
indexes when you report.
|
|
91
|
-
- Never run writes (`
|
|
93
|
+
- Never run writes (`problems feedback`, `review`,
|
|
92
94
|
`signals create|update|...`, `prompts sync|promote|ab`, `registry push`)
|
|
93
95
|
unless the user asked. Use `--dry-run` where a command offers it.
|
|
94
96
|
- An empty result in a window means "nothing in that window". Widen the window
|
|
95
97
|
once before saying something never happened.
|
|
96
|
-
- Batch reads instead of looping one id per call: `moda
|
|
97
|
-
`moda
|
|
98
|
+
- Batch reads instead of looping one id per call: `moda traces get` and
|
|
99
|
+
`moda traces spans` take up to 25 ids and report per-item results.
|
|
98
100
|
- Do not load `moda manifest --json` or `moda --json-schemas` to learn the CLI
|
|
99
|
-
(about 20k and 11k tokens). Use `moda <
|
|
101
|
+
(about 20k and 11k tokens). Use `moda <noun> <verb> --help`.
|
|
100
102
|
|
|
101
103
|
## Recipes
|
|
102
104
|
|
|
103
105
|
Find where something happened and read the turn:
|
|
104
106
|
|
|
105
107
|
```bash
|
|
106
|
-
moda search "refund not processed" --limit=5 --json \
|
|
108
|
+
moda traces search "refund not processed" --limit=5 --json \
|
|
107
109
|
| jq -r 'first(.results[] | select(.message_index != null)) | "\(.conversation_id) \(.message_index)"' \
|
|
108
|
-
| while read id idx; do moda
|
|
110
|
+
| while read id idx; do moda traces get "$id" --msg-index="$idx" --window=3; done
|
|
109
111
|
```
|
|
110
112
|
|
|
111
113
|
Open the turn behind the top frustration:
|
|
112
114
|
|
|
113
115
|
```bash
|
|
114
|
-
moda
|
|
116
|
+
moda detections list --family=frustration --limit=5 --json \
|
|
115
117
|
| jq -r 'first(.frustrations[] | select(.anchor.msg_index != null)) | .anchor | "\(.conversation_id) \(.msg_index)"' \
|
|
116
|
-
| while read id idx; do moda
|
|
118
|
+
| while read id idx; do moda traces get "$id" --msg-index="$idx" --window=2; done
|
|
117
119
|
```
|
|
118
120
|
|
|
119
121
|
Which tool fails most, and an example:
|
|
120
122
|
|
|
121
123
|
```bash
|
|
122
|
-
moda
|
|
123
|
-
moda
|
|
124
|
+
moda detections tools --json | jq '.tools[] | {tool: .tool_name, failures: .failure_count}'
|
|
125
|
+
moda detections tools <tool_name> --limit=3
|
|
124
126
|
```
|
|
125
127
|
|
|
126
128
|
## References
|
|
@@ -14,34 +14,34 @@ Loaded on demand from the moda-cli skill. This table can lag the CLI: `moda <com
|
|
|
14
14
|
| `moda auth logout` | Revoke the CLI session server-side, then delete it locally |
|
|
15
15
|
| `moda auth status` | Both credentials; exit `0` if either is usable, `4` if neither (expected, not an error) |
|
|
16
16
|
| `moda auth token` | Print the raw bearer token for `$(moda auth token)` composition |
|
|
17
|
-
| `moda search "<query>"` | **Message-grain semantic/keyword/hybrid search (incl. tool calls); primary content-discovery path** |
|
|
17
|
+
| `moda traces search "<query>"` | **Message-grain semantic/keyword/hybrid search (incl. tool calls); primary content-discovery path** |
|
|
18
18
|
| `moda overview` | Dashboard KPIs, top clusters, recent activity |
|
|
19
19
|
| `moda ask "<question>"` | Natural-language production/harness answer with evidence (exit `3` = degraded local fallback) |
|
|
20
20
|
| `moda investigate` | Rank production issues (tool failures, frustrations, top Problems) with evidence + next commands |
|
|
21
21
|
| `moda clusters` | Browse topic cluster hierarchy; `--search="q"` finds clusters by meaning, `--node-id=ID` resolves a deep link |
|
|
22
22
|
| `moda cluster-traces <node_id>` | Traces in a cluster (legacy alias: `moda cluster-conversations`) |
|
|
23
23
|
| `moda traces` | List/filter traces by structured fields (legacy alias: `moda conversations`) |
|
|
24
|
-
| `moda
|
|
25
|
-
| `moda
|
|
26
|
-
| `moda
|
|
27
|
-
| `moda
|
|
28
|
-
| `moda
|
|
29
|
-
| `moda
|
|
30
|
-
| `moda problems` | Rank cross-signal Problems by root cause (what to fix first); takes only `--days-back`/`--limit`. Filter by family with `moda problems
|
|
31
|
-
| `moda
|
|
32
|
-
| `moda
|
|
33
|
-
| `moda
|
|
34
|
-
| `moda
|
|
35
|
-
| `moda laziness` | Laziness detections per trace (`--days-back`, `--limit` 1-50, `--conversation-id`, `--family`) |
|
|
36
|
-
| `moda
|
|
37
|
-
| `moda
|
|
38
|
-
| `moda
|
|
39
|
-
| `moda
|
|
24
|
+
| `moda traces get <conversation_id>` | Windowed message context (max 5 per side); `--msg-index=N` (alias `--message-index`) anchors the window |
|
|
25
|
+
| `moda detections list --family=frustration` | User frustration detections with evidence (the frustration family of `emotions`, legacy field names) |
|
|
26
|
+
| `moda detections list --family=emotion` | Multi-family emotion detections: frustration, disapproval, sadness, confusion, anxiety, trust, positive (`--family=F`, `--user-id`, `--conversation-id`, limit 1–20) |
|
|
27
|
+
| `moda detections list --family=hallucination` | Grounding detections: contradicted/verified outputs, rule breakdown (`--kind=contradicted\|verified`, `--conversation-id=ID`) |
|
|
28
|
+
| `moda detections tools` | Tool failure overview |
|
|
29
|
+
| `moda detections tools <tool_name>` | Per-tool failure breakdown + examples |
|
|
30
|
+
| `moda problems` | Rank cross-signal Problems by root cause (what to fix first); takes only `--days-back`/`--limit`. Filter by family with `moda problems unexplained --family=F` or `moda problems get <problem_id> --traces --family=F` |
|
|
31
|
+
| `moda problems get <problem_id>` | One Problem: dossier, or `--evidence`/`--reports`/`--traces`/`--feedback` pages (`--conversations` is a legacy alias of `--traces`) (`--limit` 1–50, `--cursor` verbatim keyset token) |
|
|
32
|
+
| `moda problems feedback <problem_id>` | Write: `--action=mark_fixed\|dismiss\|flag_attribution\|rename`. `--reason` required for flag_attribution, optional for dismiss; `--new-name` for rename; `--attribution-id` (UUID) required for flag_attribution |
|
|
33
|
+
| `moda review <kind> <target>` | Write: kinds `problem\|problem-detection\|quote\|tool\|laziness\|hallucination\|custom-signal`; `--reason` optional; problem-detection needs `--conversation-id` + `--attribution-id`; quote needs `--family`, `--turn`, `--quote`; tool needs `--tool` plus `--tool-use-id` or `--msg-index`; custom-signal needs `--conversation-id`; `--verdict=clear` removes a mark (not for problem) |
|
|
34
|
+
| `moda reviews list [conversation_id]` | Current false-positive marks: all (`--kind=quote\|tool\|laziness\|hallucination\|custom-signal`, `--all` to include removed, `--limit`) or one trace with its problem detection reviews. Problem dismissals are read with `moda problems get <problem_id> --feedback`, detection reviews with `moda reviews list <conversation_id> --kind=problem-detection` |
|
|
35
|
+
| `moda detections list --family=laziness` | Laziness detections per trace (`--days-back`, `--limit` 1-50, `--conversation-id`, `--family`) |
|
|
36
|
+
| `moda problems linear <problem_id>` | The Problem's linked Linear issue (key, state, URL) or `link: null`; needs Linear connected (Integrations → Linear) |
|
|
37
|
+
| `moda problems link <problem_id>` | Write: file the Problem as a Linear issue (`--team-id`, `--title`, `--priority=0-4`), or link an existing one with `--issue=ENG-123\|URL` (exclusive with the others). Idempotent per Problem |
|
|
38
|
+
| `moda reviews list <conversation_id> --kind=problem-detection` | Problem detections on a trace with review state (`--attribution-id=UUID` narrows to one) |
|
|
39
|
+
| `moda review problem-detection <problem_id>` | Write: `--conversation-id`, `--attribution-id` (UUID), `--verdict=correct\|false_positive\|clear`, optional `--note` |
|
|
40
40
|
| `moda signals` | Custom signals: criteria, status, 7-day judged/matched stats |
|
|
41
41
|
| `moda signal <signal_id>` | One custom signal; `--detections` pages recent hits (`--days-back` 1–90, `--limit` 1–20, `--offset`) |
|
|
42
|
-
| `moda
|
|
43
|
-
| `moda signal
|
|
44
|
-
| `moda
|
|
42
|
+
| `moda detections list --family=custom_signal --conversation-id=<conversation_id>` | Custom signal hits on one trace |
|
|
43
|
+
| `moda review custom-signal <signal_id>` | Write: `--conversation-id`, `--verdict=correct\|false_positive\|clear`, optional `--note`, `--msg-index`/`--msg-dedup-token` |
|
|
44
|
+
| `moda traces state <conversation_id>` | Agent memory: slots/threads/events; `--summary-only`; `--snapshot --msg-index=N` (state at a turn); `--replay --message-count=N` (state over time) |
|
|
45
45
|
| `moda failures` | Tool failures only, ranked (no frustrations or Problems) |
|
|
46
46
|
| `moda tail` | Live tail: one JSON line per new trace/detection (`--signal=traces\|emotions\|all`, legacy alias `--signal=conversations`, `--interval=N`, `--once`, `--max-events=N`). **Emotions caveat:** `/emotions` is ranked by score with no time ordering or cursor, so the tail follows the *highest-scoring* detections rather than everything; each poll emits a `tail_coverage` line with `scanned`/`total`/`coverage_pct`/`complete`. Pass `--full-scan` for a complete window scan (many more requests, capped by the API's offset ceiling of 10000). |
|
|
47
47
|
| `moda feedback "<note>"` | Flag wrong/missing data or CLI quirks to the Moda team |
|
|
@@ -51,13 +51,13 @@ Loaded on demand from the moda-cli skill. This table can lag the CLI: `moda <com
|
|
|
51
51
|
| `moda prompts promote <key>` | Move a remote prompt label |
|
|
52
52
|
| `moda registry template\|validate\|push\|status\|coverage\|log\|show\|prompt` | Register prompts, skills, tools, and agent wiring from one agent-written manifest (`--file`, `--dry-run`, `--live`, `--message`, `--days`); every push is a commit in the Registry history; no repo scan |
|
|
53
53
|
| `moda users` / `moda user <id>` | End users with health/activity (`--page-size=N`, alias `--limit=N`; `--page=N`); one user's profile |
|
|
54
|
-
| `moda
|
|
54
|
+
| `moda problems reopen <pid>` | Reopen a Problem marked fixed |
|
|
55
55
|
| `moda signals <create\|update\|activate\|pause\|delete\|examples\|refine\|revise\|proposals\|accept\|reject\|search\|preview>` | Manage custom signals (writes take `--dry-run`; LLM-backed ones are rate-limited per key) |
|
|
56
56
|
| `moda prompts list\|show\|proposals\|usage\|replay-runs\|decide` | Read the prompt registry; merge/reject proposals |
|
|
57
57
|
| `moda replays sets\|runs\|run\|playouts\|playout` | Replay results: run history, per-seed verdicts, playout transcripts with tool provenance (read-only) |
|
|
58
58
|
| `moda skills list\|show\|policy` / `moda harness list\|show\|versions\|diff` | Skill library and synced harnesses |
|
|
59
59
|
|
|
60
|
-
`moda search` flags: `--mode=keyword|semantic|hybrid` (default `hybrid`),
|
|
60
|
+
`moda traces search` flags: `--mode=keyword|semantic|hybrid` (default `hybrid`),
|
|
61
61
|
`--user-id`, `--time-range=<range>`, `--limit=N` (1–100, default 20).
|
|
62
62
|
|
|
63
63
|
There is no flag set common to every command: each command rejects a flag it
|
|
@@ -5,15 +5,15 @@ Loaded on demand from the moda-cli skill.
|
|
|
5
5
|
### 10. Everything the dashboard does, from the CLI
|
|
6
6
|
|
|
7
7
|
Every command below needs only an API key, and the tenant comes from the key.
|
|
8
|
-
Writes listed here accept `--dry-run` (prints the exact request, sends nothing) except `
|
|
8
|
+
Writes listed here accept `--dry-run` (prints the exact request, sends nothing) except `problems feedback`, `review`, `problems link`, and `prompts promote|ab|propose`. Those refuse `--dry-run` before sending anything, so run them only when you mean it.
|
|
9
9
|
|
|
10
10
|
| Dashboard | CLI |
|
|
11
11
|
|---|---|
|
|
12
12
|
| Traces list: sort, page | `moda traces --sort=recent\|longest\|slowest --offset=N` |
|
|
13
|
-
| Trace detail: whole transcript | `moda
|
|
13
|
+
| Trace detail: whole transcript | `moda traces get <id> --all` (ordered, with tool calls/results) |
|
|
14
14
|
| Users page / user profile | `moda users [--window --cohort --sort --order --health-status]`, `moda user <id>` |
|
|
15
|
-
| Problems: unexplained remainder | `moda problems
|
|
16
|
-
| Reopen a problem marked fixed | `moda
|
|
15
|
+
| Problems: unexplained remainder | `moda problems unexplained [--family=emotion\|tool_failure\|laziness\|… --cursor]` |
|
|
16
|
+
| Reopen a problem marked fixed | `moda problems reopen <problem_id> --reason="…"` (resolved ones return only via re-detection) |
|
|
17
17
|
| Custom signals: create | `moda signals draft --note="…"`, then `moda signals create --name --description --pass-criteria --fail-criteria [--tool-names=a,b]` or `--file=body.json` |
|
|
18
18
|
| Custom signals: edit, lifecycle | `moda signals update <id> …`, `moda signals activate\|pause\|delete <id>` (delete: drafts only; archiving is dashboard-only) |
|
|
19
19
|
| Custom signals: review & tune | `moda signals examples <id> add\|update\|remove --conversation-id=…`, `refine <id> --request="…"`, `revise <id> --conversation-ids=…`, `proposals <id>`, `accept\|reject <id> <proposal_id>` |
|
|
@@ -5,11 +5,11 @@ Loaded on demand from the moda-cli skill.
|
|
|
5
5
|
### 5. Frustration analysis
|
|
6
6
|
|
|
7
7
|
```bash
|
|
8
|
-
moda
|
|
9
|
-
moda
|
|
10
|
-
moda
|
|
11
|
-
moda
|
|
12
|
-
moda
|
|
8
|
+
moda detections list --family=frustration # last 7 days
|
|
9
|
+
moda detections list --family=frustration --days-back=14 --limit=20
|
|
10
|
+
moda detections list --family=frustration --include-window --window=1 --limit=5 # agent output omits windows unless asked
|
|
11
|
+
moda detections list --family=emotion:frustration --user-id=<id> # one end user
|
|
12
|
+
moda detections list --family=emotion --conversation-id=<id> # every family, one trace
|
|
13
13
|
```
|
|
14
14
|
|
|
15
15
|
In agent output, `frustrations` and `emotions` rows leave out the embedded
|
|
@@ -47,10 +47,10 @@ half-width; `--window=1` ⇒ 3 messages total (anchor ± 1), the canonical
|
|
|
47
47
|
### 6. Tool failure investigation
|
|
48
48
|
|
|
49
49
|
```bash
|
|
50
|
-
moda
|
|
51
|
-
moda
|
|
52
|
-
moda
|
|
53
|
-
moda
|
|
50
|
+
moda detections tools # which tools are breaking
|
|
51
|
+
moda detections tools <tool_name> # subtype breakdown + examples
|
|
52
|
+
moda detections tools search --subtype=dependency.timeout --limit=10 # a value from `subtypes`
|
|
53
|
+
moda detections tools search --include-window --window=1 --limit=5
|
|
54
54
|
```
|
|
55
55
|
|
|
56
56
|
Every example row in `tool-failure-detail` output carries a top-level
|
|
@@ -63,7 +63,7 @@ directly instead of matching by `tool_use_id` against the trace.
|
|
|
63
63
|
|
|
64
64
|
By default each row embeds `conversation`: the anchor ± 2 messages (without
|
|
65
65
|
the duplicate `content_blocks`), plus `context_command`, the exact
|
|
66
|
-
`moda
|
|
66
|
+
`moda traces get <conversation_id> --msg-index=<msg_index> --window=2` read for
|
|
67
67
|
the untrimmed window. `--compact` (same as `--include-window=false` or
|
|
68
68
|
`--no-include-window`) drops `conversation` and keeps the anchor and
|
|
69
69
|
`context_command`.
|
|
@@ -39,7 +39,7 @@ moda feedback "trace is a refund request, not this problem" \
|
|
|
39
39
|
`--message-index`); they go into `refs` and let the team reproduce what
|
|
40
40
|
you saw.
|
|
41
41
|
- Feedback does not change the problem itself. To act on a misroute directly,
|
|
42
|
-
use `moda
|
|
43
|
-
--attribution-id=<uuid> --reason=TEXT` or `moda
|
|
42
|
+
use `moda problems feedback <problem_id> --action=flag_attribution
|
|
43
|
+
--attribution-id=<uuid> --reason=TEXT` or `moda review <kind> <target>
|
|
44
44
|
--reason=TEXT`.
|
|
45
45
|
- Never include secrets or full transcripts in the note.
|
|
@@ -116,7 +116,7 @@ Fields an agent should read:
|
|
|
116
116
|
- `summary.text` + `summary.confidence` — the answer and how much to trust it
|
|
117
117
|
(`high | medium | low | unknown`).
|
|
118
118
|
- `evidence_refs[]` — each `{ id, kind, label, path }`. When `path` looks like
|
|
119
|
-
`conversation/<id>`, open it with `moda
|
|
119
|
+
`conversation/<id>`, open it with `moda traces get <id>`.
|
|
120
120
|
- `next_commands[]` — suggested follow-up commands (each carries `mutability`
|
|
121
121
|
and `requires_approval`).
|
|
122
122
|
- `warnings[]`, `errors[]`.
|
|
@@ -179,7 +179,7 @@ Treat an answer as degraded if **any** of these hold:
|
|
|
179
179
|
items are dropped from `data`. When rows are dropped, the list's
|
|
180
180
|
`pagination` says what you got (`returned`, `limit`, `next_offset`; a
|
|
181
181
|
`next_cursor` that would skip rows becomes `null`) and `next_commands` has
|
|
182
|
-
the exact command for the rest (e.g. `moda
|
|
182
|
+
the exact command for the rest (e.g. `moda detections list --family=frustration --offset=3
|
|
183
183
|
--limit=7`). `truncation.data.full_output_path` (also an `artifacts[]` entry of
|
|
184
184
|
kind `full_output`) is that file; read it in line ranges or with `jq`.
|
|
185
185
|
These files are pruned after 24 hours (`MODA_AGENT_OUTPUT_RETENTION_HOURS`).
|
|
@@ -43,7 +43,7 @@ Data API fallback. Abbreviated envelope:
|
|
|
43
43
|
{ "id": "tool:lookupCustomer", "kind": "tool_failure", "label": "lookupCustomer: 12 failure(s)", "path": null }
|
|
44
44
|
],
|
|
45
45
|
"next_commands": [
|
|
46
|
-
{ "command": "moda
|
|
46
|
+
{ "command": "moda detections tools lookupCustomer --days-back=7 --include-window", "purpose": "Inspect tool failure examples and trace anchors.", "mutability": "read", "requires_approval": false }
|
|
47
47
|
],
|
|
48
48
|
"warnings": ["Cloud ask endpoint unavailable; synthesized answer from local Data API evidence."],
|
|
49
49
|
"errors": [],
|
|
@@ -55,7 +55,7 @@ Data API fallback. Abbreviated envelope:
|
|
|
55
55
|
narrows the tool-failure evidence to one tool, and `--trace` (legacy alias
|
|
56
56
|
`--conversation`) scopes only the frustration evidence (overview, tool
|
|
57
57
|
failures and Problems stay tenant-wide). Its findings include the top 1–3
|
|
58
|
-
Problems from `moda problems`, with `moda
|
|
58
|
+
Problems from `moda problems`, with `moda problems get <id>` as the next command.
|
|
59
59
|
`moda failures` is the tool-failures-only view.
|
|
60
60
|
|
|
61
61
|
After reporting the top behavioral failure and the harness layer it routes to
|
|
@@ -73,14 +73,14 @@ Problem, or before hand-editing prompt/skill files in a repo that talks to Moda.
|
|
|
73
73
|
moda investigate # rank tool failures, frustrations,
|
|
74
74
|
# and top Problems with evidence
|
|
75
75
|
moda problems # cross-signal Problems by root cause
|
|
76
|
-
moda
|
|
77
|
-
moda
|
|
78
|
-
moda
|
|
79
|
-
moda
|
|
80
|
-
moda
|
|
81
|
-
moda
|
|
76
|
+
moda problems get <problem_id> # compact dossier: what, where, how often
|
|
77
|
+
moda problems get <problem_id> --full # every dossier block (large)
|
|
78
|
+
moda problems get <problem_id> --evidence # attribution rationales and anchors
|
|
79
|
+
moda problems get <problem_id> --traces # affected traces with anchors
|
|
80
|
+
moda detections tools <tool_name> # failing calls with surrounding turns
|
|
81
|
+
moda traces get <conversation_id> --window=3 # the messages around one turn
|
|
82
82
|
# ...propose and make the change, citing the problem/trace ids in the PR...
|
|
83
|
-
moda
|
|
83
|
+
moda problems feedback <problem_id> --action=mark_fixed # after it ships
|
|
84
84
|
```
|
|
85
85
|
|
|
86
86
|
Rules that keep the fix grounded:
|
|
@@ -93,4 +93,4 @@ Rules that keep the fix grounded:
|
|
|
93
93
|
`moda prompts ab --baseline=<path> --candidate=<path> --traces=<ids>` (see
|
|
94
94
|
section 7).
|
|
95
95
|
- Mark the Problem fixed only after the change ships. If the issue is detected
|
|
96
|
-
again, the Problem comes back; `moda
|
|
96
|
+
again, the Problem comes back; `moda problems reopen` reopens it by hand.
|
|
@@ -8,43 +8,43 @@ Loaded on demand from the moda-cli skill.
|
|
|
8
8
|
|
|
9
9
|
```bash
|
|
10
10
|
# Semantic: find the moment a user expressed an intent, in their own words
|
|
11
|
-
moda search "user wants to cancel subscription" --mode=semantic --limit=5 --json \
|
|
11
|
+
moda traces search "user wants to cancel subscription" --mode=semantic --limit=5 --json \
|
|
12
12
|
| jq -r '.results[] | "\(.conversation_id) @\(.message_index) \(.snippet)"'
|
|
13
13
|
|
|
14
14
|
# Keyword: locate every tool call/result mentioning an exact identifier
|
|
15
|
-
moda search "stripe.refunds.create" --mode=keyword --limit=10
|
|
15
|
+
moda traces search "stripe.refunds.create" --mode=keyword --limit=10
|
|
16
16
|
|
|
17
17
|
# Search, then open the top message-level hit at the exact turn (skips
|
|
18
18
|
# trace-level fallback hits, which have no message_index)
|
|
19
|
-
moda search "rate limit exceeded" --limit=5 --json \
|
|
19
|
+
moda traces search "rate limit exceeded" --limit=5 --json \
|
|
20
20
|
| jq -r 'first(.results[] | select(.message_index != null)) | "\(.conversation_id) \(.message_index)"' \
|
|
21
|
-
| xargs -r -n2 sh -c 'moda
|
|
21
|
+
| xargs -r -n2 sh -c 'moda traces get "$0" --msg-index="$1"'
|
|
22
22
|
```
|
|
23
23
|
|
|
24
24
|
### Daily health check
|
|
25
25
|
|
|
26
26
|
```bash
|
|
27
27
|
moda overview --days-back=1
|
|
28
|
-
moda
|
|
29
|
-
moda
|
|
28
|
+
moda detections list --family=frustration --days-back=1 --limit=5
|
|
29
|
+
moda detections tools --days-back=1
|
|
30
30
|
```
|
|
31
31
|
|
|
32
32
|
### Debug a frustrated user
|
|
33
33
|
|
|
34
34
|
```bash
|
|
35
|
-
moda
|
|
35
|
+
moda detections list --family=frustration --days-back=7 --limit=10
|
|
36
36
|
# Each row has an anchor: {conversation_id, msg_index}. Open that turn
|
|
37
37
|
# (without --msg-index, context centers on the middle of the trace):
|
|
38
|
-
moda
|
|
38
|
+
moda traces get <anchor.conversation_id> --msg-index=<anchor.msg_index> --window=3
|
|
39
39
|
```
|
|
40
40
|
|
|
41
41
|
### Diagnose a flaky tool
|
|
42
42
|
|
|
43
43
|
```bash
|
|
44
|
-
moda
|
|
45
|
-
moda
|
|
44
|
+
moda detections tools
|
|
45
|
+
moda detections tools <tool_name>
|
|
46
46
|
# Pick an example conversation_id from the output
|
|
47
|
-
moda
|
|
47
|
+
moda traces get <conversation_id>
|
|
48
48
|
```
|
|
49
49
|
|
|
50
50
|
### Explore what users are doing
|
|
@@ -62,13 +62,13 @@ envelope, where the payload sits under `.data`.
|
|
|
62
62
|
|
|
63
63
|
```bash
|
|
64
64
|
# Pull primary causes of recent frustrations
|
|
65
|
-
moda
|
|
65
|
+
moda detections list --family=frustration --days-back=7 --json | jq -r '.frustrations[].primary_cause'
|
|
66
66
|
|
|
67
67
|
# Get trace IDs for a search and fetch context for all of them in one call
|
|
68
68
|
moda traces --search="timeout" --limit=3 --json \
|
|
69
69
|
| jq -r '.conversations[].conversation_id' \
|
|
70
|
-
| moda
|
|
70
|
+
| moda traces get --ids-from=-
|
|
71
71
|
|
|
72
72
|
# Tool failure leaderboard
|
|
73
|
-
moda
|
|
73
|
+
moda detections tools --json | jq '.tools[] | {tool: .tool_name, failures: .failure_count}'
|
|
74
74
|
```
|
|
@@ -5,12 +5,12 @@ Loaded on demand from the moda-cli skill.
|
|
|
5
5
|
### 1. Search messages (primary content-discovery path)
|
|
6
6
|
|
|
7
7
|
```bash
|
|
8
|
-
moda search "refund flow timeout" # hybrid (default)
|
|
9
|
-
moda search "customer asked for a refund" --mode=semantic
|
|
10
|
-
moda search "SEARCH_NO_RESULTS" --mode=keyword
|
|
11
|
-
moda search "checkout error" --time-range=7d --limit=10
|
|
12
|
-
moda search "stripe.charges.create failed" --user-id=<id>
|
|
13
|
-
moda search "Иван" --conversation-id=<id> --mode=keyword # inside one trace
|
|
8
|
+
moda traces search "refund flow timeout" # hybrid (default)
|
|
9
|
+
moda traces search "customer asked for a refund" --mode=semantic
|
|
10
|
+
moda traces search "SEARCH_NO_RESULTS" --mode=keyword
|
|
11
|
+
moda traces search "checkout error" --time-range=7d --limit=10
|
|
12
|
+
moda traces search "stripe.charges.create failed" --user-id=<id>
|
|
13
|
+
moda traces search "Иван" --conversation-id=<id> --mode=keyword # inside one trace
|
|
14
14
|
```
|
|
15
15
|
|
|
16
16
|
Searches trace messages at message grain. Pass `--include-tool-io` to
|
|
@@ -34,7 +34,7 @@ trails ingestion (`index_freshness`); a warning says so when your
|
|
|
34
34
|
(same as `moda traces --search`), those traces come back in `results` with
|
|
35
35
|
`match_grain: "trace"`, plus `fallback.source: "traces_search"` and a warning.
|
|
36
36
|
These rows have no score or message index. Open one with
|
|
37
|
-
`moda
|
|
37
|
+
`moda traces spans <conversation_id>` or `moda traces get <conversation_id>` to find the
|
|
38
38
|
matching turn. When more traces exist, `fallback.has_more` is true and
|
|
39
39
|
`next_commands` includes the `moda traces --search ... --offset` command to
|
|
40
40
|
page. With `--mode=keyword`, ranked traces are not substituted. If the fallback
|
|
@@ -58,16 +58,16 @@ embedding provider is reachable, so check it before trusting semantic ranking.
|
|
|
58
58
|
`score` scales differ per mode (see above) — branch on `search_mode`, never
|
|
59
59
|
compare scores across modes.
|
|
60
60
|
|
|
61
|
-
Hand the anchors straight to `moda
|
|
61
|
+
Hand the anchors straight to `moda traces get`:
|
|
62
62
|
|
|
63
63
|
```bash
|
|
64
64
|
# Find, then read the exact turn. --json prints the bare payload; piped output
|
|
65
65
|
# without it is the agent envelope, where results sit under .data.results.
|
|
66
|
-
hit=$(moda search "double charge" --limit=1 --json | jq -c '.results[0] // empty')
|
|
66
|
+
hit=$(moda traces search "double charge" --limit=1 --json | jq -c '.results[0] // empty')
|
|
67
67
|
cid=$(echo "$hit" | jq -r '.conversation_id')
|
|
68
68
|
# Trace-level fallback hits (match_grain: "trace") have no message_index.
|
|
69
69
|
idx=$(echo "$hit" | jq -r '.message_index // empty')
|
|
70
|
-
moda
|
|
70
|
+
moda traces get "$cid" ${idx:+--msg-index="$idx"}
|
|
71
71
|
```
|
|
72
72
|
|
|
73
73
|
### 3. Browse topic clusters
|
|
@@ -81,7 +81,7 @@ moda cluster-traces <node_id> # traces in cluster
|
|
|
81
81
|
### 3b. List / filter traces (structured, not semantic)
|
|
82
82
|
|
|
83
83
|
Use `moda traces` to enumerate or filter by structured fields — not to
|
|
84
|
-
search by meaning (use `moda search` for that). The `--search` flag here
|
|
84
|
+
search by meaning (use `moda traces search` for that). The `--search` flag here
|
|
85
85
|
ranks whole traces: hybrid (keyword + semantic) when embeddings exist, else
|
|
86
86
|
keyword. Check `total_kind`: `candidate_pool` means results are related traces
|
|
87
87
|
that may not contain the words and `pagination.total` is not a match count;
|
|
@@ -100,19 +100,19 @@ Filters: `--search`, `--cluster-id`, `--user-id`, `--time-range`
|
|
|
100
100
|
### 4. Read a single trace
|
|
101
101
|
|
|
102
102
|
```bash
|
|
103
|
-
moda
|
|
104
|
-
moda
|
|
105
|
-
moda
|
|
106
|
-
moda
|
|
107
|
-
moda
|
|
108
|
-
moda
|
|
109
|
-
moda
|
|
103
|
+
moda traces get <conversation_id> # default window around middle
|
|
104
|
+
moda traces get <conversation_id> --msg-index=5 # center on message 5
|
|
105
|
+
moda traces get <conversation_id> --window=3 # 3 messages each side (max 5)
|
|
106
|
+
moda traces get <conversation_id> --msg-index=40 --window=0 # just message 40 + total_messages
|
|
107
|
+
moda traces get <conversation_id> --metadata-only # summary + total_messages, no messages
|
|
108
|
+
moda traces get <conversation_id> --all # whole trace in order, first 500 messages
|
|
109
|
+
moda traces get <conversation_id> --all --from=500 --max-messages=500 # continue (max 5000)
|
|
110
110
|
```
|
|
111
111
|
|
|
112
112
|
`--all` returns `has_more`/`next_from`; when it stops at the cap, the warning
|
|
113
113
|
prints the exact `--from` command to continue.
|
|
114
114
|
|
|
115
|
-
**Several traces? Batch them in one call instead of looping `moda
|
|
115
|
+
**Several traces? Batch them in one call instead of looping `moda traces get`
|
|
116
116
|
per id.** Pass up to 25 targets as positionals or one per line with
|
|
117
117
|
`--ids-from=-` (stdin) or `--ids-from=<file>`. A target is `<id>` or
|
|
118
118
|
`<id>@<msg_index>` (the anchor; ids may contain `:`), and the shared flags
|
|
@@ -121,8 +121,8 @@ be JSON (`{"id":"...","msg_index":N}`), the only spelling for an id that itself
|
|
|
121
121
|
ends in `@<digits>`.
|
|
122
122
|
|
|
123
123
|
```bash
|
|
124
|
-
moda
|
|
125
|
-
printf '%s\n' telegram:8565459080@12 web:conv-9 | moda
|
|
124
|
+
moda traces get telegram:8565459080@12 linq:c6f88988-aa@3 web:conv-9 --window=2
|
|
125
|
+
printf '%s\n' telegram:8565459080@12 web:conv-9 | moda traces get --ids-from=- --window=1
|
|
126
126
|
```
|
|
127
127
|
|
|
128
128
|
A batch returns `data.results: [{ target, ok, data | error }]` in input order,
|
|
@@ -132,29 +132,29 @@ every target failed. When the agent byte budget would overflow, the items that
|
|
|
132
132
|
fit are returned and `remaining_targets` plus a `next_commands` entry read the
|
|
133
133
|
rest. One positional id keeps the single-trace output above.
|
|
134
134
|
|
|
135
|
-
`moda
|
|
135
|
+
`moda traces get` returns messages whole, with their tool calls and results. An
|
|
136
136
|
older Moda server cuts message text, tool inputs, and tool results at 500
|
|
137
137
|
characters; the response warns when that server is in use.
|
|
138
138
|
`--window` is at most 5 messages per side. For the raw,
|
|
139
139
|
complete picture of what was captured — every span, the parent/child hierarchy,
|
|
140
|
-
tool calls, prompt/response bodies, and `gen_ai.*` attributes — use `moda
|
|
140
|
+
tool calls, prompt/response bodies, and `gen_ai.*` attributes — use `moda traces spans`.
|
|
141
141
|
|
|
142
142
|
### 4b. Audit raw spans (completeness, orphans, duplicates)
|
|
143
143
|
|
|
144
144
|
```bash
|
|
145
|
-
moda
|
|
146
|
-
moda
|
|
147
|
-
moda
|
|
148
|
-
moda
|
|
149
|
-
moda
|
|
150
|
-
moda
|
|
151
|
-
moda
|
|
145
|
+
moda traces spans <conversation_id|trace_id> # auto-detects OTLP trace_id vs trace ID (conversation_id)
|
|
146
|
+
moda traces spans <trace_id> --kind=trace # force OTLP trace_id lookup
|
|
147
|
+
moda traces spans <conversation_id> --include-raw # attach verbatim raw_event bodies
|
|
148
|
+
moda traces spans <conversation_id> --include-bodies # keep prompt/response/attribute bodies
|
|
149
|
+
moda traces spans <conversation_id> --limit=1000 # ingest rows to read (default 200, max 1000)
|
|
150
|
+
moda traces spans <conversation_id> --cursor=<next_cursor> # next page of a capped audit
|
|
151
|
+
moda traces spans <id1> <id2> <id3> # batch: up to 25 ids, one envelope
|
|
152
152
|
```
|
|
153
153
|
|
|
154
154
|
Auditing several traces? Batch the ids (positionals or `--ids-from=-|<file>`)
|
|
155
|
-
instead of looping `moda
|
|
155
|
+
instead of looping `moda traces spans`: the result is `data.results: [{ target, ok,
|
|
156
156
|
data | error }]` with the same per-item errors and byte-budget rules as batch
|
|
157
|
-
`moda
|
|
157
|
+
`moda traces get`. Ids are taken verbatim (no `@` anchors); `--cursor` pages one
|
|
158
158
|
trace, so follow each item's `next_commands` for it.
|
|
159
159
|
|
|
160
160
|
`truncated` / `has_more` at the top level say whether the audit is complete.
|