trodo-cli 0.0.0-stage → 0.3.0

package/skills/trodo/SKILL.md

@@ -0,0 +1,121 @@
+---
+name: trodo
+version: 1.0.0
+cli_version: ">=0.3.0"
+last_updated: 2026-11-03
+description: >-
+  Read and act on an AI agent's production traces in Trodo with the `trodo` CLI:
+  find failing runs, read a run's span tree or a whole conversation, see why an
+  eval failed step by step, check traffic stats against last week, triage and
+  close issues, pull and push prompt versions, give feedback, and turn traces
+  into datasets and fine-tuning files. Use for "why did this run fail", "what
+  broke since yesterday", "show me the conversation", "which evals fail most",
+  "fix this Trodo issue", "update the prompt", "build a dataset from my traces".
+---
+
+# Trodo from the terminal
+
+Every command prints JSON (pipe it to `jq`), `--table` for a person, `--csv` for a
+sheet. Exit codes: 0 ok, 1 the API said no, 2 a usage mistake (the message says which).
+
+## Setup
+
+```bash
+npm i -g trodo-cli
+trodo login      # opens the app: check the code, choose the team, Connect
+trodo whoami
+```
+
+Without a browser (CI): `TRODO_API_KEY=trodo_mk_…` from Settings → MCP.
+
+Everything is per team. `trodo teams` lists the teams this terminal is signed in
+to, `trodo switch TEAM` changes the current one, `--team TEAM` picks one for a
+single command. Check `trodo whoami` before writing anything.
+
+## Find what went wrong
+
+```bash
+trodo traces stats --since 24h --compare            # errors, latency, cost, evals vs the day before
+trodo runs list --since 24h --where "status=error" --table
+trodo runs list --eval "any:failed" --agent support --table
+trodo issues list --status open,regression --table
+trodo runs search "refund for order 4821"           # free text over inputs, outputs and spans
+```
+
+The filter is the Traces page's: `--since 24h|7d|30d`, `--agent NAME`,
+`--where "field op value"` (`= != ~ > >= < <=` on any property:
+`total_cost>0.05`, `span.model~gpt-4o`, `metadata.tier=pro`) and
+`--eval "NAME:COND"` (`failed`, `passed`, `score<0.5`, `reason=CODE`,
+`STEP.FIELD=value`; `trodo evals list` names the evals and their steps).
+`trodo traces agents`, `trodo traces fields` and `trodo traces values FIELD`
+say what `--agent` and `--where` can name.
+
+## Read one
+
+```bash
+trodo runs tree RUN_ID                     # the span tree: names, models, time, tokens, errors
+trodo runs get RUN_ID                      # everything: spans, messages, feedback, evaluations
+trodo spans get SPAN_ID                    # one span's full input and output
+trodo conversations get ID --messages      # the thread as user / assistant turns
+trodo evals verdicts --run RUN_ID          # every eval's verdict on it
+trodo evals verdict VERDICT_ID             # each step the eval took and why
+trodo evals steps EVAL --table             # which step decides most failures
+trodo users get DISTINCT_ID                # one user's runs, cost, evals and feedback
+```
+
+Read the deepest failing span before deciding the cause: what failed above it is
+what the user saw, not why.
+
+## Fix and verify
+
+```bash
+trodo issues get ID                        # root cause, evidence, timeline
+trodo issues failures ID                   # recorded inputs to replay against the fix
+trodo traces count --since 1h --eval "NAME:failed"   # after the fix ships: should fall to 0
+trodo issues close ID                      # a new failure reopens it as a regression
+```
+
+## Prompts
+
+```bash
+trodo prompts pull support-triage --label production -o prompt.json
+# edit prompt.json
+trodo prompts push support-triage -f prompt.json --message "why" --label staging
+trodo prompts label support-triage 5 production     # agents fetching by label switch on their next call
+```
+
+Moving `production` changes what live agents run. Do it only when asked.
+
+## Feedback
+
+```bash
+trodo runs feedback RUN_ID --thumbs down --comment "quoted the wrong plan"
+trodo evals mark VERDICT_ID --wrong --should pass --reason "the refund was within policy"
+trodo evals queue --table ; trodo evals grade ITEM_ID CHOICE
+```
+
+## Datasets and training data
+
+```bash
+trodo traces export --kind spans --eval any:passed --eval any:not-failed --format openai -o sft.jsonl
+trodo traces export --kind runs --since 7d --include-trace --dir ./runs     # one JSON file per run
+trodo datasets create "Refund failures" --kind runs --eval "Refund policy:failed" --include-trace
+trodo datasets rows "Refund failures" -o rows.jsonl
+# write better answers: {"id": "...", "expected": "..."} per line
+trodo datasets update "Refund failures" -f updates.jsonl
+trodo datasets export "Refund failures" --rows expected --format openai --target expected -o train.jsonl
+```
+
+`trodo datasets append NAME -f rows.jsonl` takes `{"input": "…", "output": "…", "expected": "…"}`
+lines, or `{"run_id": "…"}` / `{"span_id": "…"}` / `{"conversation_id": "…"}` to copy traffic.
+
+## What the CLI will not do
+
+It reads everything and only adds. Nothing deletes, archives, replaces or
+reverts data, and traces cannot be changed. If a task needs that, tell the
+person to do it in the app.
+
+## Anything else
+
+`trodo api GET /runs/conversations --param q=alice` calls any `/api/v1` route with the
+signed-in key. `trodo help` lists every command.

package/skills/trodo-datasets/SKILL.md

@@ -0,0 +1,116 @@
+---
+name: trodo-datasets
+version: 1.0.0
+cli_version: ">=0.2.0"
+last_updated: 2026-11-03
+description: >-
+  Turn an agent's production traces into datasets and fine-tuning files with the
+  `trodo` CLI: pick traces by what the team's evals said about them (passed,
+  failed, a score, one step's answer), keep whole runs with their span trees,
+  process rows locally, write Expected back, and export OpenAI / Fireworks /
+  Together chat JSONL or DPO pairs. Use for "build a training set from my
+  traces", "export the runs that passed", "label these rows", "make a dataset of
+  failures", "fine-tune on what went well", and for running a Trodo labeling agent.
+---
+
+# Trodo datasets from the terminal
+
+The team's evals already say which traces went well. A supervised fine-tune
+wants exactly those: their own Output is the target, nothing to label. Rows
+that failed need a better answer first (Expected). Work in that order.
+
+## Setup
+
+```bash
+npm i -g trodo-cli
+trodo login          # opens the app: check the code, choose the team, Connect
+trodo whoami
+```
+
+In CI or anywhere without a browser: `TRODO_API_KEY=trodo_mk_…` (Settings → MCP).
+Every command prints JSON; add `--table` to read it yourself.
+
+## 1. Know the evals
+
+```bash
+trodo evals list --table
+```
+
+`--eval "NAME:COND"` takes an eval's name (or any unique part of it, or `any`):
+`failed`, `passed`, `not-failed`, `marked-wrong`, `not-marked-wrong`,
+`marked-right`, `score<0.5`, `reason=CODE`, or one step's answer
+`STEP.FIELD=value` / `STEP.FIELD<3` (STEP is the step's id or the start of its
+name). Several `--eval` flags AND together. "Passed every eval" is
+`--eval any:passed --eval any:not-failed`. People's corrections count: a
+verdict someone marked wrong is read the other way round.
+
+## 2. Look before you pull
+
+```bash
+trodo traces count --kind runs --since 30d --agent support --eval "any:passed" --eval "any:not-failed"
+```
+
+Kinds: `spans` (one model call), `runs` (one task, add `--include-trace` for
+every span), `conversations` (a whole thread), `verdicts` (eval results, for
+training the eval itself). Narrow with `--where "status=error"`,
+`--where "total_cost>0.05"`, `--where "span.model~gpt-4o"`, `--where "metadata.tier=pro"`.
+
+## 3a. Straight to a training file
+
+```bash
+trodo traces export --kind spans --since 30d --eval any:passed --eval any:not-failed \
+  --format openai --target output -o sft.jsonl
+```
+
+## 3b. Or keep it as a dataset (versioned, shared with the team, labelable)
+
+```bash
+trodo datasets create "Refunds that went well" --kind runs --since 30d \
+  --agent support --eval "Refund policy:passed" --include-trace [--live]
+trodo datasets export "Refunds that went well" --rows passed --format openai --target output -o train.jsonl
+```
+
+`--live` keeps adding new matches every 15 minutes.
+
+## 4. Process rows locally, write them back
+
+```bash
+trodo datasets export "Refund failures" > rows.jsonl          # each line has its "id"
+python label.py < rows.jsonl > updates.jsonl                   # your logic: {"id", "expected", "metadata"?}
+trodo datasets update "Refund failures" -f updates.jsonl --label derived --name "Expected from label.py"
+```
+
+One call is one dataset version (restorable in the app). `--label derived`
+records that a rule wrote Expected, not a person. Add rows with
+`trodo datasets append NAME -f rows.jsonl`: plain rows `{input, output?, expected?}`
+or references `{run_id}` / `{span_id}` / `{conversation_id}` that Trodo fills in.
+
+## 5. Or let a Trodo agent label in the product
+
+The dataset page's **Label** menu (or asking Lucid) makes a labeling agent:
+models that must agree, a model with your rules, a stronger model rewriting
+failures, or two models compared. Run one from here:
+
+```bash
+trodo agents list --table
+trodo agents run "Label Expected where models agree" --wait
+```
+
+## 6. Train
+
+```bash
+trodo datasets export "Refunds" --rows expected --format openai --target expected -o train.jsonl
+firectl dataset create refunds train.jsonl
+firectl sftj create --base-model accounts/fireworks/models/qwen3-8b --dataset refunds --output-model refunds-v1
+```
+
+DPO: `--format dpo` writes Expected as preferred and Output as rejected, for
+rows that have both and differ.
+
+## Rules
+
+- Count first, then pull with a `--limit`. Exports stream; use `-o file`.
+- Never write Expected from a guess. Expected is a person's answer or the
+  output of a rule the team agreed on.
+- The key acts as the person who made it, on one team. A 403 `missing_scope`
+  means the key needs that scope; make a new one with `trodo login`.