stageflow 0.44.0 → 0.46.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (83) hide show
  1. package/dist/agents/evalGate.js +1 -0
  2. package/dist/cli/evalCommand.d.ts +1 -1
  3. package/dist/cli/evalCommand.js +39 -2
  4. package/dist/cli/hostClient.d.ts +6 -0
  5. package/dist/cli/hostClient.js +4 -0
  6. package/dist/config/loadEval.d.ts +3 -1
  7. package/dist/config/loadEval.js +75 -7
  8. package/dist/config/validateEval.js +17 -1
  9. package/dist/evals/calibration.d.ts +21 -0
  10. package/dist/evals/calibration.js +69 -0
  11. package/dist/evals/calibrationSuggest.d.ts +29 -0
  12. package/dist/evals/calibrationSuggest.js +72 -0
  13. package/dist/evals/calibrator.d.ts +46 -0
  14. package/dist/evals/calibrator.js +148 -0
  15. package/dist/evals/compare.d.ts +1 -1
  16. package/dist/evals/compare.js +3 -0
  17. package/dist/evals/constants.d.ts +4 -2
  18. package/dist/evals/constants.js +4 -0
  19. package/dist/evals/engine/FakeEngine.d.ts +1 -0
  20. package/dist/evals/engine/FakeEngine.js +7 -2
  21. package/dist/evals/engine/promptfoo/PromptfooEngine.js +2 -1
  22. package/dist/evals/engine/promptfoo/generate.d.ts +2 -0
  23. package/dist/evals/engine/promptfoo/generate.js +33 -16
  24. package/dist/evals/engine/promptfoo/judgeProviders.js +2 -1
  25. package/dist/evals/engine/promptfoo/normalize.d.ts +4 -1
  26. package/dist/evals/engine/promptfoo/normalize.js +16 -3
  27. package/dist/evals/engine/promptfoo/rubricPrompt.d.ts +2 -0
  28. package/dist/evals/engine/promptfoo/rubricPrompt.js +10 -2
  29. package/dist/evals/errors.d.ts +7 -0
  30. package/dist/evals/errors.js +10 -0
  31. package/dist/evals/judgeProviders.d.ts +6 -0
  32. package/dist/evals/judgeProviders.js +14 -0
  33. package/dist/evals/judged.d.ts +7 -0
  34. package/dist/evals/judged.js +7 -0
  35. package/dist/evals/manager.d.ts +14 -8
  36. package/dist/evals/manager.js +63 -10
  37. package/dist/evals/plan.d.ts +2 -0
  38. package/dist/evals/plan.js +5 -4
  39. package/dist/evals/planInternalAgent.d.ts +1 -1
  40. package/dist/evals/planInternalAgent.js +2 -6
  41. package/dist/evals/read/api.d.ts +6 -0
  42. package/dist/evals/read/api.js +5 -1
  43. package/dist/evals/read/labelQueue.d.ts +45 -0
  44. package/dist/evals/read/labelQueue.js +83 -0
  45. package/dist/evals/stats.d.ts +40 -0
  46. package/dist/evals/stats.js +69 -0
  47. package/dist/evals/storeTypes.d.ts +42 -0
  48. package/dist/evals/types.d.ts +13 -0
  49. package/dist/evals/votes.d.ts +10 -0
  50. package/dist/evals/votes.js +60 -0
  51. package/dist/mcp/evalTools.js +39 -0
  52. package/dist/operatorAgent/profiles/guideConfirm.js +1 -1
  53. package/dist/operatorAgent/profiles/stageflowGuide.d.ts +1 -1
  54. package/dist/operatorAgent/profiles/stageflowGuide.js +8 -7
  55. package/dist/package-meta.d.ts +1 -1
  56. package/dist/package-meta.js +1 -1
  57. package/dist/runstore/port.d.ts +13 -1
  58. package/dist/runstore/sqlite/SqliteRunStore.d.ts +10 -1
  59. package/dist/runstore/sqlite/SqliteRunStore.js +25 -0
  60. package/dist/runstore/sqlite/calibrationStore.d.ts +10 -0
  61. package/dist/runstore/sqlite/calibrationStore.js +106 -0
  62. package/dist/runstore/sqlite/evalStore.d.ts +5 -0
  63. package/dist/runstore/sqlite/evalStore.js +7 -0
  64. package/dist/runstore/sqlite/migrations/018-eval-grader-labels.d.ts +11 -0
  65. package/dist/runstore/sqlite/migrations/018-eval-grader-labels.js +43 -0
  66. package/dist/runstore/sqlite/migrations/index.js +2 -0
  67. package/dist/server/evalRoutes.js +213 -3
  68. package/dist/toolkit/meta.js +2 -0
  69. package/dist/ui/assets/{MarkdownText-S5C_vemc.js → MarkdownText-BS2BBbCx.js} +1 -1
  70. package/dist/ui/assets/adapters-DPD-cAsy.js +17 -0
  71. package/dist/ui/assets/chat-BD1-VTT6.js +2 -0
  72. package/dist/ui/assets/chat-DXnQv9st.css +2 -0
  73. package/dist/ui/assets/main-AX5XNLtX.js +278 -0
  74. package/dist/ui/assets/{main-Del1GeLf.css → main-CFX0_n8J.css} +1 -1
  75. package/dist/ui/chat.html +3 -3
  76. package/dist/ui/index.html +3 -3
  77. package/package.json +1 -1
  78. package/skills/stageflow/references/control-surface.md +1 -1
  79. package/skills/stageflow/references/evals.md +1 -0
  80. package/dist/ui/assets/adapters-C0l4ruiR.js +0 -17
  81. package/dist/ui/assets/chat-CjnsmvqR.js +0 -2
  82. package/dist/ui/assets/chat-DFDd4UjS.css +0 -2
  83. package/dist/ui/assets/main-_yXVqcW9.js +0 -277
package/dist/ui/chat.html CHANGED
@@ -13,10 +13,10 @@
13
13
  </script>
14
14
  <link rel="icon" type="image/svg+xml" href="/stageflow-icon.svg" />
15
15
  <title>Chat</title>
16
- <script type="module" crossorigin src="/assets/chat-CjnsmvqR.js"></script>
16
+ <script type="module" crossorigin src="/assets/chat-BD1-VTT6.js"></script>
17
17
  <link rel="modulepreload" crossorigin href="/assets/jsx-runtime-iVoFAvQ_.js">
18
- <link rel="modulepreload" crossorigin href="/assets/adapters-C0l4ruiR.js">
19
- <link rel="stylesheet" crossorigin href="/assets/chat-DFDd4UjS.css">
18
+ <link rel="modulepreload" crossorigin href="/assets/adapters-DPD-cAsy.js">
19
+ <link rel="stylesheet" crossorigin href="/assets/chat-DXnQv9st.css">
20
20
  </head>
21
21
  <body>
22
22
  <div id="root"></div>
package/dist/ui/index.html CHANGED
@@ -13,11 +13,11 @@
13
13
  <link rel="stylesheet" href="https://fonts.googleapis.com/css2?family=Figtree:wght@400;500;600;700&display=swap" />
14
14
  <link rel="icon" type="image/svg+xml" href="/stageflow-icon.svg" />
15
15
  <title>Stageflow</title>
16
- <script type="module" crossorigin src="/assets/main-_yXVqcW9.js"></script>
16
+ <script type="module" crossorigin src="/assets/main-AX5XNLtX.js"></script>
17
17
  <link rel="modulepreload" crossorigin href="/assets/jsx-runtime-iVoFAvQ_.js">
18
- <link rel="modulepreload" crossorigin href="/assets/adapters-C0l4ruiR.js">
18
+ <link rel="modulepreload" crossorigin href="/assets/adapters-DPD-cAsy.js">
19
19
  <link rel="modulepreload" crossorigin href="/assets/Tooltip-DrpoRRNi.js">
20
- <link rel="stylesheet" crossorigin href="/assets/main-Del1GeLf.css">
20
+ <link rel="stylesheet" crossorigin href="/assets/main-CFX0_n8J.css">
21
21
  </head>
22
22
  <body>
23
23
  <div id="root"></div>
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "stageflow",
3
- "version": "0.44.0",
3
+ "version": "0.46.0",
4
4
  "description": "Stageflow — CLI pipeline runtime for configurable stages on Pi",
5
5
  "type": "module",
6
6
  "main": "./dist/index.js",
package/skills/stageflow/references/control-surface.md CHANGED
@@ -23,7 +23,7 @@ Start a host with `sf ui` or `sf mcp` when the user wants MCP. Do not auto-start
23
23
 
24
24
  Use MCP tools over the Streamable HTTP endpoint at `{baseUrl}/mcp`. Send `Authorization: Bearer <drive-token>` when `STAGEFLOW_CONTROL_TOKEN` (or a named drive token) is configured. Tool names and payloads live in Stageflow docs: MCP (checkout: [docs/mcp.md](../../../docs/mcp.md)). Typical talking-job tools: `list_pipelines`, `list_tasks`, `start_run`, `run_stage`, `get_run`, `wait_run`, `list_waiting`, `answer_gate`, `decide_feedback_loop`, `get_envelope`, `read_artifact`, `list_checkout_changes`, `get_run_diff`, `read_checkout_file`, `validate`, `describe_pipeline`, `get_health`, `list_providers`, `list_models`, `list_project_mcp`, `probe_project_mcp`, `export_run`, `list_skills`, `list_stage_events`, `get_stage_verification`.
25
25
 
26
- Eval tools (`list_evals`, `get_eval`, `validate_eval`, `run_eval`, `get_eval_run`, `list_eval_runs`, `get_eval_results`, `get_eval_result`, `cancel_eval_run`, `compare_eval_runs`, `get_eval_failure_suggestions`, `get_eval_failure_sort_status`, `sort_eval_failures`) follow the same rule; `run_eval` costs money, and `sort_eval_failures` sends failed results to an external classifier (manual only). See [evals.md](evals.md).
26
+ Eval tools (`list_evals`, `get_eval`, `validate_eval`, `run_eval`, `get_eval_run`, `list_eval_runs`, `get_eval_results`, `get_eval_result`, `cancel_eval_run`, `compare_eval_runs`, `get_eval_failure_suggestions`, `get_eval_failure_sort_status`, `sort_eval_failures`, `list_grader_calibrations`, `calibrate_eval_grader`) follow the same rule; `run_eval` costs money, and `sort_eval_failures` sends failed results to an external classifier (manual only), and `calibrate_eval_grader` re-judges the author's labeled outputs with the judge model (costs money). See [evals.md](evals.md).
27
27
 
28
28
  `get_health` returns capacity fields plus `stageflow_home`, `boot_providers` / `providers_live` (prefer `list_providers` for live auth), and an on-demand `disk` breakdown (`runs_bytes`, `worktrees_bytes`, `repos_bytes`, `state_db_bytes`, `a2a_artifacts_bytes`, `free_bytes`) for the durable root. When `slotsAvailable` is `0`, starts may still succeed as queued until `STAGEFLOW_MAX_QUEUED` is full — see `start_run` in the MCP docs.
29
29
 
package/skills/stageflow/references/evals.md CHANGED
@@ -19,6 +19,7 @@ An eval is `evals/<id>.eval.yaml` (target stage or agent, dataset, graders, defa
19
19
  | **Run an eval** | `sf eval run <id\|path> --json` | `run_eval` | **Calls models and costs real money** |
20
20
  | Read failure suggestions | `sf eval suggestions <evalRunId> [--status ...] --json` | `get_eval_failure_suggestions`, `get_eval_failure_sort_status` | Free |
21
21
  | **Sort failures** | `sf eval sort <evalRunId> [--classifier auto\|jev\|judge] [--limit n] [--resort] --json` | `sort_eval_failures` | **Sends failed results to TypeSafe Jev or the judge model; small cost.** Manual only; tags nothing |
22
+ | **Check a judge against the author's labels** | `sf eval calibrate <evalId> <graderId> [--split dev\|held_out] [--stability] --json` | `calibrate_eval_grader`, `list_grader_calibrations` (read-only) | **Re-judges the author's labeled outputs with the judge model (one call per label); costs money.** Never label results yourself: labels are the author's ground truth |
22
23
  | Stop a run | `sf eval runs cancel <evalRunId>` | `cancel_eval_run` | Stops spending |
23
24
 
24
25
  Never start an eval run on your own. Tell the human what it will do (cases x repeats stage calls, plus judge calls for rubric graders; for an agent, conversations of several turns, plus a simulator model for simulated scenarios and a judge for `rubric` and `goal_reached`) and the cost cap (`defaults.max_cost_usd`, default 2; `--max-cost` overrides it), and wait for a yes. Only one eval run is active per Host.