stageflow 0.42.0 → 0.46.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +12 -3
- package/dist/agent/activity.d.ts +18 -0
- package/dist/agent/claudeAdapter.js +7 -0
- package/dist/agent/claudeTools.d.ts +27 -18
- package/dist/agent/claudeTools.js +54 -15
- package/dist/agent/compactionEvents.d.ts +34 -0
- package/dist/agent/compactionEvents.js +37 -0
- package/dist/agent/openRouterRouting.d.ts +7 -0
- package/dist/agent/openRouterRouting.js +23 -0
- package/dist/agent/piAdapter.d.ts +6 -0
- package/dist/agent/piAdapter.js +128 -27
- package/dist/agent/piSettings.d.ts +7 -0
- package/dist/agent/piSettings.js +9 -0
- package/dist/agent/sandboxPiTools.d.ts +3 -1
- package/dist/agent/sandboxPiTools.js +8 -2
- package/dist/agent/stageTurnDriver.d.ts +11 -2
- package/dist/agent/stageTurnDriver.js +45 -4
- package/dist/agent/toolOutputBudget.d.ts +15 -1
- package/dist/agent/toolOutputBudget.js +18 -0
- package/dist/agent/toolOutputBudgetExtension.d.ts +2 -0
- package/dist/agent/toolOutputBudgetExtension.js +40 -3
- package/dist/agents/authorChat.d.ts +1 -0
- package/dist/agents/authorChat.js +7 -0
- package/dist/agents/authorSessionStore.d.ts +3 -0
- package/dist/agents/authorSessionStore.js +17 -2
- package/dist/agents/evalGate.js +1 -0
- package/dist/agents/hostContext.d.ts +1 -1
- package/dist/agents/runtime/piRuntime.d.ts +5 -1
- package/dist/agents/runtime/piRuntime.js +46 -12
- package/dist/agents/service.js +3 -2
- package/dist/agents/sessions.js +3 -1
- package/dist/cli/evalCommand.d.ts +1 -1
- package/dist/cli/evalCommand.js +39 -2
- package/dist/cli/hostClient.d.ts +6 -0
- package/dist/cli/hostClient.js +4 -0
- package/dist/compaction/directives.d.ts +18 -0
- package/dist/compaction/directives.js +119 -0
- package/dist/compaction/extension.d.ts +20 -0
- package/dist/compaction/extension.js +78 -0
- package/dist/compaction/mask.d.ts +18 -0
- package/dist/compaction/mask.js +155 -0
- package/dist/compaction/pin.d.ts +21 -0
- package/dist/compaction/pin.js +270 -0
- package/dist/compaction/scrub.d.ts +7 -0
- package/dist/compaction/scrub.js +26 -0
- package/dist/compaction/sessionWiring.d.ts +22 -0
- package/dist/compaction/sessionWiring.js +26 -0
- package/dist/compaction/summarize.d.ts +79 -0
- package/dist/compaction/summarize.js +171 -0
- package/dist/compaction/toolPolicy.d.ts +3 -0
- package/dist/compaction/toolPolicy.js +33 -0
- package/dist/compaction/types.d.ts +45 -0
- package/dist/compaction/types.js +7 -0
- package/dist/compaction/workingState.d.ts +56 -0
- package/dist/compaction/workingState.js +198 -0
- package/dist/config/createPipeline.js +2 -0
- package/dist/config/createTrigger.js +7 -1
- package/dist/config/ensureCatalogScanRoot.d.ts +2 -0
- package/dist/config/ensureCatalogScanRoot.js +6 -0
- package/dist/config/evalCatalog.js +30 -2
- package/dist/config/githubTriggerAction.d.ts +8 -0
- package/dist/config/githubTriggerAction.js +29 -0
- package/dist/config/hostConfig.js +2 -0
- package/dist/config/loadEval.d.ts +3 -1
- package/dist/config/loadEval.js +96 -10
- package/dist/config/pipelineNeeds.d.ts +2 -0
- package/dist/config/pipelineNeeds.js +16 -3
- package/dist/config/pipelineRoute.js +5 -4
- package/dist/config/validateCatalog.d.ts +1 -1
- package/dist/config/validateCatalog.js +12 -0
- package/dist/config/validateEval.js +60 -3
- package/dist/config/writeEval.js +2 -2
- package/dist/envelope/assertEmittedArtifacts.d.ts +12 -0
- package/dist/envelope/assertEmittedArtifacts.js +43 -0
- package/dist/envelope/preEmitChecks.d.ts +3 -0
- package/dist/evals/calibration.d.ts +21 -0
- package/dist/evals/calibration.js +69 -0
- package/dist/evals/calibrationSuggest.d.ts +29 -0
- package/dist/evals/calibrationSuggest.js +72 -0
- package/dist/evals/calibrator.d.ts +46 -0
- package/dist/evals/calibrator.js +148 -0
- package/dist/evals/compare.d.ts +1 -1
- package/dist/evals/compare.js +3 -0
- package/dist/evals/constants.d.ts +7 -3
- package/dist/evals/constants.js +7 -1
- package/dist/evals/engine/FakeEngine.d.ts +2 -1
- package/dist/evals/engine/FakeEngine.js +9 -3
- package/dist/evals/engine/promptfoo/PromptfooEngine.d.ts +1 -1
- package/dist/evals/engine/promptfoo/PromptfooEngine.js +4 -2
- package/dist/evals/engine/promptfoo/generate.d.ts +3 -1
- package/dist/evals/engine/promptfoo/generate.js +43 -23
- package/dist/evals/engine/promptfoo/judgeProviders.js +2 -1
- package/dist/evals/engine/promptfoo/normalize.d.ts +4 -1
- package/dist/evals/engine/promptfoo/normalize.js +16 -3
- package/dist/evals/engine/promptfoo/rubricPrompt.d.ts +3 -1
- package/dist/evals/engine/promptfoo/rubricPrompt.js +15 -5
- package/dist/evals/errors.d.ts +7 -0
- package/dist/evals/errors.js +10 -0
- package/dist/evals/judgeProviders.d.ts +6 -0
- package/dist/evals/judgeProviders.js +14 -0
- package/dist/evals/judged.d.ts +7 -0
- package/dist/evals/judged.js +7 -0
- package/dist/evals/manager.d.ts +27 -8
- package/dist/evals/manager.js +73 -12
- package/dist/evals/plan.d.ts +10 -1
- package/dist/evals/plan.js +33 -13
- package/dist/evals/planInternalAgent.d.ts +19 -0
- package/dist/evals/planInternalAgent.js +201 -0
- package/dist/evals/read/api.d.ts +18 -0
- package/dist/evals/read/api.js +10 -1
- package/dist/evals/read/labelQueue.d.ts +45 -0
- package/dist/evals/read/labelQueue.js +83 -0
- package/dist/evals/read/list.js +10 -1
- package/dist/evals/read/runs.d.ts +5 -0
- package/dist/evals/read/runs.js +7 -1
- package/dist/evals/runner/caseRunner.d.ts +2 -1
- package/dist/evals/runner/caseRunner.js +16 -5
- package/dist/evals/runner/internalAgentCaseRunner.d.ts +6 -0
- package/dist/evals/runner/internalAgentCaseRunner.js +250 -0
- package/dist/evals/snapshot.d.ts +10 -2
- package/dist/evals/snapshot.js +8 -3
- package/dist/evals/stats.d.ts +40 -0
- package/dist/evals/stats.js +69 -0
- package/dist/evals/storeTypes.d.ts +42 -0
- package/dist/evals/types.d.ts +25 -2
- package/dist/evals/types.js +4 -1
- package/dist/evals/votes.d.ts +10 -0
- package/dist/evals/votes.js +60 -0
- package/dist/guide/cardContract.d.ts +133 -0
- package/dist/guide/cardContract.js +156 -0
- package/dist/guide/chat.d.ts +45 -0
- package/dist/guide/chat.js +378 -0
- package/dist/guide/knowledge/index.d.ts +14 -0
- package/dist/guide/knowledge/index.js +109 -0
- package/dist/guide/knowledge/topics/agents.d.ts +2 -0
- package/dist/guide/knowledge/topics/agents.js +58 -0
- package/dist/guide/knowledge/topics/bestPractices.d.ts +2 -0
- package/dist/guide/knowledge/topics/bestPractices.js +67 -0
- package/dist/guide/knowledge/topics/connections.d.ts +2 -0
- package/dist/guide/knowledge/topics/connections.js +66 -0
- package/dist/guide/knowledge/topics/consoleMap.d.ts +2 -0
- package/dist/guide/knowledge/topics/consoleMap.js +55 -0
- package/dist/guide/knowledge/topics/evals.d.ts +2 -0
- package/dist/guide/knowledge/topics/evals.js +72 -0
- package/dist/guide/knowledge/topics/limitations.d.ts +2 -0
- package/dist/guide/knowledge/topics/limitations.js +84 -0
- package/dist/guide/knowledge/topics/mcpSkillsBrowser.d.ts +2 -0
- package/dist/guide/knowledge/topics/mcpSkillsBrowser.js +57 -0
- package/dist/guide/knowledge/topics/memories.d.ts +2 -0
- package/dist/guide/knowledge/topics/memories.js +63 -0
- package/dist/guide/knowledge/topics/modelsAndProviders.d.ts +2 -0
- package/dist/guide/knowledge/topics/modelsAndProviders.js +56 -0
- package/dist/guide/knowledge/topics/overview.d.ts +2 -0
- package/dist/guide/knowledge/topics/overview.js +56 -0
- package/dist/guide/knowledge/topics/pipelines.d.ts +2 -0
- package/dist/guide/knowledge/topics/pipelines.js +93 -0
- package/dist/guide/knowledge/topics/stages.d.ts +2 -0
- package/dist/guide/knowledge/topics/stages.js +54 -0
- package/dist/guide/knowledge/topics/tasks.d.ts +2 -0
- package/dist/guide/knowledge/topics/tasks.js +52 -0
- package/dist/guide/knowledge/topics/triggers.d.ts +2 -0
- package/dist/guide/knowledge/topics/triggers.js +68 -0
- package/dist/guide/knowledge/topics/troubleshooting.d.ts +2 -0
- package/dist/guide/knowledge/topics/troubleshooting.js +85 -0
- package/dist/guide/knowledgeTools.d.ts +2 -0
- package/dist/guide/knowledgeTools.js +56 -0
- package/dist/guide/showUi.d.ts +6 -0
- package/dist/guide/showUi.js +546 -0
- package/dist/guide/threadStore.d.ts +48 -0
- package/dist/guide/threadStore.js +189 -0
- package/dist/guide/writeRoot.d.ts +11 -0
- package/dist/guide/writeRoot.js +70 -0
- package/dist/hosted/translate.js +2 -0
- package/dist/mcp/agentTools.js +12 -0
- package/dist/mcp/catalogTools.js +18 -20
- package/dist/mcp/catalogWriteTools.d.ts +8 -0
- package/dist/mcp/catalogWriteTools.js +225 -0
- package/dist/mcp/connectionTools.d.ts +3 -0
- package/dist/mcp/connectionTools.js +85 -0
- package/dist/mcp/controlTools.js +34 -1
- package/dist/mcp/deps.d.ts +9 -0
- package/dist/mcp/evalTools.js +723 -21
- package/dist/mcp/memoryHostTools.d.ts +3 -0
- package/dist/mcp/memoryHostTools.js +72 -0
- package/dist/mcp/pipelineAuthorTools.js +14 -1
- package/dist/mcp/server.js +5 -0
- package/dist/mcp/tools.js +6 -0
- package/dist/mcp/triggerTools.js +77 -31
- package/dist/operatorAgent/evalGate.d.ts +22 -0
- package/dist/operatorAgent/evalGate.js +77 -0
- package/dist/operatorAgent/fakeHost.js +10 -2
- package/dist/operatorAgent/host.js +18 -2
- package/dist/operatorAgent/index.d.ts +1 -0
- package/dist/operatorAgent/index.js +1 -0
- package/dist/operatorAgent/internalAgents.d.ts +11 -0
- package/dist/operatorAgent/internalAgents.js +13 -0
- package/dist/operatorAgent/piModel.d.ts +5 -1
- package/dist/operatorAgent/piModel.js +15 -10
- package/dist/operatorAgent/profiles/agentAuthor.js +66 -22
- package/dist/operatorAgent/profiles/agentAuthorPlaybook.d.ts +1 -1
- package/dist/operatorAgent/profiles/agentAuthorPlaybook.js +1 -1
- package/dist/operatorAgent/profiles/guideConfirm.d.ts +21 -0
- package/dist/operatorAgent/profiles/guideConfirm.js +100 -0
- package/dist/operatorAgent/profiles/guideTools.d.ts +7 -0
- package/dist/operatorAgent/profiles/guideTools.js +123 -0
- package/dist/operatorAgent/profiles/stageflowGuide.d.ts +7 -0
- package/dist/operatorAgent/profiles/stageflowGuide.js +133 -0
- package/dist/operatorAgent/profiles/workshopAuthor.js +23 -0
- package/dist/operatorAgent/profiles/workshopAuthorPlaybook.d.ts +1 -1
- package/dist/operatorAgent/profiles/workshopAuthorPlaybook.js +9 -1
- package/dist/operatorAgent/profiles/workshopAuthorTools.d.ts +2 -0
- package/dist/operatorAgent/profiles/workshopAuthorTools.js +197 -9
- package/dist/operatorAgent/types.d.ts +23 -0
- package/dist/package-meta.d.ts +1 -1
- package/dist/package-meta.js +1 -1
- package/dist/runstore/port.d.ts +13 -1
- package/dist/runstore/retention.d.ts +2 -1
- package/dist/runstore/retention.js +3 -7
- package/dist/runstore/runStatusClock.d.ts +17 -0
- package/dist/runstore/runStatusClock.js +19 -0
- package/dist/runstore/sqlite/SqliteRunStore.d.ts +11 -1
- package/dist/runstore/sqlite/SqliteRunStore.js +70 -45
- package/dist/runstore/sqlite/calibrationStore.d.ts +10 -0
- package/dist/runstore/sqlite/calibrationStore.js +106 -0
- package/dist/runstore/sqlite/evalStore.d.ts +5 -0
- package/dist/runstore/sqlite/evalStore.js +7 -0
- package/dist/runstore/sqlite/migrations/018-eval-grader-labels.d.ts +11 -0
- package/dist/runstore/sqlite/migrations/018-eval-grader-labels.js +43 -0
- package/dist/runstore/sqlite/migrations/index.js +2 -0
- package/dist/runstore/workspaceLayout.js +6 -1
- package/dist/runtime/artifactDiskInspect.d.ts +14 -0
- package/dist/runtime/artifactDiskInspect.js +52 -0
- package/dist/runtime/artifactPath.d.ts +24 -0
- package/dist/runtime/artifactPath.js +84 -0
- package/dist/runtime/completionCheckRunner.js +15 -43
- package/dist/runtime/githubPollSource.d.ts +22 -4
- package/dist/runtime/githubPollSource.js +93 -8
- package/dist/runtime/priorArtifactGrants.d.ts +8 -0
- package/dist/runtime/priorArtifactGrants.js +105 -0
- package/dist/runtime/runActionEligibility.d.ts +46 -0
- package/dist/runtime/runActionEligibility.js +123 -0
- package/dist/runtime/runManager.d.ts +4 -1
- package/dist/runtime/runManager.js +52 -10
- package/dist/runtime/runRetryCoordinator.d.ts +2 -14
- package/dist/runtime/runRetryCoordinator.js +3 -67
- package/dist/runtime/sandboxContainer.d.ts +7 -0
- package/dist/runtime/sandboxContainer.js +48 -6
- package/dist/runtime/sandboxDesk.d.ts +2 -1
- package/dist/runtime/sandboxDesk.js +11 -12
- package/dist/runtime/stageEnvironment.js +1 -0
- package/dist/runtime/stageRoots.d.ts +12 -0
- package/dist/runtime/stageSandbox.d.ts +2 -1
- package/dist/runtime/stageSandbox.js +18 -2
- package/dist/runtime/stageWorker.js +20 -0
- package/dist/server/agentAuthorRoutes.js +12 -3
- package/dist/server/bootstrap.d.ts +5 -0
- package/dist/server/bootstrap.js +14 -1
- package/dist/server/evalRoutes.js +251 -12
- package/dist/server/guideRoutes.d.ts +5 -0
- package/dist/server/guideRoutes.js +166 -0
- package/dist/server/http.d.ts +10 -0
- package/dist/server/http.js +35 -1
- package/dist/server/internalAgentRoutes.d.ts +6 -0
- package/dist/server/internalAgentRoutes.js +12 -0
- package/dist/server/mcpHost.js +2 -0
- package/dist/toolkit/meta.d.ts +1 -1
- package/dist/toolkit/meta.js +41 -1
- package/dist/toolkit/profiles.d.ts +5 -0
- package/dist/toolkit/profiles.js +12 -0
- package/dist/tools/askOperator.js +2 -1
- package/dist/tools/emitStageEnvelope.d.ts +17 -16
- package/dist/tools/emitStageEnvelope.js +46 -32
- package/dist/tools/stageToolNames.d.ts +4 -0
- package/dist/tools/stageToolNames.js +4 -0
- package/dist/tools/workingState.d.ts +40 -0
- package/dist/tools/workingState.js +108 -0
- package/dist/tools/writeStageArtifact.d.ts +0 -9
- package/dist/tools/writeStageArtifact.js +2 -1
- package/dist/ui/assets/{MarkdownText-BTrOqu5J.js → MarkdownText-BS2BBbCx.js} +2 -2
- package/dist/ui/assets/adapters-DPD-cAsy.js +17 -0
- package/dist/ui/assets/chat-BD1-VTT6.js +2 -0
- package/dist/ui/assets/chat-DXnQv9st.css +2 -0
- package/dist/ui/assets/main-AX5XNLtX.js +278 -0
- package/dist/ui/assets/{main-CHlmTNJe.css → main-CFX0_n8J.css} +1 -1
- package/dist/ui/chat.html +3 -4
- package/dist/ui/index.html +3 -4
- package/dist/workshop/chatTurn.js +1 -1
- package/dist/workshop/workshopPin.d.ts +3 -3
- package/dist/workshop/workshopPin.js +16 -15
- package/examples/compaction-stress/README.md +30 -0
- package/examples/compaction-stress/check-run.mjs +53 -0
- package/examples/compaction-stress/collect-evidence.yaml +60 -0
- package/examples/compaction-stress/compaction-stress.pipeline.yaml +141 -0
- package/examples/compaction-stress/compaction-stress.task.yaml +15 -0
- package/examples/compaction-stress/generate-evidence.mjs +110 -0
- package/examples/compaction-stress/root-cause.yaml +26 -0
- package/examples/compaction-stress/stageflow.yaml +9 -0
- package/examples/compaction-stress/write-postmortem.yaml +40 -0
- package/examples/evals/README.md +18 -0
- package/examples/evals/guide-behavior.dataset.jsonl +15 -0
- package/examples/evals/guide-behavior.eval.yaml +35 -0
- package/examples/evals/guide-depth.dataset.jsonl +35 -0
- package/examples/evals/guide-depth.eval.yaml +169 -0
- package/examples/memory/evals/guide-behavior.dataset.jsonl +15 -0
- package/examples/memory/evals/guide-behavior.eval.yaml +35 -0
- package/examples/memory/evals/guide-depth.dataset.jsonl +35 -0
- package/examples/memory/evals/guide-depth.eval.yaml +169 -0
- package/package.json +1 -1
- package/skills/stageflow/references/control-surface.md +1 -1
- package/skills/stageflow/references/evals.md +1 -0
- package/dist/ui/assets/AssistantRuntimeProvider-qIX1xNoH.js +0 -9
- package/dist/ui/assets/ReadonlyStore-BfXqHykm.js +0 -6
- package/dist/ui/assets/chat-Df_wCpak.js +0 -5
- package/dist/ui/assets/chat-P7KQznHm.css +0 -2
- package/dist/ui/assets/main-CvVfxmMd.js +0 -271
package/dist/ui/chat.html
CHANGED
|
@@ -13,11 +13,10 @@
|
|
|
13
13
|
</script>
|
|
14
14
|
<link rel="icon" type="image/svg+xml" href="/stageflow-icon.svg" />
|
|
15
15
|
<title>Chat</title>
|
|
16
|
-
<script type="module" crossorigin src="/assets/chat-
|
|
16
|
+
<script type="module" crossorigin src="/assets/chat-BD1-VTT6.js"></script>
|
|
17
17
|
<link rel="modulepreload" crossorigin href="/assets/jsx-runtime-iVoFAvQ_.js">
|
|
18
|
-
<link rel="modulepreload" crossorigin href="/assets/
|
|
19
|
-
<link rel="
|
|
20
|
-
<link rel="stylesheet" crossorigin href="/assets/chat-P7KQznHm.css">
|
|
18
|
+
<link rel="modulepreload" crossorigin href="/assets/adapters-DPD-cAsy.js">
|
|
19
|
+
<link rel="stylesheet" crossorigin href="/assets/chat-DXnQv9st.css">
|
|
21
20
|
</head>
|
|
22
21
|
<body>
|
|
23
22
|
<div id="root"></div>
|
package/dist/ui/index.html
CHANGED
|
@@ -13,12 +13,11 @@
|
|
|
13
13
|
<link rel="stylesheet" href="https://fonts.googleapis.com/css2?family=Figtree:wght@400;500;600;700&display=swap" />
|
|
14
14
|
<link rel="icon" type="image/svg+xml" href="/stageflow-icon.svg" />
|
|
15
15
|
<title>Stageflow</title>
|
|
16
|
-
<script type="module" crossorigin src="/assets/main-
|
|
16
|
+
<script type="module" crossorigin src="/assets/main-AX5XNLtX.js"></script>
|
|
17
17
|
<link rel="modulepreload" crossorigin href="/assets/jsx-runtime-iVoFAvQ_.js">
|
|
18
|
-
<link rel="modulepreload" crossorigin href="/assets/
|
|
19
|
-
<link rel="modulepreload" crossorigin href="/assets/AssistantRuntimeProvider-qIX1xNoH.js">
|
|
18
|
+
<link rel="modulepreload" crossorigin href="/assets/adapters-DPD-cAsy.js">
|
|
20
19
|
<link rel="modulepreload" crossorigin href="/assets/Tooltip-DrpoRRNi.js">
|
|
21
|
-
<link rel="stylesheet" crossorigin href="/assets/main-
|
|
20
|
+
<link rel="stylesheet" crossorigin href="/assets/main-CFX0_n8J.css">
|
|
22
21
|
</head>
|
|
23
22
|
<body>
|
|
24
23
|
<div id="root"></div>
|
|
@@ -276,7 +276,7 @@ export async function runWorkshopChatTurn(input) {
|
|
|
276
276
|
}
|
|
277
277
|
rawEvents = await agentSession.send(prompt, {
|
|
278
278
|
onDelta: input.onDelta,
|
|
279
|
-
onActivity: (update) => {
|
|
279
|
+
onActivity: ({ args: _args, result: _result, ...update }) => {
|
|
280
280
|
const buildId = registry.getPinnedBuildId(sessionId);
|
|
281
281
|
input.onActivity?.({
|
|
282
282
|
...update,
|
|
@@ -50,7 +50,7 @@ export declare function recordPinnedWorkshopSave(context: unknown, saved: {
|
|
|
50
50
|
}): void;
|
|
51
51
|
/**
|
|
52
52
|
* Create an untitled build and move this turn's pin onto it.
|
|
53
|
-
*
|
|
53
|
+
* When a pin already exists, return that build without creating another.
|
|
54
54
|
*/
|
|
55
55
|
export declare function pinWorkshopBuildOnCreate(input: {
|
|
56
56
|
sessionId: string;
|
|
@@ -59,8 +59,8 @@ export declare function pinWorkshopBuildOnCreate(input: {
|
|
|
59
59
|
storeRoot: string;
|
|
60
60
|
}): WorkshopBuildRecord;
|
|
61
61
|
/**
|
|
62
|
-
* Focus a build.
|
|
63
|
-
*
|
|
62
|
+
* Focus a build. Mid-turn, always move the mutation pin onto it so the studio
|
|
63
|
+
* pointer and draft tools stay locked to the same build.
|
|
64
64
|
*/
|
|
65
65
|
export declare function focusWorkshopBuildPointer(input: {
|
|
66
66
|
sessionId: string;
|
|
@@ -175,20 +175,26 @@ export function recordPinnedWorkshopSave(context, saved) {
|
|
|
175
175
|
}
|
|
176
176
|
/**
|
|
177
177
|
* Create an untitled build and move this turn's pin onto it.
|
|
178
|
-
*
|
|
178
|
+
* When a pin already exists, return that build without creating another.
|
|
179
179
|
*/
|
|
180
180
|
export function pinWorkshopBuildOnCreate(input) {
|
|
181
|
-
const build = createWorkshopBuild(input.storeRoot, { draft: input.draft });
|
|
182
181
|
registerLiveWorkshopSession(input.sessionId, {
|
|
183
182
|
storeRoot: input.storeRoot,
|
|
184
183
|
registry: input.registry,
|
|
185
184
|
});
|
|
185
|
+
const existingId = input.registry.getPinnedBuildId(input.sessionId);
|
|
186
|
+
if (existingId) {
|
|
187
|
+
const existing = readWorkshopBuild(input.storeRoot, existingId);
|
|
188
|
+
if (existing)
|
|
189
|
+
return existing;
|
|
190
|
+
}
|
|
191
|
+
const build = createWorkshopBuild(input.storeRoot, { draft: input.draft });
|
|
186
192
|
movePinOntoBuild(input.registry, input.sessionId, build, input.storeRoot);
|
|
187
193
|
return build;
|
|
188
194
|
}
|
|
189
195
|
/**
|
|
190
|
-
* Focus a build.
|
|
191
|
-
*
|
|
196
|
+
* Focus a build. Mid-turn, always move the mutation pin onto it so the studio
|
|
197
|
+
* pointer and draft tools stay locked to the same build.
|
|
192
198
|
*/
|
|
193
199
|
export function focusWorkshopBuildPointer(input) {
|
|
194
200
|
const build = getWorkshopBuild(input.storeRoot, input.buildId);
|
|
@@ -196,18 +202,13 @@ export function focusWorkshopBuildPointer(input) {
|
|
|
196
202
|
storeRoot: input.storeRoot,
|
|
197
203
|
registry: input.registry,
|
|
198
204
|
});
|
|
199
|
-
const pinned = input.registry.getPinnedBuildId(input.sessionId);
|
|
200
205
|
const turnLive = input.registry.getLiveTurn(input.sessionId);
|
|
201
|
-
|
|
202
|
-
if (pinMoved) {
|
|
206
|
+
if (turnLive) {
|
|
203
207
|
movePinOntoBuild(input.registry, input.sessionId, build, input.storeRoot);
|
|
208
|
+
return { pinMoved: true, activeBuildId: build.id };
|
|
204
209
|
}
|
|
205
|
-
|
|
206
|
-
|
|
207
|
-
|
|
208
|
-
|
|
209
|
-
if (turnLive)
|
|
210
|
-
emitPointerChange(input.registry, input.sessionId, build);
|
|
211
|
-
}
|
|
212
|
-
return { pinMoved, activeBuildId: build.id };
|
|
210
|
+
enqueueSessionWrite(input.sessionId, () => {
|
|
211
|
+
updateWorkshopSessionActiveBuildId(input.storeRoot, input.sessionId, build.id);
|
|
212
|
+
});
|
|
213
|
+
return { pinMoved: false, activeBuildId: build.id };
|
|
213
214
|
}
|
|
@@ -0,0 +1,30 @@
|
|
|
1
|
+
# compaction-stress
|
|
2
|
+
|
|
3
|
+
A three-stage incident postmortem whose first stage must read about 1 MB of logs (12 shards)
|
|
4
|
+
through a ~128k-163k context window, so its session compacts several times. It checks that a
|
|
5
|
+
stage still emits a correct envelope after compaction and that the next stage uses it.
|
|
6
|
+
|
|
7
|
+
## Run
|
|
8
|
+
|
|
9
|
+
1. Generate the evidence: `node examples/compaction-stress/generate-evidence.mjs <dir>`
|
|
10
|
+
(also writes `<dir>.expected.json`, the ground truth, outside the checkout).
|
|
11
|
+
2. Make `<dir>` the task checkout (`/data/evidence` in the Docker recipe; see
|
|
12
|
+
`docker-compose.compaction-stress.yml`). It must be a plain directory, not a mount.
|
|
13
|
+
3. Start `compaction-stress` in the console, or `sf run`.
|
|
14
|
+
4. Check the result: `node examples/compaction-stress/check-run.mjs <runId> <dir>.expected.json`.
|
|
15
|
+
|
|
16
|
+
## If a model leaks tool calls
|
|
17
|
+
|
|
18
|
+
Some OpenRouter providers return a model's tool calls as plain text (for example `<|tool▁calls▁begin|>` in the reasoning), so a stage ends with `missing emit_stage_envelope`. Pin the providers in the Host's `$STAGEFLOW_HOME/agent/models.json`:
|
|
19
|
+
|
|
20
|
+
```json
|
|
21
|
+
{ "providers": { "openrouter": { "modelOverrides": { "deepseek/deepseek-chat-v3.1": {
|
|
22
|
+
"compat": { "openRouterRouting": { "order": ["SiliconFlow", "CoreWeave"], "allow_fallbacks": false } } } } } } }
|
|
23
|
+
```
|
|
24
|
+
|
|
25
|
+
## What to look for
|
|
26
|
+
|
|
27
|
+
- The stage inspector's Context row climbs and drops after each compaction.
|
|
28
|
+
- `read`/`bash` results beyond the per-turn cap are replaced by "output not returned" notices.
|
|
29
|
+
- The envelope matches `expected.json`: incident id, build, retry policy, failed checkouts,
|
|
30
|
+
duration, recovery time, all 12 shards, and the cert-expiry red herring in `ruled_out`.
|
|
@@ -0,0 +1,53 @@
|
|
|
1
|
+
#!/usr/bin/env node
|
|
2
|
+
// Compares a finished compaction-stress run with the ground truth.
|
|
3
|
+
// Usage: node check-run.mjs <runId> <expected.json> [baseUrl]
|
|
4
|
+
import { readFileSync } from "node:fs";
|
|
5
|
+
|
|
6
|
+
const [runId, expectedPath, base = "http://127.0.0.1:3850"] = process.argv.slice(2);
|
|
7
|
+
if (!runId || !expectedPath) {
|
|
8
|
+
console.error("usage: check-run.mjs <runId> <expected.json> [baseUrl]");
|
|
9
|
+
process.exit(2);
|
|
10
|
+
}
|
|
11
|
+
const expected = JSON.parse(readFileSync(expectedPath, "utf8"));
|
|
12
|
+
const body = await (await fetch(`${base}/api/runs/${runId}`)).json();
|
|
13
|
+
const run = body.run ?? body;
|
|
14
|
+
const stage = (id) => run.stages.find((s) => s.stage_id === id);
|
|
15
|
+
const payload = (id) => stage(id)?.envelope?.payload ?? {};
|
|
16
|
+
const evidence = payload("collect-evidence");
|
|
17
|
+
const rootCause = payload("root-cause");
|
|
18
|
+
const post = payload("write-postmortem");
|
|
19
|
+
const compactions = (stage("collect-evidence")?.events ?? []).filter((e) => e.event === "context_usage");
|
|
20
|
+
|
|
21
|
+
const checks = [
|
|
22
|
+
["run succeeded", run.status === "succeeded"],
|
|
23
|
+
["incident id", evidence.incident_id === expected.incident_id],
|
|
24
|
+
["build", evidence.suspected_change?.build === expected.root_cause_build],
|
|
25
|
+
["flag", evidence.suspected_change?.flag === expected.root_cause_flag],
|
|
26
|
+
[
|
|
27
|
+
"retry policy",
|
|
28
|
+
evidence.suspected_change?.retry_policy ===
|
|
29
|
+
`max_attempts=${expected.retry_policy.max_attempts} backoff=${expected.retry_policy.backoff} jitter=${expected.retry_policy.jitter}`,
|
|
30
|
+
],
|
|
31
|
+
["failed checkouts", evidence.impact?.failed_checkouts === expected.failed_checkouts],
|
|
32
|
+
["duration", evidence.impact?.duration_min === expected.duration_min],
|
|
33
|
+
["recovery time", String(evidence.impact?.recovery_ts ?? "").endsWith(expected.recovery_utc)],
|
|
34
|
+
["all 12 shards listed", (evidence.shards_reviewed ?? []).length >= 12],
|
|
35
|
+
["red herring ruled out", /cert/i.test(JSON.stringify(evidence.ruled_out ?? []))],
|
|
36
|
+
["root cause names the build or flag", [expected.root_cause_build, expected.root_cause_flag].some((x) => JSON.stringify(rootCause).includes(x))],
|
|
37
|
+
["root cause keeps the impact numbers", rootCause.impact?.failed_checkouts === expected.failed_checkouts],
|
|
38
|
+
["postmortem numbers match", post.failed_checkouts === expected.failed_checkouts && post.duration_min === expected.duration_min && String(post.recovery_ts ?? "").endsWith(expected.recovery_utc)],
|
|
39
|
+
["postmortem carries incident id", post.incident_id === expected.incident_id],
|
|
40
|
+
["postmortem has 4+ action items", (post.action_items ?? []).length >= 4],
|
|
41
|
+
["context was tracked", compactions.length > 0],
|
|
42
|
+
[
|
|
43
|
+
"context dropped at least once (compaction)",
|
|
44
|
+
compactions.some((e, i) => i > 0 && e.percent < compactions[i - 1].percent - 20),
|
|
45
|
+
],
|
|
46
|
+
];
|
|
47
|
+
let failed = 0;
|
|
48
|
+
for (const [name, ok] of checks) {
|
|
49
|
+
if (!ok) failed += 1;
|
|
50
|
+
console.log(`${ok ? "PASS" : "FAIL"} ${name}`);
|
|
51
|
+
}
|
|
52
|
+
console.log(`\n${checks.length - failed}/${checks.length} passed`);
|
|
53
|
+
process.exit(failed === 0 ? 0 : 1);
|
|
@@ -0,0 +1,60 @@
|
|
|
1
|
+
id: collect-evidence
|
|
2
|
+
system_prompt: |
|
|
3
|
+
You are an incident analyst collecting evidence for a postmortem.
|
|
4
|
+
|
|
5
|
+
Start by listing the checkout (ls logs, ls logs/*, ls runbooks) and use only
|
|
6
|
+
paths that listing shows; never guess a file name. The services are
|
|
7
|
+
api-gateway, payments-api, ledger-db, order-queue, notification-worker and
|
|
8
|
+
auth-service, each with two shards such as
|
|
9
|
+
logs/payments-api/payments-api-01.log (12 shards in total).
|
|
10
|
+
|
|
11
|
+
Read EVERY log shard under logs/ in the checkout and the runbooks. Each shard
|
|
12
|
+
is about 48 KB, so open them ONE AT A TIME: one read call per turn, never
|
|
13
|
+
several in the same turn, and never read a file you already noted. Your
|
|
14
|
+
conversation is trimmed automatically as it grows, so your notes file is the
|
|
15
|
+
only memory you can rely on. After EACH shard, call write_stage_artifact for
|
|
16
|
+
evidence-notes.md with the complete notes so far (it replaces the previous
|
|
17
|
+
version): per shard, the exact log lines that matter (errors, warnings,
|
|
18
|
+
deploys and flag changes, state changes, incident ids, counts), with their
|
|
19
|
+
timestamps copied verbatim. Keep it short: at most 5 lines per shard, and
|
|
20
|
+
skip routine INFO lines. A shard you did not open
|
|
21
|
+
must not appear in shards_reviewed.
|
|
22
|
+
|
|
23
|
+
Before you emit, read evidence-notes.md back and build the envelope only from
|
|
24
|
+
it. Never write a number, id or build that is not in your notes.
|
|
25
|
+
|
|
26
|
+
Look for the incident id, the change that started it, the first failing
|
|
27
|
+
dependency, the customer impact, and the recovery. Ignore unrelated
|
|
28
|
+
warnings, but list them in ruled_out so the next stage does not chase them.
|
|
29
|
+
|
|
30
|
+
Contract details (follow them exactly):
|
|
31
|
+
- Timeline entries use ISO-8601 UTC timestamps (YYYY-MM-DDTHH:MM:SSZ), one
|
|
32
|
+
entry per key event, oldest first, at least 8 entries.
|
|
33
|
+
- suspected_change.retry_policy is a single string of the form
|
|
34
|
+
"max_attempts=<n> backoff=<x> jitter=<y>" copied from the log.
|
|
35
|
+
- impact.failed_checkouts and duration_min are integers taken from the
|
|
36
|
+
incident summary line, not estimated.
|
|
37
|
+
- shards_reviewed lists the relative file paths you actually read.
|
|
38
|
+
- List evidence-notes.md in artifacts.
|
|
39
|
+
Call emit_stage_envelope exactly once when done.
|
|
40
|
+
model: openrouter/deepseek/deepseek-chat-v3.1
|
|
41
|
+
timeout_ms: 2400000
|
|
42
|
+
io:
|
|
43
|
+
input:
|
|
44
|
+
schema:
|
|
45
|
+
type: object
|
|
46
|
+
output:
|
|
47
|
+
schema:
|
|
48
|
+
$ref: "#/schemas/evidence-report"
|
|
49
|
+
verify:
|
|
50
|
+
- id: notes-declared
|
|
51
|
+
type: artifact
|
|
52
|
+
basename: evidence-notes.md
|
|
53
|
+
when:
|
|
54
|
+
- emit
|
|
55
|
+
- id: notes-on-disk
|
|
56
|
+
type: artifact
|
|
57
|
+
path: evidence-notes.md
|
|
58
|
+
nonempty: true
|
|
59
|
+
when:
|
|
60
|
+
- after
|
|
@@ -0,0 +1,141 @@
|
|
|
1
|
+
id: compaction-stress
|
|
2
|
+
model: openrouter/deepseek/deepseek-chat-v3.1
|
|
3
|
+
schemas:
|
|
4
|
+
evidence-report:
|
|
5
|
+
type: object
|
|
6
|
+
required:
|
|
7
|
+
- incident_id
|
|
8
|
+
- timeline
|
|
9
|
+
- shards_reviewed
|
|
10
|
+
- suspected_change
|
|
11
|
+
- impact
|
|
12
|
+
- ruled_out
|
|
13
|
+
properties:
|
|
14
|
+
incident_id:
|
|
15
|
+
type: string
|
|
16
|
+
pattern: "^INC-[0-9]+$"
|
|
17
|
+
shards_reviewed:
|
|
18
|
+
type: array
|
|
19
|
+
minItems: 12
|
|
20
|
+
items:
|
|
21
|
+
type: string
|
|
22
|
+
pattern: "^logs/[a-z-]+/[a-z-]+-0[1-2]\\.log$"
|
|
23
|
+
timeline:
|
|
24
|
+
type: array
|
|
25
|
+
minItems: 8
|
|
26
|
+
items:
|
|
27
|
+
type: object
|
|
28
|
+
required: [ts, service, event]
|
|
29
|
+
properties:
|
|
30
|
+
ts:
|
|
31
|
+
type: string
|
|
32
|
+
pattern: "^2026-03-14T[0-9]{2}:[0-9]{2}:[0-9]{2}Z$"
|
|
33
|
+
service:
|
|
34
|
+
type: string
|
|
35
|
+
event:
|
|
36
|
+
type: string
|
|
37
|
+
suspected_change:
|
|
38
|
+
type: object
|
|
39
|
+
required: [build, flag, retry_policy]
|
|
40
|
+
properties:
|
|
41
|
+
flag:
|
|
42
|
+
type: string
|
|
43
|
+
retry_policy:
|
|
44
|
+
type: string
|
|
45
|
+
pattern: "^max_attempts=[0-9]+ backoff=[a-z]+ jitter=[a-z]+$"
|
|
46
|
+
build:
|
|
47
|
+
type: string
|
|
48
|
+
pattern: "^[0-9a-f]{7}$"
|
|
49
|
+
impact:
|
|
50
|
+
type: object
|
|
51
|
+
required: [failed_checkouts, duration_min, recovery_ts]
|
|
52
|
+
properties:
|
|
53
|
+
failed_checkouts:
|
|
54
|
+
type: integer
|
|
55
|
+
duration_min:
|
|
56
|
+
type: integer
|
|
57
|
+
recovery_ts:
|
|
58
|
+
type: string
|
|
59
|
+
pattern: "^2026-03-14T[0-9]{2}:[0-9]{2}:[0-9]{2}Z$"
|
|
60
|
+
ruled_out:
|
|
61
|
+
type: array
|
|
62
|
+
items:
|
|
63
|
+
type: string
|
|
64
|
+
root-cause:
|
|
65
|
+
type: object
|
|
66
|
+
required: [incident_id, root_cause, contributing_factors, policy_violations, evidence_refs, impact]
|
|
67
|
+
properties:
|
|
68
|
+
impact:
|
|
69
|
+
type: object
|
|
70
|
+
required: [failed_checkouts, duration_min, recovery_ts]
|
|
71
|
+
properties:
|
|
72
|
+
failed_checkouts:
|
|
73
|
+
type: integer
|
|
74
|
+
duration_min:
|
|
75
|
+
type: integer
|
|
76
|
+
recovery_ts:
|
|
77
|
+
type: string
|
|
78
|
+
pattern: "^2026-03-14T[0-9]{2}:[0-9]{2}:[0-9]{2}Z$"
|
|
79
|
+
incident_id:
|
|
80
|
+
type: string
|
|
81
|
+
pattern: "^INC-[0-9]+$"
|
|
82
|
+
root_cause:
|
|
83
|
+
type: string
|
|
84
|
+
contributing_factors:
|
|
85
|
+
type: array
|
|
86
|
+
items:
|
|
87
|
+
type: string
|
|
88
|
+
policy_violations:
|
|
89
|
+
type: array
|
|
90
|
+
minItems: 1
|
|
91
|
+
items:
|
|
92
|
+
type: string
|
|
93
|
+
pattern: "^runbooks/[a-z-]+\\.md: "
|
|
94
|
+
evidence_refs:
|
|
95
|
+
type: array
|
|
96
|
+
minItems: 3
|
|
97
|
+
items:
|
|
98
|
+
type: string
|
|
99
|
+
pattern: "^logs/[a-z-]+/[a-z-]+-0[1-2]\\.log$"
|
|
100
|
+
postmortem:
|
|
101
|
+
type: object
|
|
102
|
+
required: [incident_id, action_items, postmortem_artifact, failed_checkouts, duration_min, recovery_ts]
|
|
103
|
+
properties:
|
|
104
|
+
failed_checkouts:
|
|
105
|
+
type: integer
|
|
106
|
+
duration_min:
|
|
107
|
+
type: integer
|
|
108
|
+
recovery_ts:
|
|
109
|
+
type: string
|
|
110
|
+
pattern: "^2026-03-14T[0-9]{2}:[0-9]{2}:[0-9]{2}Z$"
|
|
111
|
+
incident_id:
|
|
112
|
+
type: string
|
|
113
|
+
pattern: "^INC-[0-9]+$"
|
|
114
|
+
postmortem_artifact:
|
|
115
|
+
type: string
|
|
116
|
+
action_items:
|
|
117
|
+
type: array
|
|
118
|
+
items:
|
|
119
|
+
type: object
|
|
120
|
+
required: [title, owner, due]
|
|
121
|
+
properties:
|
|
122
|
+
title:
|
|
123
|
+
type: string
|
|
124
|
+
owner:
|
|
125
|
+
type: string
|
|
126
|
+
pattern: "^[a-z-]+$"
|
|
127
|
+
due:
|
|
128
|
+
type: string
|
|
129
|
+
pattern: "^2026-0[34]-[0-9]{2}$"
|
|
130
|
+
stages:
|
|
131
|
+
- id: collect-evidence
|
|
132
|
+
entry: true
|
|
133
|
+
route:
|
|
134
|
+
- to: root-cause
|
|
135
|
+
uses: ./collect-evidence.yaml
|
|
136
|
+
- id: root-cause
|
|
137
|
+
route:
|
|
138
|
+
- to: write-postmortem
|
|
139
|
+
uses: ./root-cause.yaml
|
|
140
|
+
- id: write-postmortem
|
|
141
|
+
uses: ./write-postmortem.yaml
|
|
@@ -0,0 +1,15 @@
|
|
|
1
|
+
id: compaction-stress
|
|
2
|
+
goal: >
|
|
3
|
+
Produce a blameless postmortem for the checkout outage on 2026-03-14
|
|
4
|
+
(02:00-08:00 UTC) from the raw service logs and runbooks in the checkout.
|
|
5
|
+
context: |
|
|
6
|
+
The checkout is an evidence bundle: logs/<service>/<service>-NN.log (six
|
|
7
|
+
services, two shards each, about 1 MB in total) and runbooks/*.md.
|
|
8
|
+
Generate it with: node examples/compaction-stress/generate-evidence.mjs <dir>
|
|
9
|
+
and point `checkout` at <dir>. The logs are deliberately larger than a
|
|
10
|
+
stage's context budget, so the first stage's session must compact.
|
|
11
|
+
constraints: |
|
|
12
|
+
Read the evidence; do not guess. Do not modify the checkout.
|
|
13
|
+
Blameless tone: name changes and systems, not people, except the oncall
|
|
14
|
+
who is named only as the responder.
|
|
15
|
+
checkout: /data/evidence
|
|
@@ -0,0 +1,110 @@
|
|
|
1
|
+
#!/usr/bin/env node
|
|
2
|
+
// Writes a deterministic ~700 KB incident log set plus expected.json (ground truth, written next to the dir so the stage cannot read it).
|
|
3
|
+
// Usage: node generate-evidence.mjs <outDir>
|
|
4
|
+
import { mkdirSync, writeFileSync } from "node:fs";
|
|
5
|
+
import path from "node:path";
|
|
6
|
+
|
|
7
|
+
const outDir = path.resolve(process.argv[2] ?? "./evidence");
|
|
8
|
+
const DATE = "2026-03-14";
|
|
9
|
+
const SERVICES = ["api-gateway", "payments-api", "ledger-db", "order-queue", "notification-worker", "auth-service"];
|
|
10
|
+
const SHARDS_PER_SERVICE = 2;
|
|
11
|
+
const LINES_PER_SHARD = 400;
|
|
12
|
+
const START = Date.parse(`${DATE}T02:00:00Z`);
|
|
13
|
+
const END = Date.parse(`${DATE}T08:00:00Z`);
|
|
14
|
+
|
|
15
|
+
let seed = 20260314;
|
|
16
|
+
const rnd = () => {
|
|
17
|
+
seed = (seed * 1664525 + 1013904223) >>> 0;
|
|
18
|
+
return seed / 2 ** 32;
|
|
19
|
+
};
|
|
20
|
+
const pick = (xs) => xs[Math.floor(rnd() * xs.length)];
|
|
21
|
+
const hex = (n) => Array.from({ length: n }, () => "0123456789abcdef"[Math.floor(rnd() * 16)]).join("");
|
|
22
|
+
const iso = (ms) => new Date(ms).toISOString().replace(".000Z", "Z");
|
|
23
|
+
|
|
24
|
+
const noise = {
|
|
25
|
+
"api-gateway": () => `INFO route=${pick(["/v2/checkout", "/v2/cart", "/v2/orders", "/v2/profile"])} status=200 latency_ms=${20 + Math.floor(rnd() * 90)} req=${hex(12)} client=${pick(["web", "ios", "android"])}`,
|
|
26
|
+
"payments-api": () => `INFO charge_authorized order=${hex(8)} amount_cents=${500 + Math.floor(rnd() * 20000)} psp=${pick(["stripe", "adyen"])} attempt=1 dur_ms=${80 + Math.floor(rnd() * 200)}`,
|
|
27
|
+
"ledger-db": () => `INFO txn_commit id=${hex(10)} rows=${1 + Math.floor(rnd() * 4)} pool_active=${20 + Math.floor(rnd() * 40)}/200 dur_ms=${2 + Math.floor(rnd() * 15)}`,
|
|
28
|
+
"order-queue": () => `INFO msg_ack topic=${pick(["order.created", "order.paid", "order.shipped"])} depth=${Math.floor(rnd() * 40)} consumer=c-${Math.floor(rnd() * 8)}`,
|
|
29
|
+
"notification-worker": () => `INFO sent channel=${pick(["email", "push", "sms"])} template=${pick(["receipt", "shipping", "promo"])} user=u${Math.floor(rnd() * 90000)}`,
|
|
30
|
+
"auth-service": () => `INFO token_issued user=u${Math.floor(rnd() * 90000)} ttl_s=3600 mfa=${pick(["totp", "none", "webauthn"])}`,
|
|
31
|
+
};
|
|
32
|
+
|
|
33
|
+
// Needles: [service, UTC time of day, line]. The shard and position follow the time.
|
|
34
|
+
const needles = [
|
|
35
|
+
["payments-api", "02:08:07", "INFO deploy build=7f3a91c author=dkaplan flag payments.retry_v2=on rollout=100% retry_policy max_attempts=7 backoff=none jitter=none"],
|
|
36
|
+
["payments-api", "02:19:30", "WARN psp_timeout psp=stripe order=ab12cd34 attempt=2 retrying immediately"],
|
|
37
|
+
["ledger-db", "02:31:44", "ERROR connection pool exhausted active=200/200 waiting=1873 first_seen=02:31:44Z"],
|
|
38
|
+
["order-queue", "03:02:10", "ERROR dlq_growth topic=order.paid dlq_depth=5000 oldest_msg_age_s=612"],
|
|
39
|
+
["api-gateway", "03:10:12", "ERROR slo_breach route=/v2/checkout error_rate=38% window=5m paged_oncall=maria.chen incident_declared=INC-4471"],
|
|
40
|
+
["notification-worker", "03:55:00", "WARN backlog receipts_pending=22914 throttled_by=order-queue"],
|
|
41
|
+
["auth-service", "04:15:00", "WARN cert_expiry host=auth-internal.example.net expires_in_days=9 (unrelated to incident)"],
|
|
42
|
+
["payments-api", "05:47:30", "INFO rollback build=7f3a91c -> 6e21b0d flag payments.retry_v2=off by=maria.chen"],
|
|
43
|
+
["ledger-db", "05:49:10", "INFO connection pool recovered active=64/200 waiting=0"],
|
|
44
|
+
["api-gateway", "05:52:03", "INFO recovery route=/v2/checkout error_rate=0.4% window=5m resolved_at=05:52:03Z"],
|
|
45
|
+
["api-gateway", "07:40:00", "INFO incident_summary id=INC-4471 failed_checkouts=48211 duration_min=201 sev=1"],
|
|
46
|
+
];
|
|
47
|
+
const needleMs = (hhmmss) => Date.parse(`${DATE}T${hhmmss}Z`);
|
|
48
|
+
|
|
49
|
+
for (const service of SERVICES) {
|
|
50
|
+
mkdirSync(path.join(outDir, "logs", service), { recursive: true });
|
|
51
|
+
for (let shard = 1; shard <= SHARDS_PER_SERVICE; shard += 1) {
|
|
52
|
+
const span = (END - START) / SHARDS_PER_SERVICE;
|
|
53
|
+
const t0 = START + (shard - 1) * span;
|
|
54
|
+
const lines = [];
|
|
55
|
+
for (let i = 0; i < LINES_PER_SHARD; i += 1) {
|
|
56
|
+
const ts = t0 + (i / LINES_PER_SHARD) * span;
|
|
57
|
+
lines.push(`${iso(Math.floor(ts / 1000) * 1000)} ${service} ${noise[service]()}`);
|
|
58
|
+
}
|
|
59
|
+
for (const [svc, time, text] of needles) {
|
|
60
|
+
const ms = needleMs(time);
|
|
61
|
+
if (svc !== service || Math.floor((ms - START) / span) + 1 !== shard) continue;
|
|
62
|
+
const at = Math.floor(((ms - t0) / span) * LINES_PER_SHARD);
|
|
63
|
+
lines.splice(at, 0, `${iso(ms)} ${service} ${text}`);
|
|
64
|
+
}
|
|
65
|
+
writeFileSync(path.join(outDir, "logs", service, `${service}-${String(shard).padStart(2, "0")}.log`), `${lines.join("\n")}\n`);
|
|
66
|
+
}
|
|
67
|
+
}
|
|
68
|
+
|
|
69
|
+
mkdirSync(path.join(outDir, "runbooks"), { recursive: true });
|
|
70
|
+
writeFileSync(
|
|
71
|
+
path.join(outDir, "runbooks", "payments-retry-policy.md"),
|
|
72
|
+
`# Payments retry policy (owner: payments-platform)
|
|
73
|
+
|
|
74
|
+
- Client retries to a PSP must use exponential backoff with full jitter.
|
|
75
|
+
- max_attempts must not exceed 3 without a circuit breaker.
|
|
76
|
+
- Feature flags that change retry behavior require a canary at 5% for 30 minutes before 100% rollout.
|
|
77
|
+
- A change that raises ledger-db connection demand needs a pool capacity review.
|
|
78
|
+
`,
|
|
79
|
+
);
|
|
80
|
+
writeFileSync(
|
|
81
|
+
path.join(outDir, "runbooks", "severity-and-comms.md"),
|
|
82
|
+
`# Severity and comms
|
|
83
|
+
|
|
84
|
+
- Sev 1: customer-facing checkout failure above 20% for 5 minutes. Page oncall, open an incident channel, post status page update within 15 minutes.
|
|
85
|
+
- Postmortems are due within 5 business days. Action items need an owner and a due date.
|
|
86
|
+
`,
|
|
87
|
+
);
|
|
88
|
+
|
|
89
|
+
writeFileSync(
|
|
90
|
+
`${outDir}.expected.json`,
|
|
91
|
+
JSON.stringify(
|
|
92
|
+
{
|
|
93
|
+
incident_id: "INC-4471",
|
|
94
|
+
root_cause_build: "7f3a91c",
|
|
95
|
+
root_cause_flag: "payments.retry_v2",
|
|
96
|
+
retry_policy: { max_attempts: 7, backoff: "none", jitter: "none" },
|
|
97
|
+
first_pool_exhaustion_utc: "02:31:44Z",
|
|
98
|
+
dlq_depth: 5000,
|
|
99
|
+
paged_oncall: "maria.chen",
|
|
100
|
+
rollback_to_build: "6e21b0d",
|
|
101
|
+
recovery_utc: "05:52:03Z",
|
|
102
|
+
failed_checkouts: 48211,
|
|
103
|
+
duration_min: 201,
|
|
104
|
+
red_herring: "auth-internal.example.net cert expiry",
|
|
105
|
+
},
|
|
106
|
+
null,
|
|
107
|
+
2,
|
|
108
|
+
),
|
|
109
|
+
);
|
|
110
|
+
console.log(`wrote evidence to ${outDir}`);
|
|
@@ -0,0 +1,26 @@
|
|
|
1
|
+
id: root-cause
|
|
2
|
+
system_prompt: |
|
|
3
|
+
You are the incident reviewer. The previous stage's evidence report is in
|
|
4
|
+
the assignment as the prior envelope. Treat it as the source of facts; open
|
|
5
|
+
the runbooks in the checkout to judge the change against policy.
|
|
6
|
+
|
|
7
|
+
Decide the root cause and the contributing factors. Open the runbooks in
|
|
8
|
+
the checkout (ls runbooks first; use only files that exist). Each
|
|
9
|
+
policy_violations entry has the form "runbooks/<file>.md: <the rule it
|
|
10
|
+
breaks, quoted from that file>". Do not restate the timeline. evidence_refs
|
|
11
|
+
lists at least 3 log shards that support the root cause, copied from the
|
|
12
|
+
prior report's shards_reviewed.
|
|
13
|
+
Do not chase anything in the prior report's ruled_out list.
|
|
14
|
+
Carry the prior report's impact object into your payload unchanged
|
|
15
|
+
(failed_checkouts, duration_min, recovery_ts): the next stage writes the
|
|
16
|
+
postmortem from your envelope alone and must not estimate those numbers.
|
|
17
|
+
Call emit_stage_envelope exactly once.
|
|
18
|
+
model: openrouter/deepseek/deepseek-chat-v3.1
|
|
19
|
+
timeout_ms: 1200000
|
|
20
|
+
io:
|
|
21
|
+
input:
|
|
22
|
+
schema:
|
|
23
|
+
$ref: "#/schemas/evidence-report"
|
|
24
|
+
output:
|
|
25
|
+
schema:
|
|
26
|
+
$ref: "#/schemas/root-cause"
|
|
@@ -0,0 +1,40 @@
|
|
|
1
|
+
id: write-postmortem
|
|
2
|
+
system_prompt: |
|
|
3
|
+
You are writing the final blameless postmortem. Inputs: the evidence report
|
|
4
|
+
and the root-cause analysis from the prior envelopes.
|
|
5
|
+
|
|
6
|
+
Write postmortem.md with write_stage_artifact containing: summary, impact
|
|
7
|
+
(exact failed checkout count, duration in minutes and recovery time),
|
|
8
|
+
timeline, root cause, contributing factors, what went well, and action
|
|
9
|
+
items. Every number, time and build id in the document must come from the
|
|
10
|
+
prior envelope; never estimate or round them, and never invent a time the
|
|
11
|
+
envelope does not give. Repeat failed_checkouts, duration_min and
|
|
12
|
+
recovery_ts in your payload exactly as the document states them.
|
|
13
|
+
|
|
14
|
+
Action items: at least 4, each with a title, an owner (a team name such as
|
|
15
|
+
payments-platform or sre, never a person) and a due date in YYYY-MM-DD,
|
|
16
|
+
within 30 days after 2026-03-14. Include one item for the retry policy, one
|
|
17
|
+
for rollout canarying, and one for ledger-db pool capacity.
|
|
18
|
+
Call emit_stage_envelope exactly once with postmortem_artifact set to the
|
|
19
|
+
path write_stage_artifact returned.
|
|
20
|
+
model: openrouter/deepseek/deepseek-chat-v3.1
|
|
21
|
+
timeout_ms: 1200000
|
|
22
|
+
io:
|
|
23
|
+
input:
|
|
24
|
+
schema:
|
|
25
|
+
$ref: "#/schemas/root-cause"
|
|
26
|
+
output:
|
|
27
|
+
schema:
|
|
28
|
+
$ref: "#/schemas/postmortem"
|
|
29
|
+
verify:
|
|
30
|
+
- id: postmortem-declared
|
|
31
|
+
type: artifact
|
|
32
|
+
basename: postmortem.md
|
|
33
|
+
when:
|
|
34
|
+
- emit
|
|
35
|
+
- id: postmortem-on-disk
|
|
36
|
+
type: artifact
|
|
37
|
+
path: postmortem.md
|
|
38
|
+
nonempty: true
|
|
39
|
+
when:
|
|
40
|
+
- after
|
package/examples/evals/README.md
CHANGED
|
@@ -121,3 +121,21 @@ graders:
|
|
|
121
121
|
```
|
|
122
122
|
|
|
123
123
|
A simulator model and the judge read the conversation, so the run costs more and varies more between repeats. See [Evals for agents](../../docs/evals.md#evals-for-agents).
|
|
124
|
+
|
|
125
|
+
# Guide behavior (internal agent)
|
|
126
|
+
|
|
127
|
+
`guide-behavior.eval.yaml` checks the built-in Guide. The Host must be started with `STAGEFLOW_SHOW_INTERNAL_AGENTS=1`. The target is `stageflow-guide`. Tools stay `read_only`: a write, send or delete is recorded and not executed, so the graders can require `confirm_action` or `show_ui` and forbid the write in the same breath.
|
|
128
|
+
|
|
129
|
+
It is strict on purpose. Every grader must pass and the pass threshold is 1. The 15 scenarios cover off-topic requests, a vague request, the wrong building block, a secret in a file, a schedule before a first run, `needs` instead of `route`, a prompt-injection instruction, planning before a multi-step build, building only after a yes, and a limit the short playbook does not list (OAuth MCP on a stage, which must be looked up). A model that builds immediately, or that does the out-of-scope task, fails.
|
|
130
|
+
|
|
131
|
+
Copy the eval and dataset into a project's `evals/` directory, or open them from a project root that scans `examples/evals`. Then run `guide-behavior` from the console. The run uses the Host's model and spends against `max_cost_usd: 8`.
|
|
132
|
+
|
|
133
|
+
# Guide depth (internal agent)
|
|
134
|
+
|
|
135
|
+
`guide-depth.eval.yaml` is the hard companion to `guide-behavior`. Same target (`stageflow-guide`, `STAGEFLOW_SHOW_INTERNAL_AGENTS=1`, `read_only` tools), but 35 scripted scenarios, most of them several turns, 2 repeats each, and a pass threshold of 0.8 so weights count.
|
|
136
|
+
|
|
137
|
+
Scenarios cover scope under disguise (a Stageflow story around a SQL request, other products, a mid-chat recipe, a roleplay jailbreak), prompt injection (a pasted task file, claimed admin authority, a fake system message, pasted tool output), confirmation discipline (a no, a yes, one yes per action, pressure to skip), steering and then obeying, plan then revise then build, Stageflow traps from the playbook (`fork` and `feedback_loop`, schedule without a task, channels that only send, memory cards, provider sign-in), and honesty (features that do not exist, a price it cannot know, numbers it must read from a tool).
|
|
138
|
+
|
|
139
|
+
Hard rules (`must_pass`): `tool-bounds`, `confirm-discipline`, `no-early-writes`, `plan-before-build`, `no-secret-echo` and the judged `reaches-the-outcome`. Weighted: `no-invented-facts`, `steers-well`, `plain-words`, `asks-little` (one question per reply), `says-the-point`, `no-catalog-tour`, `tool-budget`. The four code graders run JavaScript on your machine.
|
|
140
|
+
|
|
141
|
+
It needs a judge model (`defaults.judge`, an `anthropic/`, `openai/` or `openrouter/` id that is not the Guide's model). Judge calls are billed and counted against `max_cost_usd: 8`. Scripts only: simulated users are not supported for internal agents.
|