thumbgate 1.29.1 → 1.30.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.claude/commands/dashboard.md +11 -1
- package/.claude/commands/thumbgate-dashboard.md +23 -8
- package/.claude-plugin/plugin.json +1 -1
- package/.well-known/mcp/server-card.json +1 -1
- package/README.md +61 -1
- package/adapters/claude/.mcp.json +2 -2
- package/adapters/forge/forge.yaml +3 -3
- package/adapters/mcp/server-stdio.js +164 -7
- package/adapters/opencode/opencode.json +1 -1
- package/bin/cli.js +7 -5
- package/commands/dashboard.md +11 -1
- package/commands/thumbgate-dashboard.md +23 -8
- package/config/agent-outcome-monitor-thresholds.json +63 -0
- package/config/evals/agent-outcomes-baseline.json +17 -0
- package/config/evals/agent-outcomes-golden.json +412 -0
- package/config/evals/prompt-eval-baseline.json +23 -0
- package/config/mcp-allowlists.json +26 -2
- package/config/post-deploy-marketing-pages.json +26 -1
- package/config/schemas/task-outcome-receipt.schema.json +296 -0
- package/openapi/openapi.yaml +235 -0
- package/package.json +55 -11
- package/public/architecture.html +130 -0
- package/public/assets/diagrams/agent-integration.png +0 -0
- package/public/assets/diagrams/before-after.svg +21 -0
- package/public/assets/diagrams/decision.svg +36 -0
- package/public/assets/diagrams/feedback-pipeline.png +0 -0
- package/public/assets/diagrams/loop.svg +34 -0
- package/public/assets/diagrams/plugin-topology.png +0 -0
- package/public/assets/diagrams/pre-action-gate-loop.svg +59 -0
- package/public/assets/diagrams/stack.svg +18 -0
- package/public/assets/diagrams/thumbgate-architecture.png +0 -0
- package/public/case-studies.html +151 -0
- package/public/eval-scorecard.html +195 -0
- package/public/eval-scorecard.json +18 -0
- package/public/evaluations.html +168 -0
- package/public/index.html +6 -3
- package/public/numbers.html +2 -2
- package/public/whitepaper.html +189 -0
- package/scripts/activation-quickstart.js +1 -0
- package/scripts/agent-outcome-eval.js +130 -0
- package/scripts/agent-outcome-monitor.js +331 -0
- package/scripts/agent-reasoning-traces.js +8 -9
- package/scripts/async-job-runner.js +107 -13
- package/scripts/billing.js +3 -1
- package/scripts/claude-feedback-sync.js +3 -2
- package/scripts/cli-feedback.js +13 -7
- package/scripts/cross-encoder-reranker.js +3 -0
- package/scripts/durability/step.js +121 -12
- package/scripts/feedback-aggregate.js +5 -2
- package/scripts/feedback-loop.js +244 -182
- package/scripts/gates-engine.js +512 -22
- package/scripts/generate-case-study-outreach.js +253 -0
- package/scripts/generate-eval-scorecard.js +276 -0
- package/scripts/growth-campaigns.js +183 -0
- package/scripts/human-escalation.js +265 -0
- package/scripts/hybrid-feedback-context.js +93 -50
- package/scripts/jsonl-watcher.js +1 -0
- package/scripts/judge-reward-function.js +30 -18
- package/scripts/lesson-inference.js +23 -4
- package/scripts/lesson-retrieval.js +71 -4
- package/scripts/lesson-search.js +26 -3
- package/scripts/mcp-config.js +26 -5
- package/scripts/mcp-oauth.js +37 -2
- package/scripts/model-eval.js +308 -0
- package/scripts/parallel-workflow-orchestrator.js +86 -22
- package/scripts/prompt-eval.js +81 -4
- package/scripts/published-cli.js +11 -1
- package/scripts/refresh-proof-pack.js +261 -0
- package/scripts/risk-scorer.js +144 -15
- package/scripts/schedule-manager.js +249 -0
- package/scripts/statusline-local-stats.js +1 -1
- package/scripts/task-outcomes.js +425 -0
- package/scripts/thumbgate-bench.js +13 -0
- package/scripts/tool-contract-validator.js +287 -59
- package/scripts/tool-kpi-tracker.js +124 -0
- package/scripts/tool-registry.js +192 -1
- package/src/api/server.js +355 -89
|
@@ -1,15 +1,25 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: dashboard
|
|
3
3
|
description: Open the local HTTP dashboard for the current project in your web browser.
|
|
4
|
+
allowed-tools: Bash(npx thumbgate dashboard:*)
|
|
4
5
|
---
|
|
5
6
|
|
|
6
7
|
# Open Dashboard
|
|
7
8
|
|
|
8
9
|
Open the local HTTP dashboard for the current project in your web browser.
|
|
9
10
|
|
|
11
|
+
This command wraps existing ThumbGate capability — **no new logic**.
|
|
12
|
+
|
|
10
13
|
## Instructions
|
|
14
|
+
|
|
11
15
|
Execute the following command in the project directory to open the browser dashboard scoped to the current repository:
|
|
16
|
+
|
|
12
17
|
```bash
|
|
13
|
-
thumbgate
|
|
18
|
+
npx thumbgate dashboard --open
|
|
14
19
|
```
|
|
15
20
|
|
|
21
|
+
Standalone shortcut:
|
|
22
|
+
|
|
23
|
+
```bash
|
|
24
|
+
thumbgate-dashboard
|
|
25
|
+
```
|
|
@@ -1,15 +1,30 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: thumbgate-dashboard
|
|
3
|
-
description: Open the local HTTP dashboard for the current project in your web browser.
|
|
3
|
+
description: Open the local HTTP dashboard for the current project in your web browser. Use for "open dashboard", "thumbgate-dashboard", "show my gates in the browser", "project dashboard", or "open the local ThumbGate UI".
|
|
4
|
+
allowed-tools: Bash(npx thumbgate dashboard:*)
|
|
4
5
|
---
|
|
5
6
|
|
|
6
|
-
#
|
|
7
|
+
# ThumbGate Dashboard
|
|
7
8
|
|
|
8
|
-
Open the local HTTP dashboard for the current project in
|
|
9
|
+
Open the local HTTP dashboard for the current project so you can inspect lessons, checks, gate stats, and tokens saved in the browser.
|
|
9
10
|
|
|
10
|
-
|
|
11
|
-
|
|
12
|
-
|
|
13
|
-
|
|
14
|
-
|
|
11
|
+
This command wraps existing ThumbGate capability — **no new logic**. It runs the existing project-scoped dashboard opener.
|
|
12
|
+
|
|
13
|
+
## Steps
|
|
14
|
+
|
|
15
|
+
1. Open the project-scoped dashboard:
|
|
16
|
+
```bash
|
|
17
|
+
npx thumbgate dashboard --open
|
|
18
|
+
```
|
|
19
|
+
Equivalent standalone shortcut after a global install:
|
|
20
|
+
```bash
|
|
21
|
+
thumbgate-dashboard
|
|
22
|
+
```
|
|
23
|
+
2. Confirm the browser lands on the local dashboard for this repo (lessons, checks, gate stats, tokens saved).
|
|
24
|
+
3. If the command fails, run `npx thumbgate doctor` — a missing install is usually fixed by `npx thumbgate init`.
|
|
15
25
|
|
|
26
|
+
## Example
|
|
27
|
+
|
|
28
|
+
```
|
|
29
|
+
/thumbgate-dashboard
|
|
30
|
+
```
|
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "thumbgate",
|
|
3
3
|
"description": "One 👎 becomes a hard rule the agent cannot bypass. Captures thumbs-down feedback, distills it into PreToolUse Pre-Action Checks, enforced across every future Claude Code session.",
|
|
4
|
-
"version": "1.
|
|
4
|
+
"version": "1.30.0",
|
|
5
5
|
"author": {
|
|
6
6
|
"name": "Igor Ganapolsky",
|
|
7
7
|
"email": "ig5973700@gmail.com",
|
package/README.md
CHANGED
|
@@ -65,12 +65,15 @@ Spec-driven agent frameworks like **GSD** (get-shit-done) and **GitHub Spec Kit*
|
|
|
65
65
|
|
|
66
66
|
| Command | What it does | Wraps (existing capability) |
|
|
67
67
|
|---------|--------------|------------------------------|
|
|
68
|
+
| **`/thumbgate-dashboard`** | **Open the local project dashboard in your browser** (lessons, checks, tokens saved) | **`npx thumbgate dashboard --open`** (global bin after `npm i -g`) |
|
|
68
69
|
| `/thumbgate-guard` | Turn the last agent mistake into a hard prevention rule | `capture_feedback` + `thumbgate force-gate` |
|
|
69
70
|
| `/thumbgate-rules` | List the active prevention rules + lessons guarding this repo | `prevention_rules`, `get_reliability_rules`, `search_lessons` |
|
|
70
71
|
| `/thumbgate-blocked` | Show what's actually been blocked — gate stats + enforcement matrix | `gate_stats`, `enforcement_matrix` |
|
|
71
72
|
| `/thumbgate-protect` | Show branch/release governance; grant a scoped, expiring approval | `get_branch_governance`, `approve_protected_action` |
|
|
72
73
|
| `/thumbgate-doctor` | Health-check the wiring (hooks, MCP, agent-readiness) | `thumbgate doctor` |
|
|
73
74
|
|
|
75
|
+
> **Open the dashboard anytime:** after `npx thumbgate init`, run **`npx thumbgate dashboard --open`** (works without a global install). Type **`/thumbgate-dashboard`** in Claude Code / Cursor, or **`/project:thumbgate-dashboard`** in Grok. After `npm i -g thumbgate`, the **`thumbgate-dashboard`** bin is also on your PATH.
|
|
76
|
+
|
|
74
77
|
Each is a thin wrapper over an existing MCP tool or CLI command — **no new enforcement logic, just discoverability**.
|
|
75
78
|
|
|
76
79
|
---
|
|
@@ -196,6 +199,63 @@ Hand-rolled hooks are the right tool for a small, static denylist you maintain b
|
|
|
196
199
|
|
|
197
200
|
Prompt engineering still matters, but it is only the starting point. ThumbGate adds prompt evaluation on top: proof lanes, benchmarks, and self-heal checks produce reviewable evidence about whether a prompt and workflow held up under execution. Run `npx thumbgate eval --from-feedback --write-report=.thumbgate/prompt-eval-proof.md` to turn accepted thumbs-up/down feedback into reusable eval cases and a local proof report.
|
|
198
201
|
|
|
202
|
+
### How ThumbGate knows an AI agent is working
|
|
203
|
+
|
|
204
|
+
ThumbGate does not treat a plausible response, a successful tool call, or a
|
|
205
|
+
demo as task success. The `record_task_outcome` MCP tool and
|
|
206
|
+
`POST /v1/task-outcomes` API require a task-level receipt. A receipt is marked
|
|
207
|
+
`working: true` only when the task is completed, verification passed, evidence
|
|
208
|
+
is present, tool contracts and policy checks passed, no unsupported claim was
|
|
209
|
+
recorded, and no side effect was duplicated.
|
|
210
|
+
|
|
211
|
+
The metrics remain separate so a strong average cannot conceal an unsafe
|
|
212
|
+
failure:
|
|
213
|
+
|
|
214
|
+
| Layer | Measured signals |
|
|
215
|
+
|-------|------------------|
|
|
216
|
+
| Task | verified completion, evidence-backed completion, first-attempt success, repeated failure, recovery, rollback |
|
|
217
|
+
| Tool | contract accuracy, execution success, retry rate, duplicate side effects |
|
|
218
|
+
| Safety | unsafe escapes, policy violations, safe-action false blocks |
|
|
219
|
+
| Escalation | correct escalation rate and human decision latency |
|
|
220
|
+
| Efficiency | p50/p95 latency, total cost, cost per verified success |
|
|
221
|
+
| Business | explicit KPI values grouped by unit; no inferred revenue |
|
|
222
|
+
|
|
223
|
+
```bash
|
|
224
|
+
npm run eval:agent-outcomes # 8 reviewed golden cases; fails on regression
|
|
225
|
+
npm run monitor:agent-outcomes # local production receipts; fails on missing evidence
|
|
226
|
+
npm run monitor:agent-outcomes -- --hosted
|
|
227
|
+
npm run monitor:agent-outcomes -- --install-schedule
|
|
228
|
+
```
|
|
229
|
+
|
|
230
|
+
Prompt evaluation is deterministic first. JSON outputs are validated against
|
|
231
|
+
their declared schema. An LLM judge may add a separate score, but an unavailable
|
|
232
|
+
or failed judge is reported as `deterministic_only`; it is never converted into
|
|
233
|
+
a neutral pass. Empty feedback or task-outcome datasets return
|
|
234
|
+
`insufficient_evidence`.
|
|
235
|
+
|
|
236
|
+
Task outcome receipts are stored locally in
|
|
237
|
+
`.thumbgate/task-outcome-receipts.jsonl`. Observable tool traces exclude raw
|
|
238
|
+
hidden reasoning and deterministic tool-argument fingerprints. Human
|
|
239
|
+
escalation requests are append-only, must carry evidence and requester
|
|
240
|
+
identity, and expire. Decisions require both the ordinary API credential and
|
|
241
|
+
an independently revocable `X-ThumbGate-Human-Reviewer-Key`; the decision
|
|
242
|
+
actor comes from server-side `THUMBGATE_HUMAN_REVIEWER_ID` configuration, not
|
|
243
|
+
from caller-controlled JSON.
|
|
244
|
+
|
|
245
|
+
The installed daily monitor runs through ThumbGate's local scheduler rather
|
|
246
|
+
than consuming a GitHub-hosted cron runner. It reads operator authentication
|
|
247
|
+
from the existing environment or local operator config, never from command-line
|
|
248
|
+
arguments, and writes a machine-readable report under
|
|
249
|
+
`~/.thumbgate/reports/agent-outcome-monitor.json`. The GitHub workflow remains
|
|
250
|
+
manual for release-time verification.
|
|
251
|
+
|
|
252
|
+
The complete five-system architecture review—local RAG, tool-using agents,
|
|
253
|
+
multi-agent handoffs, MCP enterprise integration, and production
|
|
254
|
+
evaluation/observability—is maintained in
|
|
255
|
+
[`VERIFICATION_EVIDENCE.md`](./VERIFICATION_EVIDENCE.md). It answers why each
|
|
256
|
+
architecture exists, what can fail, how it is measured and secured, how it is
|
|
257
|
+
deployed, and what evidence is required before claiming it works.
|
|
258
|
+
|
|
199
259
|
### Retrieval & latency: local-first, zero network hops
|
|
200
260
|
|
|
201
261
|
ThumbGate's latency advantage is structural, not a tuned cloud cluster: there is no retrieval service and no model on the enforcement path, so the gate decision never leaves your machine.
|
|
@@ -385,7 +445,7 @@ npx thumbgate background-governance # review background-agent run risk
|
|
|
385
445
|
npx thumbgate model-candidates --workload=dashboard-analysis --provider=openai --json # evaluate GPT-5.5 routing
|
|
386
446
|
npx thumbgate native-messaging-audit # inspect local browser bridges and extension hosts
|
|
387
447
|
npx thumbgate dashboard --open # open local project-scoped dashboard in browser
|
|
388
|
-
thumbgate-dashboard #
|
|
448
|
+
thumbgate-dashboard # global bin after npm i -g thumbgate; agents: /thumbgate-dashboard (Claude/Cursor) or /project:thumbgate-dashboard (Grok)
|
|
389
449
|
npx thumbgate check-update # check if a new version is available on npm/GitHub
|
|
390
450
|
npx thumbgate self-update # update ThumbGate to the latest version globally
|
|
391
451
|
npx thumbgate serve # start MCP server on stdio
|
|
@@ -2,13 +2,13 @@
|
|
|
2
2
|
"mcpServers": {
|
|
3
3
|
"thumbgate": {
|
|
4
4
|
"command": "npx",
|
|
5
|
-
"args": ["--yes", "--package", "thumbgate@1.
|
|
5
|
+
"args": ["--yes", "--package", "thumbgate@1.30.0", "thumbgate", "serve"]
|
|
6
6
|
}
|
|
7
7
|
},
|
|
8
8
|
"hooks": {
|
|
9
9
|
"preToolUse": {
|
|
10
10
|
"command": "npx",
|
|
11
|
-
"args": ["--yes", "--package", "thumbgate@1.
|
|
11
|
+
"args": ["--yes", "--package", "thumbgate@1.30.0", "thumbgate", "gate-check"]
|
|
12
12
|
}
|
|
13
13
|
}
|
|
14
14
|
}
|
|
@@ -9,12 +9,12 @@ version: "1"
|
|
|
9
9
|
skills:
|
|
10
10
|
thumbgate-gate-check:
|
|
11
11
|
description: "ThumbGate PreToolUse gate — blocks known-bad tool calls"
|
|
12
|
-
command: "npx --yes --package thumbgate@1.
|
|
12
|
+
command: "npx --yes --package thumbgate@1.30.0 thumbgate gate-check"
|
|
13
13
|
trigger: pre_tool_use
|
|
14
14
|
|
|
15
15
|
thumbgate-feedback:
|
|
16
16
|
description: "ThumbGate feedback capture — logs user prompt context"
|
|
17
|
-
command: "npx --yes --package thumbgate@1.
|
|
17
|
+
command: "npx --yes --package thumbgate@1.30.0 thumbgate hook-auto-capture"
|
|
18
18
|
trigger: user_prompt
|
|
19
19
|
|
|
20
20
|
mcp:
|
|
@@ -23,6 +23,6 @@ mcp:
|
|
|
23
23
|
args:
|
|
24
24
|
- "--yes"
|
|
25
25
|
- "--package"
|
|
26
|
-
- "thumbgate@1.
|
|
26
|
+
- "thumbgate@1.30.0"
|
|
27
27
|
- "thumbgate"
|
|
28
28
|
- "serve"
|
|
@@ -85,6 +85,18 @@ const {
|
|
|
85
85
|
pairFeedbackWithReceipt,
|
|
86
86
|
buildReceiptContextEntries,
|
|
87
87
|
} = require('../../scripts/action-receipts');
|
|
88
|
+
const {
|
|
89
|
+
calculateTaskOutcomeMetrics,
|
|
90
|
+
getTaskOutcome,
|
|
91
|
+
readTaskOutcomes,
|
|
92
|
+
recordTaskOutcome,
|
|
93
|
+
} = require('../../scripts/task-outcomes');
|
|
94
|
+
const {
|
|
95
|
+
listEscalations,
|
|
96
|
+
requestEscalation,
|
|
97
|
+
} = require('../../scripts/human-escalation');
|
|
98
|
+
const { recordReasoningTrace } = require('../../scripts/agent-reasoning-traces');
|
|
99
|
+
const { recordToolCall } = require('../../scripts/tool-kpi-tracker');
|
|
88
100
|
const {
|
|
89
101
|
evaluateOperationalIntegrity,
|
|
90
102
|
} = require('../../scripts/operational-integrity');
|
|
@@ -305,7 +317,7 @@ const {
|
|
|
305
317
|
finalizeSession: finalizeFeedbackSession,
|
|
306
318
|
} = require('../../scripts/feedback-session');
|
|
307
319
|
|
|
308
|
-
const SERVER_INFO = { name: 'thumbgate-mcp', version: '1.
|
|
320
|
+
const SERVER_INFO = { name: 'thumbgate-mcp', version: '1.30.0' };
|
|
309
321
|
const COMMERCE_CATEGORIES = [
|
|
310
322
|
'product_recommendation',
|
|
311
323
|
'brand_compliance',
|
|
@@ -368,6 +380,9 @@ function toTextResult(payload) {
|
|
|
368
380
|
const text = typeof payload === 'string' ? payload : JSON.stringify(payload, null, 2);
|
|
369
381
|
return {
|
|
370
382
|
content: [{ type: 'text', text }],
|
|
383
|
+
...(payload !== null && typeof payload === 'object'
|
|
384
|
+
? { structuredContent: payload }
|
|
385
|
+
: {}),
|
|
371
386
|
};
|
|
372
387
|
}
|
|
373
388
|
|
|
@@ -405,7 +420,7 @@ function toCaptureFeedbackTextResult(result) {
|
|
|
405
420
|
if (reminder) {
|
|
406
421
|
blocks.push({ type: 'text', text: reminder });
|
|
407
422
|
}
|
|
408
|
-
return { content: blocks };
|
|
423
|
+
return { content: blocks, structuredContent: result };
|
|
409
424
|
}
|
|
410
425
|
|
|
411
426
|
function formatContextPack(pack) {
|
|
@@ -747,10 +762,27 @@ function buildEstimateUncertaintyResponse(args = {}) {
|
|
|
747
762
|
}
|
|
748
763
|
|
|
749
764
|
async function callTool(name, args = {}) {
|
|
765
|
+
const attemptStartMs = Date.now();
|
|
750
766
|
const activeProfile = getActiveMcpProfile();
|
|
751
|
-
|
|
767
|
+
try {
|
|
768
|
+
assertToolAllowed(name, activeProfile);
|
|
769
|
+
} catch (error) {
|
|
770
|
+
recordMcpToolTrace(name, args, {
|
|
771
|
+
success: false,
|
|
772
|
+
category: 'profile_denied',
|
|
773
|
+
evidence: [error.message],
|
|
774
|
+
latencyMs: Date.now() - attemptStartMs,
|
|
775
|
+
});
|
|
776
|
+
throw error;
|
|
777
|
+
}
|
|
752
778
|
const capability = getToolCapability(name);
|
|
753
779
|
if (!capability.available) {
|
|
780
|
+
recordMcpToolTrace(name, args, {
|
|
781
|
+
success: false,
|
|
782
|
+
category: 'capability',
|
|
783
|
+
evidence: capability.missingModules,
|
|
784
|
+
latencyMs: Date.now() - attemptStartMs,
|
|
785
|
+
});
|
|
754
786
|
if (capability.availability === 'private_core') {
|
|
755
787
|
return unavailablePrivateMcpFeature(name);
|
|
756
788
|
}
|
|
@@ -773,6 +805,12 @@ async function callTool(name, args = {}) {
|
|
|
773
805
|
const err = new Error(`Tool contract violation on '${name}': ${validation.errors.join('; ')}`);
|
|
774
806
|
err.errorCategory = 'contract';
|
|
775
807
|
err.isRetryable = false;
|
|
808
|
+
recordMcpToolTrace(name, args, {
|
|
809
|
+
success: false,
|
|
810
|
+
category: 'contract',
|
|
811
|
+
evidence: validation.errors,
|
|
812
|
+
latencyMs: Date.now() - attemptStartMs,
|
|
813
|
+
});
|
|
776
814
|
throw err;
|
|
777
815
|
}
|
|
778
816
|
}
|
|
@@ -783,12 +821,48 @@ async function callTool(name, args = {}) {
|
|
|
783
821
|
const err = new Error(`Action blocked by Semantic Firewall: ${firewallResult.message}`);
|
|
784
822
|
err.errorCategory = 'permission';
|
|
785
823
|
err.isRetryable = false;
|
|
824
|
+
recordMcpToolTrace(name, args, {
|
|
825
|
+
success: false,
|
|
826
|
+
category: 'permission',
|
|
827
|
+
evidence: [firewallResult.message],
|
|
828
|
+
latencyMs: Date.now() - attemptStartMs,
|
|
829
|
+
});
|
|
786
830
|
throw err;
|
|
787
831
|
}
|
|
788
832
|
}
|
|
789
833
|
const startMs = Date.now();
|
|
790
|
-
|
|
834
|
+
let result;
|
|
835
|
+
try {
|
|
836
|
+
result = await callToolInner(name, args);
|
|
837
|
+
} catch (err) {
|
|
838
|
+
recordMcpToolTrace(name, args, {
|
|
839
|
+
success: false,
|
|
840
|
+
category: err.errorCategory || 'execution',
|
|
841
|
+
evidence: [err.code || err.message || 'tool execution failed'],
|
|
842
|
+
latencyMs: Date.now() - attemptStartMs,
|
|
843
|
+
});
|
|
844
|
+
throw err;
|
|
845
|
+
}
|
|
791
846
|
const latencyMs = Date.now() - startMs;
|
|
847
|
+
const outputValidation = validateMcpToolOutput(toolDef, result);
|
|
848
|
+
if (!outputValidation.valid) {
|
|
849
|
+
const err = new Error(`Structured output contract violation on '${name}': ${outputValidation.errors.join('; ')}`);
|
|
850
|
+
err.errorCategory = 'output_contract';
|
|
851
|
+
err.isRetryable = false;
|
|
852
|
+
recordMcpToolTrace(name, args, {
|
|
853
|
+
success: false,
|
|
854
|
+
category: 'output_contract',
|
|
855
|
+
evidence: outputValidation.errors,
|
|
856
|
+
latencyMs,
|
|
857
|
+
});
|
|
858
|
+
throw err;
|
|
859
|
+
}
|
|
860
|
+
recordMcpToolTrace(name, args, {
|
|
861
|
+
success: true,
|
|
862
|
+
category: 'success',
|
|
863
|
+
evidence: [`tool completed in ${latencyMs}ms`],
|
|
864
|
+
latencyMs,
|
|
865
|
+
});
|
|
792
866
|
try {
|
|
793
867
|
const { recordAuditEvent } = require('../../scripts/audit-trail');
|
|
794
868
|
recordAuditEvent({
|
|
@@ -802,6 +876,65 @@ async function callTool(name, args = {}) {
|
|
|
802
876
|
return result;
|
|
803
877
|
}
|
|
804
878
|
|
|
879
|
+
function validateMcpToolOutput(toolDef, result) {
|
|
880
|
+
if (!toolDef || !toolDef.outputSchema) return { valid: true, errors: [] };
|
|
881
|
+
const { validateStructuredOutput } = require('../../scripts/tool-contract-validator');
|
|
882
|
+
if (!result || result.structuredContent === undefined) {
|
|
883
|
+
return { valid: false, errors: ['Tool response is missing structuredContent'] };
|
|
884
|
+
}
|
|
885
|
+
return validateStructuredOutput(result.structuredContent, toolDef.outputSchema);
|
|
886
|
+
}
|
|
887
|
+
|
|
888
|
+
function recordMcpToolTrace(name, args, outcome = {}) {
|
|
889
|
+
try {
|
|
890
|
+
recordToolCall({
|
|
891
|
+
toolName: name,
|
|
892
|
+
serverName: 'mcp',
|
|
893
|
+
latencyMs: Number(outcome.latencyMs || 0),
|
|
894
|
+
success: outcome.success === true,
|
|
895
|
+
agentId: args.agentId || args.processId || args.taskId || 'unknown',
|
|
896
|
+
metadata: {
|
|
897
|
+
category: outcome.category || 'unknown',
|
|
898
|
+
traceId: args.traceId || args.taskId || null,
|
|
899
|
+
},
|
|
900
|
+
});
|
|
901
|
+
} catch {
|
|
902
|
+
// KPI telemetry must not change the tool's functional outcome.
|
|
903
|
+
}
|
|
904
|
+
try {
|
|
905
|
+
const traceId = args.traceId || args.taskId || `mcp-${Date.now()}-${name}`;
|
|
906
|
+
recordReasoningTrace({
|
|
907
|
+
trace_id: traceId,
|
|
908
|
+
task_type: 'tool-use',
|
|
909
|
+
source: 'mcp-runtime',
|
|
910
|
+
success: outcome.success,
|
|
911
|
+
outcome: {
|
|
912
|
+
success: outcome.success,
|
|
913
|
+
terminalState: outcome.category,
|
|
914
|
+
},
|
|
915
|
+
messages: [
|
|
916
|
+
{
|
|
917
|
+
role: 'assistant',
|
|
918
|
+
content: `tool: ${name}`,
|
|
919
|
+
tool_calls: [{ function: { name } }],
|
|
920
|
+
},
|
|
921
|
+
{
|
|
922
|
+
role: 'tool',
|
|
923
|
+
content: `tool response: ${outcome.category}; ${outcome.evidence?.join('; ') || 'no evidence'}`,
|
|
924
|
+
success: outcome.success,
|
|
925
|
+
},
|
|
926
|
+
],
|
|
927
|
+
metadata: {
|
|
928
|
+
latencyMs: outcome.latencyMs || 0,
|
|
929
|
+
argumentFingerprintStored: false,
|
|
930
|
+
rawArgumentsStored: false,
|
|
931
|
+
},
|
|
932
|
+
});
|
|
933
|
+
} catch {
|
|
934
|
+
// Trace telemetry must not change the tool's functional outcome.
|
|
935
|
+
}
|
|
936
|
+
}
|
|
937
|
+
|
|
805
938
|
async function callToolInner(name, args) {
|
|
806
939
|
args = args || {};
|
|
807
940
|
// Semantic Aliases for high-level branding alignment
|
|
@@ -815,10 +948,13 @@ async function callToolInner(name, args) {
|
|
|
815
948
|
// action receipt (this action -> this outcome) before promotion. Returns
|
|
816
949
|
// args unchanged when there is no matching receipt (non-breaking).
|
|
817
950
|
const pairedFeedback = pairFeedbackWithReceipt(args);
|
|
818
|
-
return toCaptureFeedbackTextResult(captureFeedback(
|
|
951
|
+
return toCaptureFeedbackTextResult(captureFeedback({
|
|
952
|
+
...pairedFeedback,
|
|
953
|
+
reviewOrigin: 'automated',
|
|
954
|
+
}));
|
|
819
955
|
}
|
|
820
956
|
case 'feedback_summary':
|
|
821
|
-
return toTextResult(feedbackSummary(Number(args.recent || 20)));
|
|
957
|
+
return toTextResult(feedbackSummary(Number(args.recent || 20), { humanOnly: true }));
|
|
822
958
|
case 'search_lessons': {
|
|
823
959
|
const module = loadPrivateMcpModule('lessonSearch');
|
|
824
960
|
if (!module) return unavailablePrivateMcpFeature('search_lessons');
|
|
@@ -826,6 +962,9 @@ async function callToolInner(name, args) {
|
|
|
826
962
|
limit: Number(args.limit || 10),
|
|
827
963
|
category: args.category,
|
|
828
964
|
tags: Array.isArray(args.tags) ? args.tags : [],
|
|
965
|
+
scope: args.scope,
|
|
966
|
+
requireScope: args.requireScope === true,
|
|
967
|
+
includeShared: args.includeShared !== false,
|
|
829
968
|
}));
|
|
830
969
|
}
|
|
831
970
|
case 'suggest_fix':
|
|
@@ -845,6 +984,9 @@ async function callToolInner(name, args) {
|
|
|
845
984
|
{
|
|
846
985
|
candidateCount: 20,
|
|
847
986
|
maxResults: Number(args.maxResults || 5),
|
|
987
|
+
scope: args.scope,
|
|
988
|
+
requireScope: args.requireScope === true,
|
|
989
|
+
includeShared: args.includeShared !== false,
|
|
848
990
|
},
|
|
849
991
|
));
|
|
850
992
|
}
|
|
@@ -897,7 +1039,7 @@ async function callToolInner(name, args) {
|
|
|
897
1039
|
return toTextResult(document);
|
|
898
1040
|
}
|
|
899
1041
|
case 'feedback_stats':
|
|
900
|
-
return toTextResult(analyzeFeedback());
|
|
1042
|
+
return toTextResult(analyzeFeedback(undefined, { humanOnly: true }));
|
|
901
1043
|
case 'diagnose_failure':
|
|
902
1044
|
return buildDiagnoseFailureResponse(args);
|
|
903
1045
|
case 'reflect_on_feedback':
|
|
@@ -1075,6 +1217,7 @@ async function callToolInner(name, args) {
|
|
|
1075
1217
|
workflowContract: args.workflowContract,
|
|
1076
1218
|
repoPath: args.repoPath,
|
|
1077
1219
|
localOnly: args.localOnly === true,
|
|
1220
|
+
ttlMs: args.ttlMs,
|
|
1078
1221
|
clear: args.clear === true,
|
|
1079
1222
|
}),
|
|
1080
1223
|
});
|
|
@@ -1154,6 +1297,19 @@ async function callToolInner(name, args) {
|
|
|
1154
1297
|
? getReceiptForAction(args.actionId)
|
|
1155
1298
|
: getRecentReceipts(Number(args.limit || 20)),
|
|
1156
1299
|
);
|
|
1300
|
+
case 'record_task_outcome':
|
|
1301
|
+
return toTextResult(recordTaskOutcome(args));
|
|
1302
|
+
case 'get_task_outcomes': {
|
|
1303
|
+
if (args.taskId) return toTextResult(getTaskOutcome(args.taskId));
|
|
1304
|
+
const limit = Number(args.limit || 20);
|
|
1305
|
+
return toTextResult(readTaskOutcomes().slice(-limit));
|
|
1306
|
+
}
|
|
1307
|
+
case 'get_agent_outcome_metrics':
|
|
1308
|
+
return toTextResult(calculateTaskOutcomeMetrics(readTaskOutcomes()));
|
|
1309
|
+
case 'request_human_escalation':
|
|
1310
|
+
return toTextResult(requestEscalation(args));
|
|
1311
|
+
case 'list_human_escalations':
|
|
1312
|
+
return toTextResult(listEscalations({ status: args.status }).slice(0, Number(args.limit || 20)));
|
|
1157
1313
|
case 'verify_claim':
|
|
1158
1314
|
return toTextResult(verifyClaimEvidence(args.claim, { goalContract: args.goalContract }));
|
|
1159
1315
|
case 'require_evidence_for_claim': {
|
|
@@ -1658,5 +1814,6 @@ module.exports = {
|
|
|
1658
1814
|
listAvailableTools,
|
|
1659
1815
|
unavailablePrivateMcpFeature,
|
|
1660
1816
|
callToolInner,
|
|
1817
|
+
validateMcpToolOutput,
|
|
1661
1818
|
},
|
|
1662
1819
|
};
|
package/bin/cli.js
CHANGED
|
@@ -1185,7 +1185,7 @@ function capture() {
|
|
|
1185
1185
|
}
|
|
1186
1186
|
|
|
1187
1187
|
if (args.summary) {
|
|
1188
|
-
console.log(feedbackSummary(Number(args.recent || 20)));
|
|
1188
|
+
console.log(feedbackSummary(Number(args.recent || 20), { humanOnly: true }));
|
|
1189
1189
|
return;
|
|
1190
1190
|
}
|
|
1191
1191
|
|
|
@@ -1271,6 +1271,7 @@ function capture() {
|
|
|
1271
1271
|
whatWorked: args['what-worked'],
|
|
1272
1272
|
tags: args.tags,
|
|
1273
1273
|
gateAction: gateAction || undefined,
|
|
1274
|
+
reviewOrigin: 'human',
|
|
1274
1275
|
});
|
|
1275
1276
|
|
|
1276
1277
|
if (result.accepted) {
|
|
@@ -1386,6 +1387,7 @@ function feedbackSelfTest() {
|
|
|
1386
1387
|
? (args['what-worked'] || 'Feedback capture persisted and was verified by a self-test')
|
|
1387
1388
|
: undefined,
|
|
1388
1389
|
tags: args.tags || 'self-test,dogfood,feedback-capture',
|
|
1390
|
+
reviewOrigin: 'automated',
|
|
1389
1391
|
});
|
|
1390
1392
|
|
|
1391
1393
|
const paths = getFeedbackPaths();
|
|
@@ -1444,7 +1446,7 @@ function stats() {
|
|
|
1444
1446
|
trackEvent('cli_stats', { command: 'stats' });
|
|
1445
1447
|
const args = parseArgs(process.argv.slice(3));
|
|
1446
1448
|
const { analyzeFeedback } = require(path.join(PKG_ROOT, 'scripts', 'feedback-loop'));
|
|
1447
|
-
const data = analyzeFeedback();
|
|
1449
|
+
const data = analyzeFeedback(undefined, { humanOnly: true });
|
|
1448
1450
|
|
|
1449
1451
|
// Gate enforcement stats — runtime intercepts + configured gates
|
|
1450
1452
|
let gateData = { blocked: 0, warned: 0, passed: 0, byGate: {} };
|
|
@@ -1761,7 +1763,7 @@ function summary() {
|
|
|
1761
1763
|
const args = parseArgs(process.argv.slice(3));
|
|
1762
1764
|
const { feedbackSummary, analyzeFeedback } = require(path.join(PKG_ROOT, 'scripts', 'feedback-loop'));
|
|
1763
1765
|
if (args.json) {
|
|
1764
|
-
const data = analyzeFeedback();
|
|
1766
|
+
const data = analyzeFeedback(undefined, { humanOnly: true });
|
|
1765
1767
|
console.log(JSON.stringify({
|
|
1766
1768
|
total: data.total,
|
|
1767
1769
|
positives: data.totalPositive,
|
|
@@ -1771,7 +1773,7 @@ function summary() {
|
|
|
1771
1773
|
}, null, 2));
|
|
1772
1774
|
return;
|
|
1773
1775
|
}
|
|
1774
|
-
console.log(feedbackSummary(Number(args.recent || 20)));
|
|
1776
|
+
console.log(feedbackSummary(Number(args.recent || 20), { humanOnly: true }));
|
|
1775
1777
|
}
|
|
1776
1778
|
|
|
1777
1779
|
function lessons() {
|
|
@@ -2861,7 +2863,7 @@ function sessionStart() {
|
|
|
2861
2863
|
} catch (_) { /* best-effort fallback sync */ }
|
|
2862
2864
|
const { analyzeFeedback } = require(path.join(PKG_ROOT, 'scripts', 'feedback-loop'));
|
|
2863
2865
|
const { refreshStatuslineCache } = require(path.join(PKG_ROOT, 'scripts', 'hook-thumbgate-cache-updater'));
|
|
2864
|
-
refreshStatuslineCache(analyzeFeedback());
|
|
2866
|
+
refreshStatuslineCache(analyzeFeedback(undefined, { humanOnly: true }));
|
|
2865
2867
|
|
|
2866
2868
|
// Build a top-level <system-reminder> block that Claude Code's SessionStart
|
|
2867
2869
|
// hook surfaces to the agent as first-class context — not buried stderr.
|
package/commands/dashboard.md
CHANGED
|
@@ -1,15 +1,25 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: dashboard
|
|
3
3
|
description: Open the local HTTP dashboard for the current project in your web browser.
|
|
4
|
+
allowed-tools: Bash(npx thumbgate dashboard:*), Bash(thumbgate-dashboard:*)
|
|
4
5
|
---
|
|
5
6
|
|
|
6
7
|
# Open Dashboard
|
|
7
8
|
|
|
8
9
|
Open the local HTTP dashboard for the current project in your web browser.
|
|
9
10
|
|
|
11
|
+
This command wraps existing ThumbGate capability — **no new logic**.
|
|
12
|
+
|
|
10
13
|
## Instructions
|
|
14
|
+
|
|
11
15
|
Execute the following command in the project directory to open the browser dashboard scoped to the current repository:
|
|
16
|
+
|
|
12
17
|
```bash
|
|
13
|
-
thumbgate
|
|
18
|
+
npx thumbgate dashboard --open
|
|
14
19
|
```
|
|
15
20
|
|
|
21
|
+
Standalone shortcut:
|
|
22
|
+
|
|
23
|
+
```bash
|
|
24
|
+
thumbgate-dashboard
|
|
25
|
+
```
|
|
@@ -1,15 +1,30 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: thumbgate-dashboard
|
|
3
|
-
description: Open the local HTTP dashboard for the current project in your web browser.
|
|
3
|
+
description: Open the local HTTP dashboard for the current project in your web browser. Use for "open dashboard", "thumbgate-dashboard", "show my gates in the browser", "project dashboard", or "open the local ThumbGate UI".
|
|
4
|
+
allowed-tools: Bash(npx thumbgate dashboard:*)
|
|
4
5
|
---
|
|
5
6
|
|
|
6
|
-
#
|
|
7
|
+
# ThumbGate Dashboard
|
|
7
8
|
|
|
8
|
-
Open the local HTTP dashboard for the current project in
|
|
9
|
+
Open the local HTTP dashboard for the current project so you can inspect lessons, checks, gate stats, and tokens saved in the browser.
|
|
9
10
|
|
|
10
|
-
|
|
11
|
-
|
|
12
|
-
|
|
13
|
-
|
|
14
|
-
|
|
11
|
+
This command wraps existing ThumbGate capability — **no new logic**. It runs the existing project-scoped dashboard opener.
|
|
12
|
+
|
|
13
|
+
## Steps
|
|
14
|
+
|
|
15
|
+
1. Open the project-scoped dashboard:
|
|
16
|
+
```bash
|
|
17
|
+
npx thumbgate dashboard --open
|
|
18
|
+
```
|
|
19
|
+
Equivalent standalone shortcut after a global install:
|
|
20
|
+
```bash
|
|
21
|
+
thumbgate-dashboard
|
|
22
|
+
```
|
|
23
|
+
2. Confirm the browser lands on the local dashboard for this repo (lessons, checks, gate stats, tokens saved).
|
|
24
|
+
3. If the command fails, run `npx thumbgate doctor` — a missing install is usually fixed by `npx thumbgate init`.
|
|
15
25
|
|
|
26
|
+
## Example
|
|
27
|
+
|
|
28
|
+
```
|
|
29
|
+
/thumbgate-dashboard
|
|
30
|
+
```
|