homegraph 1.1.2 → 1.1.3

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (175) hide show
  1. package/LICENSE +21 -21
  2. package/README.md +243 -243
  3. package/dist/bin/homegraph.js +371 -9
  4. package/dist/bin/homegraph.js.map +1 -1
  5. package/dist/db/migrations.js +18 -18
  6. package/dist/db/queries.js +140 -140
  7. package/dist/db/schema.sql +152 -152
  8. package/dist/directory.js +5 -5
  9. package/dist/extraction/wasm/tree-sitter-c_sharp.wasm +0 -0
  10. package/dist/extraction/wasm/tree-sitter-pascal.wasm +0 -0
  11. package/dist/installer/instructions-template.js +9 -9
  12. package/dist/installer/targets/shared.d.ts +5 -6
  13. package/dist/installer/targets/shared.d.ts.map +1 -1
  14. package/dist/installer/targets/shared.js +5 -6
  15. package/dist/installer/targets/shared.js.map +1 -1
  16. package/dist/mcp/liveness-watchdog.js +16 -16
  17. package/dist/mcp/server-instructions.js +69 -69
  18. package/dist/mcp/tools.d.ts +17 -4
  19. package/dist/mcp/tools.d.ts.map +1 -1
  20. package/dist/mcp/tools.js +192 -22
  21. package/dist/mcp/tools.js.map +1 -1
  22. package/dist/reasoning/reasoner.js +32 -32
  23. package/dist/spec/config.d.ts +43 -0
  24. package/dist/spec/config.d.ts.map +1 -0
  25. package/dist/spec/config.js +275 -0
  26. package/dist/spec/config.js.map +1 -0
  27. package/dist/spec/db/commit-node.d.ts +23 -0
  28. package/dist/spec/db/commit-node.d.ts.map +1 -0
  29. package/dist/spec/db/commit-node.js +62 -0
  30. package/dist/spec/db/commit-node.js.map +1 -0
  31. package/dist/spec/db/fragment-node.d.ts +23 -0
  32. package/dist/spec/db/fragment-node.d.ts.map +1 -0
  33. package/dist/spec/db/fragment-node.js +120 -0
  34. package/dist/spec/db/fragment-node.js.map +1 -0
  35. package/dist/spec/db/fts.d.ts +60 -0
  36. package/dist/spec/db/fts.d.ts.map +1 -0
  37. package/dist/spec/db/fts.js +285 -0
  38. package/dist/spec/db/fts.js.map +1 -0
  39. package/dist/spec/db/index.d.ts +13 -0
  40. package/dist/spec/db/index.d.ts.map +1 -0
  41. package/dist/spec/db/index.js +50 -0
  42. package/dist/spec/db/index.js.map +1 -0
  43. package/dist/spec/db/relations.d.ts +55 -0
  44. package/dist/spec/db/relations.d.ts.map +1 -0
  45. package/dist/spec/db/relations.js +158 -0
  46. package/dist/spec/db/relations.js.map +1 -0
  47. package/dist/spec/db/schema.d.ts +30 -0
  48. package/dist/spec/db/schema.d.ts.map +1 -0
  49. package/dist/spec/db/schema.js +87 -0
  50. package/dist/spec/db/schema.js.map +1 -0
  51. package/dist/spec/db/schema.sql +107 -0
  52. package/dist/spec/db/spec-node.d.ts +41 -0
  53. package/dist/spec/db/spec-node.d.ts.map +1 -0
  54. package/dist/spec/db/spec-node.js +114 -0
  55. package/dist/spec/db/spec-node.js.map +1 -0
  56. package/dist/spec/evolve/impact-locator.d.ts +13 -0
  57. package/dist/spec/evolve/impact-locator.d.ts.map +1 -0
  58. package/dist/spec/evolve/impact-locator.js +25 -0
  59. package/dist/spec/evolve/impact-locator.js.map +1 -0
  60. package/dist/spec/evolve/llm-client.d.ts +50 -0
  61. package/dist/spec/evolve/llm-client.d.ts.map +1 -0
  62. package/dist/spec/evolve/llm-client.js +176 -0
  63. package/dist/spec/evolve/llm-client.js.map +1 -0
  64. package/dist/spec/evolve/logic-checker.d.ts +12 -0
  65. package/dist/spec/evolve/logic-checker.d.ts.map +1 -0
  66. package/dist/spec/evolve/logic-checker.js +48 -0
  67. package/dist/spec/evolve/logic-checker.js.map +1 -0
  68. package/dist/spec/evolve/pipeline.d.ts +40 -0
  69. package/dist/spec/evolve/pipeline.d.ts.map +1 -0
  70. package/dist/spec/evolve/pipeline.js +314 -0
  71. package/dist/spec/evolve/pipeline.js.map +1 -0
  72. package/dist/spec/evolve/spec-rewriter.d.ts +42 -0
  73. package/dist/spec/evolve/spec-rewriter.d.ts.map +1 -0
  74. package/dist/spec/evolve/spec-rewriter.js +254 -0
  75. package/dist/spec/evolve/spec-rewriter.js.map +1 -0
  76. package/dist/spec/graph/queries.d.ts +58 -0
  77. package/dist/spec/graph/queries.d.ts.map +1 -0
  78. package/dist/spec/graph/queries.js +215 -0
  79. package/dist/spec/graph/queries.js.map +1 -0
  80. package/dist/spec/mining/diff-parser.d.ts +33 -0
  81. package/dist/spec/mining/diff-parser.d.ts.map +1 -0
  82. package/dist/spec/mining/diff-parser.js +166 -0
  83. package/dist/spec/mining/diff-parser.js.map +1 -0
  84. package/dist/spec/mining/git-scanner.d.ts +91 -0
  85. package/dist/spec/mining/git-scanner.d.ts.map +1 -0
  86. package/dist/spec/mining/git-scanner.js +266 -0
  87. package/dist/spec/mining/git-scanner.js.map +1 -0
  88. package/dist/spec/mining/pipeline.d.ts +53 -0
  89. package/dist/spec/mining/pipeline.d.ts.map +1 -0
  90. package/dist/spec/mining/pipeline.js +165 -0
  91. package/dist/spec/mining/pipeline.js.map +1 -0
  92. package/dist/spec/mining/scope-resolver.d.ts +45 -0
  93. package/dist/spec/mining/scope-resolver.d.ts.map +1 -0
  94. package/dist/spec/mining/scope-resolver.js +103 -0
  95. package/dist/spec/mining/scope-resolver.js.map +1 -0
  96. package/dist/spec/mining/spec-extractor.d.ts +69 -0
  97. package/dist/spec/mining/spec-extractor.d.ts.map +1 -0
  98. package/dist/spec/mining/spec-extractor.js +369 -0
  99. package/dist/spec/mining/spec-extractor.js.map +1 -0
  100. package/dist/spec/types.d.ts +149 -0
  101. package/dist/spec/types.d.ts.map +1 -0
  102. package/dist/spec/types.js +15 -0
  103. package/dist/spec/types.js.map +1 -0
  104. package/dist/spec/utils.d.ts +166 -0
  105. package/dist/spec/utils.d.ts.map +1 -0
  106. package/dist/spec/utils.js +461 -0
  107. package/dist/spec/utils.js.map +1 -0
  108. package/package.json +58 -57
  109. package/scripts/add-lang/bench.sh +60 -60
  110. package/scripts/add-lang/check-grammar.mjs +75 -75
  111. package/scripts/add-lang/dump-ast.mjs +103 -103
  112. package/scripts/add-lang/verify-extraction.mjs +70 -70
  113. package/scripts/agent-eval/ab-adoption.sh +91 -91
  114. package/scripts/agent-eval/ab-hook.sh +86 -86
  115. package/scripts/agent-eval/ab-impl.sh +78 -78
  116. package/scripts/agent-eval/ab-new-vs-baseline.sh +102 -102
  117. package/scripts/agent-eval/ab-sufficiency.sh +78 -78
  118. package/scripts/agent-eval/arms-F.sh +21 -21
  119. package/scripts/agent-eval/arms-matrix.sh +37 -37
  120. package/scripts/agent-eval/audit.sh +68 -68
  121. package/scripts/agent-eval/bench-readme.sh +28 -28
  122. package/scripts/agent-eval/bench-why-repo.sh +22 -22
  123. package/scripts/agent-eval/block-read-hook.sh +19 -19
  124. package/scripts/agent-eval/hook-settings.json +15 -15
  125. package/scripts/agent-eval/itrun.sh +120 -120
  126. package/scripts/agent-eval/offload-eval-3arm.sh +72 -72
  127. package/scripts/agent-eval/offload-eval-cost.mjs +133 -133
  128. package/scripts/agent-eval/offload-eval-effort.mjs +108 -108
  129. package/scripts/agent-eval/offload-eval-frontload-matrix.sh +25 -25
  130. package/scripts/agent-eval/offload-eval-frontload.sh +47 -47
  131. package/scripts/agent-eval/offload-eval-ground-truth.json +18 -18
  132. package/scripts/agent-eval/offload-eval-hook.mjs +84 -84
  133. package/scripts/agent-eval/offload-eval-judge.mjs +103 -103
  134. package/scripts/agent-eval/offload-eval-matrix.sh +20 -20
  135. package/scripts/agent-eval/offload-eval-metrics.mjs +94 -94
  136. package/scripts/agent-eval/offload-eval-refs1.sh +50 -50
  137. package/scripts/agent-eval/offload-eval-setup.sh +24 -24
  138. package/scripts/agent-eval/offload-eval-styles.sh +71 -71
  139. package/scripts/agent-eval/offload-eval-summarize.mjs +68 -68
  140. package/scripts/agent-eval/offload-eval.md +76 -76
  141. package/scripts/agent-eval/parse-arms.mjs +116 -116
  142. package/scripts/agent-eval/parse-bench-readme.mjs +84 -84
  143. package/scripts/agent-eval/parse-run.mjs +45 -45
  144. package/scripts/agent-eval/parse-session.mjs +93 -93
  145. package/scripts/agent-eval/probe-context.mjs +21 -21
  146. package/scripts/agent-eval/probe-explore.mjs +40 -40
  147. package/scripts/agent-eval/probe-node.mjs +20 -20
  148. package/scripts/agent-eval/probe-sweep.mjs +119 -119
  149. package/scripts/agent-eval/probe-trace.mjs +20 -20
  150. package/scripts/agent-eval/redirect-read-hook.sh +38 -38
  151. package/scripts/agent-eval/run-agent.sh +34 -34
  152. package/scripts/agent-eval/run-all.sh +69 -69
  153. package/scripts/agent-eval/run-arms.sh +56 -56
  154. package/scripts/agent-eval/seq-matrix.mjs +137 -137
  155. package/scripts/build-bundle.sh +118 -118
  156. package/scripts/extract-release-notes.mjs +130 -130
  157. package/scripts/local-install.sh +41 -41
  158. package/scripts/npm-sdk.js +75 -75
  159. package/scripts/npm-shim.js +246 -246
  160. package/scripts/pack-npm.sh +119 -119
  161. package/scripts/prepare-release.mjs +270 -270
  162. package/scripts/qa_eval/README.md +407 -404
  163. package/scripts/qa_eval/_test_deveco_probe.py +41 -41
  164. package/scripts/qa_eval/agent_runner.py +526 -526
  165. package/scripts/qa_eval/data/.gitignore +4 -4
  166. package/scripts/qa_eval/data/test-set.jsonl +2 -22
  167. package/scripts/qa_eval/eval_metrics.py +274 -233
  168. package/scripts/qa_eval/external_agent.py +976 -671
  169. package/scripts/qa_eval/llm_config.py +92 -92
  170. package/scripts/qa_eval/memory_monitor.py +132 -132
  171. package/scripts/qa_eval/my_answer_accuracy.py +187 -187
  172. package/scripts/qa_eval/requirements.txt +2 -2
  173. package/scripts/qa_eval/run_pipeline.py +804 -711
  174. package/scripts/qa_eval/stats_efficiency.py +279 -279
  175. package/scripts/qa_eval/stats_scores.py +207 -207
@@ -1,108 +1,108 @@
1
- #!/usr/bin/env node
2
- // Effort A/B — does HOMEGRAPH_OFFLOAD_EFFORT=high improve offload SYNTHESIS FIDELITY vs low?
3
- // Probe-based (no agent): for each repo × effort × rep, run homegraph_explore with the offload
4
- // ON on the canonical question, capture the synthesized answer + AI tokens/cost/latency, then
5
- // Sonnet-judge that answer's fidelity vs source-verified ground truth. Isolates the synthesis
6
- // from agent/adoption noise. Requires `homegraph login` (managed offload) + indexed repos.
7
- //
8
- // Env: REPS (default 3) · CG_ENGINE (engine repo) · AGENT_EVAL_OUT (repos under /repos) · CONC (judge concurrency)
9
- import { pathToFileURL, fileURLToPath } from 'node:url';
10
- import { resolve, dirname, join } from 'node:path';
11
- import { readFileSync, writeFileSync, existsSync, rmSync } from 'node:fs';
12
- import { execFile } from 'node:child_process';
13
- import { tmpdir } from 'node:os';
14
-
15
- const HERE = dirname(fileURLToPath(import.meta.url));
16
- const ENGINE = process.env.CG_ENGINE || resolve(HERE, '..', '..');
17
- const OUT = process.env.AGENT_EVAL_OUT || '/tmp/cg-offload-eval';
18
- const REPOS = join(OUT, 'repos');
19
- const GT = JSON.parse(readFileSync(resolve(HERE, 'offload-eval-ground-truth.json'), 'utf8'));
20
- const REPS = Number(process.env.REPS || 3);
21
- const CONC = Number(process.env.CONC || 4);
22
- const EFFORTS = (process.env.EFFORTS_FILTER || 'low,high').split(',');
23
- const ONLY = process.env.REPOS_FILTER ? new Set(process.env.REPOS_FILTER.split(',')) : null;
24
- const TIER = { mtkruto: 'small', postybirb: 'medium', shapeshift: 'complex', trezor: 'large' };
25
-
26
- const load = async (rel) => import(pathToFileURL(resolve(ENGINE, rel)).href);
27
- const idx = await load('dist/index.js');
28
- const toolsMod = await load('dist/mcp/tools.js');
29
- const HomeGraph = idx.default?.default ?? idx.default ?? idx.HomeGraph;
30
- const ToolHandler = toolsMod.ToolHandler ?? toolsMod.default?.ToolHandler;
31
- if (typeof HomeGraph?.openSync !== 'function' || typeof ToolHandler !== 'function') {
32
- console.error('could not load engine from', ENGINE); process.exit(2);
33
- }
34
-
35
- const fidPrompt = (gt, ans) => `You are scoring the FIDELITY of a machine-synthesized code-exploration answer against verified ground truth. Do NOT use any tools.
36
-
37
- QUESTION: ${gt.question}
38
-
39
- VERIFIED GROUND TRUTH (the actual call path + files):
40
- ${gt.truth}
41
-
42
- SYNTHESIZED ANSWER (to score):
43
- ${ans || '(empty)'}
44
-
45
- Judge: (1) is the traced call path correct vs ground truth? (2) are the cited files/symbols correct (not fabricated)? (3) if it gave a "Coverage:" verdict, was it honest? A confident WRONG trace is the worst outcome — penalize it harder than an honest partial.
46
- Output ONLY minified JSON: {"verdict":"pass|partial|fail","score":<0-100>,"fabrication":<true|false>,"coverageHonest":<true|false>,"note":"<=20 words"}`;
47
-
48
- const askJudge = (prompt) => new Promise((res) => {
49
- execFile('claude', ['-p', prompt, '--model', 'sonnet', '--effort', 'high', '--max-budget-usd', '0.5',
50
- '--strict-mcp-config', '--mcp-config', '{"mcpServers":{}}'],
51
- { cwd: OUT, maxBuffer: 1 << 24, timeout: 120000 }, (err, stdout) => {
52
- const m = (stdout || '').match(/\{[\s\S]*\}/);
53
- if (!m) return res({ verdict: 'error', score: null, note: (err ? err.message : 'no json').slice(0, 60) });
54
- try { res(JSON.parse(m[0])); } catch { res({ verdict: 'error', score: null }); }
55
- });
56
- });
57
-
58
- // ---- 1. Probe: collect synthesized answers at each effort -------------------
59
- const records = [];
60
- for (const repo of Object.keys(GT)) {
61
- if (ONLY && !ONLY.has(repo)) continue;
62
- const dir = join(REPOS, repo);
63
- if (!existsSync(join(dir, '.homegraph'))) { console.error('skip (not indexed):', repo); continue; }
64
- const cg = HomeGraph.openSync(dir);
65
- const h = new ToolHandler(cg);
66
- for (const effort of EFFORTS) {
67
- for (let rep = 1; rep <= REPS; rep++) {
68
- process.env.HOMEGRAPH_OFFLOAD_EFFORT = effort;
69
- const usageLog = join(tmpdir(), `effort-${repo}-${effort}-${rep}.jsonl`);
70
- try { rmSync(usageLog); } catch { /* none */ }
71
- process.env.HOMEGRAPH_OFFLOAD_USAGE_LOG = usageLog;
72
- let answer = '';
73
- try { answer = (await h.execute('homegraph_explore', { query: GT[repo].question }))?.content?.[0]?.text ?? ''; }
74
- catch (e) { console.error(` ${repo}/${effort}#${rep} explore failed: ${e?.message}`); }
75
- const fired = /Synthesized by HomeGraph/.test(answer);
76
- const ai = { tokens: 0, cost: 0, ms: 0 };
77
- if (existsSync(usageLog)) for (const e of readFileSync(usageLog, 'utf8').split('\n').filter(Boolean).map(JSON.parse)) {
78
- ai.tokens += e.totalTokens || 0; ai.cost += e.costUsd || 0; ai.ms += e.ms || 0;
79
- }
80
- records.push({ repo, tier: TIER[repo], effort, rep, fired, ai, answer });
81
- console.error(` ${repo}/${effort}#${rep}: fired=${fired} ${ai.tokens}tok $${ai.cost.toFixed(4)} ${ai.ms}ms`);
82
- }
83
- }
84
- try { cg.close?.(); } catch { /* none */ }
85
- }
86
-
87
- // ---- 2. Judge fidelity (concurrency) ---------------------------------------
88
- console.error(`\njudging ${records.length} answers (concurrency ${CONC})...`);
89
- let done = 0;
90
- const q = [...records];
91
- async function worker() { while (q.length) { const r = q.shift(); r.fid = await askJudge(fidPrompt(GT[r.repo], r.answer)); console.error(` [${++done}/${records.length}] ${r.repo}/${r.effort}#${r.rep}: ${r.fid.verdict} ${r.fid.score ?? ''}`); } }
92
- await Promise.all(Array.from({ length: CONC }, worker));
93
- writeFileSync(join(OUT, 'effort-results.jsonl'), records.map((r) => JSON.stringify(r)).join('\n') + '\n');
94
-
95
- // ---- 3. Aggregate: low vs high per repo ------------------------------------
96
- const med = (a) => { a = a.filter((x) => x != null).sort((x, y) => x - y); return a.length ? (a.length % 2 ? a[(a.length - 1) / 2] : (a[a.length / 2 - 1] + a[a.length / 2]) / 2) : null; };
97
- console.log(`\n${'='.repeat(80)}\nEFFORT A/B — offload synthesis fidelity (probe, n=${REPS}/cell)\n${'='.repeat(80)}`);
98
- console.log(`${'repo'.padEnd(11)} ${'tier'.padEnd(8)} ${'effort'.padEnd(6)} fired ${'fid(med)'.padStart(8)} ${'fab%'.padStart(5)} ${'AItok'.padStart(7)} ${'AIcost'.padStart(8)} ${'ms(med)'.padStart(8)}`);
99
- for (const repo of Object.keys(GT)) {
100
- for (const effort of EFFORTS) {
101
- const rs = records.filter((r) => r.repo === repo && r.effort === effort);
102
- if (!rs.length) continue;
103
- const fids = rs.map((r) => r.fid?.score).filter((x) => x != null);
104
- const fab = rs.filter((r) => r.fid?.fabrication === true).length;
105
- console.log(`${repo.padEnd(11)} ${TIER[repo].padEnd(8)} ${effort.padEnd(6)} ${rs.filter((r) => r.fired).length}/${rs.length} ${String(med(fids) ?? '—').padStart(8)} ${String(Math.round(100 * fab / rs.length) + '%').padStart(5)} ${String(Math.round(med(rs.map((r) => r.ai.tokens)) / 1000) + 'k').padStart(7)} ${('$' + (med(rs.map((r) => r.ai.cost)) ?? 0).toFixed(4)).padStart(8)} ${String(med(rs.map((r) => r.ai.ms)) ?? '—').padStart(8)}`);
106
- }
107
- }
108
- console.log('');
1
+ #!/usr/bin/env node
2
+ // Effort A/B — does HOMEGRAPH_OFFLOAD_EFFORT=high improve offload SYNTHESIS FIDELITY vs low?
3
+ // Probe-based (no agent): for each repo × effort × rep, run homegraph_explore with the offload
4
+ // ON on the canonical question, capture the synthesized answer + AI tokens/cost/latency, then
5
+ // Sonnet-judge that answer's fidelity vs source-verified ground truth. Isolates the synthesis
6
+ // from agent/adoption noise. Requires `homegraph login` (managed offload) + indexed repos.
7
+ //
8
+ // Env: REPS (default 3) · CG_ENGINE (engine repo) · AGENT_EVAL_OUT (repos under /repos) · CONC (judge concurrency)
9
+ import { pathToFileURL, fileURLToPath } from 'node:url';
10
+ import { resolve, dirname, join } from 'node:path';
11
+ import { readFileSync, writeFileSync, existsSync, rmSync } from 'node:fs';
12
+ import { execFile } from 'node:child_process';
13
+ import { tmpdir } from 'node:os';
14
+
15
+ const HERE = dirname(fileURLToPath(import.meta.url));
16
+ const ENGINE = process.env.CG_ENGINE || resolve(HERE, '..', '..');
17
+ const OUT = process.env.AGENT_EVAL_OUT || '/tmp/cg-offload-eval';
18
+ const REPOS = join(OUT, 'repos');
19
+ const GT = JSON.parse(readFileSync(resolve(HERE, 'offload-eval-ground-truth.json'), 'utf8'));
20
+ const REPS = Number(process.env.REPS || 3);
21
+ const CONC = Number(process.env.CONC || 4);
22
+ const EFFORTS = (process.env.EFFORTS_FILTER || 'low,high').split(',');
23
+ const ONLY = process.env.REPOS_FILTER ? new Set(process.env.REPOS_FILTER.split(',')) : null;
24
+ const TIER = { mtkruto: 'small', postybirb: 'medium', shapeshift: 'complex', trezor: 'large' };
25
+
26
+ const load = async (rel) => import(pathToFileURL(resolve(ENGINE, rel)).href);
27
+ const idx = await load('dist/index.js');
28
+ const toolsMod = await load('dist/mcp/tools.js');
29
+ const HomeGraph = idx.default?.default ?? idx.default ?? idx.HomeGraph;
30
+ const ToolHandler = toolsMod.ToolHandler ?? toolsMod.default?.ToolHandler;
31
+ if (typeof HomeGraph?.openSync !== 'function' || typeof ToolHandler !== 'function') {
32
+ console.error('could not load engine from', ENGINE); process.exit(2);
33
+ }
34
+
35
+ const fidPrompt = (gt, ans) => `You are scoring the FIDELITY of a machine-synthesized code-exploration answer against verified ground truth. Do NOT use any tools.
36
+
37
+ QUESTION: ${gt.question}
38
+
39
+ VERIFIED GROUND TRUTH (the actual call path + files):
40
+ ${gt.truth}
41
+
42
+ SYNTHESIZED ANSWER (to score):
43
+ ${ans || '(empty)'}
44
+
45
+ Judge: (1) is the traced call path correct vs ground truth? (2) are the cited files/symbols correct (not fabricated)? (3) if it gave a "Coverage:" verdict, was it honest? A confident WRONG trace is the worst outcome — penalize it harder than an honest partial.
46
+ Output ONLY minified JSON: {"verdict":"pass|partial|fail","score":<0-100>,"fabrication":<true|false>,"coverageHonest":<true|false>,"note":"<=20 words"}`;
47
+
48
+ const askJudge = (prompt) => new Promise((res) => {
49
+ execFile('claude', ['-p', prompt, '--model', 'sonnet', '--effort', 'high', '--max-budget-usd', '0.5',
50
+ '--strict-mcp-config', '--mcp-config', '{"mcpServers":{}}'],
51
+ { cwd: OUT, maxBuffer: 1 << 24, timeout: 120000 }, (err, stdout) => {
52
+ const m = (stdout || '').match(/\{[\s\S]*\}/);
53
+ if (!m) return res({ verdict: 'error', score: null, note: (err ? err.message : 'no json').slice(0, 60) });
54
+ try { res(JSON.parse(m[0])); } catch { res({ verdict: 'error', score: null }); }
55
+ });
56
+ });
57
+
58
+ // ---- 1. Probe: collect synthesized answers at each effort -------------------
59
+ const records = [];
60
+ for (const repo of Object.keys(GT)) {
61
+ if (ONLY && !ONLY.has(repo)) continue;
62
+ const dir = join(REPOS, repo);
63
+ if (!existsSync(join(dir, '.homegraph'))) { console.error('skip (not indexed):', repo); continue; }
64
+ const cg = HomeGraph.openSync(dir);
65
+ const h = new ToolHandler(cg);
66
+ for (const effort of EFFORTS) {
67
+ for (let rep = 1; rep <= REPS; rep++) {
68
+ process.env.HOMEGRAPH_OFFLOAD_EFFORT = effort;
69
+ const usageLog = join(tmpdir(), `effort-${repo}-${effort}-${rep}.jsonl`);
70
+ try { rmSync(usageLog); } catch { /* none */ }
71
+ process.env.HOMEGRAPH_OFFLOAD_USAGE_LOG = usageLog;
72
+ let answer = '';
73
+ try { answer = (await h.execute('homegraph_explore', { query: GT[repo].question }))?.content?.[0]?.text ?? ''; }
74
+ catch (e) { console.error(` ${repo}/${effort}#${rep} explore failed: ${e?.message}`); }
75
+ const fired = /Synthesized by HomeGraph/.test(answer);
76
+ const ai = { tokens: 0, cost: 0, ms: 0 };
77
+ if (existsSync(usageLog)) for (const e of readFileSync(usageLog, 'utf8').split('\n').filter(Boolean).map(JSON.parse)) {
78
+ ai.tokens += e.totalTokens || 0; ai.cost += e.costUsd || 0; ai.ms += e.ms || 0;
79
+ }
80
+ records.push({ repo, tier: TIER[repo], effort, rep, fired, ai, answer });
81
+ console.error(` ${repo}/${effort}#${rep}: fired=${fired} ${ai.tokens}tok $${ai.cost.toFixed(4)} ${ai.ms}ms`);
82
+ }
83
+ }
84
+ try { cg.close?.(); } catch { /* none */ }
85
+ }
86
+
87
+ // ---- 2. Judge fidelity (concurrency) ---------------------------------------
88
+ console.error(`\njudging ${records.length} answers (concurrency ${CONC})...`);
89
+ let done = 0;
90
+ const q = [...records];
91
+ async function worker() { while (q.length) { const r = q.shift(); r.fid = await askJudge(fidPrompt(GT[r.repo], r.answer)); console.error(` [${++done}/${records.length}] ${r.repo}/${r.effort}#${r.rep}: ${r.fid.verdict} ${r.fid.score ?? ''}`); } }
92
+ await Promise.all(Array.from({ length: CONC }, worker));
93
+ writeFileSync(join(OUT, 'effort-results.jsonl'), records.map((r) => JSON.stringify(r)).join('\n') + '\n');
94
+
95
+ // ---- 3. Aggregate: low vs high per repo ------------------------------------
96
+ const med = (a) => { a = a.filter((x) => x != null).sort((x, y) => x - y); return a.length ? (a.length % 2 ? a[(a.length - 1) / 2] : (a[a.length / 2 - 1] + a[a.length / 2]) / 2) : null; };
97
+ console.log(`\n${'='.repeat(80)}\nEFFORT A/B — offload synthesis fidelity (probe, n=${REPS}/cell)\n${'='.repeat(80)}`);
98
+ console.log(`${'repo'.padEnd(11)} ${'tier'.padEnd(8)} ${'effort'.padEnd(6)} fired ${'fid(med)'.padStart(8)} ${'fab%'.padStart(5)} ${'AItok'.padStart(7)} ${'AIcost'.padStart(8)} ${'ms(med)'.padStart(8)}`);
99
+ for (const repo of Object.keys(GT)) {
100
+ for (const effort of EFFORTS) {
101
+ const rs = records.filter((r) => r.repo === repo && r.effort === effort);
102
+ if (!rs.length) continue;
103
+ const fids = rs.map((r) => r.fid?.score).filter((x) => x != null);
104
+ const fab = rs.filter((r) => r.fid?.fabrication === true).length;
105
+ console.log(`${repo.padEnd(11)} ${TIER[repo].padEnd(8)} ${effort.padEnd(6)} ${rs.filter((r) => r.fired).length}/${rs.length} ${String(med(fids) ?? '—').padStart(8)} ${String(Math.round(100 * fab / rs.length) + '%').padStart(5)} ${String(Math.round(med(rs.map((r) => r.ai.tokens)) / 1000) + 'k').padStart(7)} ${('$' + (med(rs.map((r) => r.ai.cost)) ?? 0).toFixed(4)).padStart(8)} ${String(med(rs.map((r) => r.ai.ms)) ?? '—').padStart(8)}`);
106
+ }
107
+ }
108
+ console.log('');
@@ -1,25 +1,25 @@
1
- #!/usr/bin/env bash
2
- # Run the FRONTLOAD arm across all 4 tiers (n reps), then judge + merge with the existing
3
- # matrix (offload/raw/nocg in $OUT/judged.jsonl, if present) + emit a combined summary.
4
- # Env: REPS (default 3) AGENT_EVAL_OUT=<scratch dir>
5
- set -uo pipefail
6
- HERE="$(cd "$(dirname "$0")" && pwd)"
7
- OUT="${AGENT_EVAL_OUT:-/tmp/cg-offload-eval}"
8
- GT="$HERE/offload-eval-ground-truth.json"
9
- REPS="${REPS:-3}"
10
- export RESULTS="$OUT/results-fl.jsonl"
11
- : > "$RESULTS"; rm -f "$OUT/runs/hook-debug.log"
12
- for repo in mtkruto postybirb shapeshift trezor; do
13
- case "$repo" in mtkruto) tier=small;; postybirb) tier=medium;; shapeshift) tier=complex;; trezor) tier=large;; esac
14
- Q=$(node -e "console.log(JSON.parse(require('fs').readFileSync(process.argv[1],'utf8'))[process.argv[2]].question)" "$GT" "$repo")
15
- echo ""; echo "### $repo ($tier) $(date +%H:%M:%S)"
16
- bash "$HERE/offload-eval-frontload.sh" "$OUT/repos/$repo" "$tier" "$REPS" "$Q"
17
- done
18
- echo ""
19
- echo "frontload: $(wc -l < "$RESULTS") runs | hook injections: $(grep -c INJECTED "$OUT/runs/hook-debug.log" 2>/dev/null) | errors: $(grep -c ERROR "$OUT/runs/hook-debug.log" 2>/dev/null)"
20
- echo "=== JUDGE frontload ==="
21
- node "$HERE/offload-eval-judge.mjs" --results "$RESULTS" --truth "$GT" --out "$OUT/judged-fl.jsonl" --concurrency 4 2>&1 | tail -4
22
- if [ -f "$OUT/judged.jsonl" ]; then cat "$OUT/judged.jsonl" "$OUT/judged-fl.jsonl" > "$OUT/judged-all.jsonl"; else cp "$OUT/judged-fl.jsonl" "$OUT/judged-all.jsonl"; fi
23
- echo "=== COMBINED SUMMARY ==="
24
- node "$HERE/offload-eval-summarize.mjs" "$OUT/judged-all.jsonl"
25
- echo "###### FRONTLOAD MATRIX DONE"
1
+ #!/usr/bin/env bash
2
+ # Run the FRONTLOAD arm across all 4 tiers (n reps), then judge + merge with the existing
3
+ # matrix (offload/raw/nocg in $OUT/judged.jsonl, if present) + emit a combined summary.
4
+ # Env: REPS (default 3) AGENT_EVAL_OUT=<scratch dir>
5
+ set -uo pipefail
6
+ HERE="$(cd "$(dirname "$0")" && pwd)"
7
+ OUT="${AGENT_EVAL_OUT:-/tmp/cg-offload-eval}"
8
+ GT="$HERE/offload-eval-ground-truth.json"
9
+ REPS="${REPS:-3}"
10
+ export RESULTS="$OUT/results-fl.jsonl"
11
+ : > "$RESULTS"; rm -f "$OUT/runs/hook-debug.log"
12
+ for repo in mtkruto postybirb shapeshift trezor; do
13
+ case "$repo" in mtkruto) tier=small;; postybirb) tier=medium;; shapeshift) tier=complex;; trezor) tier=large;; esac
14
+ Q=$(node -e "console.log(JSON.parse(require('fs').readFileSync(process.argv[1],'utf8'))[process.argv[2]].question)" "$GT" "$repo")
15
+ echo ""; echo "### $repo ($tier) $(date +%H:%M:%S)"
16
+ bash "$HERE/offload-eval-frontload.sh" "$OUT/repos/$repo" "$tier" "$REPS" "$Q"
17
+ done
18
+ echo ""
19
+ echo "frontload: $(wc -l < "$RESULTS") runs | hook injections: $(grep -c INJECTED "$OUT/runs/hook-debug.log" 2>/dev/null) | errors: $(grep -c ERROR "$OUT/runs/hook-debug.log" 2>/dev/null)"
20
+ echo "=== JUDGE frontload ==="
21
+ node "$HERE/offload-eval-judge.mjs" --results "$RESULTS" --truth "$GT" --out "$OUT/judged-fl.jsonl" --concurrency 4 2>&1 | tail -4
22
+ if [ -f "$OUT/judged.jsonl" ]; then cat "$OUT/judged.jsonl" "$OUT/judged-fl.jsonl" > "$OUT/judged-all.jsonl"; else cp "$OUT/judged-fl.jsonl" "$OUT/judged-all.jsonl"; fi
23
+ echo "=== COMBINED SUMMARY ==="
24
+ node "$HERE/offload-eval-summarize.mjs" "$OUT/judged-all.jsonl"
25
+ echo "###### FRONTLOAD MATRIX DONE"
@@ -1,47 +1,47 @@
1
- #!/usr/bin/env bash
2
- # FRONTLOAD arm (approach 1): homegraph attached (offload-disabled) + the front-load
3
- # UserPromptSubmit hook (offload-eval-hook.mjs), n reps, appended to $RESULTS. Compare against
4
- # the matrix's raw/nocg baselines. Usage: offload-eval-frontload.sh <indexed-repo> <tier> <reps> "<Q>"
5
- # Env: MODEL=sonnet EFFORT=high RESULTS=<file> AGENT_EVAL_OUT=<scratch dir>
6
- set -uo pipefail
7
- HERE="$(cd "$(dirname "$0")" && pwd)"
8
- ENGINE="$(cd "$HERE/../.." && pwd)"
9
- BIN="$ENGINE/dist/bin/homegraph.js"
10
- OUT="${AGENT_EVAL_OUT:-/tmp/cg-offload-eval}"
11
- TARGET="${1:?repo}"; TIER="${2:?tier}"; REPS="${3:?reps}"; Q="${4:?question}"
12
- RUNS="$OUT/runs"
13
- EXTRACT="$HERE/offload-eval-metrics.mjs"
14
- RESULTS="${RESULTS:-$OUT/results-fl.jsonl}"
15
- REPO=$(basename "$TARGET")
16
- mkdir -p "$RUNS"
17
- [ -d "$TARGET/.homegraph" ] || { echo "not indexed: $TARGET"; exit 1; }
18
- TARGET=$(cd "$TARGET" && pwd -P)
19
-
20
- CFG="$RUNS/mcp-fl-$REPO.json"
21
- printf '{"mcpServers":{"homegraph":{"command":"env","args":["HOMEGRAPH_WASM_RELAUNCHED=1","HOMEGRAPH_OFFLOAD_DISABLE=1","node","%s","serve","--mcp","--path","%s"]}}}' "$BIN" "$TARGET" > "$CFG"
22
- # Generate the hook settings pointing at the persisted hook; enable its debug log so we can
23
- # count injections (claude passes this env down to the spawned hook process).
24
- HOOKCFG="$RUNS/frontload-settings.json"
25
- printf '{"hooks":{"UserPromptSubmit":[{"hooks":[{"type":"command","command":"node %s/offload-eval-hook.mjs"}]}]}}' "$HERE" > "$HOOKCFG"
26
- export CG_FRONTLOAD_DEBUG="$RUNS/hook-debug.log"
27
-
28
- prewarm() {
29
- pkill -9 -f "serve --mcp --path $1" 2>/dev/null; rm -f "$1/.homegraph/daemon.sock" 2>/dev/null; sleep 0.6
30
- env HOMEGRAPH_OFFLOAD_DISABLE=1 HOMEGRAPH_DAEMON_IDLE_TIMEOUT_MS=1800000 node "$BIN" serve --mcp --path "$1" </dev/null >/dev/null 2>&1 &
31
- node -e 'const fs=require("fs");let n=0;const t=setInterval(()=>{if(fs.existsSync(process.argv[1]+"/.homegraph/daemon.sock")){clearInterval(t);process.exit(0)}if(n++>150){clearInterval(t);process.exit(1)}},100)' "$1" \
32
- && echo " daemon warm" || echo " WARN no daemon"
33
- }
34
-
35
- echo "###### FRONTLOAD repo=$REPO tier=$TIER reps=$REPS"
36
- prewarm "$TARGET"
37
- for r in $(seq 1 "$REPS"); do
38
- tag="$REPO-frontload-$r"
39
- ( cd "$TARGET" && claude -p "$Q" --output-format stream-json --verbose --permission-mode bypassPermissions \
40
- --model "${MODEL:-sonnet}" --effort "${EFFORT:-high}" --max-budget-usd 4 \
41
- --strict-mcp-config --mcp-config "$CFG" --settings "$HOOKCFG" \
42
- </dev/null > "$RUNS/$tag.jsonl" 2>"$RUNS/$tag.err" )
43
- node "$EXTRACT" --run "$RUNS/$tag.jsonl" --usage "-" --arm frontload --rep "$r" --repo "$REPO" --tier "$TIER" --q "$Q" >> "$RESULTS"
44
- node -e 'const o=JSON.parse(require("fs").readFileSync(process.argv[1],"utf8").trim().split("\n").pop());console.log(` [frontload #${o.rep}] ${o.durationSec}s | main $${o.costUsdMain} ${o.tokBillable}tok | read=${o.read} grep=${o.grep} agentExplore=${o.explore} | ok=${o.ok}`)' "$RESULTS"
45
- done
46
- pkill -9 -f "serve --mcp --path $TARGET" 2>/dev/null; rm -f "$TARGET/.homegraph/daemon.sock" 2>/dev/null
47
- echo "###### FRONTLOAD DONE $REPO (cumulative hook injections: $(grep -c INJECTED "$CG_FRONTLOAD_DEBUG" 2>/dev/null))"
1
+ #!/usr/bin/env bash
2
+ # FRONTLOAD arm (approach 1): homegraph attached (offload-disabled) + the front-load
3
+ # UserPromptSubmit hook (offload-eval-hook.mjs), n reps, appended to $RESULTS. Compare against
4
+ # the matrix's raw/nocg baselines. Usage: offload-eval-frontload.sh <indexed-repo> <tier> <reps> "<Q>"
5
+ # Env: MODEL=sonnet EFFORT=high RESULTS=<file> AGENT_EVAL_OUT=<scratch dir>
6
+ set -uo pipefail
7
+ HERE="$(cd "$(dirname "$0")" && pwd)"
8
+ ENGINE="$(cd "$HERE/../.." && pwd)"
9
+ BIN="$ENGINE/dist/bin/homegraph.js"
10
+ OUT="${AGENT_EVAL_OUT:-/tmp/cg-offload-eval}"
11
+ TARGET="${1:?repo}"; TIER="${2:?tier}"; REPS="${3:?reps}"; Q="${4:?question}"
12
+ RUNS="$OUT/runs"
13
+ EXTRACT="$HERE/offload-eval-metrics.mjs"
14
+ RESULTS="${RESULTS:-$OUT/results-fl.jsonl}"
15
+ REPO=$(basename "$TARGET")
16
+ mkdir -p "$RUNS"
17
+ [ -d "$TARGET/.homegraph" ] || { echo "not indexed: $TARGET"; exit 1; }
18
+ TARGET=$(cd "$TARGET" && pwd -P)
19
+
20
+ CFG="$RUNS/mcp-fl-$REPO.json"
21
+ printf '{"mcpServers":{"homegraph":{"command":"env","args":["HOMEGRAPH_WASM_RELAUNCHED=1","HOMEGRAPH_OFFLOAD_DISABLE=1","node","%s","serve","--mcp","--path","%s"]}}}' "$BIN" "$TARGET" > "$CFG"
22
+ # Generate the hook settings pointing at the persisted hook; enable its debug log so we can
23
+ # count injections (claude passes this env down to the spawned hook process).
24
+ HOOKCFG="$RUNS/frontload-settings.json"
25
+ printf '{"hooks":{"UserPromptSubmit":[{"hooks":[{"type":"command","command":"node %s/offload-eval-hook.mjs"}]}]}}' "$HERE" > "$HOOKCFG"
26
+ export CG_FRONTLOAD_DEBUG="$RUNS/hook-debug.log"
27
+
28
+ prewarm() {
29
+ pkill -9 -f "serve --mcp --path $1" 2>/dev/null; rm -f "$1/.homegraph/daemon.sock" 2>/dev/null; sleep 0.6
30
+ env HOMEGRAPH_OFFLOAD_DISABLE=1 HOMEGRAPH_DAEMON_IDLE_TIMEOUT_MS=1800000 node "$BIN" serve --mcp --path "$1" </dev/null >/dev/null 2>&1 &
31
+ node -e 'const fs=require("fs");let n=0;const t=setInterval(()=>{if(fs.existsSync(process.argv[1]+"/.homegraph/daemon.sock")){clearInterval(t);process.exit(0)}if(n++>150){clearInterval(t);process.exit(1)}},100)' "$1" \
32
+ && echo " daemon warm" || echo " WARN no daemon"
33
+ }
34
+
35
+ echo "###### FRONTLOAD repo=$REPO tier=$TIER reps=$REPS"
36
+ prewarm "$TARGET"
37
+ for r in $(seq 1 "$REPS"); do
38
+ tag="$REPO-frontload-$r"
39
+ ( cd "$TARGET" && claude -p "$Q" --output-format stream-json --verbose --permission-mode bypassPermissions \
40
+ --model "${MODEL:-sonnet}" --effort "${EFFORT:-high}" --max-budget-usd 4 \
41
+ --strict-mcp-config --mcp-config "$CFG" --settings "$HOOKCFG" \
42
+ </dev/null > "$RUNS/$tag.jsonl" 2>"$RUNS/$tag.err" )
43
+ node "$EXTRACT" --run "$RUNS/$tag.jsonl" --usage "-" --arm frontload --rep "$r" --repo "$REPO" --tier "$TIER" --q "$Q" >> "$RESULTS"
44
+ node -e 'const o=JSON.parse(require("fs").readFileSync(process.argv[1],"utf8").trim().split("\n").pop());console.log(` [frontload #${o.rep}] ${o.durationSec}s | main $${o.costUsdMain} ${o.tokBillable}tok | read=${o.read} grep=${o.grep} agentExplore=${o.explore} | ok=${o.ok}`)' "$RESULTS"
45
+ done
46
+ pkill -9 -f "serve --mcp --path $TARGET" 2>/dev/null; rm -f "$TARGET/.homegraph/daemon.sock" 2>/dev/null
47
+ echo "###### FRONTLOAD DONE $REPO (cumulative hook injections: $(grep -c INJECTED "$CG_FRONTLOAD_DEBUG" 2>/dev/null))"
@@ -1,18 +1,18 @@
1
- {
2
- "mtkruto": {
3
- "question": "How does calling the high-level client.sendMessage() method get the message serialized into a TL/MTProto request and sent over the network transport to Telegram's servers? Trace the call path.",
4
- "truth": "Verified call path (small TS Telegram-client lib):\n1. Client.sendMessage — client/6_client.ts:1852 → calls this.#messageManager.sendMessage(...) (1853)\n2. MessageManager.sendMessage — client/3_message_manager.ts:330 → builds the TL function {_:\"messages.sendMessage\", peer, message,...} and calls this.#c.invoke(...) (~377)\n3. c.invoke closure / Client.#invoke — client/6_client.ts:279/887 → resolves a ClientEncrypted and calls client.invoke(function_) (897)\n4. ClientEncrypted.invoke — client/2_client_encrypted.ts:324 → this.#send(function_) (325)\n5. ClientEncrypted.#send — client/2_client_encrypted.ts:261 → SERIALIZES via Api.serializeObject(function_) (290), then this.session.send(body) (296)\n - Serialization: Api.serializeObject (tl/2_telegram.ts:38) → new TLWriter().writeObject(...) (tl/1_tl_writer.ts) writes constructor id + fields\n6. SessionEncrypted.send — session/2_session_encrypted.ts:193 → ENQUEUES a PendingMessage and wakes the send loop (#awakeSendLoop) [DYNAMIC: async queue, not a direct call]\n7. SessionEncrypted.#sendLoopBody (AbortableLoop) — session/2_session_encrypted.ts:282 → #encryptMessage (serializeMessage tl/2_message.ts + IGE-256 AES) then this.transport.transport.send(payload) (339)\n8. Transport.send — TransportAbridged.send (transport/1_transport_abridged.ts:71) or TransportIntermediate → this.#connection.write(encrypt(framed))\n9. ConnectionTCP.write (connection/1_connection_tcp.ts:96) or ConnectionWebSocket.write — bytes exit the process.\nKEY SYMBOLS a correct answer must hit: sendMessage → MessageManager.sendMessage → invoke (ClientEncrypted.invoke/#send) → Api.serializeObject/TLWriter → SessionEncrypted.send (queue) → #sendLoopBody/#encryptMessage → Transport.send → Connection.write.\nDYNAMIC BOUNDARIES: the invoke indirection (closure→#invoke→ClientEncrypted), and the send-loop QUEUE between SessionEncrypted.send and #sendLoopBody (async, no direct edge)."
5
- },
6
- "postybirb": {
7
- "question": "How does submitting/queueing a post reach the website-specific code that actually uploads to a target website? Trace the path from the submission entry point in the NestJS server to a concrete website service's post implementation.",
8
- "truth": "Verified call path (NestJS+Electron, server under electron-app/src/server/):\n1. Entry: PostController.queue (submission/post/post.controller.ts:33, POST queue/:id) OR SubmissionService.queueScheduledSubmissions (submission/submission.service.ts, @Interval(60000) scheduler) — both call PostService.queue\n2. PostService.queue — submission/post/post.service.ts:68 → this.post(submission)\n3. PostService.post (private) — post.service.ts:206 → this.createPoster(...) for each non-default SubmissionPart\n4. PostService.createPoster — post.service.ts:373 → website = this.websites.getWebsiteModule(part.website) [part.website is a STRING]\n5. WebsiteProvider.getWebsiteModule — websites/website-provider.service.ts:86 → websiteModulesMap[name.toLowerCase()] [DYNAMIC: string-keyed registry of DI-injected Website singletons]; then new Poster(..., website, ...)\n6. Poster constructor — submission/post/poster.ts:117 → setTimeout(this.post.bind(this), delay) [DYNAMIC: async timer, min ~5s]\n7. Poster.post (poster.ts:131) → performPost (171) → attemptPost (217)\n8. Poster.attemptPost — poster.ts:217 → this.website.postFileSubmission(token, data, accountData) [DYNAMIC: polymorphic dispatch on abstract Website] (or postNotificationSubmission)\n9. Website.postFileSubmission (abstract) — websites/website.base.ts:102\n10. Concrete e.g. FurAffinity.postFileSubmission — websites/fur-affinity/fur-affinity.service.ts:231 → multi-step HTTP upload (GET /submit, POST /submit/upload multipart, POST /submit/finalize)\nKEY SYMBOLS: PostService.queue → PostService.post → createPoster → WebsiteProvider.getWebsiteModule (registry) → Poster (setTimeout) → Poster.attemptPost → Website.postFileSubmission (abstract base) → a concrete website service (e.g. FurAffinity).\nDYNAMIC BOUNDARIES: NestJS DI builds the website registry; string-keyed map lookup; setTimeout defers Poster.post; polymorphic dispatch on the abstract Website base. A correct answer must reach a concrete website's post via the registry + base class, not stop at PostService."
9
- },
10
- "shapeshift": {
11
- "question": "How does executing a swap work in ShapeShift — from the code that fetches quotes and selects a swapper down to a specific swapper's execute/trade? Name the swapper interface, the registry, and one concrete swapper, and trace the path.",
12
- "truth": "Verified call path (large multi-package monorepo; swap logic in packages/swapper + execution in src/lib):\nQUOTE/REGISTRY layer:\n- Swapper interface = the `Swapper` type (execute methods) + `SwapperApi` type (getTradeQuote/getUnsignedTx) — packages/swapper/src/types.ts (~846/897)\n- Registry = `swappers: Record<SwapperName,(SwapperApi & Swapper)>` — packages/swapper/src/constants.ts:52 (merges e.g. thorchainSwapper + thorchainApi)\n- Aggregator: swapperApi RTK endpoint getTradeQuote/getTradeRates (src/state/apis/swapper/swapperApi.ts:78/156) → getTradeQuotes (packages/swapper/src/swapper.ts:18/26): swapper = swappers[swapperName]; swapper.getTradeQuote(...) [DYNAMIC interface dispatch]\n- Concrete (THORChain): thorchainApi.getTradeQuote (swappers/ThorchainSwapper/endpoints.ts → getTradeQuote/getTradeQuote.ts:15) → getL1RateOrQuote → getQuote (thorService HTTP)\nSELECTION: tradeQuoteSlice selectors rank quotes (selectSortedTradeQuotes / selectActiveSwapperName) [DYNAMIC: winner chosen by ranking/user]\nEXECUTION layer:\n- useTradeExecution (src/components/MultiHopTrade/.../hooks/useTradeExecution.tsx:200/476) → new TradeExecution(); execution.execEvmTransaction(...) (CowSwap: execEvmMessage)\n- TradeExecution.execEvmTransaction — src/lib/tradeExecution.ts:326 → _execWalletAgnostic(...) (372)\n- TradeExecution._execWalletAgnostic — tradeExecution.ts:136 → swapper = swappers[swapperName] (149) [DYNAMIC registry]; buildSignBroadcast → swapper.getUnsignedEvmTransaction(...) (355) then swapper.executeEvmTransaction(unsignedTx, {signAndBroadcastTransaction}) (367) [DYNAMIC SwapperApi/Swapper interface]\n- Concrete (THORChain): thorchainSwapper.executeEvmTransaction (swappers/ThorchainSwapper/ThorchainSwapper.ts → utils.ts:181) delegates to callbacks.signAndBroadcastTransaction (wallet). CowSwap alt: cowSwapper.executeEvmMessage → signCowOrder + cowService.post.\nKEY SYMBOLS: Swapper/SwapperApi types, swappers registry (constants.ts), getTradeQuotes (swapper.ts), TradeExecution._execWalletAgnostic, swapper.getUnsignedEvmTransaction/executeEvmTransaction, one concrete swapper (thorchainSwapper/zrxSwapper/cowSwapper).\nDYNAMIC BOUNDARIES: swappers[name] registry lookup (2 sites); all hops into a concrete swapper are via the Swapper/SwapperApi interface, never a direct function. A correct answer must name the interface + registry and reach a concrete swapper through interface dispatch."
13
- },
14
- "trezor": {
15
- "question": "How does sending a crypto transaction flow from the send form's review/sign action through to signing it via @trezor/connect (TrezorConnect.signTransaction)? Trace the call path.",
16
- "truth": "Verified call path (trezor-suite monorepo; app in packages/suite, shared logic in suite-common/wallet-core, device API in packages/connect):\n1. ReviewButton.handleButtonReviewClick — packages/suite/src/views/wallet/send/TotalSent/ReviewButton.tsx:120 → signTransaction() (= useSendForm's sign)\n2. useSendForm.sign (exported as signTransaction) — packages/suite/src/hooks/wallet/useSendForm.ts:278 → dispatch(signAndPushSendFormTransactionThunk({formState, precomposedTransaction, selectedAccount})) [DYNAMIC: redux thunk]\n3. signAndPushSendFormTransactionThunk — packages/suite/src/actions/wallet/send/sendFormThunks.ts:237 → (first enhancePrecomposedTransactionThunk) then dispatch(signTransactionThunk({...})) [cross-package: thunk from suite-common/wallet-core]\n4. signTransactionThunk (coin-routing hub) — suite-common/wallet-core/src/send/sendFormThunks.ts:532/573 → networkType branch → dispatch(signBitcoinSendFormTransactionThunk) (ethereum→signEthereumSendFormTransactionThunk, etc.) [DYNAMIC: runtime coin dispatch]\n5. signBitcoinSendFormTransactionThunk — suite-common/wallet-core/src/send/sendFormBitcoinThunks.ts:394 → await TrezorConnect.signTransaction(signPayload)\n6. TrezorConnect.signTransaction (facade) — packages/connect-common/src/factory.ts → closure calls impl.call({method:'signTransaction',...}) [DYNAMIC: facade / iframe-or-module boundary]\n7. CoreInModule.call — packages/connect/src/impl/core-in-module.ts:171 → posts CORE_CALL to core (deferred promise)\n8. Core.onCall → getMethod — packages/connect/src/core/index.ts → getMethod resolves 'signTransaction' → new SignTransaction(message) [DYNAMIC: name→class]\n9. SignTransaction.run — packages/connect/src/api/signTransaction.ts:317 → signTx via device.getCommands().typedCall (protobuf to device). After signing, signAndPushSendFormTransactionThunk → pushSendFormTransactionThunk → TrezorConnect.pushTransaction.\nKEY SYMBOLS: ReviewButton → useSendForm.sign → signAndPushSendFormTransactionThunk (suite) → signTransactionThunk (wallet-core, coin hub) → signBitcoinSendFormTransactionThunk → TrezorConnect.signTransaction → connect factory/CoreInModule.call → SignTransaction.run.\nDYNAMIC BOUNDARIES: every suite→wallet-core hop is a redux thunk dispatch; signTransactionThunk branches by networkType at runtime; TrezorConnect is a dynamically-built facade crossing an iframe/module boundary; the SignTransaction class is resolved by name. A correct answer must cross suite→wallet-core→connect and reach TrezorConnect.signTransaction / SignTransaction.run, not stop at the UI."
17
- }
18
- }
1
+ {
2
+ "mtkruto": {
3
+ "question": "How does calling the high-level client.sendMessage() method get the message serialized into a TL/MTProto request and sent over the network transport to Telegram's servers? Trace the call path.",
4
+ "truth": "Verified call path (small TS Telegram-client lib):\n1. Client.sendMessage — client/6_client.ts:1852 → calls this.#messageManager.sendMessage(...) (1853)\n2. MessageManager.sendMessage — client/3_message_manager.ts:330 → builds the TL function {_:\"messages.sendMessage\", peer, message,...} and calls this.#c.invoke(...) (~377)\n3. c.invoke closure / Client.#invoke — client/6_client.ts:279/887 → resolves a ClientEncrypted and calls client.invoke(function_) (897)\n4. ClientEncrypted.invoke — client/2_client_encrypted.ts:324 → this.#send(function_) (325)\n5. ClientEncrypted.#send — client/2_client_encrypted.ts:261 → SERIALIZES via Api.serializeObject(function_) (290), then this.session.send(body) (296)\n - Serialization: Api.serializeObject (tl/2_telegram.ts:38) → new TLWriter().writeObject(...) (tl/1_tl_writer.ts) writes constructor id + fields\n6. SessionEncrypted.send — session/2_session_encrypted.ts:193 → ENQUEUES a PendingMessage and wakes the send loop (#awakeSendLoop) [DYNAMIC: async queue, not a direct call]\n7. SessionEncrypted.#sendLoopBody (AbortableLoop) — session/2_session_encrypted.ts:282 → #encryptMessage (serializeMessage tl/2_message.ts + IGE-256 AES) then this.transport.transport.send(payload) (339)\n8. Transport.send — TransportAbridged.send (transport/1_transport_abridged.ts:71) or TransportIntermediate → this.#connection.write(encrypt(framed))\n9. ConnectionTCP.write (connection/1_connection_tcp.ts:96) or ConnectionWebSocket.write — bytes exit the process.\nKEY SYMBOLS a correct answer must hit: sendMessage → MessageManager.sendMessage → invoke (ClientEncrypted.invoke/#send) → Api.serializeObject/TLWriter → SessionEncrypted.send (queue) → #sendLoopBody/#encryptMessage → Transport.send → Connection.write.\nDYNAMIC BOUNDARIES: the invoke indirection (closure→#invoke→ClientEncrypted), and the send-loop QUEUE between SessionEncrypted.send and #sendLoopBody (async, no direct edge)."
5
+ },
6
+ "postybirb": {
7
+ "question": "How does submitting/queueing a post reach the website-specific code that actually uploads to a target website? Trace the path from the submission entry point in the NestJS server to a concrete website service's post implementation.",
8
+ "truth": "Verified call path (NestJS+Electron, server under electron-app/src/server/):\n1. Entry: PostController.queue (submission/post/post.controller.ts:33, POST queue/:id) OR SubmissionService.queueScheduledSubmissions (submission/submission.service.ts, @Interval(60000) scheduler) — both call PostService.queue\n2. PostService.queue — submission/post/post.service.ts:68 → this.post(submission)\n3. PostService.post (private) — post.service.ts:206 → this.createPoster(...) for each non-default SubmissionPart\n4. PostService.createPoster — post.service.ts:373 → website = this.websites.getWebsiteModule(part.website) [part.website is a STRING]\n5. WebsiteProvider.getWebsiteModule — websites/website-provider.service.ts:86 → websiteModulesMap[name.toLowerCase()] [DYNAMIC: string-keyed registry of DI-injected Website singletons]; then new Poster(..., website, ...)\n6. Poster constructor — submission/post/poster.ts:117 → setTimeout(this.post.bind(this), delay) [DYNAMIC: async timer, min ~5s]\n7. Poster.post (poster.ts:131) → performPost (171) → attemptPost (217)\n8. Poster.attemptPost — poster.ts:217 → this.website.postFileSubmission(token, data, accountData) [DYNAMIC: polymorphic dispatch on abstract Website] (or postNotificationSubmission)\n9. Website.postFileSubmission (abstract) — websites/website.base.ts:102\n10. Concrete e.g. FurAffinity.postFileSubmission — websites/fur-affinity/fur-affinity.service.ts:231 → multi-step HTTP upload (GET /submit, POST /submit/upload multipart, POST /submit/finalize)\nKEY SYMBOLS: PostService.queue → PostService.post → createPoster → WebsiteProvider.getWebsiteModule (registry) → Poster (setTimeout) → Poster.attemptPost → Website.postFileSubmission (abstract base) → a concrete website service (e.g. FurAffinity).\nDYNAMIC BOUNDARIES: NestJS DI builds the website registry; string-keyed map lookup; setTimeout defers Poster.post; polymorphic dispatch on the abstract Website base. A correct answer must reach a concrete website's post via the registry + base class, not stop at PostService."
9
+ },
10
+ "shapeshift": {
11
+ "question": "How does executing a swap work in ShapeShift — from the code that fetches quotes and selects a swapper down to a specific swapper's execute/trade? Name the swapper interface, the registry, and one concrete swapper, and trace the path.",
12
+ "truth": "Verified call path (large multi-package monorepo; swap logic in packages/swapper + execution in src/lib):\nQUOTE/REGISTRY layer:\n- Swapper interface = the `Swapper` type (execute methods) + `SwapperApi` type (getTradeQuote/getUnsignedTx) — packages/swapper/src/types.ts (~846/897)\n- Registry = `swappers: Record<SwapperName,(SwapperApi & Swapper)>` — packages/swapper/src/constants.ts:52 (merges e.g. thorchainSwapper + thorchainApi)\n- Aggregator: swapperApi RTK endpoint getTradeQuote/getTradeRates (src/state/apis/swapper/swapperApi.ts:78/156) → getTradeQuotes (packages/swapper/src/swapper.ts:18/26): swapper = swappers[swapperName]; swapper.getTradeQuote(...) [DYNAMIC interface dispatch]\n- Concrete (THORChain): thorchainApi.getTradeQuote (swappers/ThorchainSwapper/endpoints.ts → getTradeQuote/getTradeQuote.ts:15) → getL1RateOrQuote → getQuote (thorService HTTP)\nSELECTION: tradeQuoteSlice selectors rank quotes (selectSortedTradeQuotes / selectActiveSwapperName) [DYNAMIC: winner chosen by ranking/user]\nEXECUTION layer:\n- useTradeExecution (src/components/MultiHopTrade/.../hooks/useTradeExecution.tsx:200/476) → new TradeExecution(); execution.execEvmTransaction(...) (CowSwap: execEvmMessage)\n- TradeExecution.execEvmTransaction — src/lib/tradeExecution.ts:326 → _execWalletAgnostic(...) (372)\n- TradeExecution._execWalletAgnostic — tradeExecution.ts:136 → swapper = swappers[swapperName] (149) [DYNAMIC registry]; buildSignBroadcast → swapper.getUnsignedEvmTransaction(...) (355) then swapper.executeEvmTransaction(unsignedTx, {signAndBroadcastTransaction}) (367) [DYNAMIC SwapperApi/Swapper interface]\n- Concrete (THORChain): thorchainSwapper.executeEvmTransaction (swappers/ThorchainSwapper/ThorchainSwapper.ts → utils.ts:181) delegates to callbacks.signAndBroadcastTransaction (wallet). CowSwap alt: cowSwapper.executeEvmMessage → signCowOrder + cowService.post.\nKEY SYMBOLS: Swapper/SwapperApi types, swappers registry (constants.ts), getTradeQuotes (swapper.ts), TradeExecution._execWalletAgnostic, swapper.getUnsignedEvmTransaction/executeEvmTransaction, one concrete swapper (thorchainSwapper/zrxSwapper/cowSwapper).\nDYNAMIC BOUNDARIES: swappers[name] registry lookup (2 sites); all hops into a concrete swapper are via the Swapper/SwapperApi interface, never a direct function. A correct answer must name the interface + registry and reach a concrete swapper through interface dispatch."
13
+ },
14
+ "trezor": {
15
+ "question": "How does sending a crypto transaction flow from the send form's review/sign action through to signing it via @trezor/connect (TrezorConnect.signTransaction)? Trace the call path.",
16
+ "truth": "Verified call path (trezor-suite monorepo; app in packages/suite, shared logic in suite-common/wallet-core, device API in packages/connect):\n1. ReviewButton.handleButtonReviewClick — packages/suite/src/views/wallet/send/TotalSent/ReviewButton.tsx:120 → signTransaction() (= useSendForm's sign)\n2. useSendForm.sign (exported as signTransaction) — packages/suite/src/hooks/wallet/useSendForm.ts:278 → dispatch(signAndPushSendFormTransactionThunk({formState, precomposedTransaction, selectedAccount})) [DYNAMIC: redux thunk]\n3. signAndPushSendFormTransactionThunk — packages/suite/src/actions/wallet/send/sendFormThunks.ts:237 → (first enhancePrecomposedTransactionThunk) then dispatch(signTransactionThunk({...})) [cross-package: thunk from suite-common/wallet-core]\n4. signTransactionThunk (coin-routing hub) — suite-common/wallet-core/src/send/sendFormThunks.ts:532/573 → networkType branch → dispatch(signBitcoinSendFormTransactionThunk) (ethereum→signEthereumSendFormTransactionThunk, etc.) [DYNAMIC: runtime coin dispatch]\n5. signBitcoinSendFormTransactionThunk — suite-common/wallet-core/src/send/sendFormBitcoinThunks.ts:394 → await TrezorConnect.signTransaction(signPayload)\n6. TrezorConnect.signTransaction (facade) — packages/connect-common/src/factory.ts → closure calls impl.call({method:'signTransaction',...}) [DYNAMIC: facade / iframe-or-module boundary]\n7. CoreInModule.call — packages/connect/src/impl/core-in-module.ts:171 → posts CORE_CALL to core (deferred promise)\n8. Core.onCall → getMethod — packages/connect/src/core/index.ts → getMethod resolves 'signTransaction' → new SignTransaction(message) [DYNAMIC: name→class]\n9. SignTransaction.run — packages/connect/src/api/signTransaction.ts:317 → signTx via device.getCommands().typedCall (protobuf to device). After signing, signAndPushSendFormTransactionThunk → pushSendFormTransactionThunk → TrezorConnect.pushTransaction.\nKEY SYMBOLS: ReviewButton → useSendForm.sign → signAndPushSendFormTransactionThunk (suite) → signTransactionThunk (wallet-core, coin hub) → signBitcoinSendFormTransactionThunk → TrezorConnect.signTransaction → connect factory/CoreInModule.call → SignTransaction.run.\nDYNAMIC BOUNDARIES: every suite→wallet-core hop is a redux thunk dispatch; signTransactionThunk branches by networkType at runtime; TrezorConnect is a dynamically-built facade crossing an iframe/module boundary; the SignTransaction class is resolved by name. A correct answer must cross suite→wallet-core→connect and reach TrezorConnect.signTransaction / SignTransaction.run, not stop at the UI."
17
+ }
18
+ }