@mknrt/autotests-overkill 1.2.2 → 1.2.5

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -1,980 +0,0 @@
1
- # FIS Platform Backend Integration Implementation Plan
2
-
3
- > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
4
-
5
- **Goal:** Add focused `fis_platform` backend knowledge and an explainable API interaction map to `autotests-overkill` so MCP answers become materially better for `autotests2` test authoring, debugging, impact analysis, and CI/API triage.
6
-
7
- **Architecture:** Extend the existing `connectors -> extractors -> SQLite FTS -> domain -> MCP tools` pipeline with one explicit bridge object, `ApiDiscoveryCatalog`, plus two narrow knowledge layers: `api-interaction` and `backend-contract`. Build the catalog from `autotests2` support code first, pass it into `fis_platform` backend discovery explicitly, then store only evidence-backed documents with confidence metadata. Runtime API captures stay on-demand triage evidence in iteration 1; persistent runtime observation indexing is deferred until static interaction indexing proves useful.
8
-
9
- **Tech Stack:** TypeScript, Node.js, Zod, built-in node:sqlite/FTS5, Vitest, MCP SDK, Java source parsing via regex/structured text extraction.
10
-
11
- ---
12
-
13
- ## File Structure
14
-
15
- ### Existing files to modify
16
-
17
- - `src/config/consumerConfig.ts`
18
- Responsibility: runtime config schema and path resolution.
19
- - `src/appContext.ts`
20
- Responsibility: wiring connectors and database into the app context.
21
- - `src/connectors/autotests2Connector.ts`
22
- Responsibility: expose request-helper and API-support sources for indexing.
23
- - `src/indexer/refreshPipeline.ts`
24
- Responsibility: building knowledge documents and syncing them into SQLite.
25
- - `src/mcp/registerTools.ts`
26
- Responsibility: MCP tool registry and tool contracts.
27
- - `src/domain/generateSpecBlueprint.ts`
28
- Responsibility: reuse-first blueprint generation for new tests.
29
- - `src/domain/analyzeDiffImpact.ts`
30
- Responsibility: mapping changed files to risky test scopes.
31
- - `src/domain/triageFailedRun.ts`
32
- Responsibility: artifact-based failure triage and recommended next actions.
33
- - `README.md`
34
- Responsibility: top-level usage, supported repos, and tool catalog summary.
35
- - `docs/tool-catalog.md`
36
- Responsibility: operator-facing description of MCP tools.
37
- - `docs/operator-cookbook.md`
38
- Responsibility: practical usage scenarios and indexing workflow.
39
-
40
- ### New files to create
41
-
42
- - `src/connectors/fisPlatformConnector.ts`
43
- Responsibility: focused backend source discovery in `fis_platform` using an explicit `ApiDiscoveryCatalog`.
44
- - `src/indexer/extractors/apiDiscoveryExtractor.ts`
45
- Responsibility: build and score `ApiDiscoveryCatalog` from `autotests2` support code.
46
- - `src/indexer/extractors/backendContractExtractor.ts`
47
- Responsibility: extract endpoint, params, async task semantics, feature flags, and error hints from selected Java sources.
48
- - `src/indexer/extractors/apiInteractionExtractor.ts`
49
- Responsibility: convert selected `ApiDiscoveryCatalog` families into normalized API interaction documents.
50
- - `src/domain/findBackendTestContext.ts`
51
- Responsibility: produce an evidence-backed answer that links `autotests2` request flows, captured API paths, and backend contracts.
52
- - `tests/connectors/fisPlatformConnector.test.ts`
53
- Responsibility: verify source selection stays narrow and relevant.
54
- - `tests/indexer/apiDiscoveryExtractor.test.ts`
55
- Responsibility: verify API family discovery, scoring, and explainability.
56
- - `tests/indexer/backendContractExtractor.test.ts`
57
- Responsibility: verify extracted backend knowledge shape.
58
- - `tests/indexer/apiInteractionExtractor.test.ts`
59
- Responsibility: verify API interaction records are normalized and linked to `autotests2` assets.
60
- - `tests/domain/findBackendTestContext.test.ts`
61
- Responsibility: verify tool output includes API interaction evidence plus backend evidence for `autotests2` scenarios.
62
- - `tests/domain/generateSpecBlueprint.backend.test.ts`
63
- Responsibility: verify backend hints appear only when relevant.
64
- - `tests/domain/analyzeDiffImpact.backend.test.ts`
65
- Responsibility: verify backend diffs map to `autotests2` tests.
66
- - `tests/domain/triageFailedRun.backend.test.ts`
67
- Responsibility: verify captured API paths enrich triage output.
68
-
69
- ### `ApiDiscoveryCatalog` contract
70
-
71
- `ApiDiscoveryCatalog` is the bridge between `autotests2`, runtime triage, and `fis_platform`. It must be built before backend discovery and saved to `.overkill-cache/snapshots/api-discovery-catalog.json`.
72
-
73
- ```ts
74
- export type ApiDiscoveryCatalog = {
75
- families: ApiDiscoveryFamily[];
76
- generatedAt: string;
77
- warnings: string[];
78
- };
79
-
80
- export type ApiDiscoveryFamily = {
81
- family: string;
82
- normalizedPaths: string[];
83
- consumerSignals: Array<{
84
- kind: 'endpoint-constant' | 'request-wrapper' | 'capture-support' | 'spec-reference';
85
- path: string;
86
- name?: string;
87
- snippet: string;
88
- }>;
89
- score: number;
90
- whySelected: string[];
91
- };
92
- ```
93
-
94
- Scoring for iteration 1 must be deterministic:
95
-
96
- ```ts
97
- score =
98
- endpointConstantHits * 4
99
- + requestWrapperHits * 3
100
- + apiCaptureSupportHits * 2
101
- + specReferenceHits;
102
- ```
103
-
104
- Only the top 2-3 families with at least two different signal kinds should be selected for backend matching. Families below that threshold may be stored in the snapshot as candidates, but they must not drive blueprint or impact answers.
105
-
106
- ### Backend discovery rules for iteration 1
107
-
108
- - Scan `fis_platform` for Java REST/resource sources using structural signals such as `@Path`, `@GET`, `@POST`, `@PUT`, `@DELETE`, `@QueryParam`, `@HeaderParam`, `@Consumes`, `@Produces`.
109
- - Boost files that are also referenced from REST/OpenAPI registration or documentation modules.
110
- - Exclude obvious noise by rule, not by handpicked file list: `thirdparty`, generated sources, frontend assets, vendor directories, and test fixtures.
111
- - Keep only endpoints whose normalized path or resource name overlaps with selected families from `ApiDiscoveryCatalog`.
112
- - Produce a `confidence` score for every backend match. Only `confidence >= 0.7` can be used in blueprint, impact, and triage recommendations; lower confidence records can be returned only as exploratory evidence.
113
-
114
- ### API discovery rules for iteration 1
115
-
116
- - Discover canonical API identities from `autotests2` support code, not from spec file names.
117
- - Scan `cypress/support/**/*.js`, then keep files with API signals such as `cy.request`, `Cypress.Commands.add`, URL/path constants, `fetch`, `XMLHttpRequest`, `apiCapture`, or `request`.
118
- - Extract consumer evidence such as endpoint constants, wrapper function names, request methods, and normalized request paths.
119
- - Treat specs as downstream evidence consumers, not as the source of truth for API aliases.
120
-
121
- ### API identities that must be discoverable in iteration 1
122
-
123
- - The system must discover API families from `autotests2` support code first, then optionally enrich matching during triage with runtime capture evidence.
124
- - Iteration 1 should implement only the top 2-3 highest-signal API families after discovery, not a predeclared domain list.
125
- - Discovery output must be saved as evidence in snapshots so the chosen families are explainable and reviewable.
126
- - The exact consumer links must be derived from indexed support code and, during triage, matched against runtime capture evidence. They must not be maintained as a hardcoded alias-to-file map.
127
- - Runtime API capture files are read on demand by `triage_failed_run` in iteration 1. They should not be persisted as knowledge documents until we know the static catalog is useful and low-noise.
128
-
129
- ## Task 1: Add `fis_platform` to runtime configuration and app context
130
-
131
- **Files:**
132
- - Modify: `src/config/consumerConfig.ts`
133
- - Modify: `src/appContext.ts`
134
- - Test: `tests/config/consumerConfig.test.ts`
135
- - Test: `tests/cli/runtimeAppContext.test.ts`
136
-
137
- - [ ] **Step 1: Write the failing config test**
138
-
139
- ```ts
140
- it('parses fisPlatform repo path when provided', () => {
141
- const config = parseConsumerConfig({
142
- repos: {
143
- autotests2: '../autotests2',
144
- caseplatformWeb: '../caseplatform-web',
145
- fisPlatform: '../fis_platform',
146
- },
147
- ci: {
148
- artifactsRoot: '../autotests2/cypress',
149
- },
150
- }, 'C:/workspace');
151
-
152
- expect(config.repos.fisPlatform).toBe('C:/fis_platform'.replaceAll('/', path.sep));
153
- });
154
- ```
155
-
156
- - [ ] **Step 2: Run the config test to verify it fails**
157
-
158
- Run: `npm test -- tests/config/consumerConfig.test.ts`
159
- Expected: FAIL with a Zod/schema mismatch because `fisPlatform` is unknown.
160
-
161
- - [ ] **Step 3: Extend the config schema and defaults**
162
-
163
- ```ts
164
- repos: z.object({
165
- autotests2: z.string().min(1),
166
- caseplatformWeb: z.string().min(1),
167
- fisPlatform: z.string().min(1).optional(),
168
- }).strict(),
169
- ```
170
-
171
- ```ts
172
- repos: {
173
- autotests2: resolveRepoPath(parsed.repos.autotests2, baseDir),
174
- caseplatformWeb: resolveRepoPath(parsed.repos.caseplatformWeb, baseDir),
175
- fisPlatform: parsed.repos.fisPlatform
176
- ? resolveRepoPath(parsed.repos.fisPlatform, baseDir)
177
- : undefined,
178
- },
179
- ```
180
-
181
- ```ts
182
- repos: {
183
- autotests2: '../autotests2',
184
- caseplatformWeb: '../caseplatform-web',
185
- fisPlatform: '../fis_platform',
186
- },
187
- ```
188
-
189
- - [ ] **Step 4: Wire the connector into app context**
190
-
191
- ```ts
192
- import { FisPlatformConnector } from './connectors/fisPlatformConnector.js';
193
- ```
194
-
195
- ```ts
196
- export type AppContext = {
197
- config: ConsumerConfig;
198
- database: ReturnType<typeof createKnowledgeDatabase>;
199
- autotests2: Autotests2Connector;
200
- caseplatform: CaseplatformConnector;
201
- fisPlatform?: FisPlatformConnector;
202
- ciArtifacts: CiArtifactsConnector;
203
- gitlab: GitLabConnector;
204
- reportPortal: ReportPortalConnector;
205
- };
206
- ```
207
-
208
- ```ts
209
- fisPlatform: config.repos.fisPlatform
210
- ? new FisPlatformConnector(config.repos.fisPlatform)
211
- : undefined,
212
- ```
213
-
214
- - [ ] **Step 5: Run the focused tests to verify they pass**
215
-
216
- Run: `npm test -- tests/config/consumerConfig.test.ts tests/cli/runtimeAppContext.test.ts`
217
- Expected: PASS
218
-
219
- - [ ] **Step 6: Commit**
220
-
221
- ```bash
222
- git add src/config/consumerConfig.ts src/appContext.ts tests/config/consumerConfig.test.ts tests/cli/runtimeAppContext.test.ts
223
- git commit -m "feat: add fis platform runtime config"
224
- ```
225
-
226
- ## Task 2: Build `ApiDiscoveryCatalog` from `autotests2`
227
-
228
- **Files:**
229
- - Modify: `src/connectors/autotests2Connector.ts`
230
- - Create: `src/indexer/extractors/apiDiscoveryExtractor.ts`
231
- - Test: `tests/indexer/apiDiscoveryExtractor.test.ts`
232
-
233
- - [ ] **Step 1: Write the failing API discovery test**
234
-
235
- ```ts
236
- it('builds an explainable api discovery catalog from consumer support code', async () => {
237
- const catalog = await buildApiDiscoveryCatalog({
238
- root: fixturePaths.autotests2Root,
239
- maxSelectedFamilies: 3,
240
- });
241
-
242
- expect(catalog.families.length).toBeGreaterThan(0);
243
- expect(catalog.families.every((family) => family.score > 0)).toBe(true);
244
- expect(catalog.families.every((family) => family.whySelected.length > 0)).toBe(true);
245
- expect(catalog.families.every((family) => family.consumerSignals.length >= 2)).toBe(true);
246
- });
247
- ```
248
-
249
- - [ ] **Step 2: Run the API discovery test to verify it fails**
250
-
251
- Run: `npm test -- tests/indexer/apiDiscoveryExtractor.test.ts`
252
- Expected: FAIL with module not found for `buildApiDiscoveryCatalog`.
253
-
254
- - [ ] **Step 3: Add API support source discovery to `Autotests2Connector`**
255
-
256
- ```ts
257
- async findApiSupportSources() {
258
- const files = await this.fs.glob(this.root, ['cypress/support/**/*.js']);
259
-
260
- return files
261
- .map((filePath) => ({ path: filePath, body: this.fs.read(filePath) }))
262
- .filter((source) => /cy\.request|Cypress\.Commands\.add|fetch\(|XMLHttpRequest|apiCapture|request|\/platform\/|\/rs2\//i.test(source.body));
263
- }
264
- ```
265
-
266
- - [ ] **Step 4: Implement `ApiDiscoveryCatalog` and deterministic scoring**
267
-
268
- ```ts
269
- export type ApiDiscoveryCatalog = {
270
- families: ApiDiscoveryFamily[];
271
- generatedAt: string;
272
- warnings: string[];
273
- };
274
-
275
- export type ApiDiscoveryFamily = {
276
- family: string;
277
- normalizedPaths: string[];
278
- consumerSignals: Array<{
279
- kind: 'endpoint-constant' | 'request-wrapper' | 'capture-support' | 'spec-reference';
280
- path: string;
281
- name?: string;
282
- snippet: string;
283
- }>;
284
- score: number;
285
- whySelected: string[];
286
- };
287
- ```
288
-
289
- ```ts
290
- const scoreFamily = (signals: ApiDiscoveryFamily['consumerSignals']) => {
291
- const count = (kind: ApiDiscoveryFamily['consumerSignals'][number]['kind']) =>
292
- signals.filter((signal) => signal.kind === kind).length;
293
-
294
- return count('endpoint-constant') * 4
295
- + count('request-wrapper') * 3
296
- + count('capture-support') * 2
297
- + count('spec-reference');
298
- };
299
- ```
300
-
301
- ```ts
302
- const hasEnoughEvidence = (family: ApiDiscoveryFamily) =>
303
- new Set(family.consumerSignals.map((signal) => signal.kind)).size >= 2;
304
- ```
305
-
306
- - [ ] **Step 5: Run the API discovery test**
307
-
308
- Run: `npm test -- tests/indexer/apiDiscoveryExtractor.test.ts`
309
- Expected: PASS
310
-
311
- - [ ] **Step 6: Commit**
312
-
313
- ```bash
314
- git add src/connectors/autotests2Connector.ts src/indexer/extractors/apiDiscoveryExtractor.ts tests/indexer/apiDiscoveryExtractor.test.ts
315
- git commit -m "feat: discover autotests2 api families"
316
- ```
317
-
318
- ## Task 3: Discover matching `fis_platform` backend sources from `ApiDiscoveryCatalog`
319
-
320
- **Files:**
321
- - Create: `src/connectors/fisPlatformConnector.ts`
322
- - Create: `src/indexer/extractors/backendContractExtractor.ts`
323
- - Test: `tests/connectors/fisPlatformConnector.test.ts`
324
- - Test: `tests/indexer/backendContractExtractor.test.ts`
325
-
326
- - [ ] **Step 1: Write the failing backend discovery test**
327
-
328
- ```ts
329
- it('selects backend rest sources using an explicit api discovery catalog', async () => {
330
- const connector = new FisPlatformConnector(fixturePaths.fisPlatformRoot);
331
- const sources = await connector.listFocusedSources({
332
- families: [
333
- {
334
- family: 'sample',
335
- normalizedPaths: ['/platform/rs2/sample/action'],
336
- consumerSignals: [
337
- { kind: 'endpoint-constant', path: 'fixture/apiEndpoints.js', snippet: '/platform/rs2/sample/action' },
338
- { kind: 'request-wrapper', path: 'fixture/api.commands.js', name: 'sampleAction', snippet: 'cy.request' },
339
- ],
340
- score: 7,
341
- whySelected: ['fixture family with two signal kinds'],
342
- },
343
- ],
344
- generatedAt: '2026-04-20T00:00:00.000Z',
345
- warnings: [],
346
- });
347
-
348
- expect(sources.some((item) => item.path.includes('thirdparty'))).toBe(false);
349
- expect(sources.every((item) => item.confidence >= 0 && item.confidence <= 1)).toBe(true);
350
- });
351
- ```
352
-
353
- - [ ] **Step 2: Run the backend discovery test to verify it fails**
354
-
355
- Run: `npm test -- tests/connectors/fisPlatformConnector.test.ts`
356
- Expected: FAIL with module not found for `FisPlatformConnector`.
357
-
358
- - [ ] **Step 3: Implement backend discovery without consumer repo access**
359
-
360
- ```ts
361
- export type FocusedFisSource = {
362
- path: string;
363
- body: string;
364
- sourceKind: 'backend-resource' | 'backend-interface' | 'backend-openapi-config';
365
- matchedFamily: string;
366
- confidence: number;
367
- whyMatched: string[];
368
- };
369
- ```
370
-
371
- ```ts
372
- async listFocusedSources(apiCatalog: ApiDiscoveryCatalog): Promise<FocusedFisSource[]> {
373
- const files = await this.fs.glob(this.root, ['lib/**/*.java', 'lib/**/pom.xml']);
374
-
375
- return files
376
- .filter((filePath) => !/(thirdparty|extjs|oryx|generated|node_modules)/i.test(filePath))
377
- .map((filePath) => ({ path: filePath, body: this.fs.read(filePath) }))
378
- .filter((source) => /@Path|@(GET|POST|PUT|DELETE)|@QueryParam|@HeaderParam/.test(source.body))
379
- .flatMap((source) => matchBackendSourceToFamilies(source, apiCatalog.families))
380
- .filter((source) => source.confidence >= 0.4);
381
- }
382
- ```
383
-
384
- - [ ] **Step 4: Write the failing backend contract extractor test**
385
-
386
- ```ts
387
- it('extracts backend endpoint records with confidence and evidence paths', async () => {
388
- const extraction = await extractBackendContracts({
389
- sources: fixtureFocusedFisSources,
390
- });
391
-
392
- const endpointRecord = extraction.documents[0];
393
-
394
- expect(endpointRecord.metadata.endpointPath).toBeTruthy();
395
- expect(endpointRecord.metadata.evidencePaths.length).toBeGreaterThan(0);
396
- expect(endpointRecord.metadata.confidence).toBeGreaterThanOrEqual(0.4);
397
- });
398
- ```
399
-
400
- - [ ] **Step 5: Implement backend contract records**
401
-
402
- ```ts
403
- export type BackendKnowledgeRecord = {
404
- id: string;
405
- title: string;
406
- path: string;
407
- body: string;
408
- sourceKind: 'backend-endpoint';
409
- repoKind: 'fis-platform';
410
- metadata: {
411
- httpMethod?: 'GET' | 'POST' | 'PUT' | 'DELETE';
412
- endpointPath: string;
413
- apiFamily: string;
414
- paramNames: string[];
415
- ruleHints: string[];
416
- evidencePaths: string[];
417
- confidence: number;
418
- whyMatched: string[];
419
- };
420
- };
421
- ```
422
-
423
- ```ts
424
- const endpointMatches = [
425
- ...body.matchAll(/@(GET|POST|PUT|DELETE)|@Path\("([^"]+)"\)|@QueryParam\("([^"]+)"\)|@HeaderParam\("([^"]+)"\)/g),
426
- ];
427
- ```
428
-
429
- - [ ] **Step 6: Run backend discovery and extractor tests**
430
-
431
- Run: `npm test -- tests/connectors/fisPlatformConnector.test.ts tests/indexer/backendContractExtractor.test.ts`
432
- Expected: PASS
433
-
434
- - [ ] **Step 7: Commit**
435
-
436
- ```bash
437
- git add src/connectors/fisPlatformConnector.ts src/indexer/extractors/backendContractExtractor.ts tests/connectors/fisPlatformConnector.test.ts tests/indexer/backendContractExtractor.test.ts
438
- git commit -m "feat: discover backend contracts from api catalog"
439
- ```
440
-
441
- ## Task 4: Index API interactions and backend contracts
442
-
443
- **Files:**
444
- - Create: `src/indexer/extractors/apiInteractionExtractor.ts`
445
- - Modify: `src/indexer/refreshPipeline.ts`
446
- - Test: `tests/indexer/apiInteractionExtractor.test.ts`
447
- - Test: `tests/indexer/refreshPipeline.caseplatformIndex.test.ts`
448
-
449
- - [ ] **Step 1: Write the failing API interaction test**
450
-
451
- ```ts
452
- it('converts selected api discovery families into api interaction documents', async () => {
453
- const extraction = extractApiInteractions({
454
- catalog: fixtureApiDiscoveryCatalog,
455
- });
456
-
457
- const record = extraction.documents[0];
458
-
459
- expect(record.sourceKind).toBe('api-interaction');
460
- expect(record.metadata.apiFamily).toBeTruthy();
461
- expect(record.metadata.consumerSignalIds.length).toBeGreaterThan(0);
462
- expect(record.metadata.confidence).toBeGreaterThanOrEqual(0.7);
463
- });
464
- ```
465
-
466
- - [ ] **Step 2: Run the API interaction test to verify it fails**
467
-
468
- Run: `npm test -- tests/indexer/apiInteractionExtractor.test.ts`
469
- Expected: FAIL with module not found for `extractApiInteractions`.
470
-
471
- - [ ] **Step 3: Implement API interaction records**
472
-
473
- ```ts
474
- export type ApiInteractionRecord = {
475
- id: string;
476
- title: string;
477
- path: string;
478
- body: string;
479
- sourceKind: 'api-interaction';
480
- repoKind: 'autotests2';
481
- metadata: {
482
- apiFamily: string;
483
- normalizedPaths: string[];
484
- consumerSignalIds: string[];
485
- evidencePaths: string[];
486
- confidence: number;
487
- whySelected: string[];
488
- };
489
- };
490
- ```
491
-
492
- - [ ] **Step 4: Integrate catalog, interactions, and backend contracts into refresh pipeline**
493
-
494
- ```ts
495
- const apiCatalog = await buildApiDiscoveryCatalog({
496
- root: context.config.repos.autotests2,
497
- maxSelectedFamilies: 3,
498
- });
499
- snapshotStore.write('api-discovery-catalog', apiCatalog);
500
-
501
- const apiInteractions = extractApiInteractions({ catalog: apiCatalog });
502
- snapshotStore.write('api-interactions', apiInteractions);
503
-
504
- for (const record of apiInteractions.documents) {
505
- documents.push(documentFrom({
506
- sourceKind: record.sourceKind,
507
- repoKind: record.repoKind,
508
- path: record.path,
509
- title: record.title,
510
- body: record.body,
511
- metadata: record.metadata,
512
- updatedAt: now,
513
- }));
514
- }
515
- ```
516
-
517
- ```ts
518
- if (context.fisPlatform && context.config.repos.fisPlatform && fs.existsSync(context.config.repos.fisPlatform)) {
519
- repoKindsToSync.push('fis-platform');
520
- const backendSources = await context.fisPlatform.listFocusedSources(apiCatalog);
521
- const backendContracts = await extractBackendContracts({ sources: backendSources });
522
- snapshotStore.write('backend-contracts', backendContracts);
523
-
524
- for (const record of backendContracts.documents) {
525
- documents.push(documentFrom({
526
- sourceKind: record.sourceKind,
527
- repoKind: record.repoKind,
528
- path: record.path,
529
- title: record.title,
530
- body: record.body,
531
- metadata: record.metadata,
532
- updatedAt: now,
533
- }));
534
- }
535
- }
536
- ```
537
-
538
- - [ ] **Step 5: Run indexing tests**
539
-
540
- Run: `npm test -- tests/indexer/apiInteractionExtractor.test.ts tests/indexer/refreshPipeline.caseplatformIndex.test.ts`
541
- Expected: PASS
542
-
543
- - [ ] **Step 6: Commit**
544
-
545
- ```bash
546
- git add src/indexer/extractors/apiInteractionExtractor.ts src/indexer/refreshPipeline.ts tests/indexer/apiInteractionExtractor.test.ts
547
- git commit -m "feat: index api interactions and backend contracts"
548
- ```
549
-
550
- ## Task 5: Add `find_backend_test_context` MCP tool
551
-
552
- **Files:**
553
- - Create: `src/domain/findBackendTestContext.ts`
554
- - Modify: `src/mcp/registerTools.ts`
555
- - Test: `tests/domain/findBackendTestContext.test.ts`
556
- - Test: `tests/mcp/wave1Tools.test.ts`
557
-
558
- - [ ] **Step 1: Write the failing domain test**
559
-
560
- ```ts
561
- it('returns api interaction evidence, backend evidence, and linked autotests2 specs for a discovered api flow', async () => {
562
- const result = await findBackendTestContext({ query: 'discovered api flow' }, context);
563
-
564
- expect(result.summary).toContain('API interaction');
565
- expect(result.evidence.some((item) => item.label === 'api interaction')).toBe(true);
566
- expect(result.evidence.some((item) => item.path?.includes('fis_platform'))).toBe(true);
567
- expect(result.evidence.some((item) => item.path?.includes('autotests2'))).toBe(true);
568
- expect(result.recommended_actions.some((item) => item.includes('required params'))).toBe(true);
569
- });
570
- ```
571
-
572
- - [ ] **Step 2: Run the domain test to verify it fails**
573
-
574
- Run: `npm test -- tests/domain/findBackendTestContext.test.ts`
575
- Expected: FAIL with module not found for `findBackendTestContext`.
576
-
577
- - [ ] **Step 3: Implement the domain service**
578
-
579
- ```ts
580
- const interactionHits = queryDocuments(context.database, input.query, {
581
- repoKind: 'autotests2',
582
- sourceKinds: ['api-interaction'],
583
- limit: 6,
584
- }).filter((item) => {
585
- const metadata = JSON.parse(item.metadata_json) as { confidence?: number };
586
- return (metadata.confidence ?? 0) >= 0.7;
587
- });
588
- ```
589
-
590
- ```ts
591
- const backendHits = queryDocuments(context.database, input.query, {
592
- repoKind: 'fis-platform',
593
- sourceKinds: ['backend-endpoint'],
594
- limit: 8,
595
- }).filter((item) => {
596
- const metadata = JSON.parse(item.metadata_json) as { confidence?: number };
597
- return (metadata.confidence ?? 0) >= 0.7;
598
- });
599
- ```
600
-
601
- ```ts
602
- const linkedSpecs = queryDocuments(context.database, [ ...interactionHits, ...backendHits ].flatMap((item) => {
603
- const metadata = JSON.parse(item.metadata_json) as { consumerSignalIds?: string[]; evidencePaths?: string[] };
604
- return [ ...(metadata.consumerSignalIds ?? []), ...(metadata.evidencePaths ?? []) ];
605
- }).join(' '), {
606
- repoKind: 'autotests2',
607
- sourceKinds: ['spec', 'command', 'helper'],
608
- limit: 8,
609
- });
610
- ```
611
-
612
- ```ts
613
- return output(
614
- `Found ${interactionHits.length} API interaction record(s), ${backendHits.length} backend context record(s), and ${linkedSpecs.length} autotests2 asset(s) for '${input.query}'.`,
615
- [
616
- ...interactionHits.map((item) => evidenceFile('api interaction', item.path, truncate(item.body, 220), { retrieval: 'knowledge-store' })),
617
- ...backendHits.map((item) => evidenceFile('backend endpoint', item.path, truncate(item.body, 220), { retrieval: 'knowledge-store' })),
618
- ...linkedSpecs.map((item) => evidenceFile('linked autotest asset', item.path, truncate(item.body, 180), { retrieval: 'knowledge-store' })),
619
- ],
620
- [
621
- 'Reuse the indexed API interaction flow before reconstructing request order by hand.',
622
- 'Check required params and backend feature flags before debugging selectors.',
623
- 'Reuse the linked autotests2 request flow instead of rebuilding API assumptions from scratch.',
624
- ],
625
- );
626
- ```
627
-
628
- - [ ] **Step 4: Register the MCP tool**
629
-
630
- ```ts
631
- {
632
- name: 'find_backend_test_context',
633
- description: 'Finds autotests2 API interaction flows plus backend endpoint, validation, async-task, and error hints.',
634
- inputSchema: z.object({ query: z.string().min(1) }),
635
- outputSchema: toolOutputSchema,
636
- execute: findBackendTestContext,
637
- },
638
- ```
639
-
640
- - [ ] **Step 5: Run the domain and MCP tests**
641
-
642
- Run: `npm test -- tests/domain/findBackendTestContext.test.ts tests/mcp/wave1Tools.test.ts`
643
- Expected: PASS
644
-
645
- - [ ] **Step 6: Commit**
646
-
647
- ```bash
648
- git add src/domain/findBackendTestContext.ts src/mcp/registerTools.ts tests/domain/findBackendTestContext.test.ts tests/mcp/wave1Tools.test.ts
649
- git commit -m "feat: add backend test context mcp tool"
650
- ```
651
-
652
- ## Task 6: Upgrade `generate_spec_blueprint` with API and backend hints for API-heavy flows
653
-
654
- **Files:**
655
- - Modify: `src/domain/generateSpecBlueprint.ts`
656
- - Test: `tests/domain/generateSpecBlueprint.backend.test.ts`
657
-
658
- - [ ] **Step 1: Write the failing blueprint test**
659
-
660
- ```ts
661
- it('adds backend hints for api-heavy scenarios', async () => {
662
- const result = await generateSpecBlueprint({ feature: 'api-heavy scenario from discovered consumer flow', area: 'constructor' }, context);
663
-
664
- expect(result.summary).toContain('spec draft');
665
- expect(result.evidence.some((item) => item.label === 'backend hint')).toBe(true);
666
- expect(result.recommended_actions.some((item) => item.includes('completion'))).toBe(true);
667
- });
668
- ```
669
-
670
- - [ ] **Step 2: Run the blueprint test to verify it fails**
671
-
672
- Run: `npm test -- tests/domain/generateSpecBlueprint.backend.test.ts`
673
- Expected: FAIL because backend hints are absent.
674
-
675
- - [ ] **Step 3: Add optional backend retrieval inside blueprint generation**
676
-
677
- ```ts
678
- const backendContext = queryDocuments(context.database, input.feature, {
679
- repoKind: 'fis-platform',
680
- sourceKinds: ['backend-endpoint'],
681
- limit: 3,
682
- }).filter((item) => {
683
- const metadata = JSON.parse(item.metadata_json) as { confidence?: number };
684
- return (metadata.confidence ?? 0) >= 0.7;
685
- });
686
- ```
687
-
688
- ```ts
689
- const apiContext = queryDocuments(context.database, input.feature, {
690
- repoKind: 'autotests2',
691
- sourceKinds: ['api-interaction'],
692
- limit: 3,
693
- }).filter((item) => {
694
- const metadata = JSON.parse(item.metadata_json) as { confidence?: number };
695
- return (metadata.confidence ?? 0) >= 0.7;
696
- });
697
- ```
698
-
699
- ```ts
700
- const backendGuidance = backendContext.length > 0 || apiContext.length > 0
701
- ? [
702
- `API flow: ${apiContext[0]?.title ?? backendContext[0]?.title}`,
703
- 'Preserve request order, required query params, and async task polling from the indexed flow.',
704
- 'If the scenario depends on async backend processing, verify completion signals before UI assertions.',
705
- ]
706
- : [];
707
- ```
708
-
709
- ```ts
710
- {
711
- type: 'symbol',
712
- label: 'backend hint',
713
- snippet: backendGuidance.join('\n'),
714
- }
715
- ```
716
-
717
- - [ ] **Step 4: Run the blueprint tests**
718
-
719
- Run: `npm test -- tests/domain/generateSpecBlueprint.integration.test.ts tests/domain/generateSpecBlueprint.backend.test.ts`
720
- Expected: PASS
721
-
722
- - [ ] **Step 5: Commit**
723
-
724
- ```bash
725
- git add src/domain/generateSpecBlueprint.ts tests/domain/generateSpecBlueprint.backend.test.ts
726
- git commit -m "feat: enrich spec blueprint with backend hints"
727
- ```
728
-
729
- ## Task 7: Upgrade diff impact analysis for backend changes
730
-
731
- **Files:**
732
- - Modify: `src/domain/analyzeDiffImpact.ts`
733
- - Test: `tests/domain/analyzeDiffImpact.backend.test.ts`
734
-
735
- - [ ] **Step 1: Write the failing backend impact test**
736
-
737
- ```ts
738
- it('maps discovered backend api changes to relevant autotests2 tests', async () => {
739
- const result = await analyzeDiffImpact({
740
- changedFiles: [
741
- `${fixturePaths.fisPlatformRoot}/lib/some-module/src/main/java/example/TransferResource.java`,
742
- ],
743
- }, context);
744
-
745
- expect(result.summary).toContain('affected spec');
746
- expect(result.evidence.some((item) => item.path?.includes('autotests2'))).toBe(true);
747
- });
748
- ```
749
-
750
- - [ ] **Step 2: Run the impact test to verify it fails**
751
-
752
- Run: `npm test -- tests/domain/analyzeDiffImpact.backend.test.ts`
753
- Expected: FAIL because backend repo paths are ignored.
754
-
755
- - [ ] **Step 3: Extend impact analysis to use API interaction records as the bridge**
756
-
757
- ```ts
758
- const backendFeatureText = deriveFeatureTextFromBackendSource({
759
- filePath,
760
- fileBody: maybeReadChangedFile(filePath),
761
- });
762
- ```
763
-
764
- ```ts
765
- const interactionHits = queryDocuments(context.database, backendFeatureText, {
766
- repoKind: 'autotests2',
767
- sourceKinds: ['api-interaction'],
768
- limit: 8,
769
- }).filter((item) => {
770
- const metadata = JSON.parse(item.metadata_json) as { confidence?: number };
771
- return (metadata.confidence ?? 0) >= 0.7;
772
- });
773
- ```
774
-
775
- ```ts
776
- const linkedSpecs = queryDocuments(context.database, interactionHits.flatMap((item) => {
777
- const metadata = JSON.parse(item.metadata_json) as { consumerSignalIds?: string[]; evidencePaths?: string[] };
778
- return [ ...(metadata.consumerSignalIds ?? []), ...(metadata.evidencePaths ?? []) ];
779
- }).join(' '), {
780
- repoKind: 'autotests2',
781
- sourceKinds: ['spec', 'command', 'helper'],
782
- limit: 8,
783
- });
784
- ```
785
-
786
- - [ ] **Step 4: Run the impact tests**
787
-
788
- Run: `npm test -- tests/domain/analyzeDiffImpact.test.ts tests/domain/analyzeDiffImpact.backend.test.ts`
789
- Expected: PASS
790
-
791
- - [ ] **Step 5: Commit**
792
-
793
- ```bash
794
- git add src/domain/analyzeDiffImpact.ts tests/domain/analyzeDiffImpact.backend.test.ts
795
- git commit -m "feat: map backend diffs to autotests2 impact"
796
- ```
797
-
798
- ## Task 8: Upgrade failure triage with API-interaction-aware backend interpretation
799
-
800
- **Files:**
801
- - Modify: `src/domain/triageFailedRun.ts`
802
- - Test: `tests/domain/triageFailedRun.backend.test.ts`
803
-
804
- - [ ] **Step 1: Write the failing triage test**
805
-
806
- ```ts
807
- it('adds backend hints when captured api traffic matches indexed interaction records', async () => {
808
- const result = await triageFailedRun({
809
- statsFile: fixturePaths.statsFile,
810
- screenshotsRoot: fixturePaths.screenshotsRoot,
811
- videosRoot: fixturePaths.videosRoot,
812
- apiCaptureRoot: fixturePaths.apiCaptureRoot,
813
- }, context);
814
-
815
- expect(result.recommended_actions.some((item) => item.includes('backend contract'))).toBe(true);
816
- });
817
- ```
818
-
819
- - [ ] **Step 2: Run the triage test to verify it fails**
820
-
821
- Run: `npm test -- tests/domain/triageFailedRun.backend.test.ts`
822
- Expected: FAIL because triage ignores backend knowledge.
823
-
824
- - [ ] **Step 3: Add API-interaction-aware endpoint enrichment**
825
-
826
- ```ts
827
- const backendSignals = apiCaptures.flatMap((filePath) => {
828
- const raw = JSON.parse(fs.readFileSync(filePath, 'utf8')) as Array<{ path?: string }>;
829
- return raw.map((entry) => entry.path ?? '').filter(Boolean);
830
- });
831
- ```
832
-
833
- ```ts
834
- const interactionHits = queryDocuments(context.database, backendSignals.join(' '), {
835
- repoKind: 'autotests2',
836
- sourceKinds: ['api-interaction'],
837
- limit: 6,
838
- }).filter((item) => {
839
- const metadata = JSON.parse(item.metadata_json) as { normalizedPaths?: string[]; confidence?: number };
840
- return (metadata.confidence ?? 0) >= 0.7
841
- && (metadata.normalizedPaths ?? []).some((path) => backendSignals.some((signal) => signal.includes(path) || path.includes(signal)));
842
- });
843
- ```
844
-
845
- ```ts
846
- const backendHits = queryDocuments(context.database, interactionHits.map((item) => item.title).join(' '), {
847
- repoKind: 'fis-platform',
848
- sourceKinds: ['backend-endpoint'],
849
- limit: 6,
850
- }).filter((item) => {
851
- const metadata = JSON.parse(item.metadata_json) as { confidence?: number };
852
- return (metadata.confidence ?? 0) >= 0.7;
853
- });
854
- ```
855
-
856
- ```ts
857
- if (interactionHits.length > 0 || backendHits.length > 0) {
858
- actions.push('Compare the failing API capture with the indexed API interaction flow and backend contract before editing the test flow.');
859
- }
860
- ```
861
-
862
- - [ ] **Step 4: Run the triage tests**
863
-
864
- Run: `npm test -- tests/domain/triageFailedRun.test.ts tests/domain/triageFailedRun.backend.test.ts`
865
- Expected: PASS
866
-
867
- - [ ] **Step 5: Commit**
868
-
869
- ```bash
870
- git add src/domain/triageFailedRun.ts tests/domain/triageFailedRun.backend.test.ts
871
- git commit -m "feat: enrich triage with backend api context"
872
- ```
873
-
874
- ## Task 9: Update documentation and verification coverage
875
-
876
- **Files:**
877
- - Modify: `README.md`
878
- - Modify: `docs/tool-catalog.md`
879
- - Modify: `docs/operator-cookbook.md`
880
- - Modify: `overkill.config.example.json`
881
- - Test: `tests/docs/readmeExamples.test.ts`
882
-
883
- - [ ] **Step 1: Write the failing docs expectation test**
884
-
885
- ```ts
886
- it('documents fis_platform as an optional backend knowledge source', async () => {
887
- const readme = fs.readFileSync('README.md', 'utf8');
888
-
889
- expect(readme).toContain('fis_platform');
890
- expect(readme).toContain('find_backend_test_context');
891
- expect(readme).toContain('api-interaction');
892
- expect(readme).toContain('ApiDiscoveryCatalog');
893
- });
894
- ```
895
-
896
- - [ ] **Step 2: Run the docs test to verify it fails**
897
-
898
- Run: `npm test -- tests/docs/readmeExamples.test.ts`
899
- Expected: FAIL because README has no backend section.
900
-
901
- - [ ] **Step 3: Update config example and docs**
902
-
903
- ```json
904
- {
905
- "repos": {
906
- "autotests2": "../autotests2",
907
- "caseplatformWeb": "../caseplatform-web",
908
- "fisPlatform": "../fis_platform"
909
- }
910
- }
911
- ```
912
-
913
- ```md
914
- - `find_backend_test_context`: explain discovered API behavior with evidence from `fis_platform` and linked `autotests2` assets
915
- ```
916
-
917
- ```md
918
- - `api-interaction` indexing: normalize request helpers and endpoint aliases from `autotests2` so MCP can connect failing API captures to backend contracts
919
- ```
920
-
921
- ```md
922
- - `ApiDiscoveryCatalog`: explain which API families were selected, why they were selected, and which consumer evidence supports them
923
- ```
924
-
925
- ```md
926
- Use backend indexing when:
927
- - debugging request flows discovered from support code or API capture
928
- - generating blueprints for API-heavy specs
929
- - triaging failures with API captures
930
- ```
931
-
932
- - [ ] **Step 4: Run the docs and full verification suite**
933
-
934
- Run: `npm run verify`
935
- Expected: PASS
936
-
937
- Run: `npm run mcp:smoke`
938
- Expected: PASS
939
-
940
- - [ ] **Step 5: Commit**
941
-
942
- ```bash
943
- git add README.md docs/tool-catalog.md docs/operator-cookbook.md overkill.config.example.json tests/docs/readmeExamples.test.ts
944
- git commit -m "docs: add fis platform backend integration guidance"
945
- ```
946
-
947
- ## Sequencing Notes
948
-
949
- - Implement Tasks 1-4 first. That produces the minimal useful index slice:
950
- config + `ApiDiscoveryCatalog` + focused backend discovery + `api-interaction` and `backend-endpoint` documents.
951
- - Implement Task 5 only after Task 4 passes, because `find_backend_test_context` depends on indexed interaction and backend documents.
952
- - Implement Tasks 6-8 after the new backend tool is stable.
953
- - Keep complex backend resources narrow in iteration 1. Only infer path-level and validation-level hints; do not attempt full semantic parsing.
954
-
955
- ## Verification Checklist
956
-
957
- - Unit tests use fixtures under `tests/fixtures/backend-integration/**`; real `C:\gitrep\autotests2` and `C:\gitrep\fis_platform` paths are allowed only in optional integration smoke checks.
958
- - `find_backend_test_context` returns at least one `fis_platform` evidence item for:
959
- - a discovered API family from consumer support code
960
- - a discovered API family with a high-confidence backend match
961
- - `find_backend_test_context` also returns at least one `api-interaction` evidence item for:
962
- - a discovered API family from consumer support code
963
- - `triage_failed_run` uses runtime API capture evidence on demand without persisting raw captures into the knowledge store.
964
- - `generate_spec_blueprint` includes backend guidance for console/API-heavy scenarios and stays unchanged for pure frontend widget flows.
965
- - `analyze_diff_impact` maps discovered backend REST/resource changes to concrete `autotests2` specs through indexed API interactions.
966
- - `triage_failed_run` adds API-aware and backend-aware actions when captured API paths match indexed interaction records and backend endpoints.
967
- - No documents from `thirdparty`, `extjs`, or `oryx` are inserted into the knowledge store.
968
-
969
- ## Deferred Work
970
-
971
- - Persist aggregated runtime API-capture observations into the knowledge store only after static API interaction indexing proves useful and low-noise.
972
- - Parse generated OpenAPI JSON/YAML if `fis_platform/lib/doc-rest` starts emitting build artifacts reliably.
973
- - Add a dedicated relation table for exact backend-to-test links only if FTS metadata becomes insufficient.
974
- - Expand beyond the first discovered high-signal API families only after iteration 1 proves useful in daily `autotests2` workflows.
975
-
976
- ## Self-Review
977
-
978
- - Spec coverage: this plan covers config, connectors, extractor/indexer, API interaction mapping, backend knowledge reuse, MCP tooling, blueprint support, impact analysis, triage, docs, and tests.
979
- - Placeholder scan: no `TODO`, `TBD`, or “implement later” markers remain in executable tasks.
980
- - Type consistency: repo kind is consistently `fis-platform` for backend contracts and `autotests2` for API interaction records, source kinds are consistently `backend-endpoint` and `api-interaction`, and the new MCP entrypoint is consistently `find_backend_test_context`.