@skyramp/mcp 0.1.4 → 0.1.5
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/build/prompts/test-maintenance/driftAnalysisSections.js +2 -2
- package/build/prompts/test-recommendation/analysisOutputPrompt.js +2 -2
- package/build/prompts/test-recommendation/recommendationSections.js +10 -42
- package/build/prompts/test-recommendation/registerRecommendTestsPrompt.js +5 -2
- package/build/prompts/test-recommendation/test-recommendation-prompt.js +152 -67
- package/build/prompts/test-recommendation/test-recommendation-prompt.test.js +18 -111
- package/build/prompts/testbot/testbot-prompts.js +9 -17
- package/build/services/ScenarioGenerationService.js +1 -2
- package/build/tools/generate-tests/generateBatchScenarioRestTool.js +4 -3
- package/build/tools/generate-tests/generateBatchScenarioRestTool.test.js +0 -9
- package/build/tools/test-management/analyzeChangesTool.js +5 -10
- package/build/types/TestRecommendation.js +0 -2
- package/build/utils/featureFlags.js +0 -25
- package/build/utils/scenarioDrafting.js +505 -116
- package/build/utils/scenarioDrafting.test.js +480 -260
- package/package.json +1 -1
- package/build/utils/httpDefaults.js +0 -12
|
@@ -143,8 +143,8 @@ When a diff adds a new HTTP method to a resource, UPDATE covers **all** existing
|
|
|
143
143
|
|
|
144
144
|
### PATCH/PUT with child collections (MANDATORY)
|
|
145
145
|
When updating a contract or integration test for a PATCH or PUT endpoint whose request/response includes a child collection array (e.g. \`items\`, \`products\`, \`line_items\`):
|
|
146
|
-
1. The request body MUST include the child array with at least one item containing the
|
|
147
|
-
2. Assert each item's
|
|
146
|
+
1. The request body MUST include the child array with at least one item containing the FK field (e.g. \`product_id\`) and a \`quantity\` field.
|
|
147
|
+
2. Assert each item's FK field and \`quantity\` match the sent values.
|
|
148
148
|
3. Assert the top-level computed total (e.g. \`total_amount\`) equals the expected math from the items.
|
|
149
149
|
A test that only sends/asserts metadata (discount, status, notes) without asserting the items array is INCOMPLETE and will produce false passes even when the items/total logic is broken.
|
|
150
150
|
|
|
@@ -52,10 +52,10 @@ The ranked test recommendation catalog is pre-built and shown below (after the s
|
|
|
52
52
|
**Your only job is to present it.**
|
|
53
53
|
|
|
54
54
|
1. Fill in every \`<…from source>\` placeholder using the field names, computed formulas, and auth details you found in Steps 1–2.
|
|
55
|
-
2. Output the completed catalog **exactly as formatted
|
|
55
|
+
2. Output the completed catalog **exactly as formatted — grouped by test type (### E2E / ### UI / ### Integration / ### Contract)**. Do NOT restructure, reorder, rename sections, or generate a new format.
|
|
56
56
|
3. Do NOT call any Skyramp generation tools. The catalog shows ready-to-use tool calls that can be executed on demand.
|
|
57
57
|
|
|
58
|
-
**If** Steps 1–2 revealed additional scenarios the catalog does not cover (e.g. a computed formula or
|
|
58
|
+
**If** Steps 1–2 revealed additional scenarios the catalog does not cover (e.g. a computed formula or FK relationship that was missed), you may optionally call \`skyramp_recommend_tests\` with \`stateFile: "${p.stateFile ?? p.sessionId}"\` and \`enrichedScenarios\` to regenerate a more complete catalog — but only after presenting the current one.`;
|
|
59
59
|
const hasJavaFiles = p.candidateRouteFiles?.some(f => /\.(java|kt)$/.test(f)) ?? false;
|
|
60
60
|
const routeFilesSection = p.candidateRouteFiles && p.candidateRouteFiles.length > 0
|
|
61
61
|
? `\nRoute/controller files found by static scan (read these to discover endpoints — the regex-based catalog below may be incomplete for your framework):\n${p.candidateRouteFiles.map(f => `- ${f}`).join("\n")}\n`
|
|
@@ -1,4 +1,4 @@
|
|
|
1
|
-
import { isContractConsumerModeEnabled
|
|
1
|
+
import { isContractConsumerModeEnabled } from "../../utils/featureFlags.js";
|
|
2
2
|
import { WorkspaceAuthType, getAuthScheme, isAuthorizationHeaderName, AUTH_MIDDLEWARE_PATTERNS_STR } from "../../utils/workspaceAuth.js";
|
|
3
3
|
// Cached at module-load — the flag is process-wide and cannot change per call.
|
|
4
4
|
const CONSUMER_MODE_ENABLED = isContractConsumerModeEnabled();
|
|
@@ -42,45 +42,13 @@ Before calling any tool, replace every \`<from source>\` placeholder in the tool
|
|
|
42
42
|
}
|
|
43
43
|
export function buildReasoningProtocol() {
|
|
44
44
|
return `<reasoning_protocol>
|
|
45
|
-
## Coverage Reasoning Block (MANDATORY — complete BEFORE your Budget Plan)
|
|
46
|
-
|
|
47
|
-
Before committing to a Budget Plan and test list, produce a <thinking> block that enumerates ALL testable surfaces introduced or affected by this PR. This prevents narrow focus on a single endpoint/method.
|
|
48
|
-
|
|
49
|
-
**For backend-only PRs**, your thinking MUST cover:
|
|
50
|
-
1. **All HTTP methods affected** — if a new validation/service method is added, trace ALL callers (not just createOne — also updateOne, updateMany, deleteOne). List every HTTP method × endpoint pair.
|
|
51
|
-
2. **Error paths per method** — for each endpoint-method, what error codes does the source code return? (400, 401, 403, 404, 409, 422). Each distinct error path is a potential test.
|
|
52
|
-
3. **Cross-service impact** — does the change affect other services that import the modified module? Those endpoints need coverage too.
|
|
53
|
-
4. **Data migrations** — if a migration exists, can its effect be verified via an API call? (e.g. backfill → GET should return the backfilled value)
|
|
54
|
-
|
|
55
|
-
**For frontend-only PRs**, your thinking MUST cover:
|
|
56
|
-
1. **Component integration** — which routes render the changed component? Each route is a test target.
|
|
57
|
-
2. **User interactions** — what actions can a user perform on the changed component? (click, type, select, drag). Each distinct action flow is a test.
|
|
58
|
-
3. **State variations** — what different states does the component render? (empty, loading, error, populated, edge values)
|
|
59
|
-
|
|
60
|
-
**For mixed (frontend + backend) PRs**, your thinking MUST cover:
|
|
61
|
-
1. All backend surfaces (methods 1–4 above)
|
|
62
|
-
2. All frontend surfaces (methods 1–3 above)
|
|
63
|
-
3. **E2E bridges** — which frontend components call the changed backend endpoints? Those are E2E test candidates that cover both layers in one test.
|
|
64
|
-
|
|
65
|
-
**Output format in your thinking block:**
|
|
66
|
-
\`\`\`
|
|
67
|
-
Testable surfaces:
|
|
68
|
-
- POST /permissions → happy path (201), invalid fields (422), missing collection (400)
|
|
69
|
-
- PATCH /permissions/:id → update with valid fields (200), update with invalid fields (422)
|
|
70
|
-
- GET /items/:collection?aggregate → with allowed fields (200), with forbidden fields (403)
|
|
71
|
-
- UI: permissions field selector → add field, remove field, wildcard toggle
|
|
72
|
-
Total distinct surfaces: N
|
|
73
|
-
\`\`\`
|
|
74
|
-
|
|
75
|
-
Your Budget Plan total MUST be ≥ the number of GENERATE slots and reflect the breadth of surfaces found. If you found 8 distinct surfaces but only budget 3 tests, you are under-covering the PR.
|
|
76
|
-
|
|
77
45
|
## Parameter Grounding Rule
|
|
78
46
|
Before each GENERATE tool call, confirm WHERE each key value comes from:
|
|
79
47
|
|
|
80
48
|
- **requestBody / responseBody fields** → source code schema (Zod, Pydantic, DTO), enriched scenario, or OpenAPI spec. **The generation tool rejects empty \`{}\` request bodies for POST/PUT/PATCH** — read the source schema first if the fields are unknown.
|
|
81
49
|
- **endpointURL** → workspace \`baseUrl\` + endpoint path (both required — never path alone)
|
|
82
50
|
- **authHeader / authScheme** → workspace config or OpenAPI \`securitySchemes\`
|
|
83
|
-
- **
|
|
51
|
+
- **FK path params** → chained from a prior step's response (check the actual field name — it may be \`id\`, \`uuid\`, \`_id\`, or a resource-specific \`*_id\` field). The chaining source can be a response body (POST or GET), a response header (e.g. \`Location\`), or a cookie — not hardcoded
|
|
84
52
|
- **Names / string values** → realistic; append timestamp suffix to avoid re-run conflicts
|
|
85
53
|
|
|
86
54
|
## Ranking Rule
|
|
@@ -142,11 +110,11 @@ export function buildTestPatternGuidelines() {
|
|
|
142
110
|
- **Middleware chains**: If auth/rate-limit/logging middleware exists, test the chain (e.g., rate limit hit → auth still checked → correct error returned)
|
|
143
111
|
- **N+1 query risk**: If list endpoints join related data (e.g., orders with products), test with large datasets
|
|
144
112
|
- **State machines**: If resources have status transitions (draft→published→archived), test invalid transitions (e.g., archived→draft should fail)
|
|
145
|
-
- **Cascade deletes**: Only recommend after reading source code to confirm which resource holds the
|
|
113
|
+
- **Cascade deletes**: Only recommend after reading source code to confirm which resource holds the FK. The resource with the FK is the child; the one it points to is the parent. Example: if orders.product_id references products, then products is the parent — deleting a product tests whether orders are protected or cascade-deleted. Getting this backwards (treating the child as the parent) produces a nonsensical test.
|
|
146
114
|
- **Race conditions**: If concurrent writes are possible (inventory deduction, counter increment), test concurrent requests
|
|
147
115
|
- **Computed fields**: If response contains derived values (total, average, count), verify computation with known inputs (e.g., total_cost = compute_seconds * rate + memory_mb * rate + external_cost)
|
|
148
116
|
- **Mutation with collection modification**: If PUT/PATCH endpoints accept arrays of child items (e.g., order line items, cart products, invoice entries), test adding/removing items and verify that derived totals (e.g., total_amount, subtotal, item_count) are recalculated correctly. This is the most common source of user-reported bugs — always prioritize it for GENERATE over simple field-update tests.
|
|
149
|
-
The PATCH/PUT request body should include the child collection array field(s) defined for that endpoint (e.g., "items" with
|
|
117
|
+
The PATCH/PUT request body should include the child collection array field(s) defined for that endpoint (e.g., "items" with FK references like "product_id" and a quantity field) chained from prior POST responses. A PATCH that only sends metadata fields (e.g., discount_type, status, notes) without modifying the child collection is NOT a valid mutation-recalc test — it will pass even when the item/total logic is broken. Before writing assertions, inspect the source code or OpenAPI spec to identify (1) the actual child collection field name and its FK/quantity/price sub-fields, and (2) how derived totals are calculated (including any discounts, taxes, or fees). Then assert: the child FK fields match chained IDs, quantities match sent values, and totals match the computation from the source code
|
|
150
118
|
- **Webhook/event side effects**: If endpoints trigger async operations, test that side effects occur (e.g., POST /orders triggers notification)
|
|
151
119
|
- **Cross-user isolation**: If resources are owned by users, test that user B cannot access/modify user A's resources (GET /users/{other_id}/data → 403 Forbidden)
|
|
152
120
|
- **Range/boundary invariants**: If business rules cap values (max retries, min balance, discount ≤ subtotal), test the boundary (e.g., set retries to max+1 → expect rejection)
|
|
@@ -160,7 +128,7 @@ that step B depends on (e.g., create product → create order referencing that p
|
|
|
160
128
|
verify order contains correct product). Single-resource CRUD alone is not an integration test.
|
|
161
129
|
Use actual field names and values from the source code schema or OpenAPI schema (not \`{}\` or invented field names); verify response data, not just status codes.
|
|
162
130
|
When a PUT/PATCH updates a resource with child collections (e.g., order items), the request body
|
|
163
|
-
MUST include the child array with
|
|
131
|
+
MUST include the child array with FK references chained from prior steps — and assertions MUST
|
|
164
132
|
verify the actual child items in the response (product_id, quantity, unit_price), not just
|
|
165
133
|
top-level metadata like discount or status.
|
|
166
134
|
|
|
@@ -214,7 +182,7 @@ Before finalizing your output, verify:
|
|
|
214
182
|
6. **Real request shapes**: requestBody for POST/PUT/PATCH uses actual field names from source (not \`{}\`). GET search/filter uses \`queryParams\`, not \`requestBody\`.
|
|
215
183
|
7. **scenarioFile**: \`skyramp_integration_test_generation\` uses the exact \`filePath\` returned by \`skyramp_batch_scenario_test_generation\` — not a guessed or hardcoded filename.
|
|
216
184
|
8. **bugCatchingTarget**: Every GENERATE integration test that targets a business rule, formula, or constraint has a non-empty \`bugCatchingTarget\`.
|
|
217
|
-
9. **
|
|
185
|
+
9. **FK chaining**: In multi-step integration tests, path params sourced from a prior step's response (e.g. \`order_id\` from step 1) use \`chainsFrom\` — not hardcoded IDs.
|
|
218
186
|
10. **Concrete scenario names**: No GENERATE item uses a placeholder name ending in a numeric suffix (e.g. \`ui-test-for-changed-component-1\`, \`ui-test-from-trace-2\`). Derive the name from the actual changed component or flow: if the diff touches \`LinkCard.tsx\`, the scenario name should be \`link-card-pin-toggle\` or \`link-card-edit-description\`, not \`ui-test-for-changed-component-1\`. The changed file list is available above — use it.
|
|
219
187
|
</verification>`;
|
|
220
188
|
}
|
|
@@ -225,7 +193,7 @@ export function buildFewShotExamples() {
|
|
|
225
193
|
**Parameter grounding**:
|
|
226
194
|
- baseURL: "http://localhost:8000" (workspace api.baseUrl)
|
|
227
195
|
- steps[0].requestBody fields "name", "price": ProductCreate schema fields (src/models/product.py)
|
|
228
|
-
- steps[1].requestBody "product_id":
|
|
196
|
+
- steps[1].requestBody "product_id": FK to products — chained from step 0 response id
|
|
229
197
|
- steps[1].requestBody "quantity": OrderCreate schema field (src/models/order.py)
|
|
230
198
|
- responseBody "total_amount": 89.97 = 29.99 × 3 — from order total formula (src/services/order_service.py: total = sum(item.price * item.quantity))
|
|
231
199
|
- authHeader/authScheme: workspace config (Authorization / Bearer)
|
|
@@ -343,7 +311,7 @@ ${authGuidance}
|
|
|
343
311
|
**For multi-endpoint workflows (integration tests) — Batch Scenario → Integration pipeline:**
|
|
344
312
|
1. Call \`skyramp_batch_scenario_test_generation\` with ALL steps in a single call: \`scenarioName\`, \`destination\`,
|
|
345
313
|
\`baseURL\`, \`${authCallParams}\`, and a \`steps\` array where each element has \`method\`, \`path\`, \`requestBody\` OR \`queryParams\`, \`responseBody\`, \`statusCode\`.
|
|
346
|
-
\`statusCode\` is
|
|
314
|
+
\`statusCode\` is optional — defaults: POST→201, DELETE→204, GET/PUT/PATCH→200. Only override for non-standard codes.
|
|
347
315
|
**OpenAPI spec is NOT required.** \`apiSchema\` is OPTIONAL — omit it if no spec exists.
|
|
348
316
|
**CRITICAL — Query params vs request body:**
|
|
349
317
|
- For **POST/PUT/PATCH**: use \`requestBody\` with realistic field values from source code schemas.
|
|
@@ -383,12 +351,12 @@ ${CONSUMER_MODE_ENABLED ? `**Contract test mode selection — set based on this
|
|
|
383
351
|
Only provider-side contract tests are supported. Pass \`providerMode: true\` for new or modified endpoints this codebase owns.`}
|
|
384
352
|
|
|
385
353
|
**For UI tests:**
|
|
386
|
-
1. \`browser_navigate\` to the target URL (from
|
|
354
|
+
1. \`browser_navigate\` to the target URL (from workspace \`api.baseUrl\`)
|
|
387
355
|
2. \`browser_snapshot\` to see the page (ARIA tree)
|
|
388
356
|
3. Interact using \`browser_click\`, \`browser_type\`, \`browser_fill_form\`, etc.
|
|
389
357
|
4. \`browser_snapshot\` after each interaction that changes the page
|
|
390
358
|
5. \`skyramp_export_zip\` with an **absolute** output path: \`<repositoryPath>/.skyramp/<test_name>_trace.zip\`
|
|
391
|
-
6. \`skyramp_ui_test_generation\` with \`playwrightInput\` = the **absolute** path of the exported zip, and \`outputDir\` =
|
|
359
|
+
6. \`skyramp_ui_test_generation\` with \`playwrightInput\` = the **absolute** path of the exported zip, and \`outputDir\` = the **frontend** service's \`testDirectory\` from workspace.yml (e.g. \`frontend/tests\`). Do NOT use the backend service's testDirectory — UI tests must go in the frontend service's test directory.
|
|
392
360
|
|
|
393
361
|
Tips: For custom dropdowns (Radix, MUI): click combobox → snapshot → click option (NOT \`browser_select_option\`).
|
|
394
362
|
|
|
@@ -4,7 +4,6 @@ import { logger } from "../../utils/logger.js";
|
|
|
4
4
|
import { buildRecommendationPrompt } from "./test-recommendation-prompt.js";
|
|
5
5
|
import { ScenarioSource, AnalysisScope } from "../../types/RepositoryAnalysis.js";
|
|
6
6
|
import { SCENARIO_CATEGORIES } from "../../types/TestRecommendation.js";
|
|
7
|
-
import { inferExpectedStatus } from "../../utils/httpDefaults.js";
|
|
8
7
|
export function mergeEnrichedScenarios(serverScenarios, raw) {
|
|
9
8
|
const rejectionNotes = [];
|
|
10
9
|
let parsed;
|
|
@@ -55,7 +54,11 @@ export function mergeEnrichedScenarios(serverScenarios, raw) {
|
|
|
55
54
|
requestBody: st.requestBody,
|
|
56
55
|
queryParams: st.queryParams,
|
|
57
56
|
responseBody: st.responseBody,
|
|
58
|
-
|
|
57
|
+
// Default status code by method if omitted to avoid `statusCode: undefined` in tool calls
|
|
58
|
+
expectedStatusCode: st.expectedStatusCode ??
|
|
59
|
+
(String(st.method ?? "").toUpperCase() === "POST" ? 201
|
|
60
|
+
: String(st.method ?? "").toUpperCase() === "DELETE" ? 204
|
|
61
|
+
: 200),
|
|
59
62
|
expectedResponseFields: st.expectedResponseFields,
|
|
60
63
|
bodyMustInclude: st.bodyMustInclude,
|
|
61
64
|
chainsFrom: st.chainsFrom,
|
|
@@ -4,9 +4,8 @@ import { WorkspaceAuthType, getDefaultAuthHeader, AUTH_MIDDLEWARE_PATTERNS_STR }
|
|
|
4
4
|
import { logger } from "../../utils/logger.js";
|
|
5
5
|
import { extractResourceFromPath } from "../../utils/routeParsers.js";
|
|
6
6
|
import { buildArchitectPreamble, buildContextFetchingGuidance, buildReasoningProtocol, buildToolWorkflows, buildTestPatternGuidelines, buildTestQualityCriteria, buildFewShotExamples, buildVerificationChecklist, buildGenerationRules, getAuthSnippets, MAX_TESTS_TO_GENERATE, MAX_RECOMMENDATIONS, MAX_CRITICAL_TESTS, } from "./recommendationSections.js";
|
|
7
|
-
import { CATEGORY_PRIORITY,
|
|
7
|
+
import { CATEGORY_PRIORITY, TEST_CATEGORIES } from "../../types/TestRecommendation.js";
|
|
8
8
|
import { buildScopeAssessmentSection, isFrontendFile } from "./scopeAssessment.js";
|
|
9
|
-
import { resolveServiceDetailsRef } from "../../utils/featureFlags.js";
|
|
10
9
|
function formatTestLocations(locs) {
|
|
11
10
|
const entries = Object.entries(locs || {});
|
|
12
11
|
if (entries.length === 0)
|
|
@@ -27,8 +26,8 @@ function formatTestLocations(locs) {
|
|
|
27
26
|
// Categories map to HIGH / MEDIUM / LOW tiers.
|
|
28
27
|
// Within a tier, novelty (new > modified > existing) breaks ties,
|
|
29
28
|
// then cross-resource, step count, and finally the deterministic SHA-256 seed.
|
|
30
|
-
//
|
|
31
|
-
const PRIORITY_ORDER =
|
|
29
|
+
// CATEGORY_PRIORITY and PriorityTier imported from ../../types/TestRecommendation.js
|
|
30
|
+
const PRIORITY_ORDER = { CRITICAL: 4, HIGH: 3, MEDIUM: 2, LOW: 1 };
|
|
32
31
|
const NOVELTY_ORDER = { new: 3, modified: 2, existing: 1 };
|
|
33
32
|
function classifyNovelty(scenario, diffContext) {
|
|
34
33
|
if (!diffContext)
|
|
@@ -134,6 +133,12 @@ function buildExternalCoverageSet(testLocations) {
|
|
|
134
133
|
}
|
|
135
134
|
// ── Execution Plan (replaces pre-ranked + scenarios + heuristic sections) ──
|
|
136
135
|
function buildFullRepoRecommendations(scored, topN, baseUrl, authHeaderValue, authSchemeSnippet, authTypeValue, isFrontendProject = false, isFrontendOnlyProject = false, externalCoverage = new Set()) {
|
|
136
|
+
// Full-repo mode only — percentage-based UI/E2E slot targets (15% each, floor 1).
|
|
137
|
+
const rawE2E = isFrontendProject ? Math.max(1, Math.round(topN * 0.15)) : 0;
|
|
138
|
+
const rawUI = isFrontendProject ? Math.max(1, Math.round(topN * 0.15)) : 0;
|
|
139
|
+
const slotsFloor = Math.floor(topN / 2);
|
|
140
|
+
const minE2ESlots = Math.min(rawE2E, slotsFloor);
|
|
141
|
+
const minUISlots = Math.min(rawUI, Math.max(0, topN - minE2ESlots));
|
|
137
142
|
const authRef = authHeaderValue
|
|
138
143
|
? `, authHeader: "${authHeaderValue}"${authSchemeSnippet}`
|
|
139
144
|
: `, authHeader: <check OpenAPI securitySchemes or auth middleware; "" if confirmed unauthenticated>`;
|
|
@@ -162,9 +167,11 @@ function buildFullRepoRecommendations(scored, topN, baseUrl, authHeaderValue, au
|
|
|
162
167
|
return true;
|
|
163
168
|
})
|
|
164
169
|
: scored;
|
|
165
|
-
//
|
|
166
|
-
|
|
167
|
-
|
|
170
|
+
// For full-stack repos, carve out E2E and UI slots before filling with backend tests.
|
|
171
|
+
const backendSlotCount = isFrontendProject
|
|
172
|
+
? Math.max(0, topN - minE2ESlots - minUISlots)
|
|
173
|
+
: topN;
|
|
174
|
+
const allItems = scoredFiltered.slice(0, backendSlotCount);
|
|
168
175
|
const byType = new Map();
|
|
169
176
|
for (const t of TYPE_ORDER)
|
|
170
177
|
byType.set(t, []);
|
|
@@ -188,7 +195,7 @@ function buildFullRepoRecommendations(scored, topN, baseUrl, authHeaderValue, au
|
|
|
188
195
|
return [
|
|
189
196
|
`**${rank}. ${title}**`,
|
|
190
197
|
` ${s.description}`,
|
|
191
|
-
` ${step.method} ${step.path}
|
|
198
|
+
` ${step.method} ${step.path} \u2192 ${step.expectedStatusCode}`,
|
|
192
199
|
` Tool: \`skyramp_contract_test_generation({ endpointURL: "${endpointURL}", method: "${step.method}"${authRef}${dataParam} })\``,
|
|
193
200
|
` From source: fill in requestData field names and the specific production boundary this validates`,
|
|
194
201
|
].join("\n");
|
|
@@ -197,7 +204,7 @@ function buildFullRepoRecommendations(scored, topN, baseUrl, authHeaderValue, au
|
|
|
197
204
|
const stepLines = s.steps.map(st => {
|
|
198
205
|
const isBody = ["POST", "PUT", "PATCH"].includes(st.method);
|
|
199
206
|
const bodyHint = isBody ? ` \u2014 body: <${st.method} ${st.path} required fields from source>` : "";
|
|
200
|
-
return ` ${st.order}. ${st.method} ${st.path}
|
|
207
|
+
return ` ${st.order}. ${st.method} ${st.path} \u2192 ${st.expectedStatusCode}: ${st.description}${bodyHint}`;
|
|
201
208
|
}).join("\n");
|
|
202
209
|
const isTraceBased = testType === "e2e" || testType === "ui";
|
|
203
210
|
let toolCallsBlock;
|
|
@@ -243,7 +250,7 @@ function buildFullRepoRecommendations(scored, topN, baseUrl, authHeaderValue, au
|
|
|
243
250
|
dataParam = `, requestBody: <${st.method} ${st.path} required fields from source code>`;
|
|
244
251
|
}
|
|
245
252
|
}
|
|
246
|
-
return ` { method: "${st.method}", path: "${st.path}"
|
|
253
|
+
return ` { method: "${st.method}", path: "${st.path}", statusCode: ${st.expectedStatusCode}${dataParam} }`;
|
|
247
254
|
}).join(",\n");
|
|
248
255
|
toolCallsBlock = [
|
|
249
256
|
` skyramp_batch_scenario_test_generation({ scenarioName: "${s.scenarioName}", destination: "${destinationHost}", baseURL: "${baseUrl}"${scenarioAuthRef}, steps: [\n${batchSteps}\n ] })`,
|
|
@@ -275,11 +282,55 @@ function buildFullRepoRecommendations(scored, topN, baseUrl, authHeaderValue, au
|
|
|
275
282
|
const entries = items.map((item, i) => renderItem(item, globalRank + i + 1)).join("\n\n");
|
|
276
283
|
return `### ${label} (${items.length})\n\n${entries}`;
|
|
277
284
|
});
|
|
278
|
-
|
|
285
|
+
// Pre-allocate E2E and UI placeholder sections for full-stack repos.
|
|
286
|
+
const e2eSectionParts = [];
|
|
287
|
+
const uiSectionParts = [];
|
|
288
|
+
if (isFrontendProject) {
|
|
289
|
+
for (let i = 0; i < minE2ESlots; i++) {
|
|
290
|
+
const rank = i + 1;
|
|
291
|
+
e2eSectionParts.push(`**${rank}. E2E User Journey ${i + 1}**\n` +
|
|
292
|
+
` End-to-end test covering a complete user journey through the frontend and backend.\n` +
|
|
293
|
+
` To generate: record a browser trace, then call the generation tool.\n` +
|
|
294
|
+
` browser_navigate({ url: "${baseUrl}" }) \u2192 exercise key user flow \u2192 skyramp_export_zip({ outputPath: "<repo>/.skyramp/e2e_journey_${i + 1}.zip" })\n` +
|
|
295
|
+
` Tool: \`skyramp_e2e_test_generation({ playwrightInput: "<repo>/.skyramp/e2e_journey_${i + 1}.zip"${authHeaderOnlyRef} })\`\n` +
|
|
296
|
+
` From source: read frontend components and their API calls to identify the highest-value user journey`);
|
|
297
|
+
}
|
|
298
|
+
for (let i = 0; i < minUISlots; i++) {
|
|
299
|
+
const rank = minE2ESlots + i + 1;
|
|
300
|
+
uiSectionParts.push(`**${rank}. UI Component Test ${i + 1}**\n` +
|
|
301
|
+
` Test key UI component interactions and state changes.\n` +
|
|
302
|
+
` To generate: record a browser trace, then call the generation tool.\n` +
|
|
303
|
+
` browser_navigate({ url: "${baseUrl}" }) \u2192 interact with UI components \u2192 skyramp_export_zip({ outputPath: "<repo>/.skyramp/ui_component_${i + 1}.zip" })\n` +
|
|
304
|
+
` Tool: \`skyramp_ui_test_generation({ playwrightInput: "<repo>/.skyramp/ui_component_${i + 1}.zip"${authHeaderOnlyRef} })\`\n` +
|
|
305
|
+
` From source: read frontend component files to identify interactions, form submissions, and state transitions`);
|
|
306
|
+
}
|
|
307
|
+
// Offset backend section ranks by the number of E2E + UI placeholders
|
|
308
|
+
const offset = minE2ESlots + minUISlots;
|
|
309
|
+
backendSections.forEach((_, idx) => {
|
|
310
|
+
const t = TYPE_ORDER.filter(t => (byType.get(t) ?? []).length > 0)[idx];
|
|
311
|
+
if (!t)
|
|
312
|
+
return;
|
|
313
|
+
const items = byType.get(t);
|
|
314
|
+
const label = TYPE_LABEL[t];
|
|
315
|
+
let globalRank = offset;
|
|
316
|
+
for (const prev of TYPE_ORDER) {
|
|
317
|
+
if (prev === t)
|
|
318
|
+
break;
|
|
319
|
+
globalRank += (byType.get(prev) ?? []).length;
|
|
320
|
+
}
|
|
321
|
+
backendSections[idx] = `### ${label} (${items.length})\n\n${items.map((item, i) => renderItem(item, globalRank + i + 1)).join("\n\n")}`;
|
|
322
|
+
});
|
|
323
|
+
}
|
|
324
|
+
const allSections = [
|
|
325
|
+
...(e2eSectionParts.length > 0 ? [`### E2E (${e2eSectionParts.length})\n\n${e2eSectionParts.join("\n\n")}`] : []),
|
|
326
|
+
...(uiSectionParts.length > 0 ? [`### UI (${uiSectionParts.length})\n\n${uiSectionParts.join("\n\n")}`] : []),
|
|
327
|
+
...backendSections,
|
|
328
|
+
];
|
|
329
|
+
const sections = allSections.join("\n\n");
|
|
279
330
|
const frontendTierNote = isFrontendOnlyProject
|
|
280
|
-
? `\n\n**Frontend repo:**
|
|
331
|
+
? `\n\n**Frontend repo:** supplement MUST include at least ${minE2ESlots} E2E test${minE2ESlots > 1 ? "s" : ""} (\`skyramp_e2e_test_generation\`) and at least ${minUISlots} UI test${minUISlots > 1 ? "s" : ""} (\`skyramp_ui_test_generation\`). Do NOT add integration or contract tests.`
|
|
281
332
|
: isFrontendProject
|
|
282
|
-
? `\n\n**Full-stack repo:**
|
|
333
|
+
? `\n\n**Full-stack repo:** supplement MUST include at least ${minE2ESlots} E2E test${minE2ESlots > 1 ? "s" : ""} (\`skyramp_e2e_test_generation\`) and at least ${minUISlots} UI test${minUISlots > 1 ? "s" : ""} (\`skyramp_ui_test_generation\`). Add these before exhausting backend tiers.`
|
|
283
334
|
: "";
|
|
284
335
|
const repoSupplementNote = supplementCount > 0
|
|
285
336
|
? `
|
|
@@ -306,9 +357,9 @@ function buildFullRepoRecommendations(scored, topN, baseUrl, authHeaderValue, au
|
|
|
306
357
|
</supplement_guidance>`
|
|
307
358
|
: "";
|
|
308
359
|
const typeMixText = isFrontendOnlyProject
|
|
309
|
-
? `This is a frontend repo. Focus on E2E and UI tests only.
|
|
360
|
+
? `This is a frontend repo. Focus on E2E and UI tests only. Include at least ${minE2ESlots} E2E test${minE2ESlots > 1 ? "s" : ""} (\`skyramp_e2e_test_generation\`) and at least ${minUISlots} UI test${minUISlots > 1 ? "s" : ""} (\`skyramp_ui_test_generation\`). Do NOT add integration or contract tests.`
|
|
310
361
|
: isFrontendProject
|
|
311
|
-
? `This is a full-stack repo. Coverage ranking: E2E > UI > Integration > Contract.
|
|
362
|
+
? `This is a full-stack repo. Coverage ranking: E2E > UI > Integration > Contract. Include at least ${minE2ESlots} E2E test${minE2ESlots > 1 ? "s" : ""} (\`skyramp_e2e_test_generation\`) and at least ${minUISlots} UI test${minUISlots > 1 ? "s" : ""} (\`skyramp_ui_test_generation\`), in addition to backend integration and contract tests.`
|
|
312
363
|
: `Focus on integration and contract tests for all API endpoints.`;
|
|
313
364
|
return `## Test Recommendations — ${topN} total (grouped by test type)
|
|
314
365
|
|
|
@@ -337,7 +388,7 @@ Before filling in tool call parameters for each item, use the analysis data alre
|
|
|
337
388
|
- Computed/derived response fields and their formulas — assert exact values; read source for formula details not captured in the analysis
|
|
338
389
|
- Auth middleware — set authHeader/authScheme from the repository context above; FastAPI HTTPBearer → 403 not 401
|
|
339
390
|
- Storage backend — if Redis or schema-less, discard unique-constraint and cascade-delete scenarios
|
|
340
|
-
- Delete behavior —
|
|
391
|
+
- Delete behavior — hard-delete → 204; soft-delete/cancel → 200
|
|
341
392
|
|
|
342
393
|
${buildTestQualityCriteria()}
|
|
343
394
|
|
|
@@ -351,6 +402,16 @@ ${buildTestQualityCriteria()}
|
|
|
351
402
|
</enrichment_notes>`;
|
|
352
403
|
}
|
|
353
404
|
function buildExecutionPlan(scored, maxGen, topN, baseUrl, authHeaderValue, authSchemeSnippet, authTypeValue, seed, endpointCount, isUIOnlyPR, hasFrontendChanges = false, hasTraces = false, externalCoverage = new Set(), relevantExternalTestPaths = []) {
|
|
405
|
+
const frontendUrl = "<frontend_url>";
|
|
406
|
+
// Slot allocation:
|
|
407
|
+
// - UI-only PR: all GENERATE slots are UI placeholders (no pre-ranked backend scenarios)
|
|
408
|
+
// - Mixed PR: last GENERATE slot is a UI placeholder; remaining slots are backend
|
|
409
|
+
// - Backend-only PR: all GENERATE slots are backend scenarios
|
|
410
|
+
const backendGenerateCount = isUIOnlyPR
|
|
411
|
+
? 0
|
|
412
|
+
: hasFrontendChanges
|
|
413
|
+
? Math.max(0, maxGen - 1)
|
|
414
|
+
: maxGen;
|
|
354
415
|
// Filter out scenarios whose primary method + resource + test type is already covered by external tests.
|
|
355
416
|
// Method-aware: an external test covering GET /orders won't block PUT /orders scenarios.
|
|
356
417
|
// This is the programmatic complement to the prompt-level Step 0 dedup instructions.
|
|
@@ -364,10 +425,8 @@ function buildExecutionPlan(scored, maxGen, topN, baseUrl, authHeaderValue, auth
|
|
|
364
425
|
return true;
|
|
365
426
|
})
|
|
366
427
|
: scored;
|
|
367
|
-
|
|
368
|
-
|
|
369
|
-
const generateItems = scoredAfterExternalDedup.slice(0, Math.min(maxGen, scoredAfterExternalDedup.length));
|
|
370
|
-
const rawAdditionalItems = scoredAfterExternalDedup.slice(maxGen, topN);
|
|
428
|
+
const generateItems = scoredAfterExternalDedup.slice(0, Math.min(backendGenerateCount, scoredAfterExternalDedup.length));
|
|
429
|
+
const rawAdditionalItems = scoredAfterExternalDedup.slice(backendGenerateCount, topN);
|
|
371
430
|
// Filter additional items whose primary resource + test type already appear in GENERATE
|
|
372
431
|
const generatedCoverage = new Set(generateItems.map(item => scenarioCoverageKey(item.scenario)));
|
|
373
432
|
const additionalItems = rawAdditionalItems.filter(item => !generatedCoverage.has(scenarioCoverageKey(item.scenario)));
|
|
@@ -380,10 +439,47 @@ function buildExecutionPlan(scored, maxGen, topN, baseUrl, authHeaderValue, auth
|
|
|
380
439
|
: authHeaderValue
|
|
381
440
|
? `, authHeader: "${authHeaderValue}"`
|
|
382
441
|
: `, authHeader: <check OpenAPI securitySchemes or auth middleware; "" if confirmed unauthenticated>`;
|
|
383
|
-
// UI-only
|
|
384
|
-
|
|
385
|
-
|
|
386
|
-
|
|
442
|
+
// UI-only: all GENERATE slots are UI test placeholders (one per changed component/flow)
|
|
443
|
+
const uiGenerateBlocks = isUIOnlyPR
|
|
444
|
+
? Array.from({ length: maxGen }, (_, i) => {
|
|
445
|
+
const rank = i + 1;
|
|
446
|
+
const zipPath = `<repositoryPath>/.skyramp/ui_test_${rank}_trace.zip`;
|
|
447
|
+
return hasTraces
|
|
448
|
+
? (`**#${rank} — GENERATE** | ui | workflow | new\n` +
|
|
449
|
+
`Scenario: ui-test-from-trace-${rank} (rename from the actual changed component/flow)\n` +
|
|
450
|
+
`Validates: UI interactions for a changed frontend component or flow.\n\n` +
|
|
451
|
+
`**Tool**: \`skyramp_ui_test_generation({ playwrightInput: "<discovered_trace_file_path>", outputDir: "<frontend service testDirectory from workspace.yml e.g. frontend/tests>" })\``)
|
|
452
|
+
: (`**#${rank} — GENERATE** | ui | workflow | new\n` +
|
|
453
|
+
`Scenario: ui-test-for-changed-component-${rank} (rename from the actual changed component/flow)\n` +
|
|
454
|
+
`Validates: UI interactions for changed frontend component/flow ${rank}.\n\n` +
|
|
455
|
+
`**Tool workflow:**\n` +
|
|
456
|
+
` 1. \`browser_navigate({ url: "${frontendUrl}" })\`\n` +
|
|
457
|
+
` 2. Interact with the changed component (read the diff to identify which component changed and what interactions it supports)\n` +
|
|
458
|
+
` 3. \`browser_snapshot()\` after each key interaction\n` +
|
|
459
|
+
` 4. \`skyramp_export_zip({ outputPath: "${zipPath}" })\` — absolute path\n` +
|
|
460
|
+
` 5. \`skyramp_ui_test_generation({ playwrightInput: "${zipPath}", outputDir: "<frontend service testDirectory from workspace.yml e.g. frontend/tests>" })\`\n\n` +
|
|
461
|
+
`Each item must target a distinct changed component or user flow.`);
|
|
462
|
+
}).join("\n\n")
|
|
463
|
+
: "";
|
|
464
|
+
// Mixed PR: reserve the last GENERATE slot for a UI test for the changed frontend components.
|
|
465
|
+
// Guard: skip when maxGen=0 (caller explicitly requested no generation)
|
|
466
|
+
const uiRank = generateItems.length + 1;
|
|
467
|
+
const uiPlaceholderBlock = (hasFrontendChanges && !isUIOnlyPR && maxGen > 0)
|
|
468
|
+
? hasTraces
|
|
469
|
+
? (`**#${uiRank} — GENERATE** | ui | workflow | new\n` +
|
|
470
|
+
`Scenario: ui-test-for-changed-components (rename from the actual changed component/flow)\n` +
|
|
471
|
+
`Validates: UI interactions for the changed frontend components in this PR.\n\n` +
|
|
472
|
+
`**Tool**: \`skyramp_ui_test_generation({ playwrightInput: "<discovered_trace_file_path>", outputDir: "<frontend service testDirectory from workspace.yml e.g. frontend/tests>" })\``)
|
|
473
|
+
: (`**#${uiRank} — GENERATE** | ui | workflow | new\n` +
|
|
474
|
+
`Scenario: ui-test-for-changed-components (rename from the actual changed component/flow)\n` +
|
|
475
|
+
`Validates: UI interactions for the changed frontend components in this PR.\n\n` +
|
|
476
|
+
`**Tool workflow:**\n` +
|
|
477
|
+
` 1. \`browser_navigate({ url: "${frontendUrl}" })\`\n` +
|
|
478
|
+
` 2. Interact with the changed component (read the diff to identify which component changed and what interactions it supports)\n` +
|
|
479
|
+
` 3. \`browser_snapshot()\` after each key interaction\n` +
|
|
480
|
+
` 4. \`skyramp_export_zip({ outputPath: "<repositoryPath>/.skyramp/ui_mixed_pr_trace.zip" })\` — absolute path\n` +
|
|
481
|
+
` 5. \`skyramp_ui_test_generation({ playwrightInput: "<repositoryPath>/.skyramp/ui_mixed_pr_trace.zip", outputDir: "<frontend service testDirectory from workspace.yml e.g. frontend/tests>" })\`\n\n` +
|
|
482
|
+
`Derive scenario name and steps from the actual changed frontend files.`)
|
|
387
483
|
: "";
|
|
388
484
|
const generateBlocks = generateItems.map((item, i) => {
|
|
389
485
|
const rank = i + 1;
|
|
@@ -400,7 +496,7 @@ function buildExecutionPlan(scored, maxGen, topN, baseUrl, authHeaderValue, auth
|
|
|
400
496
|
? `\n authHeader: "${authHeaderValue}"${authSchemeSnippet}`
|
|
401
497
|
: `\n authHeader: <resolve from workspace or OpenAPI securitySchemes>; authScheme: <if Authorization>`;
|
|
402
498
|
return (`**#${rank} — GENERATE** | ${testType} | ${s.category} | ${item.novelty}\n` +
|
|
403
|
-
`${step.method} ${step.path}
|
|
499
|
+
`${step.method} ${step.path} → ${step.expectedStatusCode}\n` +
|
|
404
500
|
`Validates: ${s.description}\n\n` +
|
|
405
501
|
`**Context for generation**:\n` +
|
|
406
502
|
` Endpoint URL: ${endpointURL}${requestBodyData}${authContext}\n\n` +
|
|
@@ -423,7 +519,7 @@ function buildExecutionPlan(scored, maxGen, topN, baseUrl, authHeaderValue, auth
|
|
|
423
519
|
const bodyData = st.requestBody && Object.keys(st.requestBody).length > 0
|
|
424
520
|
? ` [use requestBody: ${JSON.stringify(st.requestBody)} — pass as JSON string in tool call]`
|
|
425
521
|
: "";
|
|
426
|
-
return ` ${st.order}. ${st.method} ${st.path}
|
|
522
|
+
return ` ${st.order}. ${st.method} ${st.path} → ${st.expectedStatusCode}: ${st.description}${chains}${bodyHint}${bodyData}${responseHint}`;
|
|
427
523
|
}).join("\n");
|
|
428
524
|
let destinationHost = "localhost";
|
|
429
525
|
try {
|
|
@@ -434,7 +530,9 @@ function buildExecutionPlan(scored, maxGen, topN, baseUrl, authHeaderValue, auth
|
|
|
434
530
|
const authContext = authHeaderValue
|
|
435
531
|
? `authHeader: "${authHeaderValue}"${authSchemeSnippet}`
|
|
436
532
|
: "authHeader: <resolve from workspace or OpenAPI securitySchemes>; authScheme: <if Authorization>";
|
|
437
|
-
const prereqNote =
|
|
533
|
+
const prereqNote = s.category === "new_endpoint"
|
|
534
|
+
? `\n**Prerequisite discovery**: Check for FK fields (product_id, user_id, order_id) in the endpoint's request body. If found, prepend a step to create that prerequisite resource first, then chain its primary key field into the dependent step using template variable syntax. Check the actual field name from the response body (\`id\`, \`uuid\`, \`_id\`, etc.), response header (\`Location\`), or cookie — do not assume \`id\`.`
|
|
535
|
+
: "";
|
|
438
536
|
const bugLine = s.bugCatchingTarget
|
|
439
537
|
? `**Bug to catch**: ${s.bugCatchingTarget}\n`
|
|
440
538
|
: "";
|
|
@@ -463,16 +561,17 @@ function buildExecutionPlan(scored, maxGen, topN, baseUrl, authHeaderValue, auth
|
|
|
463
561
|
const s = item.scenario;
|
|
464
562
|
const testType = s.testType ?? (s.steps.length === 1 ? "contract" : "integration");
|
|
465
563
|
const target = s.steps.length === 1
|
|
466
|
-
? `${s.steps[0].method} ${s.steps[0].path}
|
|
564
|
+
? `${s.steps[0].method} ${s.steps[0].path} → ${s.steps[0].expectedStatusCode}`
|
|
467
565
|
: `Scenario: ${s.scenarioName} (${s.steps.map(st => `${st.method} ${st.path}`).join(" → ")})`;
|
|
468
566
|
return `#${rank} [ADDITIONAL] | ${testType} | ${s.category} | ${item.novelty}\n ${target}\n Validates: ${s.description}`;
|
|
469
567
|
}).join("\n\n");
|
|
470
|
-
// UI/E2E guidance — the LLM adds
|
|
471
|
-
//
|
|
472
|
-
|
|
473
|
-
|
|
568
|
+
// UI/E2E guidance — the LLM adds as many as its Budget Plan calls for.
|
|
569
|
+
// Note: if a UI test already occupies a GENERATE slot (uiPlaceholderBlock), that slot
|
|
570
|
+
// satisfies the UI generate count — do not add it again in ADDITIONAL.
|
|
571
|
+
const uiGuidance = !isUIOnlyPR ? `
|
|
572
|
+
**UI/E2E tests (add per your Budget Plan):** If your Budget Plan requires UI/E2E items beyond what is already in your GENERATE list, append an [ADDITIONAL] entry for each. If a UI test already occupies a GENERATE slot above, that slot satisfies your UI/E2E generate count — do NOT add it again to ADDITIONAL. Tool workflow for each new item:
|
|
474
573
|
- **E2E**: ${hasTraces ? "Use discovered trace/recording files with `skyramp_e2e_test_generation`." : "Add to additionalRecommendations with a note that both a backend API trace (`skyramp_start_trace_collection` / `skyramp_stop_trace_collection`) and a browser Playwright recording must be collected in a live environment first. Do NOT attempt `skyramp_e2e_test_generation` without both traces present."}
|
|
475
|
-
- **UI**: ${hasTraces ? "Use an existing Playwright `.zip` trace with `skyramp_ui_test_generation`." :
|
|
574
|
+
- **UI**: ${hasTraces ? "Use an existing Playwright `.zip` trace with `skyramp_ui_test_generation`." : "Record a trace using `browser_navigate` + `browser_snapshot` + `skyramp_export_zip`, then call `skyramp_ui_test_generation({ playwrightInput: \"<zip_path>\", outputDir: \"<frontend testDirectory from workspace.yml>\" })`."}
|
|
476
575
|
Derive scenario names and steps from the actual changed frontend files. If your Budget Plan calls for 0% UI/E2E, omit this entirely.` : "";
|
|
477
576
|
const supplementNote = `\n**If your Budget Plan total exceeds the pre-ranked items listed above:** draft additional tests from source-code enrichment (Step 1). For each new or changed endpoint, identify boundary or variation scenarios — formula parameters, search/filter constraints, required field validation. Only after exhausting PR-specific scenarios, add generic patterns (auth boundary → 401, non-existent ID → 404). Do NOT supplement with tests whose endpoint + test type match a GENERATE item.`;
|
|
478
577
|
// ── PR / branch-diff mode: execution plan ────────────────────────────────
|
|
@@ -499,7 +598,7 @@ ${externalTestFilesList}For every GENERATE item below, check its endpoint path a
|
|
|
499
598
|
|
|
500
599
|
**Step 1 — Source-Code Enrichment (before executing anything)**
|
|
501
600
|
Read the source code for ALL changed files. Before generating each recommendation, quote the relevant source code in a <source_evidence> block — include the route handler signature, request body schema fields, response shape, and any computed field formulas. Use these quotes to derive tool call parameters. Look for:
|
|
502
|
-
- **Auth middleware** — check for known signals (${AUTH_MIDDLEWARE_PATTERNS_STR}). If any match, override \`authHeader\` and \`authScheme\` even if
|
|
601
|
+
- **Auth middleware** — check for known signals (${AUTH_MIDDLEWARE_PATTERNS_STR}). If any match, override \`authHeader\` and \`authScheme\` even if workspace.yml says authType: none. **If no known signal matches but the diff shows security-adjacent code** (decorators like \`@requiresRole\`/\`@Protected\`, function names like \`validateToken\`/\`checkPermission\`/\`verifyHMAC\`, or imports from auth/security packages), read the relevant source file to determine the actual auth scheme before proceeding. Auth handling for \`skyramp_integration_test_generation\` with \`scenarioFile\` is covered in the Tool Workflows section below.
|
|
503
602
|
- Business rules and formulas (e.g. total_cost = compute * rate + memory * rate)
|
|
504
603
|
- State transitions and domain constraints (e.g. budget cannot drop below current spend)
|
|
505
604
|
- Validation logic (field constraints, cross-field dependencies)
|
|
@@ -534,7 +633,7 @@ If these conditions are not met, add it to ADDITIONAL only — do NOT displace a
|
|
|
534
633
|
When a qualifying candidate is inserted: place it HIGH before MEDIUM before LOW; within the same priority, source-code-derived candidates go BEFORE structural ones. Re-number ranks after insertion. The top ${maxGen} ranked items become GENERATE candidates.
|
|
535
634
|
|
|
536
635
|
**Source-code validation gates (apply during Step 1):**
|
|
537
|
-
- **Cascade vs referential integrity**: If both a cascade-delete and a delete-blocked scenario appear for the same resource pair, keep only the one matching the source
|
|
636
|
+
- **Cascade vs referential integrity**: If both a cascade-delete and a delete-blocked scenario appear for the same resource pair, keep only the one matching the source FK delete policy (ON DELETE CASCADE / cascade=True / onDelete: 'CASCADE' → keep cascade-delete; RESTRICT/PROTECT/no annotation → keep delete-blocked). Remove the inapplicable variant.
|
|
538
637
|
- **Unique constraints**: Unique-constraint scenarios (duplicate POST → 409) are pre-drafted for all resources. Confirm enforcement before keeping: SQL UNIQUE index, Mongoose unique: true, Prisma @unique, or explicit duplicate-check code. If the backend is Redis, schema-less, or has no explicit constraint in the changed files, move to ADDITIONAL with a note — do NOT generate.
|
|
539
638
|
|
|
540
639
|
**Step 2 — Diversity check (using enriched knowledge from Step 1)**
|
|
@@ -550,7 +649,7 @@ For each pair of GENERATE items, ask: same HTTP method + path + step sequence +
|
|
|
550
649
|
Same step sequence with only payload differences (e.g. 10% vs 5% discount both returning 200) = same code path = duplicate. Different scenario names do not make duplicate tests distinct.
|
|
551
650
|
|
|
552
651
|
**Step 3 — Execute merged plan in rank order**
|
|
553
|
-
Replace any scenario that pairs unrelated resources with one reflecting actual
|
|
652
|
+
Replace any scenario that pairs unrelated resources with one reflecting actual FK relationships in the codebase.
|
|
554
653
|
Use the field names and values from the \`<source_evidence>\` blocks you quoted in Step 1 to fill all tool call parameters. Prefer reusing Step 1 evidence when it already resolves a placeholder, but if a placeholder cannot be replaced with concrete values from files already read, you may read the specific schema, model, or handler file needed to resolve it. Assert response field values, not just status codes.
|
|
555
654
|
|
|
556
655
|
${buildTestQualityCriteria()}
|
|
@@ -566,10 +665,10 @@ ${buildGenerationRules(isUIOnlyPR)}
|
|
|
566
665
|
### GENERATE (process these EXACTLY as listed, in order — after completing Steps 0–2 above; if Step 0 converts an item to UPDATE, backfill the ADD slot from ADDITIONAL following the priority order in Step 0)
|
|
567
666
|
|
|
568
667
|
${isUIOnlyPR
|
|
569
|
-
? (
|
|
570
|
-
: (generateBlocks || " (no pre-ranked generate items — draft your own based on endpoint analysis)")}
|
|
668
|
+
? (uiGenerateBlocks || " (no UI generate items — derive scenarios from changed frontend files)")
|
|
669
|
+
: ([generateBlocks, uiPlaceholderBlock].filter(Boolean).join("\n\n") || " (no pre-ranked generate items — draft your own based on endpoint analysis)")}
|
|
571
670
|
|
|
572
|
-
**
|
|
671
|
+
**COMPLIANCE CHECK**: Before proceeding, verify your generate list matches the items above. If you plan to generate a scenario with a different name than what is listed (e.g. you want to generate "order-update-discount-calculation" but the plan says "orders-patch-add-items-recalculate"), STOP — use the plan's scenario name and steps. Add your alternative to ADDITIONAL instead. One retry on failure then skip to next item.
|
|
573
672
|
|
|
574
673
|
### ADDITIONAL (list in additionalRecommendations in this order after Step 1 insertion)
|
|
575
674
|
|
|
@@ -604,17 +703,10 @@ export function buildRecommendationPrompt(analysis, analysisScope = AnalysisScop
|
|
|
604
703
|
const hasFrontendChanges = isDiffScope && diffContext
|
|
605
704
|
? filteredChangedFiles.some(f => isFrontendFile(f))
|
|
606
705
|
: false;
|
|
607
|
-
// Backend changes detected if:
|
|
608
|
-
// 1. Endpoints directly matched from changed files (new/modified/removed), OR
|
|
609
|
-
// 2. Changed files are in backend service/model/middleware directories (affectedServices non-empty)
|
|
610
|
-
// but couldn't be mapped to specific endpoints (service-layer changes like services/items.ts)
|
|
611
706
|
const hasApiChanges = isDiffScope && diffContext
|
|
612
707
|
? (diffContext.newEndpoints.length > 0 || diffContext.modifiedEndpoints.length > 0 || (diffContext.removedEndpoints?.length ?? 0) > 0)
|
|
613
708
|
: false;
|
|
614
|
-
const
|
|
615
|
-
? (diffContext.affectedServices.length > 0 && filteredChangedFiles.some(f => !isFrontendFile(f) && /\.(ts|js|py|java|go|rb|rs|cs)$/.test(f)))
|
|
616
|
-
: false;
|
|
617
|
-
const isUIOnlyPR = hasFrontendChanges && !hasApiChanges && !hasBackendServiceChanges;
|
|
709
|
+
const isUIOnlyPR = hasFrontendChanges && !hasApiChanges;
|
|
618
710
|
const hasTraces = (analysis.artifacts?.traceFiles?.length ?? 0) > 0 ||
|
|
619
711
|
(analysis.artifacts?.playwrightRecordings?.length ?? 0) > 0;
|
|
620
712
|
// ── Mode preamble ──
|
|
@@ -814,11 +906,17 @@ ${detailBlocks}
|
|
|
814
906
|
const na = NOVELTY_ORDER[a.novelty], nb = NOVELTY_ORDER[b.novelty];
|
|
815
907
|
if (nb !== na)
|
|
816
908
|
return nb - na;
|
|
817
|
-
|
|
818
|
-
|
|
819
|
-
|
|
820
|
-
|
|
821
|
-
|
|
909
|
+
const crossA = a.scenario.steps.length > 2 ? 1 : 0;
|
|
910
|
+
const crossB = b.scenario.steps.length > 2 ? 1 : 0;
|
|
911
|
+
if (crossB !== crossA)
|
|
912
|
+
return crossB - crossA;
|
|
913
|
+
if (b.scenario.steps.length !== a.scenario.steps.length)
|
|
914
|
+
return b.scenario.steps.length - a.scenario.steps.length;
|
|
915
|
+
const errorA = a.scenario.steps.some(s => s.interactionType === "error" || s.interactionType === "edge-case") ? 1 : 0;
|
|
916
|
+
const errorB = b.scenario.steps.some(s => s.interactionType === "error" || s.interactionType === "edge-case") ? 1 : 0;
|
|
917
|
+
if (errorB !== errorA)
|
|
918
|
+
return errorB - errorA;
|
|
919
|
+
// Use locale-independent comparison to avoid runtime-locale non-determinism
|
|
822
920
|
const nameA = a.scenario.scenarioName;
|
|
823
921
|
const nameB = b.scenario.scenarioName;
|
|
824
922
|
if (nameA < nameB)
|
|
@@ -853,23 +951,12 @@ ${detailBlocks}
|
|
|
853
951
|
mainSection = buildExecutionPlan(scored, maxGen, topN, analysis.apiEndpoints.baseUrl, authHeaderValue, authSchemeSnippet, authTypeValue, seed, endpointCount, isUIOnlyPR, hasFrontendChanges, hasTraces, externalCoverage, analysis.existingTests.relevantExternalTestPaths ?? []);
|
|
854
952
|
}
|
|
855
953
|
else {
|
|
856
|
-
// Endpoint discovery hint: when backend service files changed but no endpoints were
|
|
857
|
-
// directly matched, guide the LLM to trace from service files → controllers → routes.
|
|
858
|
-
const endpointDiscoveryHint = hasBackendServiceChanges && diffContext
|
|
859
|
-
? `
|
|
860
|
-
**Endpoint Discovery Required** — the diff modifies backend service files (affected services: ${diffContext.affectedServices.join(", ")}) that don't directly define routes. You MUST:
|
|
861
|
-
1. Read the Routing entry-point files listed above
|
|
862
|
-
2. Trace which controllers/routers import the affected services
|
|
863
|
-
3. Identify the specific HTTP endpoints those controllers register
|
|
864
|
-
4. Use discovered endpoints as your GENERATE targets (contract + integration tests)
|
|
865
|
-
Do NOT default to UI-only tests — this PR has backend logic changes that require API-level testing.`
|
|
866
|
-
: "";
|
|
867
954
|
mainSection = `
|
|
868
955
|
## Draft Your Execution Plan
|
|
869
956
|
|
|
870
|
-
No pre-drafted scenarios available
|
|
957
|
+
No pre-drafted scenarios available.
|
|
871
958
|
|
|
872
|
-
${buildScopeAssessmentSection(topN, maxGen
|
|
959
|
+
${buildScopeAssessmentSection(topN, maxGen)}
|
|
873
960
|
|
|
874
961
|
Draft tests from the endpoint interactions and source code above, following the same tool pipeline described in Tool Workflows below. Prioritize critical categories: security_boundary > data_integrity > business_rule > workflow > crud.
|
|
875
962
|
|
|
@@ -877,8 +964,6 @@ For each test: pick the highest-impact endpoint(s), draft a realistic scenario w
|
|
|
877
964
|
|
|
878
965
|
**Honor your Budget Plan: produce exactly the total you committed to (GENERATE + ADDITIONAL). No fewer, no padding with low-value tests.**
|
|
879
966
|
|
|
880
|
-
**Coverage breadth enforcement:** Your GENERATE items must span DIFFERENT HTTP methods or endpoints from your Coverage Reasoning surfaces. If you identified 5+ testable surfaces but all GENERATE items target the same method + path (e.g. all POST /permissions), you are violating diversity. Spread GENERATE slots across distinct surfaces; put remaining surfaces in ADDITIONAL recommendations.
|
|
881
|
-
|
|
882
967
|
## Recommendation Stability
|
|
883
968
|
- **Carry forward** previous additionalRecommendations that still apply — match by scenarioName (multi-step) or endpoint (single-endpoint). Re-derive category and priority from test content.
|
|
884
969
|
- **Only drop** a previous recommendation if its target endpoint was removed, its business logic changed, or it is now covered by a generated test.
|