@harness-engineering/cli 1.13.0 → 1.13.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/dist/agents/skills/claude-code/add-harness-component/skill.yaml +1 -0
- package/dist/agents/skills/claude-code/align-documentation/skill.yaml +1 -0
- package/dist/agents/skills/claude-code/check-mechanical-constraints/skill.yaml +1 -0
- package/dist/agents/skills/claude-code/cleanup-dead-code/skill.yaml +1 -0
- package/dist/agents/skills/claude-code/detect-doc-drift/skill.yaml +1 -0
- package/dist/agents/skills/claude-code/enforce-architecture/skill.yaml +1 -0
- package/dist/agents/skills/claude-code/harness-accessibility/skill.yaml +1 -0
- package/dist/agents/skills/claude-code/harness-api-design/SKILL.md +304 -0
- package/dist/agents/skills/claude-code/harness-api-design/skill.yaml +74 -0
- package/dist/agents/skills/claude-code/harness-architecture-advisor/skill.yaml +1 -0
- package/dist/agents/skills/claude-code/harness-auth/SKILL.md +279 -0
- package/dist/agents/skills/claude-code/harness-auth/skill.yaml +81 -0
- package/dist/agents/skills/claude-code/harness-autopilot/skill.yaml +1 -0
- package/dist/agents/skills/claude-code/harness-brainstorming/skill.yaml +1 -0
- package/dist/agents/skills/claude-code/harness-caching/SKILL.md +309 -0
- package/dist/agents/skills/claude-code/harness-caching/skill.yaml +73 -0
- package/dist/agents/skills/claude-code/harness-chaos/SKILL.md +295 -0
- package/dist/agents/skills/claude-code/harness-chaos/skill.yaml +72 -0
- package/dist/agents/skills/claude-code/harness-code-review/skill.yaml +1 -0
- package/dist/agents/skills/claude-code/harness-codebase-cleanup/skill.yaml +1 -0
- package/dist/agents/skills/claude-code/harness-compliance/SKILL.md +303 -0
- package/dist/agents/skills/claude-code/harness-compliance/skill.yaml +78 -0
- package/dist/agents/skills/claude-code/harness-containerization/SKILL.md +284 -0
- package/dist/agents/skills/claude-code/harness-containerization/skill.yaml +80 -0
- package/dist/agents/skills/claude-code/harness-data-pipeline/SKILL.md +274 -0
- package/dist/agents/skills/claude-code/harness-data-pipeline/skill.yaml +81 -0
- package/dist/agents/skills/claude-code/harness-data-validation/SKILL.md +343 -0
- package/dist/agents/skills/claude-code/harness-data-validation/skill.yaml +75 -0
- package/dist/agents/skills/claude-code/harness-database/SKILL.md +258 -0
- package/dist/agents/skills/claude-code/harness-database/skill.yaml +80 -0
- package/dist/agents/skills/claude-code/harness-debugging/skill.yaml +1 -0
- package/dist/agents/skills/claude-code/harness-dependency-health/skill.yaml +1 -0
- package/dist/agents/skills/claude-code/harness-deployment/SKILL.md +255 -0
- package/dist/agents/skills/claude-code/harness-deployment/skill.yaml +77 -0
- package/dist/agents/skills/claude-code/harness-design/skill.yaml +1 -0
- package/dist/agents/skills/claude-code/harness-design-mobile/skill.yaml +1 -0
- package/dist/agents/skills/claude-code/harness-design-system/skill.yaml +1 -0
- package/dist/agents/skills/claude-code/harness-design-web/skill.yaml +1 -0
- package/dist/agents/skills/claude-code/harness-diagnostics/skill.yaml +1 -0
- package/dist/agents/skills/claude-code/harness-docs-pipeline/skill.yaml +1 -0
- package/dist/agents/skills/claude-code/harness-dx/SKILL.md +276 -0
- package/dist/agents/skills/claude-code/harness-dx/skill.yaml +76 -0
- package/dist/agents/skills/claude-code/harness-e2e/SKILL.md +245 -0
- package/dist/agents/skills/claude-code/harness-e2e/skill.yaml +78 -0
- package/dist/agents/skills/claude-code/harness-event-driven/SKILL.md +280 -0
- package/dist/agents/skills/claude-code/harness-event-driven/skill.yaml +77 -0
- package/dist/agents/skills/claude-code/harness-execution/skill.yaml +1 -0
- package/dist/agents/skills/claude-code/harness-feature-flags/SKILL.md +287 -0
- package/dist/agents/skills/claude-code/harness-feature-flags/skill.yaml +74 -0
- package/dist/agents/skills/claude-code/harness-git-workflow/skill.yaml +1 -0
- package/dist/agents/skills/claude-code/harness-hotspot-detector/skill.yaml +1 -0
- package/dist/agents/skills/claude-code/harness-i18n/skill.yaml +1 -0
- package/dist/agents/skills/claude-code/harness-i18n-process/skill.yaml +1 -0
- package/dist/agents/skills/claude-code/harness-i18n-workflow/skill.yaml +1 -0
- package/dist/agents/skills/claude-code/harness-impact-analysis/skill.yaml +1 -0
- package/dist/agents/skills/claude-code/harness-incident-response/SKILL.md +223 -0
- package/dist/agents/skills/claude-code/harness-incident-response/skill.yaml +78 -0
- package/dist/agents/skills/claude-code/harness-infrastructure-as-code/SKILL.md +279 -0
- package/dist/agents/skills/claude-code/harness-infrastructure-as-code/skill.yaml +80 -0
- package/dist/agents/skills/claude-code/harness-integration-test/SKILL.md +271 -0
- package/dist/agents/skills/claude-code/harness-integration-test/skill.yaml +73 -0
- package/dist/agents/skills/claude-code/harness-integrity/skill.yaml +1 -0
- package/dist/agents/skills/claude-code/harness-knowledge-mapper/skill.yaml +1 -0
- package/dist/agents/skills/claude-code/harness-load-testing/SKILL.md +274 -0
- package/dist/agents/skills/claude-code/harness-load-testing/skill.yaml +79 -0
- package/dist/agents/skills/claude-code/harness-ml-ops/SKILL.md +341 -0
- package/dist/agents/skills/claude-code/harness-ml-ops/skill.yaml +79 -0
- package/dist/agents/skills/claude-code/harness-mobile-patterns/SKILL.md +326 -0
- package/dist/agents/skills/claude-code/harness-mobile-patterns/skill.yaml +82 -0
- package/dist/agents/skills/claude-code/harness-mutation-test/SKILL.md +251 -0
- package/dist/agents/skills/claude-code/harness-mutation-test/skill.yaml +70 -0
- package/dist/agents/skills/claude-code/harness-observability/SKILL.md +283 -0
- package/dist/agents/skills/claude-code/harness-observability/skill.yaml +78 -0
- package/dist/agents/skills/claude-code/harness-onboarding/skill.yaml +1 -0
- package/dist/agents/skills/claude-code/harness-parallel-agents/skill.yaml +1 -0
- package/dist/agents/skills/claude-code/harness-perf/skill.yaml +1 -0
- package/dist/agents/skills/claude-code/harness-perf-tdd/skill.yaml +1 -0
- package/dist/agents/skills/claude-code/harness-planning/skill.yaml +1 -0
- package/dist/agents/skills/claude-code/harness-pre-commit-review/skill.yaml +1 -0
- package/dist/agents/skills/claude-code/harness-product-spec/SKILL.md +285 -0
- package/dist/agents/skills/claude-code/harness-product-spec/skill.yaml +72 -0
- package/dist/agents/skills/claude-code/harness-property-test/SKILL.md +281 -0
- package/dist/agents/skills/claude-code/harness-property-test/skill.yaml +71 -0
- package/dist/agents/skills/claude-code/harness-refactoring/skill.yaml +1 -0
- package/dist/agents/skills/claude-code/harness-release-readiness/skill.yaml +1 -0
- package/dist/agents/skills/claude-code/harness-resilience/SKILL.md +255 -0
- package/dist/agents/skills/claude-code/harness-resilience/skill.yaml +76 -0
- package/dist/agents/skills/claude-code/harness-roadmap/skill.yaml +1 -0
- package/dist/agents/skills/claude-code/harness-secrets/SKILL.md +293 -0
- package/dist/agents/skills/claude-code/harness-secrets/skill.yaml +76 -0
- package/dist/agents/skills/claude-code/harness-security-review/skill.yaml +1 -0
- package/dist/agents/skills/claude-code/harness-security-scan/skill.yaml +1 -0
- package/dist/agents/skills/claude-code/harness-skill-authoring/skill.yaml +1 -0
- package/dist/agents/skills/claude-code/harness-soundness-review/skill.yaml +1 -0
- package/dist/agents/skills/claude-code/harness-sql-review/SKILL.md +315 -0
- package/dist/agents/skills/claude-code/harness-sql-review/skill.yaml +74 -0
- package/dist/agents/skills/claude-code/harness-state-management/skill.yaml +1 -0
- package/dist/agents/skills/claude-code/harness-tdd/skill.yaml +1 -0
- package/dist/agents/skills/claude-code/harness-test-advisor/skill.yaml +1 -0
- package/dist/agents/skills/claude-code/harness-test-data/SKILL.md +268 -0
- package/dist/agents/skills/claude-code/harness-test-data/skill.yaml +74 -0
- package/dist/agents/skills/claude-code/harness-ux-copy/SKILL.md +271 -0
- package/dist/agents/skills/claude-code/harness-ux-copy/skill.yaml +77 -0
- package/dist/agents/skills/claude-code/harness-verification/skill.yaml +1 -0
- package/dist/agents/skills/claude-code/harness-verify/skill.yaml +1 -0
- package/dist/agents/skills/claude-code/harness-visual-regression/SKILL.md +257 -0
- package/dist/agents/skills/claude-code/harness-visual-regression/skill.yaml +74 -0
- package/dist/agents/skills/claude-code/initialize-harness-project/skill.yaml +1 -0
- package/dist/agents/skills/claude-code/validate-context-engineering/skill.yaml +1 -0
- package/dist/agents/skills/gemini-cli/add-harness-component/skill.yaml +1 -0
- package/dist/agents/skills/gemini-cli/align-documentation/skill.yaml +1 -0
- package/dist/agents/skills/gemini-cli/check-mechanical-constraints/skill.yaml +1 -0
- package/dist/agents/skills/gemini-cli/cleanup-dead-code/skill.yaml +1 -0
- package/dist/agents/skills/gemini-cli/detect-doc-drift/skill.yaml +1 -0
- package/dist/agents/skills/gemini-cli/enforce-architecture/skill.yaml +1 -0
- package/dist/agents/skills/gemini-cli/harness-accessibility/skill.yaml +1 -0
- package/dist/agents/skills/gemini-cli/harness-api-design/SKILL.md +304 -0
- package/dist/agents/skills/gemini-cli/harness-api-design/skill.yaml +74 -0
- package/dist/agents/skills/gemini-cli/harness-architecture-advisor/skill.yaml +1 -0
- package/dist/agents/skills/gemini-cli/harness-auth/SKILL.md +279 -0
- package/dist/agents/skills/gemini-cli/harness-auth/skill.yaml +81 -0
- package/dist/agents/skills/gemini-cli/harness-autopilot/skill.yaml +1 -0
- package/dist/agents/skills/gemini-cli/harness-brainstorming/skill.yaml +1 -0
- package/dist/agents/skills/gemini-cli/harness-caching/SKILL.md +309 -0
- package/dist/agents/skills/gemini-cli/harness-caching/skill.yaml +73 -0
- package/dist/agents/skills/gemini-cli/harness-chaos/SKILL.md +295 -0
- package/dist/agents/skills/gemini-cli/harness-chaos/skill.yaml +72 -0
- package/dist/agents/skills/gemini-cli/harness-code-review/skill.yaml +1 -0
- package/dist/agents/skills/gemini-cli/harness-codebase-cleanup/skill.yaml +1 -0
- package/dist/agents/skills/gemini-cli/harness-compliance/SKILL.md +303 -0
- package/dist/agents/skills/gemini-cli/harness-compliance/skill.yaml +78 -0
- package/dist/agents/skills/gemini-cli/harness-containerization/SKILL.md +284 -0
- package/dist/agents/skills/gemini-cli/harness-containerization/skill.yaml +80 -0
- package/dist/agents/skills/gemini-cli/harness-data-pipeline/SKILL.md +274 -0
- package/dist/agents/skills/gemini-cli/harness-data-pipeline/skill.yaml +81 -0
- package/dist/agents/skills/gemini-cli/harness-data-validation/SKILL.md +343 -0
- package/dist/agents/skills/gemini-cli/harness-data-validation/skill.yaml +75 -0
- package/dist/agents/skills/gemini-cli/harness-database/SKILL.md +258 -0
- package/dist/agents/skills/gemini-cli/harness-database/skill.yaml +80 -0
- package/dist/agents/skills/gemini-cli/harness-debugging/skill.yaml +1 -0
- package/dist/agents/skills/gemini-cli/harness-dependency-health/skill.yaml +1 -0
- package/dist/agents/skills/gemini-cli/harness-deployment/SKILL.md +255 -0
- package/dist/agents/skills/gemini-cli/harness-deployment/skill.yaml +77 -0
- package/dist/agents/skills/gemini-cli/harness-design/skill.yaml +1 -0
- package/dist/agents/skills/gemini-cli/harness-design-mobile/skill.yaml +1 -0
- package/dist/agents/skills/gemini-cli/harness-design-system/skill.yaml +1 -0
- package/dist/agents/skills/gemini-cli/harness-design-web/skill.yaml +1 -0
- package/dist/agents/skills/gemini-cli/harness-diagnostics/skill.yaml +1 -0
- package/dist/agents/skills/gemini-cli/harness-docs-pipeline/skill.yaml +1 -0
- package/dist/agents/skills/gemini-cli/harness-dx/SKILL.md +276 -0
- package/dist/agents/skills/gemini-cli/harness-dx/skill.yaml +76 -0
- package/dist/agents/skills/gemini-cli/harness-e2e/SKILL.md +245 -0
- package/dist/agents/skills/gemini-cli/harness-e2e/skill.yaml +78 -0
- package/dist/agents/skills/gemini-cli/harness-event-driven/SKILL.md +280 -0
- package/dist/agents/skills/gemini-cli/harness-event-driven/skill.yaml +77 -0
- package/dist/agents/skills/gemini-cli/harness-execution/skill.yaml +1 -0
- package/dist/agents/skills/gemini-cli/harness-feature-flags/SKILL.md +287 -0
- package/dist/agents/skills/gemini-cli/harness-feature-flags/skill.yaml +74 -0
- package/dist/agents/skills/gemini-cli/harness-git-workflow/skill.yaml +1 -0
- package/dist/agents/skills/gemini-cli/harness-hotspot-detector/skill.yaml +1 -0
- package/dist/agents/skills/gemini-cli/harness-i18n/skill.yaml +1 -0
- package/dist/agents/skills/gemini-cli/harness-i18n-process/skill.yaml +1 -0
- package/dist/agents/skills/gemini-cli/harness-i18n-workflow/skill.yaml +1 -0
- package/dist/agents/skills/gemini-cli/harness-impact-analysis/skill.yaml +1 -0
- package/dist/agents/skills/gemini-cli/harness-incident-response/SKILL.md +223 -0
- package/dist/agents/skills/gemini-cli/harness-incident-response/skill.yaml +78 -0
- package/dist/agents/skills/gemini-cli/harness-infrastructure-as-code/SKILL.md +279 -0
- package/dist/agents/skills/gemini-cli/harness-infrastructure-as-code/skill.yaml +80 -0
- package/dist/agents/skills/gemini-cli/harness-integration-test/SKILL.md +271 -0
- package/dist/agents/skills/gemini-cli/harness-integration-test/skill.yaml +73 -0
- package/dist/agents/skills/gemini-cli/harness-integrity/skill.yaml +1 -0
- package/dist/agents/skills/gemini-cli/harness-knowledge-mapper/skill.yaml +1 -0
- package/dist/agents/skills/gemini-cli/harness-load-testing/SKILL.md +274 -0
- package/dist/agents/skills/gemini-cli/harness-load-testing/skill.yaml +79 -0
- package/dist/agents/skills/gemini-cli/harness-ml-ops/SKILL.md +341 -0
- package/dist/agents/skills/gemini-cli/harness-ml-ops/skill.yaml +79 -0
- package/dist/agents/skills/gemini-cli/harness-mobile-patterns/SKILL.md +326 -0
- package/dist/agents/skills/gemini-cli/harness-mobile-patterns/skill.yaml +82 -0
- package/dist/agents/skills/gemini-cli/harness-mutation-test/SKILL.md +251 -0
- package/dist/agents/skills/gemini-cli/harness-mutation-test/skill.yaml +70 -0
- package/dist/agents/skills/gemini-cli/harness-observability/SKILL.md +283 -0
- package/dist/agents/skills/gemini-cli/harness-observability/skill.yaml +78 -0
- package/dist/agents/skills/gemini-cli/harness-onboarding/skill.yaml +1 -0
- package/dist/agents/skills/gemini-cli/harness-parallel-agents/skill.yaml +1 -0
- package/dist/agents/skills/gemini-cli/harness-perf/skill.yaml +1 -0
- package/dist/agents/skills/gemini-cli/harness-perf-tdd/skill.yaml +1 -0
- package/dist/agents/skills/gemini-cli/harness-planning/skill.yaml +1 -0
- package/dist/agents/skills/gemini-cli/harness-pre-commit-review/skill.yaml +1 -0
- package/dist/agents/skills/gemini-cli/harness-product-spec/SKILL.md +285 -0
- package/dist/agents/skills/gemini-cli/harness-product-spec/skill.yaml +72 -0
- package/dist/agents/skills/gemini-cli/harness-property-test/SKILL.md +281 -0
- package/dist/agents/skills/gemini-cli/harness-property-test/skill.yaml +71 -0
- package/dist/agents/skills/gemini-cli/harness-refactoring/skill.yaml +1 -0
- package/dist/agents/skills/gemini-cli/harness-release-readiness/skill.yaml +1 -0
- package/dist/agents/skills/gemini-cli/harness-resilience/SKILL.md +255 -0
- package/dist/agents/skills/gemini-cli/harness-resilience/skill.yaml +76 -0
- package/dist/agents/skills/gemini-cli/harness-roadmap/skill.yaml +1 -0
- package/dist/agents/skills/gemini-cli/harness-secrets/SKILL.md +293 -0
- package/dist/agents/skills/gemini-cli/harness-secrets/skill.yaml +76 -0
- package/dist/agents/skills/gemini-cli/harness-security-review/SKILL.md +240 -0
- package/dist/agents/skills/gemini-cli/harness-security-review/skill.yaml +1 -0
- package/dist/agents/skills/gemini-cli/harness-security-scan/skill.yaml +1 -0
- package/dist/agents/skills/gemini-cli/harness-skill-authoring/skill.yaml +1 -0
- package/dist/agents/skills/gemini-cli/harness-soundness-review/skill.yaml +1 -0
- package/dist/agents/skills/gemini-cli/harness-sql-review/SKILL.md +315 -0
- package/dist/agents/skills/gemini-cli/harness-sql-review/skill.yaml +74 -0
- package/dist/agents/skills/gemini-cli/harness-state-management/skill.yaml +1 -0
- package/dist/agents/skills/gemini-cli/harness-tdd/skill.yaml +1 -0
- package/dist/agents/skills/gemini-cli/harness-test-advisor/skill.yaml +1 -0
- package/dist/agents/skills/gemini-cli/harness-test-data/SKILL.md +268 -0
- package/dist/agents/skills/gemini-cli/harness-test-data/skill.yaml +74 -0
- package/dist/agents/skills/gemini-cli/harness-ux-copy/SKILL.md +271 -0
- package/dist/agents/skills/gemini-cli/harness-ux-copy/skill.yaml +77 -0
- package/dist/agents/skills/gemini-cli/harness-verification/skill.yaml +1 -0
- package/dist/agents/skills/gemini-cli/harness-verify/skill.yaml +1 -0
- package/dist/agents/skills/gemini-cli/harness-visual-regression/SKILL.md +257 -0
- package/dist/agents/skills/gemini-cli/harness-visual-regression/skill.yaml +74 -0
- package/dist/agents/skills/gemini-cli/initialize-harness-project/skill.yaml +1 -0
- package/dist/agents/skills/gemini-cli/validate-context-engineering/skill.yaml +1 -0
- package/dist/{agents-md-P2RHSUV7.js → agents-md-XU3BHE22.js} +1 -1
- package/dist/{architecture-ESOOE26S.js → architecture-2R5Z4ZAF.js} +2 -2
- package/dist/bin/harness-mcp.js +14 -13
- package/dist/bin/harness.js +22 -21
- package/dist/{check-phase-gate-S2MZKLFQ.js → check-phase-gate-2OFZ7OWW.js} +3 -2
- package/dist/{chunk-LD3DKUK5.js → chunk-4ZMOCPYO.js} +1 -1
- package/dist/{chunk-5VY23YK3.js → chunk-65FRIL4D.js} +2 -2
- package/dist/{chunk-L2KLU56K.js → chunk-AOZRDOIP.js} +2 -2
- package/dist/{chunk-MACVXDZK.js → chunk-DZS7CJKL.js} +4 -4
- package/dist/{chunk-7PZWR4LI.js → chunk-IM32EEDM.js} +9 -9
- package/dist/{chunk-2YPZKGAG.js → chunk-IMFVFNJE.js} +1 -1
- package/dist/{chunk-HD4IBGLA.js → chunk-N5G5QMS3.js} +24 -1
- package/dist/{chunk-MI5XJQDY.js → chunk-ND6PNADU.js} +23 -9
- package/dist/{chunk-7KQSUZVG.js → chunk-NERR4TAO.js} +729 -436
- package/dist/{chunk-PSNN4LWX.js → chunk-NOPU4RZ4.js} +2 -2
- package/dist/{chunk-KELT6K6M.js → chunk-PQ5YK4AY.js} +287 -258
- package/dist/{chunk-WPPDRIJL.js → chunk-QY4T6YAZ.js} +3 -3
- package/dist/{chunk-RZSUJBZZ.js → chunk-SSKDAOX5.js} +31 -28
- package/dist/{chunk-2VU4MFM3.js → chunk-TKJZKICB.js} +6 -6
- package/dist/{chunk-GNGELAXY.js → chunk-TS3XWPW5.js} +1 -1
- package/dist/chunk-UAX4I5ZE.js +217 -0
- package/dist/{chunk-VRFZWGMS.js → chunk-XYLGHKG6.js} +5 -1
- package/dist/{chunk-6N4R6FVX.js → chunk-YBJ262QL.js} +1 -1
- package/dist/{chunk-3KOLLWWE.js → chunk-Z77YQRQT.js} +11 -207
- package/dist/{ci-workflow-4NYBUG6R.js → ci-workflow-EHV65NQB.js} +1 -1
- package/dist/{create-skill-WPXHSLX2.js → create-skill-XSWHMSM5.js} +2 -2
- package/dist/{dist-WF4C7A4A.js → dist-2B363XUH.js} +1 -1
- package/dist/{dist-M6BQODWC.js → dist-HXHWB7SV.js} +2 -2
- package/dist/{docs-BPYCN2DR.js → docs-FZOPM4GK.js} +4 -2
- package/dist/{engine-LXLIWQQ3.js → engine-OL4T6NZS.js} +1 -1
- package/dist/{entropy-4VDVV5CR.js → entropy-LVHJMFGH.js} +2 -2
- package/dist/{feedback-63QB5RCA.js → feedback-IHLVLMRD.js} +1 -1
- package/dist/{generate-agent-definitions-QABOJG56.js → generate-agent-definitions-64S3CLEZ.js} +3 -3
- package/dist/{glob-helper-5OHBUQAI.js → glob-helper-R5FXNUPS.js} +1 -1
- package/dist/{graph-loader-KO4GJ5N2.js → graph-loader-GJZ4FN4Y.js} +1 -1
- package/dist/index.d.ts +35 -8
- package/dist/index.js +23 -21
- package/dist/{loader-Z2IT7QX3.js → loader-DPYFB6R6.js} +1 -1
- package/dist/{mcp-KQHEL5IF.js → mcp-JQUI7BVZ.js} +14 -13
- package/dist/{performance-26BH47O4.js → performance-ZTVSUANN.js} +2 -2
- package/dist/{review-pipeline-GHR3WFBI.js → review-pipeline-76JHKGSV.js} +1 -1
- package/dist/{runtime-PDWD7UIK.js → runtime-X7U6SC7K.js} +1 -1
- package/dist/{security-UQFUZXEN.js → security-FWQZF2IZ.js} +1 -1
- package/dist/skill-executor-XZLYZYAK.js +8 -0
- package/dist/{validate-N7QJOKFZ.js → validate-GCHZJIL7.js} +2 -2
- package/dist/{validate-cross-check-EDQ5QGTM.js → validate-cross-check-STFHYMAZ.js} +1 -1
- package/package.json +3 -3
- package/dist/skill-executor-RG45LUO5.js +0 -8
|
@@ -0,0 +1,271 @@
|
|
|
1
|
+
# Harness Integration Test
|
|
2
|
+
|
|
3
|
+
> Service boundary testing, API contract verification, and consumer-driven contract validation. Ensures services communicate correctly without requiring full end-to-end infrastructure.
|
|
4
|
+
|
|
5
|
+
## When to Use
|
|
6
|
+
|
|
7
|
+
- Testing API endpoints with real HTTP requests against a running service
|
|
8
|
+
- Validating consumer-driven contracts between microservices (Pact, Spring Cloud Contract)
|
|
9
|
+
- Verifying database interactions through repository or data access layers
|
|
10
|
+
- NOT when testing pure business logic with no I/O (use unit tests or harness-tdd instead)
|
|
11
|
+
- NOT when testing full user flows through a browser (use harness-e2e instead)
|
|
12
|
+
- NOT when performing load or stress testing on APIs (use harness-load-testing instead)
|
|
13
|
+
|
|
14
|
+
## Process
|
|
15
|
+
|
|
16
|
+
### Phase 1: DISCOVER -- Map Service Boundaries and Dependencies
|
|
17
|
+
|
|
18
|
+
1. **Identify service boundaries.** Scan the project structure for:
|
|
19
|
+
- API route definitions (Express routers, FastAPI endpoints, Spring controllers, Go HTTP handlers)
|
|
20
|
+
- Service client code (HTTP clients, gRPC stubs, message queue publishers/consumers)
|
|
21
|
+
- Shared type definitions or API schemas (OpenAPI specs, proto files, GraphQL schemas)
|
|
22
|
+
|
|
23
|
+
2. **Map inter-service dependencies.** For each service, catalog:
|
|
24
|
+
- Upstream dependencies: services this service calls
|
|
25
|
+
- Downstream consumers: services that call this service
|
|
26
|
+
- Shared resources: databases, message queues, caches
|
|
27
|
+
|
|
28
|
+
3. **Inventory existing integration tests.** Glob for test files in `tests/integration/`, `__integration__/`, `tests/api/`, and `contract-tests/`. Classify by type:
|
|
29
|
+
- API tests: send HTTP requests and assert responses
|
|
30
|
+
- Contract tests: verify provider/consumer agreements
|
|
31
|
+
- Repository tests: test data access against a real database
|
|
32
|
+
|
|
33
|
+
4. **Identify coverage gaps.** Cross-reference discovered endpoints and service boundaries against existing tests. Flag untested:
|
|
34
|
+
- API endpoints with no request/response validation
|
|
35
|
+
- Service boundaries with no contract tests
|
|
36
|
+
- Error scenarios (4xx responses, timeout handling, retry behavior)
|
|
37
|
+
|
|
38
|
+
5. **Select test strategy.** Based on the architecture:
|
|
39
|
+
- Monolith: API tests with supertest/httptest against the running application
|
|
40
|
+
- Microservices: consumer-driven contract tests with Pact plus API tests per service
|
|
41
|
+
- Event-driven: message contract tests plus async handler integration tests
|
|
42
|
+
|
|
43
|
+
### Phase 2: MOCK -- Configure Test Doubles and Infrastructure
|
|
44
|
+
|
|
45
|
+
1. **Set up test database.** Choose the appropriate strategy:
|
|
46
|
+
- **Testcontainers:** Spin up a real database in Docker for each test suite. Preferred for PostgreSQL, MySQL, MongoDB.
|
|
47
|
+
- **In-memory database:** SQLite in-memory for lightweight tests. Only when schema compatibility is confirmed.
|
|
48
|
+
- **Transaction rollback:** Wrap each test in a transaction and roll back. Fast but requires careful connection management.
|
|
49
|
+
|
|
50
|
+
2. **Configure mock services for external dependencies.** For each upstream dependency:
|
|
51
|
+
- Create a mock server using the framework's built-in tools (Pact mock, WireMock, nock, MSW)
|
|
52
|
+
- Define request/response pairs from the API contract or OpenAPI spec
|
|
53
|
+
- Configure realistic error responses (500, 503, timeout) for error path testing
|
|
54
|
+
|
|
55
|
+
3. **Set up contract broker (if using Pact).** Configure:
|
|
56
|
+
- Pact broker URL and authentication
|
|
57
|
+
- Consumer and provider version tagging strategy
|
|
58
|
+
- Webhook configuration for provider verification on deploy
|
|
59
|
+
|
|
60
|
+
4. **Create test fixtures and seed data.** Generate:
|
|
61
|
+
- Database seed scripts for required reference data
|
|
62
|
+
- Request/response fixtures for common API payloads
|
|
63
|
+
- Factory functions for building test entities with sensible defaults
|
|
64
|
+
|
|
65
|
+
5. **Verify mock infrastructure starts.** Run a smoke test that:
|
|
66
|
+
- Starts the test database and confirms connectivity
|
|
67
|
+
- Starts mock services and confirms they respond
|
|
68
|
+
- Seeds baseline data and confirms it is queryable
|
|
69
|
+
|
|
70
|
+
### Phase 3: IMPLEMENT -- Write Integration Tests
|
|
71
|
+
|
|
72
|
+
1. **Write API endpoint tests.** For each endpoint, test:
|
|
73
|
+
- Happy path: valid request returns expected response with correct status code and body
|
|
74
|
+
- Validation: invalid input returns 400 with descriptive error messages
|
|
75
|
+
- Authentication: unauthenticated requests return 401, unauthorized return 403
|
|
76
|
+
- Not found: requests for non-existent resources return 404
|
|
77
|
+
- Edge cases: empty collections, pagination boundaries, large payloads
|
|
78
|
+
|
|
79
|
+
2. **Write consumer-driven contract tests (when applicable).** For each consumer-provider pair:
|
|
80
|
+
- Consumer side: define interactions (request/response pairs) the consumer expects
|
|
81
|
+
- Provider side: verify the provider satisfies all consumer contracts
|
|
82
|
+
- Use Pact matchers for flexible verification (type matching, regex, array-like)
|
|
83
|
+
|
|
84
|
+
3. **Write repository/data access tests.** For each data access layer:
|
|
85
|
+
- CRUD operations with valid data
|
|
86
|
+
- Constraint violations (unique, foreign key, not-null)
|
|
87
|
+
- Query correctness (filters, sorting, pagination)
|
|
88
|
+
- Transaction behavior (isolation, rollback on error)
|
|
89
|
+
|
|
90
|
+
4. **Write error handling and resilience tests.** Verify:
|
|
91
|
+
- Timeout behavior: service responds within SLA when dependency is slow
|
|
92
|
+
- Retry logic: transient failures trigger retries with backoff
|
|
93
|
+
- Circuit breaker: repeated failures open the circuit and return fallback
|
|
94
|
+
- Graceful degradation: partial dependency failure does not crash the service
|
|
95
|
+
|
|
96
|
+
5. **Organize tests by execution speed.** Tag or separate:
|
|
97
|
+
- Fast integration tests (in-memory mocks, < 5 seconds): run on every commit
|
|
98
|
+
- Slow integration tests (testcontainers, external services, > 5 seconds): run on PR
|
|
99
|
+
|
|
100
|
+
### Phase 4: VALIDATE -- Execute and Verify Contract Compliance
|
|
101
|
+
|
|
102
|
+
1. **Run the full integration test suite.** Execute all tests with verbose output. Collect:
|
|
103
|
+
- Pass/fail counts per test category (API, contract, repository)
|
|
104
|
+
- Execution time per test and per suite
|
|
105
|
+
- Any tests that require external services to be running
|
|
106
|
+
|
|
107
|
+
2. **Verify contract compliance.** If using Pact:
|
|
108
|
+
- Publish consumer pacts to the broker
|
|
109
|
+
- Run provider verification against published pacts
|
|
110
|
+
- Confirm the can-i-deploy check passes for the target environment
|
|
111
|
+
|
|
112
|
+
3. **Validate test isolation.** Run tests in random order (if the framework supports it). Any test that fails only when run after a specific other test has a shared-state bug. Fix immediately.
|
|
113
|
+
|
|
114
|
+
4. **Run `harness validate`.** Confirm the project passes all harness checks including the new integration test infrastructure.
|
|
115
|
+
|
|
116
|
+
5. **Generate coverage report.** Summarize:
|
|
117
|
+
- Endpoints tested vs. total endpoints discovered
|
|
118
|
+
- Contract coverage: consumer-provider pairs with verified contracts
|
|
119
|
+
- Error scenarios covered vs. identified
|
|
120
|
+
- Recommended next steps for remaining gaps
|
|
121
|
+
|
|
122
|
+
### Graph Refresh
|
|
123
|
+
|
|
124
|
+
If a knowledge graph exists at `.harness/graph/`, refresh it after code changes to keep graph queries accurate:
|
|
125
|
+
|
|
126
|
+
```
|
|
127
|
+
harness scan [path]
|
|
128
|
+
```
|
|
129
|
+
|
|
130
|
+
## Harness Integration
|
|
131
|
+
|
|
132
|
+
- **`harness validate`** -- Run in VALIDATE phase after all integration tests are implemented. Confirms project-wide health.
|
|
133
|
+
- **`harness check-deps`** -- Run after MOCK phase to verify test infrastructure dependencies do not leak into production bundles.
|
|
134
|
+
- **`emit_interaction`** -- Used at checkpoints to present contract verification results and coverage gaps to the human.
|
|
135
|
+
- **Grep** -- Used in DISCOVER phase to find route definitions, HTTP client usage, and service boundary patterns.
|
|
136
|
+
- **Glob** -- Used to catalog existing integration tests and contract files.
|
|
137
|
+
|
|
138
|
+
## Success Criteria
|
|
139
|
+
|
|
140
|
+
- Every API endpoint has at least one integration test covering the happy path
|
|
141
|
+
- Every consumer-provider boundary has a verified contract (when using microservices)
|
|
142
|
+
- Error scenarios (400, 401, 403, 404, 500, timeout) are tested for all public endpoints
|
|
143
|
+
- All integration tests pass with test doubles -- no dependency on external staging environments for CI
|
|
144
|
+
- Test isolation is verified: tests pass in any execution order
|
|
145
|
+
- `harness validate` passes with the integration test suite in place
|
|
146
|
+
|
|
147
|
+
## Examples
|
|
148
|
+
|
|
149
|
+
### Example: Express API with Supertest and Testcontainers
|
|
150
|
+
|
|
151
|
+
**DISCOVER output:**
|
|
152
|
+
|
|
153
|
+
```
|
|
154
|
+
Framework: Express 4.18 with TypeScript
|
|
155
|
+
Database: PostgreSQL via Prisma
|
|
156
|
+
Endpoints: 14 routes across 4 controllers (users, projects, tasks, auth)
|
|
157
|
+
Existing tests: 3 integration tests in tests/integration/ (auth only)
|
|
158
|
+
Coverage gaps: projects CRUD, tasks filtering, user profile update
|
|
159
|
+
```
|
|
160
|
+
|
|
161
|
+
**IMPLEMENT -- API endpoint test with supertest:**
|
|
162
|
+
|
|
163
|
+
```typescript
|
|
164
|
+
// tests/integration/projects.test.ts
|
|
165
|
+
import request from 'supertest';
|
|
166
|
+
import { app } from '../../src/app';
|
|
167
|
+
import { prisma } from '../../src/db';
|
|
168
|
+
import { createTestUser, generateAuthToken } from '../helpers/auth';
|
|
169
|
+
|
|
170
|
+
describe('POST /api/projects', () => {
|
|
171
|
+
let authToken: string;
|
|
172
|
+
|
|
173
|
+
beforeAll(async () => {
|
|
174
|
+
const user = await createTestUser(prisma);
|
|
175
|
+
authToken = generateAuthToken(user.id);
|
|
176
|
+
});
|
|
177
|
+
|
|
178
|
+
afterAll(async () => {
|
|
179
|
+
await prisma.project.deleteMany();
|
|
180
|
+
await prisma.user.deleteMany();
|
|
181
|
+
});
|
|
182
|
+
|
|
183
|
+
it('creates a project with valid data', async () => {
|
|
184
|
+
const response = await request(app)
|
|
185
|
+
.post('/api/projects')
|
|
186
|
+
.set('Authorization', `Bearer ${authToken}`)
|
|
187
|
+
.send({ name: 'Test Project', description: 'Integration test' });
|
|
188
|
+
|
|
189
|
+
expect(response.status).toBe(201);
|
|
190
|
+
expect(response.body).toMatchObject({
|
|
191
|
+
name: 'Test Project',
|
|
192
|
+
description: 'Integration test',
|
|
193
|
+
});
|
|
194
|
+
expect(response.body.id).toBeDefined();
|
|
195
|
+
});
|
|
196
|
+
|
|
197
|
+
it('returns 400 when name is missing', async () => {
|
|
198
|
+
const response = await request(app)
|
|
199
|
+
.post('/api/projects')
|
|
200
|
+
.set('Authorization', `Bearer ${authToken}`)
|
|
201
|
+
.send({ description: 'No name' });
|
|
202
|
+
|
|
203
|
+
expect(response.status).toBe(400);
|
|
204
|
+
expect(response.body.errors).toContainEqual(expect.objectContaining({ field: 'name' }));
|
|
205
|
+
});
|
|
206
|
+
|
|
207
|
+
it('returns 401 without auth token', async () => {
|
|
208
|
+
const response = await request(app).post('/api/projects').send({ name: 'Unauthorized' });
|
|
209
|
+
|
|
210
|
+
expect(response.status).toBe(401);
|
|
211
|
+
});
|
|
212
|
+
});
|
|
213
|
+
```
|
|
214
|
+
|
|
215
|
+
### Example: Pact Consumer-Driven Contract Test
|
|
216
|
+
|
|
217
|
+
**IMPLEMENT -- Consumer side (frontend):**
|
|
218
|
+
|
|
219
|
+
```typescript
|
|
220
|
+
// contract-tests/consumer/project-service.pact.ts
|
|
221
|
+
import { PactV3, MatchersV3 } from '@pact-foundation/pact';
|
|
222
|
+
import { ProjectClient } from '../../src/clients/project-client';
|
|
223
|
+
|
|
224
|
+
const { like, eachLike, uuid } = MatchersV3;
|
|
225
|
+
|
|
226
|
+
const provider = new PactV3({
|
|
227
|
+
consumer: 'Dashboard',
|
|
228
|
+
provider: 'ProjectService',
|
|
229
|
+
});
|
|
230
|
+
|
|
231
|
+
describe('ProjectService contract', () => {
|
|
232
|
+
it('returns a list of projects', async () => {
|
|
233
|
+
await provider
|
|
234
|
+
.given('projects exist for user')
|
|
235
|
+
.uponReceiving('a request for user projects')
|
|
236
|
+
.withRequest({
|
|
237
|
+
method: 'GET',
|
|
238
|
+
path: '/api/projects',
|
|
239
|
+
headers: { Authorization: like('Bearer token-123') },
|
|
240
|
+
})
|
|
241
|
+
.willRespondWith({
|
|
242
|
+
status: 200,
|
|
243
|
+
body: eachLike({
|
|
244
|
+
id: uuid(),
|
|
245
|
+
name: like('Project Alpha'),
|
|
246
|
+
createdAt: like('2026-01-15T10:30:00Z'),
|
|
247
|
+
}),
|
|
248
|
+
})
|
|
249
|
+
.executeTest(async (mockServer) => {
|
|
250
|
+
const client = new ProjectClient(mockServer.url);
|
|
251
|
+
const projects = await client.listProjects('token-123');
|
|
252
|
+
expect(projects).toHaveLength(1);
|
|
253
|
+
expect(projects[0].name).toBe('Project Alpha');
|
|
254
|
+
});
|
|
255
|
+
});
|
|
256
|
+
});
|
|
257
|
+
```
|
|
258
|
+
|
|
259
|
+
## Gates
|
|
260
|
+
|
|
261
|
+
- **No integration tests that require external staging environments for CI.** Every integration test must run with local test doubles (mocks, containers, in-memory databases). Tests that fail without a staging VPN are not integration tests -- they are environment tests.
|
|
262
|
+
- **No shared mutable state between tests.** Each test must set up and tear down its own data. If tests fail when run in random order, shared state exists. Fix it before proceeding.
|
|
263
|
+
- **No testing implementation details.** Integration tests assert on API contracts (status codes, response shapes, headers) and observable data changes -- not on internal function calls or database column values that are not part of the public contract.
|
|
264
|
+
- **Contract changes must be coordinated.** If a provider contract test reveals a breaking change, do not silently update the consumer expectation. Flag it as a coordination point between teams.
|
|
265
|
+
|
|
266
|
+
## Escalation
|
|
267
|
+
|
|
268
|
+
- **When a service dependency has no API documentation or schema:** Cannot write accurate contract tests without knowing the contract. Escalate to the dependency team to provide an OpenAPI spec, proto file, or at minimum a Pact broker with published contracts.
|
|
269
|
+
- **When Testcontainers fails in CI (Docker-in-Docker issues, resource limits):** Fall back to in-memory alternatives where possible. For databases that have no in-memory mode, escalate to DevOps to configure CI runners with Docker support.
|
|
270
|
+
- **When contract verification fails on the provider side:** This indicates a real incompatibility between consumer expectations and provider implementation. Do not adjust the consumer test to match the provider bug. Escalate to the provider team with the failing interaction details.
|
|
271
|
+
- **When integration tests exceed 5 minutes for a single service:** Triage by separating fast tests (mocked dependencies) from slow tests (testcontainers). Run fast tests on every commit, slow tests on PR only.
|
|
@@ -0,0 +1,73 @@
|
|
|
1
|
+
name: harness-integration-test
|
|
2
|
+
version: "1.0.0"
|
|
3
|
+
description: Service boundary testing, API integration testing, and consumer-driven contract validation
|
|
4
|
+
cognitive_mode: meticulous-verifier
|
|
5
|
+
triggers:
|
|
6
|
+
- manual
|
|
7
|
+
- on_new_feature
|
|
8
|
+
- on_pr
|
|
9
|
+
platforms:
|
|
10
|
+
- claude-code
|
|
11
|
+
- gemini-cli
|
|
12
|
+
tools:
|
|
13
|
+
- Bash
|
|
14
|
+
- Read
|
|
15
|
+
- Write
|
|
16
|
+
- Edit
|
|
17
|
+
- Glob
|
|
18
|
+
- Grep
|
|
19
|
+
- emit_interaction
|
|
20
|
+
cli:
|
|
21
|
+
command: harness skill run harness-integration-test
|
|
22
|
+
args:
|
|
23
|
+
- name: path
|
|
24
|
+
description: Project root path
|
|
25
|
+
required: false
|
|
26
|
+
- name: scope
|
|
27
|
+
description: "Test scope: api, contract, or full. Defaults to full."
|
|
28
|
+
required: false
|
|
29
|
+
- name: service
|
|
30
|
+
description: "Target service name for focused testing. Tests all services when omitted."
|
|
31
|
+
required: false
|
|
32
|
+
mcp:
|
|
33
|
+
tool: run_skill
|
|
34
|
+
input:
|
|
35
|
+
skill: harness-integration-test
|
|
36
|
+
path: string
|
|
37
|
+
type: rigid
|
|
38
|
+
tier: 3
|
|
39
|
+
internal: false
|
|
40
|
+
keywords:
|
|
41
|
+
- integration test
|
|
42
|
+
- contract test
|
|
43
|
+
- Pact
|
|
44
|
+
- service boundary
|
|
45
|
+
- API test
|
|
46
|
+
- mock service
|
|
47
|
+
- test container
|
|
48
|
+
- supertest
|
|
49
|
+
- httptest
|
|
50
|
+
- consumer-driven
|
|
51
|
+
stack_signals:
|
|
52
|
+
- "tests/integration/"
|
|
53
|
+
- "src/**/__integration__/"
|
|
54
|
+
- "pact/"
|
|
55
|
+
- "contract-tests/"
|
|
56
|
+
- "tests/api/"
|
|
57
|
+
phases:
|
|
58
|
+
- name: discover
|
|
59
|
+
description: Map service boundaries, API endpoints, and inter-service dependencies
|
|
60
|
+
required: true
|
|
61
|
+
- name: mock
|
|
62
|
+
description: Configure mock services, test containers, and contract stubs
|
|
63
|
+
required: true
|
|
64
|
+
- name: implement
|
|
65
|
+
description: Write integration tests covering API contracts, error scenarios, and boundary conditions
|
|
66
|
+
required: true
|
|
67
|
+
- name: validate
|
|
68
|
+
description: Execute tests against real or containerized services and verify contract compliance
|
|
69
|
+
required: true
|
|
70
|
+
state:
|
|
71
|
+
persistent: false
|
|
72
|
+
files: []
|
|
73
|
+
depends_on: []
|
|
@@ -0,0 +1,274 @@
|
|
|
1
|
+
# Harness Load Testing
|
|
2
|
+
|
|
3
|
+
> Stress testing, capacity planning, and performance benchmarking with k6, Artillery, and Gatling. Detects existing load test infrastructure, designs test scenarios for critical paths, executes tests, and analyzes results against defined thresholds.
|
|
4
|
+
|
|
5
|
+
## When to Use
|
|
6
|
+
|
|
7
|
+
- Before major releases or migrations to validate capacity and identify breaking points
|
|
8
|
+
- At milestone boundaries to establish performance baselines for critical endpoints
|
|
9
|
+
- When scaling infrastructure and needing to verify new capacity meets demand projections
|
|
10
|
+
- NOT for unit-level microbenchmarks (use harness-perf for function-level performance)
|
|
11
|
+
- NOT for real-time production monitoring (use observability tooling like Datadog or Grafana)
|
|
12
|
+
- NOT for frontend performance audits (use Lighthouse or harness-perf for client-side metrics)
|
|
13
|
+
|
|
14
|
+
## Process
|
|
15
|
+
|
|
16
|
+
### Phase 1: DETECT -- Inventory Existing Tests and Infrastructure
|
|
17
|
+
|
|
18
|
+
1. **Discover load testing tooling.** Scan the project for load test infrastructure:
|
|
19
|
+
- k6: `*.k6.js`, `k6/` directory, `import { check } from 'k6'` patterns
|
|
20
|
+
- Artillery: `artillery.yml`, `artillery/` directory, `config.target` in YAML files
|
|
21
|
+
- Gatling: `gatling/`, `*Simulation.scala`, `gatling.conf`
|
|
22
|
+
- JMeter: `*.jmx` files, `jmeter/` directory
|
|
23
|
+
- Custom: `load-tests/`, `perf/`, `benchmark/` directories
|
|
24
|
+
|
|
25
|
+
2. **Inventory existing test scenarios.** For each discovered test file:
|
|
26
|
+
- Extract target endpoints and their HTTP methods
|
|
27
|
+
- Identify virtual user counts, ramp-up profiles, and duration
|
|
28
|
+
- Note defined thresholds (p95, p99, error rate, throughput)
|
|
29
|
+
- Check for test data generation or seed data requirements
|
|
30
|
+
|
|
31
|
+
3. **Map critical endpoints.** Identify endpoints that should be load tested:
|
|
32
|
+
- High-traffic endpoints from route definitions and API documentation
|
|
33
|
+
- Endpoints involved in revenue-critical flows (checkout, payment, subscription)
|
|
34
|
+
- Endpoints with known performance sensitivity (search, aggregation, file upload)
|
|
35
|
+
- Recently changed endpoints from `git log --oneline --since="30 days ago"`
|
|
36
|
+
|
|
37
|
+
4. **Identify coverage gaps.** Compare critical endpoints against existing test scenarios:
|
|
38
|
+
- Endpoints with no load test coverage
|
|
39
|
+
- Tests with outdated thresholds or missing scenarios (spike, soak)
|
|
40
|
+
- Tests that do not match current API contracts (changed parameters, new endpoints)
|
|
41
|
+
|
|
42
|
+
5. **Check test infrastructure.** Verify the testing environment:
|
|
43
|
+
- Target environment configuration (staging URL, test database, mock services)
|
|
44
|
+
- CI/CD integration (is load testing part of the pipeline?)
|
|
45
|
+
- Test data availability (seed scripts, fixtures, data generators)
|
|
46
|
+
|
|
47
|
+
---
|
|
48
|
+
|
|
49
|
+
### Phase 2: DESIGN -- Define Scenarios and Thresholds
|
|
50
|
+
|
|
51
|
+
1. **Select test profiles.** Design scenarios for each critical endpoint based on the testing goal:
|
|
52
|
+
- **Smoke test:** 1-5 VUs for 1 minute. Validates the test script works correctly.
|
|
53
|
+
- **Load test:** Expected production traffic for 5-15 minutes. Validates normal operation.
|
|
54
|
+
- **Stress test:** 2-3x expected traffic with ramp-up. Finds the breaking point.
|
|
55
|
+
- **Spike test:** Sudden burst from baseline to 10x traffic. Tests auto-scaling and recovery.
|
|
56
|
+
- **Soak test:** Expected traffic for 1-4 hours. Detects memory leaks and connection pool exhaustion.
|
|
57
|
+
|
|
58
|
+
2. **Define virtual user profiles.** Model realistic user behavior:
|
|
59
|
+
- Think time between requests (sleep 1-3 seconds between page interactions)
|
|
60
|
+
- Request sequences that mirror real user journeys (browse -> search -> add to cart -> checkout)
|
|
61
|
+
- Authentication flows (login, token refresh, session maintenance)
|
|
62
|
+
- Data variation (different product IDs, user accounts, search queries)
|
|
63
|
+
|
|
64
|
+
3. **Set performance thresholds.** Define pass/fail criteria per endpoint:
|
|
65
|
+
- **Latency:** p95 < 500ms, p99 < 1000ms (adjust per endpoint SLO)
|
|
66
|
+
- **Error rate:** < 1% for load tests, < 5% for stress tests
|
|
67
|
+
- **Throughput:** Minimum requests per second for the target VU count
|
|
68
|
+
- Thresholds should be derived from SLOs if they exist, or from baseline measurements
|
|
69
|
+
|
|
70
|
+
4. **Generate test scripts.** Produce test files in the detected tool format:
|
|
71
|
+
- k6: JavaScript test with `stages`, `thresholds`, and `checks`
|
|
72
|
+
- Artillery: YAML config with `phases`, `ensure`, and scenario `flow`
|
|
73
|
+
- Include parameterized data, proper assertions, and tagged metrics
|
|
74
|
+
|
|
75
|
+
5. **Design ramp-up stages.** For each profile, define the VU ramp schedule:
|
|
76
|
+
- Gradual ramp-up to avoid thundering herd at test start
|
|
77
|
+
- Plateau at target load for measurement stability
|
|
78
|
+
- Optional step-up stages for stress tests to find the exact breaking point
|
|
79
|
+
- Cool-down period to observe recovery behavior
|
|
80
|
+
|
|
81
|
+
---
|
|
82
|
+
|
|
83
|
+
### Phase 3: EXECUTE -- Run Tests and Collect Metrics
|
|
84
|
+
|
|
85
|
+
1. **Validate test environment.** Before executing:
|
|
86
|
+
- Confirm the target URL is a staging/test environment (never run load tests against production without explicit approval)
|
|
87
|
+
- Verify test database is seeded and isolated
|
|
88
|
+
- Check that external dependencies are stubbed or rate-limited appropriately
|
|
89
|
+
- Confirm monitoring is active on the target environment to correlate metrics
|
|
90
|
+
|
|
91
|
+
2. **Run smoke test first.** Execute each test script with minimal load:
|
|
92
|
+
- 1-2 VUs for 30 seconds
|
|
93
|
+
- Verify all requests return expected status codes
|
|
94
|
+
- Confirm test data and authentication work correctly
|
|
95
|
+
- If smoke test fails, fix the script before proceeding to load tests
|
|
96
|
+
|
|
97
|
+
3. **Execute the selected test profile.** Run the load test and capture:
|
|
98
|
+
- Request latency distribution (p50, p90, p95, p99, max)
|
|
99
|
+
- Throughput (requests per second over time)
|
|
100
|
+
- Error rate and error type distribution (4xx vs 5xx)
|
|
101
|
+
- Virtual user count over time
|
|
102
|
+
- If using k6: `k6 run --out json=results.json test.k6.js`
|
|
103
|
+
- If using Artillery: `artillery run --output report.json artillery.yml`
|
|
104
|
+
|
|
105
|
+
4. **Monitor system resources during execution.** If accessible:
|
|
106
|
+
- CPU and memory utilization on target servers
|
|
107
|
+
- Database connection pool usage and query times
|
|
108
|
+
- Queue depths and consumer lag
|
|
109
|
+
- Network I/O and connection counts
|
|
110
|
+
|
|
111
|
+
5. **Capture baseline or compare to previous run.** Store results for trend analysis:
|
|
112
|
+
- Save raw results to `load-tests/results/YYYY-MM-DD-<profile>.json`
|
|
113
|
+
- If a previous baseline exists, prepare a comparison dataset
|
|
114
|
+
- Tag results with git commit SHA for traceability
|
|
115
|
+
|
|
116
|
+
---
|
|
117
|
+
|
|
118
|
+
### Phase 4: ANALYZE -- Interpret Results and Report
|
|
119
|
+
|
|
120
|
+
1. **Evaluate against thresholds.** For each endpoint tested:
|
|
121
|
+
- Did p95 latency meet the threshold? If not, by how much?
|
|
122
|
+
- Did error rate stay within acceptable bounds?
|
|
123
|
+
- Did throughput meet the minimum target?
|
|
124
|
+
- At what VU count did performance degrade (stress test breaking point)?
|
|
125
|
+
|
|
126
|
+
2. **Identify bottlenecks.** Correlate performance data with system metrics:
|
|
127
|
+
- High latency + high CPU = compute-bound (optimize algorithm or scale horizontally)
|
|
128
|
+
- High latency + normal CPU = I/O-bound (database queries, external API calls, disk)
|
|
129
|
+
- Increasing errors at specific VU count = resource exhaustion (connection pools, file descriptors, memory)
|
|
130
|
+
- Latency spike at ramp-up = cold start issue (JIT compilation, cache warming, connection establishment)
|
|
131
|
+
|
|
132
|
+
3. **Calculate capacity projections.** Based on stress test results:
|
|
133
|
+
- Maximum sustainable throughput before p95 exceeds threshold
|
|
134
|
+
- Current headroom as percentage of maximum capacity
|
|
135
|
+
- Projected capacity needed for growth targets (e.g., 2x traffic in 6 months)
|
|
136
|
+
- Scaling recommendation: vertical (bigger instances) vs horizontal (more instances)
|
|
137
|
+
|
|
138
|
+
4. **Compare to baseline.** If previous results exist:
|
|
139
|
+
- Latency delta per endpoint (regression or improvement)
|
|
140
|
+
- Throughput delta (higher or lower capacity)
|
|
141
|
+
- Error rate delta
|
|
142
|
+
- Flag any regressions greater than 10% as requiring investigation
|
|
143
|
+
|
|
144
|
+
5. **Produce the load test report.** Output a structured summary:
|
|
145
|
+
|
|
146
|
+
```
|
|
147
|
+
Load Test Report: <profile> — <date>
|
|
148
|
+
Target: <URL> | Tool: <k6/Artillery/Gatling> | Duration: <time>
|
|
149
|
+
Commit: <SHA>
|
|
150
|
+
|
|
151
|
+
Results:
|
|
152
|
+
Endpoint | p50 | p95 | p99 | Errors | RPS | Status
|
|
153
|
+
GET /api/products | 45ms | 120ms | 340ms | 0.1% | 850 | PASS
|
|
154
|
+
POST /api/orders | 180ms | 520ms | 1100ms | 0.8% | 120 | WARN (p99 > 1000ms)
|
|
155
|
+
GET /api/search | 95ms | 680ms | 2100ms | 2.3% | 340 | FAIL (p99 > 1000ms, errors > 1%)
|
|
156
|
+
|
|
157
|
+
Capacity: Max sustainable 1200 RPS at p95 < 500ms (current production: ~400 RPS, 3x headroom)
|
|
158
|
+
Bottleneck: /api/search becomes I/O-bound at 500 RPS — database full-text search query
|
|
159
|
+
Recommendation: Add search index or migrate to Elasticsearch for /api/search
|
|
160
|
+
```
|
|
161
|
+
|
|
162
|
+
6. **Archive results.** Save the full report and raw data for historical comparison.
|
|
163
|
+
|
|
164
|
+
---
|
|
165
|
+
|
|
166
|
+
## Harness Integration
|
|
167
|
+
|
|
168
|
+
- **`harness skill run harness-load-testing`** -- Primary CLI entry point. Runs all four phases.
|
|
169
|
+
- **`harness validate`** -- Run after generating test scripts to verify project structure.
|
|
170
|
+
- **`harness check-deps`** -- Verify load testing tool dependencies are declared (k6, artillery npm package).
|
|
171
|
+
- **`emit_interaction`** -- Used before execution (checkpoint:human-verify) to confirm target environment and test profile.
|
|
172
|
+
- **`Glob`** -- Discover existing load test files, result archives, and configuration.
|
|
173
|
+
- **`Grep`** -- Search for endpoint definitions, route handlers, and threshold configurations.
|
|
174
|
+
- **`Write`** -- Generate load test scripts and result reports.
|
|
175
|
+
- **`Edit`** -- Update existing test scripts with new scenarios or adjusted thresholds.
|
|
176
|
+
|
|
177
|
+
## Success Criteria
|
|
178
|
+
|
|
179
|
+
- All critical endpoints are identified and have corresponding load test scenarios
|
|
180
|
+
- Test scripts are syntactically valid and pass a smoke test before full execution
|
|
181
|
+
- Thresholds are derived from SLOs or documented baseline measurements, not arbitrary values
|
|
182
|
+
- Results include latency percentiles (p50, p95, p99), error rates, and throughput
|
|
183
|
+
- Bottlenecks are identified with specific causes (CPU, I/O, connection pool, memory)
|
|
184
|
+
- Capacity projections include headroom percentage and scaling recommendations
|
|
185
|
+
|
|
186
|
+
## Examples
|
|
187
|
+
|
|
188
|
+
### Example: k6 Load Test for Express.js REST API
|
|
189
|
+
|
|
190
|
+
```
|
|
191
|
+
Phase 1: DETECT
|
|
192
|
+
Tool: k6 (found k6/ directory with 2 existing scripts)
|
|
193
|
+
Existing tests:
|
|
194
|
+
- k6/smoke-api.k6.js: GET /api/health (1 VU, 10s)
|
|
195
|
+
- k6/load-products.k6.js: GET /api/products (50 VUs, 5m, p95 < 300ms)
|
|
196
|
+
Coverage gaps:
|
|
197
|
+
- POST /api/orders — revenue-critical, no load test
|
|
198
|
+
- GET /api/search — high-traffic, no load test
|
|
199
|
+
- No stress or soak test profiles exist
|
|
200
|
+
|
|
201
|
+
Phase 2: DESIGN
|
|
202
|
+
New scenarios:
|
|
203
|
+
- k6/load-orders.k6.js: POST /api/orders
|
|
204
|
+
Stages: ramp to 100 VUs over 2m, hold 5m, ramp down 1m
|
|
205
|
+
Thresholds: p95 < 800ms, errors < 1%, RPS > 80
|
|
206
|
+
- k6/stress-api.k6.js: All endpoints
|
|
207
|
+
Stages: step-up from 50 to 500 VUs in 50-VU increments, 2m per step
|
|
208
|
+
Thresholds: find breaking point, record p95 at each step
|
|
209
|
+
- k6/soak-api.k6.js: Critical endpoints at expected load
|
|
210
|
+
Duration: 2 hours at 200 VUs
|
|
211
|
+
Thresholds: p95 < 500ms, memory growth < 50MB/hour
|
|
212
|
+
|
|
213
|
+
Phase 3: EXECUTE
|
|
214
|
+
Environment: https://staging.example.com (verified non-production)
|
|
215
|
+
Smoke: All scripts pass with 1 VU
|
|
216
|
+
Load test results captured to load-tests/results/2026-03-27-load.json
|
|
217
|
+
|
|
218
|
+
Phase 4: ANALYZE
|
|
219
|
+
Results: POST /api/orders p95=620ms (PASS), GET /api/search p99=2100ms (FAIL)
|
|
220
|
+
Bottleneck: Full-text search on PostgreSQL LIKE query at 300+ RPS
|
|
221
|
+
Capacity: 800 RPS sustainable, current production 250 RPS (3.2x headroom)
|
|
222
|
+
Recommendation: Add pg_trgm index or migrate search to Elasticsearch
|
|
223
|
+
```
|
|
224
|
+
|
|
225
|
+
### Example: Artillery Test for NestJS GraphQL API
|
|
226
|
+
|
|
227
|
+
```
|
|
228
|
+
Phase 1: DETECT
|
|
229
|
+
Tool: Artillery (found artillery.yml with 1 scenario)
|
|
230
|
+
Existing: query { products } at 20 RPS for 60s
|
|
231
|
+
Gaps: mutations not tested, no spike profile, thresholds not defined
|
|
232
|
+
|
|
233
|
+
Phase 2: DESIGN
|
|
234
|
+
New config: artillery/graphql-load.yml
|
|
235
|
+
Phases:
|
|
236
|
+
- warm-up: 5 RPS for 30s
|
|
237
|
+
- load: 50 RPS for 5m
|
|
238
|
+
- spike: jump to 200 RPS for 30s, back to 50 RPS
|
|
239
|
+
Scenarios:
|
|
240
|
+
- query { products(limit: 20) } — 60% weight
|
|
241
|
+
- mutation { createOrder(input: $input) } — 25% weight
|
|
242
|
+
- query { user(id: $id) { orders } } — 15% weight
|
|
243
|
+
Ensure:
|
|
244
|
+
- p99 < 1500ms
|
|
245
|
+
- maxErrorRate < 2
|
|
246
|
+
|
|
247
|
+
Phase 3: EXECUTE
|
|
248
|
+
Target: http://staging.internal:3000/graphql
|
|
249
|
+
Smoke: PASS (all queries resolve, auth tokens valid)
|
|
250
|
+
Full run: artillery run --output report.json artillery/graphql-load.yml
|
|
251
|
+
|
|
252
|
+
Phase 4: ANALYZE
|
|
253
|
+
Results:
|
|
254
|
+
query products: p95=89ms, p99=210ms — PASS
|
|
255
|
+
mutation createOrder: p95=340ms, p99=890ms — PASS
|
|
256
|
+
query user.orders: p95=520ms, p99=2400ms — FAIL
|
|
257
|
+
Bottleneck: N+1 query in user.orders resolver (no DataLoader)
|
|
258
|
+
Spike recovery: System recovered to baseline within 15s after spike — PASS
|
|
259
|
+
Recommendation: Add DataLoader for orders resolver, re-test after fix
|
|
260
|
+
```
|
|
261
|
+
|
|
262
|
+
## Gates
|
|
263
|
+
|
|
264
|
+
- **No load tests against production without explicit human approval.** Load tests can cause real outages. The target environment must be verified as non-production before execution. If production testing is required, a `[checkpoint:human-verify]` must be passed with documented approval.
|
|
265
|
+
- **No test execution with failing smoke tests.** If the smoke test fails, the test script is broken. Fix the script before running at load. Running broken scripts at scale wastes time and produces meaningless results.
|
|
266
|
+
- **No capacity claims without stress test data.** Capacity projections require finding the actual breaking point. A load test that passes at the expected level does not establish maximum capacity. Stress test data is required for credible projections.
|
|
267
|
+
- **No threshold changes without documented justification.** If thresholds are relaxed from a previous baseline, the reason must be documented (e.g., "p95 threshold increased from 300ms to 500ms due to added encryption overhead per SEC-2026-04 requirement").
|
|
268
|
+
|
|
269
|
+
## Escalation
|
|
270
|
+
|
|
271
|
+
- **When the staging environment does not match production configuration:** Load test results against a differently-sized environment are misleading. Report: "Staging has [N] instances with [M] CPU/RAM. Production has [X] instances with [Y] CPU/RAM. Results should be scaled by a factor of [ratio], but this is approximate. Recommend matching staging to production configuration for accurate capacity planning."
|
|
272
|
+
- **When external dependencies cannot be stubbed for load testing:** Real external APIs may rate-limit or block load test traffic. Report: "The [dependency] cannot be stubbed and will rate-limit at [N] RPS. Options: (1) use a mock service like WireMock, (2) test with a dedicated sandbox API key, (3) test only the internal service layer with the external call short-circuited."
|
|
273
|
+
- **When load test results are inconsistent between runs:** Noisy results indicate environmental interference. Report: "Variance between runs exceeds 15% for [endpoint]. Possible causes: shared staging environment, garbage collection pauses, network congestion. Recommend running 3 iterations and using the median, or running on a dedicated isolated environment."
|
|
274
|
+
- **When the breaking point is below expected production traffic:** This is a capacity emergency. Report: "The system breaks at [N] RPS but production receives [M] RPS. Immediate scaling action required. Short-term: increase instance count by [factor]. Long-term: address the identified bottleneck in [component]."
|