@causyne/agent 0.0.0-stage → 1.10.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/LICENSE +102 -0
- package/README.md +31 -2
- package/SECURITY.md +24 -0
- package/bin/causyne.js +13 -0
- package/docs/ARCHITECTURE.md +63 -0
- package/docs/PROVENANCE.md +29 -0
- package/docs/SECURITY_REVIEW.md +89 -0
- package/docs/THIRD_PARTY_NOTICES.md +49 -0
- package/native-skills/ai-engineering/context-window-management/SKILL.md +43 -0
- package/native-skills/ai-engineering/llm-evaluation/SKILL.md +46 -0
- package/native-skills/ai-engineering/model-cost-and-latency/SKILL.md +46 -0
- package/native-skills/ai-engineering/prompt-construction/SKILL.md +42 -0
- package/native-skills/ai-engineering/retrieval-augmented-generation/SKILL.md +42 -0
- package/native-skills/ai-engineering/structured-output-and-schemas/SKILL.md +45 -0
- package/native-skills/ai-engineering/tool-calling-agents/SKILL.md +43 -0
- package/native-skills/android/app-performance-profiling/SKILL.md +42 -0
- package/native-skills/android/background-work-scheduling/SKILL.md +47 -0
- package/native-skills/android/compose-state-and-recomposition/SKILL.md +43 -0
- package/native-skills/android/gradle-build-configuration/SKILL.md +41 -0
- package/native-skills/android/lifecycle-and-leaks/SKILL.md +46 -0
- package/native-skills/android/play-store-release/SKILL.md +43 -0
- package/native-skills/android/room-and-persistence/SKILL.md +40 -0
- package/native-skills/backend/authentication-and-sessions/SKILL.md +45 -0
- package/native-skills/backend/background-jobs-and-queues/SKILL.md +42 -0
- package/native-skills/backend/caching-strategy/SKILL.md +44 -0
- package/native-skills/backend/graceful-shutdown/SKILL.md +42 -0
- package/native-skills/backend/rate-limiting-and-quotas/SKILL.md +40 -0
- package/native-skills/backend/rest-api-design/SKILL.md +40 -0
- package/native-skills/backend/webhook-delivery-and-receipt/SKILL.md +43 -0
- package/native-skills/cloud/cost-optimisation/SKILL.md +46 -0
- package/native-skills/cloud/iam-least-privilege/SKILL.md +43 -0
- package/native-skills/cloud/infrastructure-as-code/SKILL.md +43 -0
- package/native-skills/cloud/multi-region-and-failover/SKILL.md +43 -0
- package/native-skills/cloud/serverless-patterns/SKILL.md +47 -0
- package/native-skills/data/data-quality-monitoring/SKILL.md +51 -0
- package/native-skills/data/etl-pipeline-reliability/SKILL.md +43 -0
- package/native-skills/data/exploratory-analysis/SKILL.md +43 -0
- package/native-skills/data/metric-definition/SKILL.md +42 -0
- package/native-skills/data/notebook-to-production/SKILL.md +43 -0
- package/native-skills/database/backup-and-recovery/SKILL.md +43 -0
- package/native-skills/database/connection-pooling/SKILL.md +42 -0
- package/native-skills/database/data-integrity-and-constraints/SKILL.md +42 -0
- package/native-skills/database/query-optimisation/SKILL.md +38 -0
- package/native-skills/database/safe-migrations/SKILL.md +47 -0
- package/native-skills/database/schema-design/SKILL.md +38 -0
- package/native-skills/database/transactions-and-isolation/SKILL.md +41 -0
- package/native-skills/design/colour-system-construction/SKILL.md +38 -0
- package/native-skills/design/design-direction-brief/SKILL.md +41 -0
- package/native-skills/design/motion-and-transitions/SKILL.md +38 -0
- package/native-skills/design/responsive-strategy/SKILL.md +39 -0
- package/native-skills/design/spacing-and-rhythm/SKILL.md +38 -0
- package/native-skills/design/type-scale-and-hierarchy/SKILL.md +39 -0
- package/native-skills/design/visual-qa-loop/SKILL.md +42 -0
- package/native-skills/devops/docker-multi-stage-builds/SKILL.md +60 -0
- package/native-skills/devops/github-actions-ci/SKILL.md +60 -0
- package/native-skills/devops/incident-triage/SKILL.md +38 -0
- package/native-skills/devops/kubernetes-deployment-hardening/SKILL.md +64 -0
- package/native-skills/devops/observability-baseline/SKILL.md +38 -0
- package/native-skills/devops/secret-management/SKILL.md +40 -0
- package/native-skills/devops/terraform-state-and-modules/SKILL.md +36 -0
- package/native-skills/devops/zero-downtime-deploys/SKILL.md +40 -0
- package/native-skills/git/bisect-regression-hunt/SKILL.md +43 -0
- package/native-skills/git/branch-strategy/SKILL.md +39 -0
- package/native-skills/git/commit-hygiene/SKILL.md +41 -0
- package/native-skills/git/history-recovery/SKILL.md +44 -0
- package/native-skills/git/large-repository-surgery/SKILL.md +46 -0
- package/native-skills/git/merge-conflict-resolution/SKILL.md +42 -0
- package/native-skills/media/cinematic-3d-shot/SKILL.md +104 -0
- package/native-skills/mobile/app-startup-optimisation/SKILL.md +41 -0
- package/native-skills/mobile/cross-platform-tradeoffs/SKILL.md +42 -0
- package/native-skills/mobile/deep-linking/SKILL.md +39 -0
- package/native-skills/mobile/mobile-storage-security/SKILL.md +45 -0
- package/native-skills/mobile/offline-first-sync/SKILL.md +42 -0
- package/native-skills/mobile/push-notifications/SKILL.md +38 -0
- package/native-skills/performance/algorithmic-complexity/SKILL.md +43 -0
- package/native-skills/performance/concurrency-and-parallelism/SKILL.md +42 -0
- package/native-skills/performance/load-testing/SKILL.md +42 -0
- package/native-skills/performance/memory-and-leak-hunting/SKILL.md +41 -0
- package/native-skills/performance/network-and-payload-efficiency/SKILL.md +42 -0
- package/native-skills/performance/profiling-methodology/SKILL.md +46 -0
- package/native-skills/python/async-correctness/SKILL.md +40 -0
- package/native-skills/python/cli-tool-construction/SKILL.md +44 -0
- package/native-skills/python/data-processing-pipelines/SKILL.md +38 -0
- package/native-skills/python/environment-isolation/SKILL.md +41 -0
- package/native-skills/python/packaging-and-distribution/SKILL.md +56 -0
- package/native-skills/python/pytest-suite-design/SKILL.md +39 -0
- package/native-skills/python/type-hints-and-static-checking/SKILL.md +42 -0
- package/native-skills/security/authorization-model/SKILL.md +42 -0
- package/native-skills/security/crypto-usage/SKILL.md +48 -0
- package/native-skills/security/dependency-supply-chain/SKILL.md +42 -0
- package/native-skills/security/input-validation-and-injection/SKILL.md +44 -0
- package/native-skills/security/secure-file-uploads/SKILL.md +46 -0
- package/native-skills/security/security-headers-and-csp/SKILL.md +44 -0
- package/native-skills/security/threat-modelling/SKILL.md +46 -0
- package/native-skills/testing/end-to-end-test-stability/SKILL.md +42 -0
- package/native-skills/testing/property-based-testing/SKILL.md +46 -0
- package/native-skills/testing/regression-test-from-a-bug/SKILL.md +41 -0
- package/native-skills/testing/snapshot-and-visual-regression/SKILL.md +42 -0
- package/native-skills/testing/test-doubles-and-mocking/SKILL.md +43 -0
- package/native-skills/testing/test-strategy-and-pyramid/SKILL.md +39 -0
- package/native-skills/testing/writing-good-assertions/SKILL.md +42 -0
- package/native-skills/web/accessible-components/SKILL.md +45 -0
- package/native-skills/web/api-client-resilience/SKILL.md +39 -0
- package/native-skills/web/bundle-size-reduction/SKILL.md +38 -0
- package/native-skills/web/core-web-vitals/SKILL.md +40 -0
- package/native-skills/web/css-layout-systems/SKILL.md +41 -0
- package/native-skills/web/form-validation-ux/SKILL.md +39 -0
- package/native-skills/web/react-state-architecture/SKILL.md +41 -0
- package/native-skills/web/ssr-hydration-correctness/SKILL.md +45 -0
- package/package.json +68 -4
- package/src/agentLoop.js +2458 -0
- package/src/appChannel.js +109 -0
- package/src/approvalGrants.js +66 -0
- package/src/auditLog.js +52 -0
- package/src/autoUpdate.js +12 -0
- package/src/capabilities.js +36 -0
- package/src/checkpoints.js +1199 -0
- package/src/cliMode.js +638 -0
- package/src/commands.js +805 -0
- package/src/compiledSkills.js +540 -0
- package/src/config.js +130 -0
- package/src/contextManager.js +498 -0
- package/src/cortex/autoCondenser.js +192 -0
- package/src/cortex/provisioner.js +27 -0
- package/src/cortex/vectorDb.js +447 -0
- package/src/designPalettes.js +99 -0
- package/src/devMode.js +6 -0
- package/src/embodimentSession.js +180 -0
- package/src/entrypoint.js +194 -0
- package/src/execSafe.js +21 -0
- package/src/gitSnapshot.js +378 -0
- package/src/mcp/jsonRpc.js +165 -0
- package/src/mcp/mcpClient.js +136 -0
- package/src/mcp/mcpRegistry.js +207 -0
- package/src/memory.js +62 -0
- package/src/memoryManager.js +124 -0
- package/src/modelCatalogue.js +132 -0
- package/src/modelIds.js +42 -0
- package/src/modelRouter.js +145 -0
- package/src/outputStore.js +246 -0
- package/src/perception.js +124 -0
- package/src/planCache.js +437 -0
- package/src/procSignal.js +32 -0
- package/src/providers/anthropic.js +408 -0
- package/src/providers/base.js +328 -0
- package/src/providers/dashscope-byok.js +86 -0
- package/src/providers/deepseek.js +80 -0
- package/src/providers/factory.js +37 -0
- package/src/providers/gemini.js +346 -0
- package/src/providers/moonshot.js +73 -0
- package/src/providers/muse.js +169 -0
- package/src/providers/ollama.js +104 -0
- package/src/providers/openai-compatible.js +84 -0
- package/src/providers/openai.js +20 -0
- package/src/providers/qwen-byok.js +76 -0
- package/src/providers/xai.js +73 -0
- package/src/provisioning/capabilityConsent.js +24 -0
- package/src/provisioning/capabilityProvisioner.js +271 -0
- package/src/provisioning/capabilityTools.js +29 -0
- package/src/reporting.js +373 -0
- package/src/sandbox.js +529 -0
- package/src/sessionStore.js +279 -0
- package/src/skillsManager.js +288 -0
- package/src/slopDetector.js +307 -0
- package/src/taskBuffer.js +45 -0
- package/src/taskShape.js +241 -0
- package/src/textDiff.js +150 -0
- package/src/tokenTracker.js +560 -0
- package/src/tool-scripts/ax_snapshot.applescript +39 -0
- package/src/tool-scripts/ax_snapshot.ps1 +10 -0
- package/src/tool-scripts/desktop_act.applescript +58 -0
- package/src/tool-scripts/resolve_automator.py +119 -0
- package/src/toolPacks.js +524 -0
- package/src/tools/archive.js +63 -0
- package/src/tools/astryxOracle.js +78 -0
- package/src/tools/axSnapshot.js +165 -0
- package/src/tools/blenderAutomator.js +349 -0
- package/src/tools/browserAutomator.js +252 -0
- package/src/tools/checkSystemSpecs.js +64 -0
- package/src/tools/clipboard.js +36 -0
- package/src/tools/codeSearchAst.js +101 -0
- package/src/tools/codeSearcher.js +230 -0
- package/src/tools/codeTools.js +262 -0
- package/src/tools/codebaseSearch.js +75 -0
- package/src/tools/createSkill.js +129 -0
- package/src/tools/deferredMcpSearch.js +246 -0
- package/src/tools/desktopAct.js +208 -0
- package/src/tools/desktopOpen.js +34 -0
- package/src/tools/dynamicLoader.js +44 -0
- package/src/tools/executeProceduralMedia.js +93 -0
- package/src/tools/fsDelete.js +51 -0
- package/src/tools/fsEdit.js +105 -0
- package/src/tools/fsFind.js +128 -0
- package/src/tools/fsList.js +77 -0
- package/src/tools/fsMove.js +57 -0
- package/src/tools/fsRead.js +87 -0
- package/src/tools/fsSearch.js +198 -0
- package/src/tools/fsWrite.js +41 -0
- package/src/tools/gameAiBehaviorTree.js +208 -0
- package/src/tools/gameAnimationRig.js +194 -0
- package/src/tools/gameAssetFetcher.js +372 -0
- package/src/tools/gameAssetHarvester.js +210 -0
- package/src/tools/gameAudioSynth.js +267 -0
- package/src/tools/gameBuilder.js +886 -0
- package/src/tools/gameCanvasAudit.js +101 -0
- package/src/tools/gameCinematicSequencer.js +151 -0
- package/src/tools/gameDataForge.js +124 -0
- package/src/tools/gameDialogueTreeGen.js +166 -0
- package/src/tools/gameEconomyBalancer.js +161 -0
- package/src/tools/gameEntitySpawner.js +207 -0
- package/src/tools/gameLoreForge.js +153 -0
- package/src/tools/gameQuestArcBuilder.js +152 -0
- package/src/tools/gameSceneBuilder.js +158 -0
- package/src/tools/gameShaderForge.js +181 -0
- package/src/tools/gameTilemapGenerator.js +164 -0
- package/src/tools/gameUiHudBuilder.js +292 -0
- package/src/tools/gameVfxEmitterGen.js +108 -0
- package/src/tools/gameWebRuntimeInspector.js +91 -0
- package/src/tools/generateMediaViaApi.js +358 -0
- package/src/tools/gitExec.js +54 -0
- package/src/tools/godotAutomator.js +437 -0
- package/src/tools/manageLocalAiDependencies.js +220 -0
- package/src/tools/memoryWrite.js +99 -0
- package/src/tools/multiEdit.js +107 -0
- package/src/tools/palmierAutomator.js +104 -0
- package/src/tools/pdfReader.js +112 -0
- package/src/tools/persistentShell.js +148 -0
- package/src/tools/printer.js +316 -0
- package/src/tools/procTerm.js +91 -0
- package/src/tools/processList.js +52 -0
- package/src/tools/readNativeSkill.js +39 -0
- package/src/tools/registry.js +277 -0
- package/src/tools/resolveAutomator.js +102 -0
- package/src/tools/screenshot.js +75 -0
- package/src/tools/subagentOrchestrator.js +305 -0
- package/src/tools/sysRun.js +125 -0
- package/src/tools/todoWrite.js +75 -0
- package/src/tools/toolDispatch.js +127 -0
- package/src/tools/toolForge.js +124 -0
- package/src/tools/updatePlan.js +88 -0
- package/src/tools/viewImage.js +82 -0
- package/src/tools/visionInspector.js +154 -0
- package/src/tools/webFetch.js +74 -0
- package/src/tools/webSearch.js +82 -0
- package/src/tui/ask.js +164 -0
- package/src/tui/banner.js +158 -0
- package/src/tui/block.js +152 -0
- package/src/tui/chatRenderer.js +953 -0
- package/src/tui/completion.js +424 -0
- package/src/tui/dialogs.js +606 -0
- package/src/tui/diff.js +296 -0
- package/src/tui/history.js +135 -0
- package/src/tui/home.js +217 -0
- package/src/tui/invocation.js +105 -0
- package/src/tui/keys.js +443 -0
- package/src/tui/lineEditor.js +256 -0
- package/src/tui/overlay.js +256 -0
- package/src/tui/prompt.js +1258 -0
- package/src/tui/selectList.js +335 -0
- package/src/tui/stage.js +330 -0
- package/src/tui/stageSurface.js +609 -0
- package/src/tui/statusLine.js +254 -0
- package/src/tui/text.js +345 -0
- package/src/tui/theme.js +393 -0
- package/src/tui/toolDisplay.js +176 -0
- package/src/tui/transcript.js +235 -0
- package/src/tui/viewport.js +337 -0
- package/src/uncertainty.js +271 -0
- package/src/userSkills.js +265 -0
- package/src/utils/cli_ui.js +0 -0
- package/src/utils/inquirer.js +12 -0
- package/src/utils/paths.js +37 -0
- package/src/version.js +38 -0
- package/src/vfeTracker.js +220 -0
|
@@ -0,0 +1,56 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: packaging-and-distribution
|
|
3
|
+
description: Package a Python project for distribution or make it installable. Use when a project has setup.py only, when imports fail after install, or before publishing to an index.
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
# Packaging and Distribution
|
|
7
|
+
|
|
8
|
+
## The current shape
|
|
9
|
+
|
|
10
|
+
`pyproject.toml` with a PEP 621 `[project]` table. `setup.py` is legacy; `setup.cfg` is superseded. A new project should have neither.
|
|
11
|
+
|
|
12
|
+
## Workflow
|
|
13
|
+
|
|
14
|
+
1. Choose the layout.
|
|
15
|
+
- `src/` layout: the package lives in `src/mypkg/`. This is the correct default, because it makes it impossible to import the package from the working directory — so the tests exercise the INSTALLED package, which is what users get.
|
|
16
|
+
- Flat layout imports from the source tree and hides packaging bugs until release.
|
|
17
|
+
- **Done when:** the package is under `src/` and the tests still pass.
|
|
18
|
+
|
|
19
|
+
2. Declare the metadata once.
|
|
20
|
+
- Name, version, description, requires-python, dependencies, and the entry points. Read the version from a single source — either the package or the manifest — never both.
|
|
21
|
+
- **Done when:** `python -m build` produces a wheel and a source distribution without warnings.
|
|
22
|
+
|
|
23
|
+
3. Pin ranges, not exact versions, for a library.
|
|
24
|
+
- A library that pins exact versions makes itself uninstallable alongside anything else. An application may pin exactly, and should, via a lock file.
|
|
25
|
+
- **Done when:** the dependency specifiers match the kind of project.
|
|
26
|
+
|
|
27
|
+
4. Test the built artefact, not the source tree.
|
|
28
|
+
- Install the wheel into a clean environment and import it. A package that works in-tree and fails on install has a missing `package-data` entry or a wrong layout, and only this step catches it.
|
|
29
|
+
- **Done when:** the wheel installs and imports in a fresh environment.
|
|
30
|
+
|
|
31
|
+
5. Check what is inside the artefact.
|
|
32
|
+
- `python -m zipfile -l dist/*.whl`. Look for missing data files and for accidentally included test or secret files.
|
|
33
|
+
- **Done when:** the wheel contains exactly what should ship.
|
|
34
|
+
|
|
35
|
+
## Template
|
|
36
|
+
|
|
37
|
+
```toml
|
|
38
|
+
[build-system]
|
|
39
|
+
requires = ["hatchling"]
|
|
40
|
+
build-backend = "hatchling.build"
|
|
41
|
+
|
|
42
|
+
[project]
|
|
43
|
+
name = "mypkg"
|
|
44
|
+
version = "0.1.0"
|
|
45
|
+
requires-python = ">=3.11"
|
|
46
|
+
dependencies = ["httpx>=0.27,<1.0"]
|
|
47
|
+
|
|
48
|
+
[project.scripts]
|
|
49
|
+
mypkg = "mypkg.cli:main"
|
|
50
|
+
```
|
|
51
|
+
|
|
52
|
+
## Verification
|
|
53
|
+
|
|
54
|
+
- `python -m build` succeeds with no warnings.
|
|
55
|
+
- The wheel installs into a clean environment and the entry point runs.
|
|
56
|
+
- The archive listing contains no test files and no credentials.
|
|
@@ -0,0 +1,39 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: pytest-suite-design
|
|
3
|
+
description: Write or restructure Python tests. Use when tests share mutable state, when a suite is slow, or when failures are hard to read.
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
# Pytest Suite Design
|
|
7
|
+
|
|
8
|
+
## Workflow
|
|
9
|
+
|
|
10
|
+
1. Make each test independent.
|
|
11
|
+
- A test that depends on another having run first will fail under `-p no:randomly`, under `-n auto`, and in any order but the current one. Shared module-level state is the usual cause.
|
|
12
|
+
- **Done when:** the suite passes with tests shuffled and with parallel execution.
|
|
13
|
+
|
|
14
|
+
2. Use fixtures for setup, not for cleverness.
|
|
15
|
+
- A fixture should produce one thing and clean it up with `yield`. Scope it as narrowly as correctness allows: `function` by default, wider only when construction is genuinely expensive AND the object is immutable.
|
|
16
|
+
- A mutable `session`-scoped fixture is shared state with extra steps.
|
|
17
|
+
- **Done when:** no fixture wider than `function` scope is mutated by a test.
|
|
18
|
+
|
|
19
|
+
3. Parametrise instead of duplicating.
|
|
20
|
+
- `@pytest.mark.parametrize` turns eight near-identical tests into one with eight cases, each reported separately. Give the cases `ids` so a failure names the case rather than an index.
|
|
21
|
+
- **Done when:** no two tests differ only by literal values.
|
|
22
|
+
|
|
23
|
+
4. Assert on behaviour, not implementation.
|
|
24
|
+
- Assert what the function returns and what it does to the world. Asserting that a private helper was called with particular arguments locks the implementation and makes every refactor a test rewrite.
|
|
25
|
+
- **Done when:** the tests still pass after an internal refactor that preserves behaviour.
|
|
26
|
+
|
|
27
|
+
5. Isolate the outside world.
|
|
28
|
+
- No test touches the real network, the real clock, or a real database unless it is explicitly an integration test and marked as one. `monkeypatch`, `freezegun`, and `responses` cover most of it.
|
|
29
|
+
- **Done when:** the unit suite passes with the network disabled.
|
|
30
|
+
|
|
31
|
+
6. Make failures self-explanatory.
|
|
32
|
+
- One logical assertion per test, and a name that states the expected behaviour: `test_expired_token_is_rejected`, not `test_auth_2`.
|
|
33
|
+
- **Done when:** a failure name alone communicates what broke.
|
|
34
|
+
|
|
35
|
+
## Verification
|
|
36
|
+
|
|
37
|
+
- `pytest -p no:randomly --randomly-seed=last` and `pytest -n auto` both pass.
|
|
38
|
+
- The unit suite passes with no network access.
|
|
39
|
+
- Every test name reads as a statement about behaviour.
|
|
@@ -0,0 +1,42 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: type-hints-and-static-checking
|
|
3
|
+
description: Add or fix type annotations in Python. Use when a codebase has no hints, when a type checker reports many errors, or when a function signature is ambiguous.
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
# Type Hints and Static Checking
|
|
7
|
+
|
|
8
|
+
## Where annotations pay
|
|
9
|
+
|
|
10
|
+
Public function signatures and data structures. Annotating every local variable adds noise without adding information — the checker infers those.
|
|
11
|
+
|
|
12
|
+
## Workflow
|
|
13
|
+
|
|
14
|
+
1. Turn the checker on before writing annotations.
|
|
15
|
+
- `mypy` or `pyright`. Start non-strict on an existing codebase; a first run in strict mode on untyped code produces thousands of errors and gets switched off.
|
|
16
|
+
- **Done when:** the checker runs and its current error count is recorded.
|
|
17
|
+
|
|
18
|
+
2. Annotate boundaries first.
|
|
19
|
+
- Public functions, class attributes, and anything crossing a module edge. This is where a wrong assumption actually costs something.
|
|
20
|
+
- **Done when:** every public function has annotated parameters and a return type.
|
|
21
|
+
|
|
22
|
+
3. Use the modern syntax.
|
|
23
|
+
- `list[str]` not `List[str]`. `str | None` not `Optional[str]`. `dict[str, int]` not `Dict[str, int]`. The `typing` aliases are deprecated.
|
|
24
|
+
- **Done when:** no import from `typing` exists that a builtin generic replaces.
|
|
25
|
+
|
|
26
|
+
4. Never reach for `Any` to silence an error.
|
|
27
|
+
- `Any` disables checking for everything it touches and spreads. If the type is genuinely unknown, `object` is honest and forces a narrowing check at the point of use.
|
|
28
|
+
- **Done when:** every remaining `Any` has a comment saying why.
|
|
29
|
+
|
|
30
|
+
5. Ratchet strictness upward.
|
|
31
|
+
- Enable one strict flag at a time and fix the errors it produces. `disallow_untyped_defs` module by module via per-module settings is the usual path.
|
|
32
|
+
- **Done when:** the error count is zero at the current strictness and one more flag is enabled.
|
|
33
|
+
|
|
34
|
+
6. Type the tricky shapes properly.
|
|
35
|
+
- `TypedDict` for a dictionary with known keys, `Protocol` for structural typing instead of an abstract base class, `Literal` for a fixed set of strings, and `TypeVar` for a function whose return type depends on its input.
|
|
36
|
+
- **Done when:** no function returns a bare `dict` where a `TypedDict` describes it.
|
|
37
|
+
|
|
38
|
+
## Verification
|
|
39
|
+
|
|
40
|
+
- The type checker reports zero errors at the configured strictness.
|
|
41
|
+
- CI runs the checker and fails on a new error.
|
|
42
|
+
- No new `Any` was introduced without a justification.
|
|
@@ -0,0 +1,42 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: authorization-model
|
|
3
|
+
description: Decide who may do what. Use when adding permissions, when a user can access another user`s data, or when authorization checks are scattered across handlers.
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
# Authorization Model
|
|
7
|
+
|
|
8
|
+
## The distinction that gets missed
|
|
9
|
+
|
|
10
|
+
Authentication is who you are. Authorization is what you may do. A system with excellent authentication and no authorization lets any logged-in user read every record — and that class of bug (broken object-level authorization) is consistently the most common serious web vulnerability.
|
|
11
|
+
|
|
12
|
+
## Workflow
|
|
13
|
+
|
|
14
|
+
1. Check ownership on every single object access.
|
|
15
|
+
- Not just on the list endpoint. `GET /orders/123` must verify that order 123 belongs to the caller. Relying on the UI not to show the link is not a control — the identifier is guessable and the endpoint is public.
|
|
16
|
+
- **Done when:** every by-identifier endpoint verifies the caller's relationship to that object.
|
|
17
|
+
|
|
18
|
+
2. Deny by default.
|
|
19
|
+
- The absence of a rule means no. A system where an unmatched route falls through to allow will leak on the first forgotten annotation.
|
|
20
|
+
- **Done when:** a new endpoint with no rule is inaccessible.
|
|
21
|
+
|
|
22
|
+
3. Centralise the decision, distribute the enforcement.
|
|
23
|
+
- One function answers "may this subject perform this action on this object". Handlers call it; they do not reimplement it. Scattered inline checks drift, and the one that drifts is the hole.
|
|
24
|
+
- **Done when:** there is one authorization entry point and every handler uses it.
|
|
25
|
+
|
|
26
|
+
4. Enforce at the data layer where you can.
|
|
27
|
+
- A query scoped to the caller's tenant by construction cannot leak, even from a handler that forgot to check. Row-level security or a mandatory scope on the repository turns a class of bug into an impossibility.
|
|
28
|
+
- **Done when:** the default query path cannot return another tenant's rows.
|
|
29
|
+
|
|
30
|
+
5. Model roles as permissions, not as titles.
|
|
31
|
+
- Check `may_refund_order`, not `is_admin`. Title checks scattered through the code make adding a role a codebase-wide change.
|
|
32
|
+
- **Done when:** no branch tests a role name directly.
|
|
33
|
+
|
|
34
|
+
6. Log the denials.
|
|
35
|
+
- A spike in denials is either a bug or an attack. Both are worth seeing.
|
|
36
|
+
- **Done when:** denials are logged with subject, action and object.
|
|
37
|
+
|
|
38
|
+
## Verification
|
|
39
|
+
|
|
40
|
+
- A test attempts to read another user's object by identifier and receives 404 or 403.
|
|
41
|
+
- A new endpoint without a rule is denied.
|
|
42
|
+
- The default repository query is tenant-scoped.
|
|
@@ -0,0 +1,48 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: crypto-usage
|
|
3
|
+
description: Encrypt, hash, sign, or generate randomness. Use whenever cryptography is being written or reviewed.
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
# Cryptography Usage
|
|
7
|
+
|
|
8
|
+
## The rule
|
|
9
|
+
|
|
10
|
+
Do not build it. Use a vetted library at the highest level of abstraction available. Cryptographic code fails silently — it produces output that looks correct and provides no security — so "it works" tells you nothing.
|
|
11
|
+
|
|
12
|
+
## The correct primitives
|
|
13
|
+
|
|
14
|
+
- **Password storage** — argon2id, or bcrypt. Never a general-purpose hash: they are fast by design, which is exactly wrong here.
|
|
15
|
+
- **Symmetric encryption** — an authenticated mode: AES-GCM or XChaCha20-Poly1305. Unauthenticated encryption is malleable, and the attack is practical.
|
|
16
|
+
- **Random values** — the platform's cryptographically secure generator. Never a general-purpose random function for a token, an identifier, or a nonce.
|
|
17
|
+
- **Comparing secrets** — constant-time comparison. A normal comparison returns early on the first differing byte and leaks the secret through timing.
|
|
18
|
+
- **Signing** — Ed25519, or ECDSA on P-256. Verify with the algorithm you expect, never with the one named in the token.
|
|
19
|
+
- **Key derivation** — HKDF from a high-entropy secret, argon2 or scrypt from a password.
|
|
20
|
+
|
|
21
|
+
## Workflow
|
|
22
|
+
|
|
23
|
+
1. State what you are protecting against.
|
|
24
|
+
- Encryption at rest protects a stolen disk, not a compromised application. Naming the threat prevents solving the wrong problem.
|
|
25
|
+
- **Done when:** the threat and what the mechanism does not cover are both written down.
|
|
26
|
+
|
|
27
|
+
2. Never reuse a nonce with a key.
|
|
28
|
+
- Nonce reuse in GCM is catastrophic — it reveals the plaintext relationship and the authentication key. Generate randomly, or use a counter that provably never repeats. XChaCha20's larger nonce makes random generation safe.
|
|
29
|
+
- **Done when:** nonce generation is shown never to repeat for a given key.
|
|
30
|
+
|
|
31
|
+
3. Plan key rotation from the start.
|
|
32
|
+
- Store a key identifier with the ciphertext so old data stays readable while new data uses a new key. Retrofitting this requires decrypting everything.
|
|
33
|
+
- **Done when:** ciphertext carries a key identifier.
|
|
34
|
+
|
|
35
|
+
4. Keep keys out of the application.
|
|
36
|
+
- A key management service or the platform keystore. A key in the environment is a key in every crash dump, log aggregator, and process listing.
|
|
37
|
+
- **Done when:** no key material appears in source, config, or environment.
|
|
38
|
+
|
|
39
|
+
5. Verify before you use.
|
|
40
|
+
- Check the authentication tag or signature BEFORE decrypting or parsing. Acting on unverified data is the whole vulnerability.
|
|
41
|
+
- **Done when:** every verification precedes every use.
|
|
42
|
+
|
|
43
|
+
## Verification
|
|
44
|
+
|
|
45
|
+
- No hand-rolled primitive exists.
|
|
46
|
+
- Secret comparison is constant-time.
|
|
47
|
+
- Nonce generation cannot repeat.
|
|
48
|
+
- Keys are external to the application and rotatable.
|
|
@@ -0,0 +1,42 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: dependency-supply-chain
|
|
3
|
+
description: Add, update, or audit dependencies. Use when introducing a package, when an advisory appears, or when reviewing what a project pulls in.
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
# Dependency Supply Chain
|
|
7
|
+
|
|
8
|
+
## The exposure
|
|
9
|
+
|
|
10
|
+
A dependency runs with your process's full privileges. Its dependencies do too, and a typical project has hundreds it never chose. Most real compromises come through this path rather than through your own code.
|
|
11
|
+
|
|
12
|
+
## Workflow
|
|
13
|
+
|
|
14
|
+
1. Justify each direct dependency before adding it.
|
|
15
|
+
- What does it do, how much does it weigh, how many transitive dependencies does it bring, when was it last released, how many maintainers. A one-function package with forty dependencies is a bad trade.
|
|
16
|
+
- **Done when:** the addition is justified against writing it yourself.
|
|
17
|
+
|
|
18
|
+
2. Check the package is the one you mean.
|
|
19
|
+
- Typosquatting relies on a plausible name. Confirm the exact name, the repository link, and the download count. A brand-new package with a familiar name is the classic attack.
|
|
20
|
+
- **Done when:** the package's repository and history are confirmed.
|
|
21
|
+
|
|
22
|
+
3. Commit the lock file and install from it.
|
|
23
|
+
- `npm ci`, `pip install -r` with hashes, `poetry install`, `cargo build --locked`. An install that resolves fresh versions in CI means the artefact you tested is not the artefact you shipped.
|
|
24
|
+
- **Done when:** CI installs from the lock file and fails if it is out of date.
|
|
25
|
+
|
|
26
|
+
4. Refuse install scripts you have not read.
|
|
27
|
+
- A lifecycle script runs arbitrary code at install time, in your CI, with your credentials. Where the ecosystem allows disabling them by default, do.
|
|
28
|
+
- **Done when:** the install-time script surface is known and minimised.
|
|
29
|
+
|
|
30
|
+
5. Automate advisory scanning, and triage rather than ignore.
|
|
31
|
+
- Run the ecosystem audit in CI. For each finding, decide: upgrade, replace, or accept with a written reason and an expiry. A permanently ignored advisory is a decision nobody revisits.
|
|
32
|
+
- **Done when:** every open advisory has an action or a dated acceptance.
|
|
33
|
+
|
|
34
|
+
6. Pin what you cannot verify.
|
|
35
|
+
- Actions and containers by digest, not by tag. A mutable tag can be repointed after review.
|
|
36
|
+
- **Done when:** CI actions and base images are pinned to digests.
|
|
37
|
+
|
|
38
|
+
## Verification
|
|
39
|
+
|
|
40
|
+
- The lock file is committed and CI installs from it.
|
|
41
|
+
- The audit runs in CI and has no unreviewed findings.
|
|
42
|
+
- Every CI action is pinned by digest.
|
|
@@ -0,0 +1,44 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: input-validation-and-injection
|
|
3
|
+
description: Handle data that came from a user, a request, or a third party. Use when building a query, a command, a template, or a path from external input.
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
# Input Validation and Injection
|
|
7
|
+
|
|
8
|
+
## The single principle
|
|
9
|
+
|
|
10
|
+
Never build a command out of a string containing untrusted data. Every injection class — SQL, shell, template, path, header, log — is the same mistake: data crossing into a position where it is read as instructions.
|
|
11
|
+
|
|
12
|
+
## By interpreter
|
|
13
|
+
|
|
14
|
+
- **SQL** — parameterised queries only. Bind the value; never interpolate. Identifiers that must be dynamic come from an allowlist, because they cannot be bound.
|
|
15
|
+
- **Shell** — pass an argument array to the spawn API, never a joined string, and never through a shell. Where a shell is unavoidable, an allowlist of complete commands is the only safe form.
|
|
16
|
+
- **Path** — resolve, then verify the result is inside the permitted root. `../` traversal survives naive prefix checks and survives decoding twice.
|
|
17
|
+
- **HTML** — contextual escaping, applied by the template engine. Never build markup by concatenation. Untrusted markup goes through a sanitiser with an allowlist.
|
|
18
|
+
- **Headers and logs** — strip carriage returns and newlines. A newline in a header splits the response; in a log it forges an entry.
|
|
19
|
+
- **Deserialisation** — never deserialise untrusted data into arbitrary types. Parse into a declared schema.
|
|
20
|
+
|
|
21
|
+
## Workflow
|
|
22
|
+
|
|
23
|
+
1. Map every entry point.
|
|
24
|
+
- Request bodies, query strings, headers, cookies, uploads, webhooks, environment, third-party API responses, and data read back from your own database that was written by a user.
|
|
25
|
+
- **Done when:** every source of external data is listed.
|
|
26
|
+
|
|
27
|
+
2. Validate on the way in, against a schema.
|
|
28
|
+
- Type, range, length, format, and an allowlist where the set is finite. Reject rather than sanitise: sanitising guesses at intent and is bypassable.
|
|
29
|
+
- **Done when:** every entry point parses into a validated type before use.
|
|
30
|
+
|
|
31
|
+
3. Escape on the way out, per context.
|
|
32
|
+
- The same value needs different escaping in HTML, in an attribute, in JavaScript, in a URL. One "escape" function applied everywhere is wrong in at least three of those.
|
|
33
|
+
- **Done when:** escaping is applied by the sink, matched to the sink.
|
|
34
|
+
|
|
35
|
+
4. Enforce size limits everywhere.
|
|
36
|
+
- Body size, field length, array length, upload size, and nesting depth. Without them, one request exhausts memory.
|
|
37
|
+
- **Done when:** every unbounded input has a stated limit.
|
|
38
|
+
|
|
39
|
+
## Verification
|
|
40
|
+
|
|
41
|
+
- No query, command, or path is built by string concatenation with external data.
|
|
42
|
+
- Traversal payloads are rejected.
|
|
43
|
+
- A newline in a header value does not split the response.
|
|
44
|
+
- Oversized input is rejected before allocation.
|
|
@@ -0,0 +1,46 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: secure-file-uploads
|
|
3
|
+
description: Accept file uploads from users. Use when adding an upload feature, or when uploaded files are served back to users.
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
# Secure File Uploads
|
|
7
|
+
|
|
8
|
+
## Why this is its own topic
|
|
9
|
+
|
|
10
|
+
An upload endpoint lets a stranger place a file of their choosing on your infrastructure. The failure modes span storage exhaustion, remote code execution, stored cross-site scripting, and server-side request forgery — and the naive implementation hits several of them.
|
|
11
|
+
|
|
12
|
+
## Workflow
|
|
13
|
+
|
|
14
|
+
1. Limit size before reading.
|
|
15
|
+
- Enforce at the reverse proxy AND in the application. Streaming a body into memory to check its length is already the denial of service.
|
|
16
|
+
- **Done when:** an oversized upload is rejected before the body is buffered.
|
|
17
|
+
|
|
18
|
+
2. Determine the type from the content, not the name.
|
|
19
|
+
- Read the magic bytes. The extension and the declared content type are both attacker-controlled. Validate against an allowlist of types you actually support — never a denylist.
|
|
20
|
+
- **Done when:** a file renamed to a permitted extension is rejected on its real content.
|
|
21
|
+
|
|
22
|
+
3. Never use the supplied filename.
|
|
23
|
+
- Generate your own identifier and store the original name only as a display label. The supplied name can contain traversal sequences, null bytes, or a name that means something to the filesystem.
|
|
24
|
+
- **Done when:** the stored path is derived entirely from server-generated values.
|
|
25
|
+
|
|
26
|
+
4. Store outside the web root, and serve deliberately.
|
|
27
|
+
- Files in a served directory can be requested directly, which turns an uploaded script into an executed one. Store in object storage or outside the root, and serve through a handler that sets the content type and `Content-Disposition: attachment`, plus `X-Content-Type-Options: nosniff`.
|
|
28
|
+
- **Done when:** an uploaded HTML file downloads rather than renders.
|
|
29
|
+
|
|
30
|
+
5. Serve user content from a separate origin.
|
|
31
|
+
- Same-origin user content means a served file can script against your session. A distinct domain contains it.
|
|
32
|
+
- **Done when:** user uploads are served from a domain that holds no session.
|
|
33
|
+
|
|
34
|
+
6. Process images out-of-process and re-encode.
|
|
35
|
+
- Image parsers are a large attack surface. Re-encoding strips embedded payloads and metadata, including location data users did not intend to share.
|
|
36
|
+
- **Done when:** stored images are re-encoded and stripped of metadata.
|
|
37
|
+
|
|
38
|
+
7. Scan, and quarantine until clean.
|
|
39
|
+
- **Done when:** a file is unreachable until it has passed scanning.
|
|
40
|
+
|
|
41
|
+
## Verification
|
|
42
|
+
|
|
43
|
+
- An oversized upload is rejected early.
|
|
44
|
+
- A renamed executable is rejected on content inspection.
|
|
45
|
+
- An uploaded HTML file downloads rather than executes.
|
|
46
|
+
- Uploaded images carry no location metadata.
|
|
@@ -0,0 +1,44 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: security-headers-and-csp
|
|
3
|
+
description: Configure HTTP security headers or a Content Security Policy. Use when hardening a web application, or when a security scan reports missing headers.
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
# Security Headers and CSP
|
|
7
|
+
|
|
8
|
+
## The headers that matter, in order of value
|
|
9
|
+
|
|
10
|
+
1. **Content-Security-Policy** — the only one that meaningfully mitigates cross-site scripting. Also the only one requiring real work.
|
|
11
|
+
2. **Strict-Transport-Security** — forces https for a long max-age. Include subdomains once you are sure.
|
|
12
|
+
3. **X-Content-Type-Options: nosniff** — stops the browser guessing a content type, which is how an uploaded text file becomes a script.
|
|
13
|
+
4. **Referrer-Policy: strict-origin-when-cross-origin** — stops paths and query strings leaking to third parties.
|
|
14
|
+
5. **X-Frame-Options / frame-ancestors** — clickjacking. Prefer the CSP directive.
|
|
15
|
+
6. **Permissions-Policy** — deny camera, microphone and geolocation unless used.
|
|
16
|
+
|
|
17
|
+
## Workflow
|
|
18
|
+
|
|
19
|
+
1. Deploy CSP in report-only mode first.
|
|
20
|
+
- A strict policy applied directly will break the site. Report-only collects violations without enforcing, and the report tells you what the site actually depends on.
|
|
21
|
+
- **Done when:** a week of reports has been collected and reviewed.
|
|
22
|
+
|
|
23
|
+
2. Build the policy from a nonce, not from an allowlist.
|
|
24
|
+
- Host allowlists are routinely bypassable through a permitted CDN. A per-response nonce with `strict-dynamic` is the current guidance and is far more robust.
|
|
25
|
+
- **Done when:** the policy uses a per-response nonce and no host wildcard.
|
|
26
|
+
|
|
27
|
+
3. Remove what the policy forbids, rather than weakening the policy.
|
|
28
|
+
- Inline handlers and inline styles are what force `unsafe-inline`, and `unsafe-inline` removes most of the protection. Move them to files or nonce them.
|
|
29
|
+
- **Done when:** the policy contains no `unsafe-inline` and no `unsafe-eval`.
|
|
30
|
+
|
|
31
|
+
4. Keep a reporting endpoint after enforcement.
|
|
32
|
+
- Violations in production are either a bug you introduced or an injection attempt. Both are worth seeing.
|
|
33
|
+
- **Done when:** violations continue to be collected after enforcement begins.
|
|
34
|
+
|
|
35
|
+
5. Set the headers at one layer.
|
|
36
|
+
- The reverse proxy or a single middleware. Headers set in several places conflict, and the browser takes the most restrictive — usually not the one you were reading.
|
|
37
|
+
- **Done when:** each header is emitted exactly once.
|
|
38
|
+
|
|
39
|
+
## Verification
|
|
40
|
+
|
|
41
|
+
- An observatory or header scan reports no missing header.
|
|
42
|
+
- The CSP contains no `unsafe-inline`, no `unsafe-eval`, and no wildcard host.
|
|
43
|
+
- An injected inline script is blocked and reported.
|
|
44
|
+
- Every response carries each header exactly once.
|
|
@@ -0,0 +1,46 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: threat-modelling
|
|
3
|
+
description: Assess the security of a feature or system. Use before building anything that handles money, personal data, or authentication, and during a security review.
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
# Threat Modelling
|
|
7
|
+
|
|
8
|
+
## The four questions
|
|
9
|
+
|
|
10
|
+
What are we building, what can go wrong, what will we do about it, and did we do a good job. Everything else is elaboration.
|
|
11
|
+
|
|
12
|
+
## Workflow
|
|
13
|
+
|
|
14
|
+
1. Draw the data flow.
|
|
15
|
+
- Components, data stores, external entities, and the trust boundaries between them. A boundary is anywhere data crosses from less trusted to more trusted — that is where controls belong and where they are usually missing.
|
|
16
|
+
- **Done when:** every trust boundary in the feature is marked.
|
|
17
|
+
|
|
18
|
+
2. Enumerate threats per boundary.
|
|
19
|
+
- Walk the categories: spoofing identity, tampering with data, repudiation, information disclosure, denial of service, elevation of privilege. Apply each to each boundary rather than brainstorming freely — free brainstorming misses whole categories.
|
|
20
|
+
- **Done when:** every boundary has been walked through all six.
|
|
21
|
+
|
|
22
|
+
3. Rank by realistic impact and likelihood.
|
|
23
|
+
- Not everything gets fixed. Say which are accepted and why, and record the acceptance.
|
|
24
|
+
- **Done when:** each threat is mitigated, transferred, or accepted in writing.
|
|
25
|
+
|
|
26
|
+
4. Name a specific control per mitigated threat.
|
|
27
|
+
- "Validate input" is not a control. "Reject requests whose body exceeds 1MB at the reverse proxy" is.
|
|
28
|
+
- **Done when:** every mitigation names a concrete, testable mechanism.
|
|
29
|
+
|
|
30
|
+
5. Test the controls.
|
|
31
|
+
- A control nobody attempted to bypass is a hypothesis. Write the test that tries.
|
|
32
|
+
- **Done when:** each control has a test attempting to defeat it.
|
|
33
|
+
|
|
34
|
+
## The threats people forget
|
|
35
|
+
|
|
36
|
+
- The admin interface, which usually has weaker controls than the user-facing one.
|
|
37
|
+
- The background job, which often runs with elevated privilege and no authorization check.
|
|
38
|
+
- The error path, which frequently leaks internal detail.
|
|
39
|
+
- The third-party integration, which is trusted implicitly.
|
|
40
|
+
- The recovery flow, which is frequently weaker than the login it can bypass.
|
|
41
|
+
|
|
42
|
+
## Verification
|
|
43
|
+
|
|
44
|
+
- A diagram exists with boundaries marked.
|
|
45
|
+
- Every boundary has been walked through all six categories.
|
|
46
|
+
- Every mitigation names a mechanism and has a test.
|
|
@@ -0,0 +1,42 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: end-to-end-test-stability
|
|
3
|
+
description: Write or fix browser or device end-to-end tests. Use when tests fail intermittently, when they rely on sleeps, or when a suite is disabled because it is unreliable.
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
# End-to-End Test Stability
|
|
7
|
+
|
|
8
|
+
## Where flakiness comes from
|
|
9
|
+
|
|
10
|
+
Almost always timing: the test acts before the application is ready. Fixed sleeps do not fix this — they make it rarer and slower, which is the worst of both.
|
|
11
|
+
|
|
12
|
+
## Workflow
|
|
13
|
+
|
|
14
|
+
1. Replace every sleep with a condition.
|
|
15
|
+
- Wait for the element to be visible and enabled, for the request to complete, for the loading state to disappear. Modern frameworks auto-wait on assertions; use them rather than hand-rolled polling.
|
|
16
|
+
- **Done when:** the suite contains no fixed-duration sleep.
|
|
17
|
+
|
|
18
|
+
2. Select by what the user perceives.
|
|
19
|
+
- Role and accessible name, then visible text, then a dedicated test identifier. Never a CSS class or an XPath through the DOM structure — those break on every styling change and test nothing meaningful.
|
|
20
|
+
- **Done when:** no selector depends on styling or DOM nesting.
|
|
21
|
+
|
|
22
|
+
3. Isolate each test's data.
|
|
23
|
+
- Every test creates what it needs with unique values and cleans up. Shared fixtures mean tests interfere and cannot run in parallel — and the failure appears in whichever test ran second.
|
|
24
|
+
- **Done when:** the suite passes with tests run in a random order and in parallel.
|
|
25
|
+
|
|
26
|
+
4. Control the network.
|
|
27
|
+
- For anything but a deliberate integration test, intercept and stub external calls. A third party's rate limit or outage should not fail your build.
|
|
28
|
+
- **Done when:** the suite passes with external network access blocked.
|
|
29
|
+
|
|
30
|
+
5. Capture evidence on failure.
|
|
31
|
+
- Screenshot, DOM snapshot, video, and network log. A CI failure with no artefact costs an hour of local reproduction, and some failures do not reproduce locally at all.
|
|
32
|
+
- **Done when:** a failing run leaves artefacts sufficient to diagnose without re-running.
|
|
33
|
+
|
|
34
|
+
6. Keep the suite small and critical.
|
|
35
|
+
- Cover the handful of journeys whose breakage would be an emergency. An end-to-end test for every form is a suite nobody waits for.
|
|
36
|
+
- **Done when:** every end-to-end test covers a journey that would be an incident if broken.
|
|
37
|
+
|
|
38
|
+
## Verification
|
|
39
|
+
|
|
40
|
+
- Twenty consecutive runs pass without a retry.
|
|
41
|
+
- The suite passes in a random order and in parallel.
|
|
42
|
+
- A deliberate failure leaves a screenshot and a network log.
|
|
@@ -0,0 +1,46 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: property-based-testing
|
|
3
|
+
description: Test a function with many valid inputs. Use when example tests keep missing edge cases, when testing a parser, serialiser, or algorithm, or when input space is large.
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
# Property-Based Testing
|
|
7
|
+
|
|
8
|
+
## The shift
|
|
9
|
+
|
|
10
|
+
Instead of "for this input, expect this output", state a property that must hold for EVERY input, and let the framework generate hundreds of them — including the pathological ones nobody thinks to write: empty, one element, duplicates, maximum integers, unusual Unicode.
|
|
11
|
+
|
|
12
|
+
## The properties worth reaching for
|
|
13
|
+
|
|
14
|
+
- **Round trip** — `decode(encode(x)) == x`. The single most valuable property, and it applies to every serialiser, parser and codec.
|
|
15
|
+
- **Invariant** — sorting preserves length and multiset; a transfer preserves the total balance.
|
|
16
|
+
- **Idempotence** — `f(f(x)) == f(x)`. Normalisers, formatters, deduplicators.
|
|
17
|
+
- **Oracle** — the fast implementation agrees with the obviously correct slow one.
|
|
18
|
+
- **Commutativity or ordering** — order of independent operations does not change the result.
|
|
19
|
+
|
|
20
|
+
## Workflow
|
|
21
|
+
|
|
22
|
+
1. Pick the property before the generator.
|
|
23
|
+
- Write the invariant as a sentence first. A generator without a property is just random input.
|
|
24
|
+
- **Done when:** the property is stated in one sentence.
|
|
25
|
+
|
|
26
|
+
2. Generate the real input domain.
|
|
27
|
+
- Constrain the generator to what the function actually accepts, but no further. Over-constraining hides the bugs; the framework's default string generator producing surrogate pairs and combining characters is a feature, not a nuisance.
|
|
28
|
+
- **Done when:** the generator covers the documented domain and nothing outside it.
|
|
29
|
+
|
|
30
|
+
3. Use the shrunk counterexample.
|
|
31
|
+
- On failure the framework reduces the input to a minimal failing case. That minimal case is usually the whole diagnosis.
|
|
32
|
+
- **Done when:** the reported failure is minimal and reproducible from its seed.
|
|
33
|
+
|
|
34
|
+
4. Keep the failing case as a regression test.
|
|
35
|
+
- Add the specific counterexample as an ordinary example test. Property tests are randomised; a pinned example guarantees this exact bug stays fixed.
|
|
36
|
+
- **Done when:** each discovered failure has a pinned example test.
|
|
37
|
+
|
|
38
|
+
5. Combine, do not replace.
|
|
39
|
+
- Properties catch classes of bugs; examples document intent and pin known cases. A suite needs both.
|
|
40
|
+
- **Done when:** the module has both property and example tests.
|
|
41
|
+
|
|
42
|
+
## Verification
|
|
43
|
+
|
|
44
|
+
- Each property runs several hundred cases in CI.
|
|
45
|
+
- A deliberately introduced off-by-one is caught.
|
|
46
|
+
- Every past failure has a pinned regression test.
|
|
@@ -0,0 +1,41 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: regression-test-from-a-bug
|
|
3
|
+
description: Fix a reported bug. Use whenever a defect is being fixed, before writing the fix itself.
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
# Regression Test from a Bug
|
|
7
|
+
|
|
8
|
+
## The order
|
|
9
|
+
|
|
10
|
+
Test first, then fix. A test written after the fix passes immediately, and you never learn whether it would have caught the bug. This is the single highest-value testing discipline and the most commonly skipped.
|
|
11
|
+
|
|
12
|
+
## Workflow
|
|
13
|
+
|
|
14
|
+
1. Reproduce it before you understand it.
|
|
15
|
+
- Get the exact inputs, state and steps. A bug you cannot reproduce cannot be verified as fixed, and "I could not reproduce it" is not a resolution.
|
|
16
|
+
- **Done when:** you can trigger the bug on demand.
|
|
17
|
+
|
|
18
|
+
2. Write a failing test at the lowest level that captures it.
|
|
19
|
+
- Reduce the reproduction to the smallest unit that still fails. A bug reproduced through the whole UI is usually a unit-level bug once narrowed.
|
|
20
|
+
- **Done when:** the test fails, and it fails with the same symptom the user reported.
|
|
21
|
+
|
|
22
|
+
3. Understand the cause before changing anything.
|
|
23
|
+
- The first plausible explanation is often wrong. Confirm it by predicting a second observable consequence and checking for it.
|
|
24
|
+
- **Done when:** you can explain why the code produced this exact wrong result.
|
|
25
|
+
|
|
26
|
+
4. Fix the cause, not the symptom.
|
|
27
|
+
- A null check that suppresses a crash leaves the wrong value flowing through the system. Ask where the null came from.
|
|
28
|
+
- **Done when:** the fix addresses where the wrong value originated.
|
|
29
|
+
|
|
30
|
+
5. Look for siblings.
|
|
31
|
+
- The same mistake is usually made in more than one place. Search for the pattern before closing.
|
|
32
|
+
- **Done when:** the codebase has been searched for the same shape and the results addressed.
|
|
33
|
+
|
|
34
|
+
6. Confirm the test now passes and the rest still does.
|
|
35
|
+
- **Done when:** the new test is green, the suite is green, and reverting the fix turns the new test red.
|
|
36
|
+
|
|
37
|
+
## Verification
|
|
38
|
+
|
|
39
|
+
- The test failed before the fix and passes after.
|
|
40
|
+
- Reverting the fix makes it fail again.
|
|
41
|
+
- A search for the same pattern elsewhere has been done and reported.
|
|
@@ -0,0 +1,42 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: snapshot-and-visual-regression
|
|
3
|
+
description: Use snapshot or visual regression tests. Use when UI changes break silently, when snapshots are updated without review, or when a large snapshot file has appeared.
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
# Snapshot and Visual Regression Testing
|
|
7
|
+
|
|
8
|
+
## The failure mode of snapshots
|
|
9
|
+
|
|
10
|
+
They are trivial to update. When a snapshot fails, the fastest action is to accept the new one, and that becomes reflexive — at which point the snapshot records what the code does rather than what it should do, and catches nothing.
|
|
11
|
+
|
|
12
|
+
## Workflow
|
|
13
|
+
|
|
14
|
+
1. Snapshot small, meaningful units.
|
|
15
|
+
- A component's rendered output, a serialised payload, a formatted error. Never a whole page — a large snapshot fails on every unrelated change and is never actually read.
|
|
16
|
+
- **Done when:** every snapshot is small enough to review in a diff.
|
|
17
|
+
|
|
18
|
+
2. Review every snapshot change deliberately.
|
|
19
|
+
- The diff is the test result. Accepting it without reading is the same as deleting the test.
|
|
20
|
+
- **Done when:** each snapshot change in a review has a stated reason.
|
|
21
|
+
|
|
22
|
+
3. Remove non-determinism before snapshotting.
|
|
23
|
+
- Timestamps, identifiers, ordering, and floating-point formatting. Replace them with stable placeholders — otherwise the snapshot fails randomly and gets updated reflexively.
|
|
24
|
+
- **Done when:** ten consecutive runs produce identical snapshots.
|
|
25
|
+
|
|
26
|
+
4. Render visual tests in a controlled environment.
|
|
27
|
+
- A container with pinned fonts and a fixed device pixel ratio. Font rendering differs between operating systems, so screenshots taken on a laptop will never match CI.
|
|
28
|
+
- **Done when:** the same commit produces identical screenshots locally and in CI.
|
|
29
|
+
|
|
30
|
+
5. Set a sensible pixel tolerance.
|
|
31
|
+
- Zero tolerance flags anti-aliasing noise. Too high hides real regressions. Tune it against a known-good pair.
|
|
32
|
+
- **Done when:** the threshold catches a one-pixel layout shift but ignores anti-aliasing.
|
|
33
|
+
|
|
34
|
+
6. Prefer explicit assertions where they are clearer.
|
|
35
|
+
- "The submit button is disabled" is a better test than a snapshot containing a disabled attribute somewhere. Reach for snapshots when the output is large and the shape is the point.
|
|
36
|
+
- **Done when:** behavioural rules are asserted explicitly, not implied by a snapshot.
|
|
37
|
+
|
|
38
|
+
## Verification
|
|
39
|
+
|
|
40
|
+
- Ten runs produce no spurious diff.
|
|
41
|
+
- Local and CI screenshots match on the same commit.
|
|
42
|
+
- Each snapshot is small enough to review.
|