vanara 0.1.0 → 0.3.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +14 -0
- package/bin/vanara.js +66 -2
- package/catalog/agents/api-designer/AGENT.md +156 -0
- package/catalog/agents/api-designer/examples/openapi-snippet.yaml +130 -0
- package/catalog/agents/api-designer/examples/review-notes.md +61 -0
- package/catalog/agents/api-designer/references/contract-and-openapi.md +82 -0
- package/catalog/agents/api-designer/references/design-checklist.md +68 -0
- package/catalog/agents/api-designer/references/versioning-and-evolution.md +63 -0
- package/catalog/agents/api-designer/scripts/lint-openapi.mjs +180 -0
- package/catalog/agents/code-reviewer/AGENT.md +184 -0
- package/catalog/agents/code-reviewer/examples/pr-comment-template.md +48 -0
- package/catalog/agents/code-reviewer/examples/sample-review-output.md +38 -0
- package/catalog/agents/code-reviewer/references/review-checklist.md +46 -0
- package/catalog/agents/code-reviewer/references/security-review.md +62 -0
- package/catalog/agents/code-reviewer/references/severity-rubric.md +63 -0
- package/catalog/agents/code-reviewer/scripts/review-guard.mjs +102 -0
- package/catalog/agents/debugger/AGENT.md +128 -0
- package/catalog/agents/debugger/examples/postmortem-template.md +59 -0
- package/catalog/agents/debugger/examples/repro-template.md +50 -0
- package/catalog/agents/debugger/references/bisection.md +63 -0
- package/catalog/agents/debugger/references/debugging-method.md +57 -0
- package/catalog/agents/debugger/references/observability.md +61 -0
- package/catalog/agents/debugger/scripts/parse-stacktrace.mjs +109 -0
- package/catalog/agents/pr-summarizer/AGENT.md +172 -0
- package/catalog/agents/pr-summarizer/examples/sample-pr-summary.md +46 -0
- package/catalog/agents/pr-summarizer/examples/weak-vs-strong-summary.md +68 -0
- package/catalog/agents/pr-summarizer/references/identifying-risk.md +68 -0
- package/catalog/agents/pr-summarizer/references/reading-the-diff.md +68 -0
- package/catalog/agents/pr-summarizer/references/test-plan-and-missing-tests.md +64 -0
- package/catalog/agents/pr-summarizer/references/writing-the-summary.md +63 -0
- package/catalog/agents/pr-summarizer/scripts/diff-risk.mjs +111 -0
- package/catalog/agents/refactoring-specialist/AGENT.md +201 -0
- package/catalog/agents/refactoring-specialist/examples/before-after.md +91 -0
- package/catalog/agents/refactoring-specialist/examples/refactor-plan.md +97 -0
- package/catalog/agents/refactoring-specialist/references/large-scale-refactor.md +50 -0
- package/catalog/agents/refactoring-specialist/references/safe-workflow.md +75 -0
- package/catalog/agents/refactoring-specialist/references/smell-catalog.md +76 -0
- package/catalog/agents/refactoring-specialist/scripts/detect-smells.mjs +150 -0
- package/catalog/agents/security-auditor/AGENT.md +184 -0
- package/catalog/agents/security-auditor/examples/audit-report.md +68 -0
- package/catalog/agents/security-auditor/examples/finding-template.md +31 -0
- package/catalog/agents/security-auditor/references/audit-workflow.md +116 -0
- package/catalog/agents/security-auditor/references/severity-and-reporting.md +56 -0
- package/catalog/agents/security-auditor/references/vuln-classes.md +129 -0
- package/catalog/agents/security-auditor/scripts/scan-secrets.mjs +86 -0
- package/catalog/agents/technical-writer/AGENT.md +179 -0
- package/catalog/agents/technical-writer/examples/how-to-example.md +76 -0
- package/catalog/agents/technical-writer/examples/reference-example.md +75 -0
- package/catalog/agents/technical-writer/references/clarity-and-style.md +52 -0
- package/catalog/agents/technical-writer/references/diataxis-and-doc-types.md +61 -0
- package/catalog/agents/technical-writer/references/structure-and-maintenance.md +59 -0
- package/catalog/agents/technical-writer/scripts/readability.mjs +103 -0
- package/catalog/agents/test-author/AGENT.md +193 -0
- package/catalog/agents/test-author/examples/test-plan.md +54 -0
- package/catalog/agents/test-author/examples/unit-test-example.ts +55 -0
- package/catalog/agents/test-author/references/aaa-and-naming.md +75 -0
- package/catalog/agents/test-author/references/mocking-and-fixtures.md +94 -0
- package/catalog/agents/test-author/references/test-types-and-pyramid.md +60 -0
- package/catalog/agents/test-author/scripts/check-coverage.mjs +145 -0
- package/catalog/agents/threat-modeler/AGENT.md +179 -0
- package/catalog/agents/threat-modeler/examples/dfd-example.md +55 -0
- package/catalog/agents/threat-modeler/examples/threat-model.md +81 -0
- package/catalog/agents/threat-modeler/references/dfd-and-trust-boundaries.md +76 -0
- package/catalog/agents/threat-modeler/references/mitigation-catalog.md +77 -0
- package/catalog/agents/threat-modeler/references/stride-method.md +68 -0
- package/catalog/agents/threat-modeler/scripts/stride-checklist.mjs +183 -0
- package/catalog/agents/vuln-scanner/AGENT.md +162 -0
- package/catalog/agents/vuln-scanner/examples/finding-template.md +57 -0
- package/catalog/agents/vuln-scanner/examples/scan-report.md +49 -0
- package/catalog/agents/vuln-scanner/references/remediation-and-severity.md +68 -0
- package/catalog/agents/vuln-scanner/references/scan-types-and-tools.md +83 -0
- package/catalog/agents/vuln-scanner/references/triage-and-false-positives.md +73 -0
- package/catalog/agents/vuln-scanner/scripts/parse-scan-results.mjs +133 -0
- package/catalog/packs/code-review-pack/PACK.md +100 -0
- package/catalog/packs/code-review-pack/examples/pr-flow-example.md +50 -0
- package/catalog/packs/code-review-pack/references/rollout-guide.md +40 -0
- package/catalog/packs/security-pack/PACK.md +128 -0
- package/catalog/packs/security-pack/examples/owasp-vuln-triage-and-fix.md +66 -0
- package/catalog/packs/security-pack/examples/threat-model-then-audit.md +77 -0
- package/catalog/packs/security-pack/references/choosing-the-right-tool.md +37 -0
- package/catalog/packs/security-pack/references/threat-model-to-remediation.md +69 -0
- package/catalog/packs/security-pack/references/workflow.md +56 -0
- package/catalog/skills/api-pagination/SKILL.md +83 -0
- package/catalog/skills/api-pagination/examples/keyset-queries.sql +38 -0
- package/catalog/skills/api-pagination/examples/response-shapes.md +51 -0
- package/catalog/skills/api-pagination/references/consuming-pages.md +39 -0
- package/catalog/skills/api-pagination/references/counting-and-totals.md +41 -0
- package/catalog/skills/api-pagination/references/cursor-encoding.md +34 -0
- package/catalog/skills/api-pagination/scripts/check-cursor.mjs +51 -0
- package/catalog/skills/caching-strategies/SKILL.md +180 -0
- package/catalog/skills/caching-strategies/examples/http-cache-headers.md +82 -0
- package/catalog/skills/caching-strategies/examples/redis-cache-aside.js +110 -0
- package/catalog/skills/caching-strategies/references/cache-patterns.md +93 -0
- package/catalog/skills/caching-strategies/references/eviction-and-ttl.md +67 -0
- package/catalog/skills/caching-strategies/references/invalidation-and-stampede.md +99 -0
- package/catalog/skills/caching-strategies/scripts/ttl-jitter.mjs +99 -0
- package/catalog/skills/conventional-commits/SKILL.md +135 -0
- package/catalog/skills/conventional-commits/examples/commit-examples.md +60 -0
- package/catalog/skills/conventional-commits/examples/commitlint.config.js +31 -0
- package/catalog/skills/conventional-commits/references/breaking-changes-semver.md +50 -0
- package/catalog/skills/conventional-commits/references/scoping-commits.md +42 -0
- package/catalog/skills/conventional-commits/references/spec.md +78 -0
- package/catalog/skills/conventional-commits/scripts/lint-commit.mjs +97 -0
- package/catalog/skills/database-migrations/SKILL.md +137 -0
- package/catalog/skills/database-migrations/examples/add-column-safe.sql +34 -0
- package/catalog/skills/database-migrations/examples/migration-config.example.json +27 -0
- package/catalog/skills/database-migrations/references/expand-contract.md +74 -0
- package/catalog/skills/database-migrations/references/rollback-and-safety.md +71 -0
- package/catalog/skills/database-migrations/references/zero-downtime-changes.md +84 -0
- package/catalog/skills/database-migrations/scripts/check-migration-reversible.mjs +139 -0
- package/catalog/skills/error-handling-patterns/SKILL.md +136 -0
- package/catalog/skills/error-handling-patterns/examples/express-error-middleware.js +93 -0
- package/catalog/skills/error-handling-patterns/examples/result-pattern.ts +81 -0
- package/catalog/skills/error-handling-patterns/references/error-taxonomy.md +91 -0
- package/catalog/skills/error-handling-patterns/references/language-patterns.md +102 -0
- package/catalog/skills/error-handling-patterns/references/retry-and-backoff.md +76 -0
- package/catalog/skills/error-handling-patterns/scripts/lint-empty-catch.mjs +113 -0
- package/catalog/skills/git-collaboration-workflows/SKILL.md +104 -0
- package/catalog/skills/git-collaboration-workflows/examples/branch-protection.md +48 -0
- package/catalog/skills/git-collaboration-workflows/examples/pr-review-playbook.md +48 -0
- package/catalog/skills/git-collaboration-workflows/references/branching-models.md +52 -0
- package/catalog/skills/git-collaboration-workflows/references/history-hygiene.md +44 -0
- package/catalog/skills/git-collaboration-workflows/scripts/check-branch-hygiene.mjs +62 -0
- package/catalog/skills/owasp-top10/SKILL.md +112 -0
- package/catalog/skills/owasp-top10/examples/sql-injection-fix.md +33 -0
- package/catalog/skills/owasp-top10/examples/xss-fix.md +27 -0
- package/catalog/skills/owasp-top10/references/access-control.md +36 -0
- package/catalog/skills/owasp-top10/references/crypto-failures.md +29 -0
- package/catalog/skills/owasp-top10/references/injection.md +39 -0
- package/catalog/skills/owasp-top10/references/ssrf-and-design.md +35 -0
- package/catalog/skills/owasp-top10/scripts/scan-injection.mjs +73 -0
- package/catalog/skills/prompt-engineering/SKILL.md +100 -0
- package/catalog/skills/prompt-engineering/examples/classification-prompt.md +29 -0
- package/catalog/skills/prompt-engineering/examples/extraction-prompt.md +25 -0
- package/catalog/skills/prompt-engineering/references/prompt-injection.md +34 -0
- package/catalog/skills/prompt-engineering/references/structured-output.md +34 -0
- package/catalog/skills/prompt-engineering/references/techniques.md +33 -0
- package/catalog/skills/prompt-engineering/scripts/validate-output.mjs +71 -0
- package/catalog/skills/readme-writing/SKILL.md +176 -0
- package/catalog/skills/readme-writing/examples/README.template.md +73 -0
- package/catalog/skills/readme-writing/examples/good-readme-example.md +95 -0
- package/catalog/skills/readme-writing/references/anatomy.md +107 -0
- package/catalog/skills/readme-writing/references/maintenance.md +68 -0
- package/catalog/skills/readme-writing/references/writing-style.md +79 -0
- package/catalog/skills/readme-writing/scripts/lint-readme.mjs +117 -0
- package/catalog/skills/refactoring-patterns/SKILL.md +178 -0
- package/catalog/skills/refactoring-patterns/examples/extract-function.before-after.js +57 -0
- package/catalog/skills/refactoring-patterns/examples/replace-conditional-with-polymorphism.before-after.ts +63 -0
- package/catalog/skills/refactoring-patterns/references/code-smells.md +92 -0
- package/catalog/skills/refactoring-patterns/references/refactoring-catalog.md +78 -0
- package/catalog/skills/refactoring-patterns/references/safe-workflow.md +74 -0
- package/catalog/skills/refactoring-patterns/scripts/detect-smells.mjs +137 -0
- package/catalog/skills/rest-api-design/SKILL.md +147 -0
- package/catalog/skills/rest-api-design/examples/error-envelope.json +26 -0
- package/catalog/skills/rest-api-design/examples/orders-api.openapi.yaml +106 -0
- package/catalog/skills/rest-api-design/references/error-handling.md +43 -0
- package/catalog/skills/rest-api-design/references/pagination.md +41 -0
- package/catalog/skills/rest-api-design/references/status-codes.md +32 -0
- package/catalog/skills/rest-api-design/references/versioning.md +31 -0
- package/catalog/skills/rest-api-design/scripts/check-envelope.mjs +62 -0
- package/catalog/skills/secrets-management/SKILL.md +134 -0
- package/catalog/skills/secrets-management/examples/env-example.md +41 -0
- package/catalog/skills/secrets-management/examples/rotation-runbook.md +50 -0
- package/catalog/skills/secrets-management/references/leak-response.md +47 -0
- package/catalog/skills/secrets-management/references/rotation.md +46 -0
- package/catalog/skills/secrets-management/references/secret-managers.md +52 -0
- package/catalog/skills/secrets-management/scripts/detect-hardcoded.mjs +96 -0
- package/catalog/skills/secure-auth/SKILL.md +135 -0
- package/catalog/skills/secure-auth/examples/auth-flow.md +60 -0
- package/catalog/skills/secure-auth/examples/jwt-verify.ts +89 -0
- package/catalog/skills/secure-auth/references/auth-attacks.md +76 -0
- package/catalog/skills/secure-auth/references/oauth2-oidc.md +65 -0
- package/catalog/skills/secure-auth/references/password-hashing.md +70 -0
- package/catalog/skills/secure-auth/references/sessions-vs-jwt.md +69 -0
- package/catalog/skills/secure-auth/scripts/check-password-policy.mjs +114 -0
- package/catalog/skills/sql-index-tuning/SKILL.md +171 -0
- package/catalog/skills/sql-index-tuning/examples/explain-walkthrough.sql +91 -0
- package/catalog/skills/sql-index-tuning/examples/index-ddl.sql +67 -0
- package/catalog/skills/sql-index-tuning/references/btree-internals.md +58 -0
- package/catalog/skills/sql-index-tuning/references/composite-and-covering.md +60 -0
- package/catalog/skills/sql-index-tuning/references/reading-explain.md +54 -0
- package/catalog/skills/sql-index-tuning/references/selectivity-and-cardinality.md +62 -0
- package/catalog/skills/sql-index-tuning/scripts/suggest-index.mjs +213 -0
- package/catalog/skills/test-plan-design/SKILL.md +154 -0
- package/catalog/skills/test-plan-design/examples/test-case-template.md +45 -0
- package/catalog/skills/test-plan-design/examples/test-plan.md +57 -0
- package/catalog/skills/test-plan-design/references/case-design.md +79 -0
- package/catalog/skills/test-plan-design/references/risk-prioritization.md +58 -0
- package/catalog/skills/test-plan-design/references/test-pyramid.md +58 -0
- package/catalog/skills/test-plan-design/scripts/coverage-gaps.mjs +109 -0
- package/free-tier.json +15 -0
- package/package.json +15 -4
- package/premium-index.json +1 -0
- package/src/config.js +31 -1
- package/src/license.js +143 -0
|
@@ -0,0 +1,128 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: security-pack
|
|
3
|
+
description: Build and ship secure software — threat modeling at design time, OWASP code audits, dependency/secret scanning, and secure auth and secrets handling, sequenced into one shift-left workflow.
|
|
4
|
+
type: pack
|
|
5
|
+
version: 2.0.0
|
|
6
|
+
updated: 2026-07-06
|
|
7
|
+
agents: [threat-modeler, security-auditor, vuln-scanner]
|
|
8
|
+
skills: [owasp-top10, secure-auth, secrets-management]
|
|
9
|
+
---
|
|
10
|
+
# Security Pack
|
|
11
|
+
|
|
12
|
+
Security is cheapest when it moves **left** — designed in at the whiteboard, enforced while the code
|
|
13
|
+
is written, verified before merge, and monitored in the supply chain. This pack bundles three agents
|
|
14
|
+
and three skills that cover that whole arc and, more importantly, teaches **which one to reach for at
|
|
15
|
+
each stage of a feature's life**.
|
|
16
|
+
|
|
17
|
+
The value of a pack is not the list of items — it is the sequencing. A threat model with no audit is
|
|
18
|
+
a document nobody enforces; an audit with no threat model is a checklist with no context. This
|
|
19
|
+
document ties them into one repeatable loop: **model → build → audit → scan → ship**.
|
|
20
|
+
|
|
21
|
+
## Who this is for
|
|
22
|
+
|
|
23
|
+
- Security engineers who own the security posture of a product or service.
|
|
24
|
+
- Developers who own the security of the features they build — the ones expected to threat-model,
|
|
25
|
+
self-audit, and keep their dependencies clean before asking for review.
|
|
26
|
+
- Tech leads introducing a shift-left practice to a team that currently bolts security on at the end.
|
|
27
|
+
|
|
28
|
+
If you only need a one-off review of an existing diff, the `security-auditor` agent alone is enough.
|
|
29
|
+
Reach for the full pack when you are carrying a feature **from design through to a clean merge** and
|
|
30
|
+
want security enforced at every handoff instead of a single gate at the finish line.
|
|
31
|
+
|
|
32
|
+
## What's included
|
|
33
|
+
|
|
34
|
+
| Item | Kind | Job in the workflow |
|
|
35
|
+
|---|---|---|
|
|
36
|
+
| `threat-modeler` | agent | STRIDE-style threat modeling at design time — enumerate attack surface, trust boundaries, and abuse cases before code exists |
|
|
37
|
+
| `security-auditor` | agent | OWASP-focused code review of the diff — injection, auth flaws, broken access control, unsafe crypto |
|
|
38
|
+
| `vuln-scanner` | agent | Dependency CVE scanning and secret detection across the tree and git history |
|
|
39
|
+
| `owasp-top10` | skill | Per-risk defenses for the OWASP Top 10 — the reference the auditor and you apply |
|
|
40
|
+
| `secure-auth` | skill | Correct password hashing, session/token handling, and MFA construction |
|
|
41
|
+
| `secrets-management` | skill | Keep secrets out of source, load them safely, and rotate without downtime |
|
|
42
|
+
|
|
43
|
+
Agents *do* the work; skills are the *reference* they (and you) apply while doing it. Pair an agent
|
|
44
|
+
with its matching skill — `security-auditor` + `owasp-top10`, or building auth with `secure-auth` —
|
|
45
|
+
so the output is both produced and grounded in a standard.
|
|
46
|
+
|
|
47
|
+
## The end-to-end workflow
|
|
48
|
+
|
|
49
|
+
A secure feature moves through five stages. The pack maps one primary item to each:
|
|
50
|
+
|
|
51
|
+
```text
|
|
52
|
+
1. Model → threat-modeler (+ owasp-top10) enumerate threats before code
|
|
53
|
+
2. Build → secure-auth / secrets-management implement the risky parts correctly
|
|
54
|
+
3. Audit → security-auditor (+ owasp-top10) review the diff, security-first
|
|
55
|
+
4. Scan → vuln-scanner CVEs in deps + secrets in the tree
|
|
56
|
+
5. Ship → gate on: no open CRITICAL/HIGH, clean scan
|
|
57
|
+
```
|
|
58
|
+
|
|
59
|
+
Model **first**, before any code. A threat model produced after the feature is built is a
|
|
60
|
+
rationalization, not a design tool — its whole purpose is to change what you build. The threats it
|
|
61
|
+
surfaces become the audit checklist in stage 3 and the abuse-case tests you write while building.
|
|
62
|
+
|
|
63
|
+
### Worked example: adding "password reset via email link"
|
|
64
|
+
|
|
65
|
+
```text
|
|
66
|
+
threat-modeler → surfaces: token guessability, token reuse/replay, host-header poisoning in the
|
|
67
|
+
reset link, user enumeration on the "email sent" response, rate-limit bypass
|
|
68
|
+
owasp-top10 → maps those to A01 (access control), A07 (auth failures), A04 (insecure design)
|
|
69
|
+
secure-auth → reset token = 256-bit random, single-use, 15-min TTL, hashed at rest; response is
|
|
70
|
+
identical whether or not the email exists (no enumeration)
|
|
71
|
+
secrets-management → the email-provider API key is loaded from the secret store, never in the repo
|
|
72
|
+
security-auditor → confirms token is compared in constant time, is invalidated on use, and the
|
|
73
|
+
"sent" response leaks nothing; flags the missing rate limit as HIGH
|
|
74
|
+
vuln-scanner → the email SDK pulls a transitive dep with a known SSRF CVE — flagged, bump required
|
|
75
|
+
```
|
|
76
|
+
|
|
77
|
+
Every item earned its place: the threat model set the agenda, the skills shaped the implementation,
|
|
78
|
+
the auditor verified it, and the scanner caught a supply-chain issue no human review would have.
|
|
79
|
+
|
|
80
|
+
## How to choose the right item
|
|
81
|
+
|
|
82
|
+
- **Designing a feature that touches auth, money, or user data?** `threat-modeler` + `owasp-top10`
|
|
83
|
+
first — enumerate abuse cases before a line is written.
|
|
84
|
+
- **Implementing login, sessions, or MFA?** `secure-auth` — do not hand-roll password hashing or
|
|
85
|
+
token generation.
|
|
86
|
+
- **About to reference a credential, key, or token?** `secrets-management` — it never belongs in
|
|
87
|
+
source, config committed to git, or a client bundle.
|
|
88
|
+
- **Diff ready for review?** `security-auditor` — security pass before correctness, every time.
|
|
89
|
+
- **Before merge, always?** `vuln-scanner` — CVEs and leaked secrets are invisible to code review.
|
|
90
|
+
|
|
91
|
+
See `references/choosing-the-right-tool.md` for the full decision table, `references/workflow.md` for
|
|
92
|
+
the stage-by-stage playbook, and `references/threat-model-to-remediation.md` for the discipline
|
|
93
|
+
playbook that connects a threat model to concrete fixes. Two complete runs are in `examples/`.
|
|
94
|
+
|
|
95
|
+
## Common pitfalls (anti-patterns this pack prevents)
|
|
96
|
+
|
|
97
|
+
- **Threat-modeling after the build.** A model written to justify existing code enumerates the
|
|
98
|
+
threats you already handled and misses the ones you didn't. The workflow puts it at stage 1 so it
|
|
99
|
+
can actually change the design.
|
|
100
|
+
- **Auditing without a threat model.** A generic OWASP checklist misses domain abuse cases (e.g. a
|
|
101
|
+
refund flow that can go negative). The threat model *is* the audit's context.
|
|
102
|
+
- **Treating secret scanning as one-time.** A secret committed once lives in git history forever even
|
|
103
|
+
after it's deleted from HEAD. `vuln-scanner` checks history, and the fix is rotation, not just
|
|
104
|
+
removal.
|
|
105
|
+
- **Rolling your own crypto/auth.** Hand-rolled password hashing, JWT verification, or token
|
|
106
|
+
comparison is a reliable source of criticals. `secure-auth` points to vetted constructions.
|
|
107
|
+
- **Fixing the secret, forgetting to rotate.** Deleting a leaked key from source does not un-leak it.
|
|
108
|
+
Any exposed secret must be rotated, not just removed.
|
|
109
|
+
|
|
110
|
+
## When NOT to use this pack
|
|
111
|
+
|
|
112
|
+
- **Compliance/audit-trail work** (GDPR, SOC 2 evidence) — that's `compliance-auditor` and the
|
|
113
|
+
privacy/audit-logging skills, not this pack's application-security focus.
|
|
114
|
+
- **Infrastructure and network hardening** — firewall rules, TLS termination, and cloud IAM are
|
|
115
|
+
DevOps/network territory, not application-code security.
|
|
116
|
+
- **A one-line, no-risk change** — a copy tweak or a CSS fix does not need a threat model; a single
|
|
117
|
+
`security-auditor` pass (or nothing) is proportionate.
|
|
118
|
+
- **Smart-contract security** — use `smart-contract-auditor` and `smart-contract-security`; the
|
|
119
|
+
threat surface there is different.
|
|
120
|
+
|
|
121
|
+
## Files in this package
|
|
122
|
+
|
|
123
|
+
- `references/workflow.md` — the five-stage security workflow in detail, with entry/exit criteria.
|
|
124
|
+
- `references/choosing-the-right-tool.md` — full decision table: symptom → item.
|
|
125
|
+
- `references/threat-model-to-remediation.md` — the discipline playbook connecting STRIDE threats to
|
|
126
|
+
audited, tested remediations.
|
|
127
|
+
- `examples/threat-model-then-audit.md` — a feature threat-modeled at design time, then audited.
|
|
128
|
+
- `examples/owasp-vuln-triage-and-fix.md` — an OWASP-class vulnerability triaged and remediated.
|
|
@@ -0,0 +1,66 @@
|
|
|
1
|
+
# Walkthrough: triaging and fixing an OWASP-class vulnerability
|
|
2
|
+
|
|
3
|
+
Not every task starts at design time. This shows the pack applied to a vulnerability found in
|
|
4
|
+
existing code — where `security-auditor` leads and the workflow compresses.
|
|
5
|
+
|
|
6
|
+
## The finding
|
|
7
|
+
|
|
8
|
+
A routine `security-auditor` pass on a diff flags:
|
|
9
|
+
|
|
10
|
+
```md
|
|
11
|
+
### CRITICAL
|
|
12
|
+
- `api/documents.js:73` — GET /documents/:id loads the record by id and returns it with no check
|
|
13
|
+
that the document belongs to the requesting user. Any authenticated user can read any document by
|
|
14
|
+
guessing/enumerating ids. (A01 Broken Access Control — IDOR)
|
|
15
|
+
```
|
|
16
|
+
|
|
17
|
+
Intermittent guessing isn't needed — the ids are sequential integers, so this is trivially
|
|
18
|
+
enumerable. CRITICAL is correct.
|
|
19
|
+
|
|
20
|
+
## Classify (`owasp-top10`)
|
|
21
|
+
|
|
22
|
+
This is **A01 Broken Access Control**, the IDOR variant: the endpoint authenticates ("are you logged
|
|
23
|
+
in?") but does not authorize ("is this *your* document?"). The reference is clear — authorization must
|
|
24
|
+
be checked server-side against the resource owner on every request, deny-by-default.
|
|
25
|
+
|
|
26
|
+
## Reproduce as a failing test first (`security-auditor` direction)
|
|
27
|
+
|
|
28
|
+
Before touching the handler, lock the bug down as a regression test:
|
|
29
|
+
|
|
30
|
+
```text
|
|
31
|
+
✗ user B requesting user A's document id receives 404, not 200 + the document
|
|
32
|
+
```
|
|
33
|
+
|
|
34
|
+
It fails against current code — confirming the vulnerability and giving the fix a target. (404, not
|
|
35
|
+
403, so the endpoint doesn't confirm the id exists — avoids the enumeration side channel too.)
|
|
36
|
+
|
|
37
|
+
## Fix the access-control check
|
|
38
|
+
|
|
39
|
+
Add the ownership guard *before* the record is returned:
|
|
40
|
+
|
|
41
|
+
```text
|
|
42
|
+
const doc = await Documents.findById(id);
|
|
43
|
+
if (!doc || doc.ownerId !== session.userId) return res.status(404).end();
|
|
44
|
+
return res.json({ data: doc, error: null });
|
|
45
|
+
```
|
|
46
|
+
|
|
47
|
+
The check runs for every request and denies by default. Test goes green. A grep for the same pattern
|
|
48
|
+
across sibling routes (`/documents/:id/download`, `/documents/:id/versions`) confirms whether the same
|
|
49
|
+
gap exists elsewhere — IDOR is rarely a single-endpoint problem.
|
|
50
|
+
|
|
51
|
+
## Re-audit + scan (`security-auditor` + `vuln-scanner`)
|
|
52
|
+
|
|
53
|
+
- `security-auditor` re-reviews: the guard is present on all three routes, the 404 leaks nothing, no
|
|
54
|
+
new finding introduced.
|
|
55
|
+
- `vuln-scanner` runs on the branch: dependencies clean, no secret introduced by the change.
|
|
56
|
+
|
|
57
|
+
## Ship
|
|
58
|
+
|
|
59
|
+
CRITICAL closed, sibling routes patched, regression tests in place, scan clean. The fix ships.
|
|
60
|
+
|
|
61
|
+
## The lesson
|
|
62
|
+
|
|
63
|
+
The pack turned a single flagged line into a *class* fix: classify the vulnerability (A01/IDOR), lock
|
|
64
|
+
it with a failing test, apply the deny-by-default mitigation, then sweep for the same pattern
|
|
65
|
+
elsewhere. Fixing only line 73 would have left the download and versions routes exploitable — the
|
|
66
|
+
OWASP framing is what prompts checking the whole class, not just the reported instance.
|
|
@@ -0,0 +1,77 @@
|
|
|
1
|
+
# Walkthrough: threat-modeling "password reset via email", then auditing it
|
|
2
|
+
|
|
3
|
+
A complete run through the five stages for a classic high-risk feature, showing which pack item leads
|
|
4
|
+
at each point.
|
|
5
|
+
|
|
6
|
+
## Stage 1 — Model (`threat-modeler` + `owasp-top10`)
|
|
7
|
+
|
|
8
|
+
Trust boundaries: browser → API → email provider → token store. Walking STRIDE surfaces:
|
|
9
|
+
|
|
10
|
+
```text
|
|
11
|
+
- Token guessability → tokens must be cryptographically random (A07)
|
|
12
|
+
- Token reuse / replay → single-use, short TTL, invalidated on use (A07)
|
|
13
|
+
- Host-header poisoning → reset link host must not come from the request header (A04)
|
|
14
|
+
- User enumeration → "email sent" response identical for known/unknown emails (A01)
|
|
15
|
+
- No rate limit → attacker can brute-force or spam resets (A04/A07)
|
|
16
|
+
```
|
|
17
|
+
|
|
18
|
+
Each threat is tagged with its OWASP class and a planned mitigation. This list becomes the audit
|
|
19
|
+
agenda.
|
|
20
|
+
|
|
21
|
+
## Stage 2 — Build (`secure-auth` + `secrets-management`)
|
|
22
|
+
|
|
23
|
+
Implement against the vetted construction:
|
|
24
|
+
|
|
25
|
+
- Reset token = 256-bit CSPRNG value, stored **hashed** at rest, single-use, 15-minute TTL.
|
|
26
|
+
- The "we sent you an email" response is byte-identical whether or not the account exists.
|
|
27
|
+
- The reset link's host is built from server config, not the incoming `Host` header.
|
|
28
|
+
- The email-provider API key is loaded from the secret store via `secrets-management` — it never
|
|
29
|
+
appears in source or committed config.
|
|
30
|
+
|
|
31
|
+
Abuse-case tests are written alongside:
|
|
32
|
+
|
|
33
|
+
```text
|
|
34
|
+
✗ a used reset token cannot be redeemed a second time
|
|
35
|
+
✗ /forgot-password returns the same body for known and unknown emails
|
|
36
|
+
✗ /forgot-password returns 429 after 5 attempts in the window
|
|
37
|
+
```
|
|
38
|
+
|
|
39
|
+
Implement until green.
|
|
40
|
+
|
|
41
|
+
## Stage 3 — Audit (`security-auditor` + `owasp-top10`)
|
|
42
|
+
|
|
43
|
+
The auditor reviews the diff against the stage-1 list:
|
|
44
|
+
|
|
45
|
+
```md
|
|
46
|
+
### HIGH
|
|
47
|
+
- `auth/reset.js:41` — token compared with `===`; use a constant-time compare to avoid a timing
|
|
48
|
+
oracle on the token value.
|
|
49
|
+
- `auth/reset.js:12` — no rate limiter on the request-reset endpoint (threat listed in the model,
|
|
50
|
+
not yet implemented). Add per-IP + per-account throttling.
|
|
51
|
+
|
|
52
|
+
### LOW
|
|
53
|
+
- `auth/reset.js:58` — TTL is 60 min; model called for 15. Tighten.
|
|
54
|
+
```
|
|
55
|
+
|
|
56
|
+
The timing-compare and rate-limit findings are fixed and re-reviewed. No CRITICAL/HIGH remains.
|
|
57
|
+
|
|
58
|
+
## Stage 4 — Scan (`vuln-scanner`)
|
|
59
|
+
|
|
60
|
+
```text
|
|
61
|
+
CVE: email-sdk@2.3.0 → transitive dep `node-fetch@2.6.1` has a known SSRF advisory.
|
|
62
|
+
Fix: bump to email-sdk@2.4.1 (pulls patched fetch).
|
|
63
|
+
Secrets: none in tree; git history clean.
|
|
64
|
+
```
|
|
65
|
+
|
|
66
|
+
Dependency bumped; scan re-run clean.
|
|
67
|
+
|
|
68
|
+
## Stage 5 — Ship
|
|
69
|
+
|
|
70
|
+
Audit clean, scan clean, no committed secret. The feature ships with its abuse-case tests as
|
|
71
|
+
permanent regression guards.
|
|
72
|
+
|
|
73
|
+
## The lesson
|
|
74
|
+
|
|
75
|
+
The threat model did the heavy lifting: it predicted the rate-limit gap and the enumeration risk
|
|
76
|
+
*before* code existed, and it gave the auditor a precise agenda. Without it, stage 3 would have been a
|
|
77
|
+
generic pass that could easily have missed the missing rate limiter.
|
|
@@ -0,0 +1,37 @@
|
|
|
1
|
+
# Choosing the right item: symptom → tool
|
|
2
|
+
|
|
3
|
+
When you're mid-feature and not sure what to reach for, match the symptom to the item.
|
|
4
|
+
|
|
5
|
+
| Symptom / task | Reach for | Why |
|
|
6
|
+
|---|---|---|
|
|
7
|
+
| "New feature touches auth, money, or PII" | `threat-modeler` + `owasp-top10` | Enumerate abuse cases before code exists |
|
|
8
|
+
| "Which OWASP risk applies here?" | `owasp-top10` | The per-risk defense reference |
|
|
9
|
+
| "I'm building login / sessions / MFA" | `secure-auth` | Vetted hashing, tokens, session handling |
|
|
10
|
+
| "I need to reset passwords / issue tokens" | `secure-auth` | Single-use, TTL'd, constant-time compared |
|
|
11
|
+
| "Where does this API key / credential go?" | `secrets-management` | Never in source; loaded from a store |
|
|
12
|
+
| "Diff is ready for review" | `security-auditor` + `owasp-top10` | Security-first code review before merge |
|
|
13
|
+
| "Is a dependency vulnerable?" | `vuln-scanner` | CVEs, including transitive, are invisible to review |
|
|
14
|
+
| "Did a secret get committed?" | `vuln-scanner` | Scans tree *and* git history |
|
|
15
|
+
| "A secret leaked — now what?" | `secrets-management` | Rotate, don't just delete |
|
|
16
|
+
|
|
17
|
+
## Agent + skill pairings
|
|
18
|
+
|
|
19
|
+
Each agent has a natural skill partner. Use them together:
|
|
20
|
+
|
|
21
|
+
- `threat-modeler` + `owasp-top10` — enumerate threats *and* tag each with the risk category that
|
|
22
|
+
drives its defense.
|
|
23
|
+
- `security-auditor` + `owasp-top10` — the auditor reviews against a concrete Top 10 checklist rather
|
|
24
|
+
than a vibe.
|
|
25
|
+
- Building auth or handling secrets — apply `secure-auth` / `secrets-management` *while writing*, not
|
|
26
|
+
as an afterthought the auditor has to catch.
|
|
27
|
+
|
|
28
|
+
## Escalation boundaries
|
|
29
|
+
|
|
30
|
+
- Regulatory/compliance evidence (GDPR, SOC 2, audit trails) → `compliance-auditor` and the
|
|
31
|
+
privacy/audit-logging skills, not this pack.
|
|
32
|
+
- Infrastructure, network, TLS, and cloud IAM hardening → DevOps/network specialists; this pack is
|
|
33
|
+
application-code security.
|
|
34
|
+
- Smart-contract vulnerabilities → `smart-contract-auditor` + `smart-contract-security`; the threat
|
|
35
|
+
surface differs from web app security.
|
|
36
|
+
- A CVE with no available patch → this is a risk-acceptance decision (document + mitigate), not
|
|
37
|
+
something `vuln-scanner` resolves for you.
|
|
@@ -0,0 +1,69 @@
|
|
|
1
|
+
# From threat model to remediation: the discipline playbook
|
|
2
|
+
|
|
3
|
+
A threat model only pays off if its threats become fixes. This playbook is the connective tissue
|
|
4
|
+
between `threat-modeler`'s output and a clean `security-auditor` pass — how a STRIDE finding turns
|
|
5
|
+
into a mitigation, a test, and an audited change.
|
|
6
|
+
|
|
7
|
+
## Step 1 — Enumerate with STRIDE (`threat-modeler`)
|
|
8
|
+
|
|
9
|
+
Walk each trust boundary in the feature and ask the six STRIDE questions. For every "yes", write a
|
|
10
|
+
one-line abuse case in the attacker's voice:
|
|
11
|
+
|
|
12
|
+
```text
|
|
13
|
+
Spoofing → "I replay another user's session token to act as them."
|
|
14
|
+
Tampering → "I change the price field in the checkout request."
|
|
15
|
+
Repudiation → "I perform a refund and there's no log tying it to me."
|
|
16
|
+
Info disclosure → "The error response tells me whether an email is registered."
|
|
17
|
+
DoS → "I hit the reset endpoint 10k times with no rate limit."
|
|
18
|
+
Elevation → "I call the admin endpoint with a normal user's token."
|
|
19
|
+
```
|
|
20
|
+
|
|
21
|
+
## Step 2 — Classify and prioritize (`owasp-top10`)
|
|
22
|
+
|
|
23
|
+
Tag each abuse case with its OWASP category and a severity driven by *impact × likelihood*:
|
|
24
|
+
|
|
25
|
+
| Abuse case | OWASP | Severity |
|
|
26
|
+
|---|---|---|
|
|
27
|
+
| Session token replay | A07 Auth failures | HIGH |
|
|
28
|
+
| Price tampering | A04 Insecure design / A08 | CRITICAL |
|
|
29
|
+
| Missing refund audit log | A09 Logging failures | MEDIUM |
|
|
30
|
+
| Email enumeration | A01 Access control | MEDIUM |
|
|
31
|
+
| No rate limit on reset | A04 / A07 | HIGH |
|
|
32
|
+
| Privilege escalation | A01 Broken access control | CRITICAL |
|
|
33
|
+
|
|
34
|
+
Fix CRITICAL/HIGH before shipping; schedule MEDIUM/LOW with an owner.
|
|
35
|
+
|
|
36
|
+
## Step 3 — Design the mitigation (skills)
|
|
37
|
+
|
|
38
|
+
Each category has a known defense — reach for the skill, don't improvise:
|
|
39
|
+
|
|
40
|
+
- Access control (A01) → deny by default; authorize every request server-side against the *resource
|
|
41
|
+
owner*, not just "is logged in".
|
|
42
|
+
- Auth failures (A07) → `secure-auth`: strong hashing, single-use tokens, constant-time compare,
|
|
43
|
+
rotate on privilege change.
|
|
44
|
+
- Insecure design (A04) → re-derive trusted values (price, role) server-side; never trust the client.
|
|
45
|
+
- Sensitive data (A02) → `secrets-management` for keys; encrypt at rest; identical responses to
|
|
46
|
+
prevent enumeration.
|
|
47
|
+
|
|
48
|
+
## Step 4 — Test the abuse case, then implement
|
|
49
|
+
|
|
50
|
+
Turn each abuse case into a failing test *first*, so the fix has a target and a regression guard:
|
|
51
|
+
|
|
52
|
+
```text
|
|
53
|
+
✗ a request with user B's token cannot mutate user A's resource (A01)
|
|
54
|
+
✗ POST /checkout ignores a client-supplied price field (A04)
|
|
55
|
+
✗ /password-reset returns 429 after N attempts per window (A07)
|
|
56
|
+
```
|
|
57
|
+
|
|
58
|
+
Implement until green. The test now permanently encodes the threat.
|
|
59
|
+
|
|
60
|
+
## Step 5 — Audit and scan (`security-auditor` + `vuln-scanner`)
|
|
61
|
+
|
|
62
|
+
Hand the diff to `security-auditor` with the stage-2 severity table as the agenda — it verifies each
|
|
63
|
+
mitigation is present and correct, citing `file:line`. Then `vuln-scanner` sweeps dependencies and
|
|
64
|
+
history. Ship only when the CRITICAL/HIGH rows are all closed and the scan is clean.
|
|
65
|
+
|
|
66
|
+
## The through-line
|
|
67
|
+
|
|
68
|
+
Threat → OWASP class → mitigation skill → failing abuse-case test → audited fix. When a finding
|
|
69
|
+
skips a step — a threat with no test, or a fix with no audit — that's where regressions re-enter.
|
|
@@ -0,0 +1,56 @@
|
|
|
1
|
+
# The five-stage security workflow
|
|
2
|
+
|
|
3
|
+
Each stage has an owner (the primary item), the skills it applies, and an explicit **exit criterion**
|
|
4
|
+
— you don't advance until the criterion is met. This is what keeps security from collapsing into a
|
|
5
|
+
single rushed review at the end.
|
|
6
|
+
|
|
7
|
+
## Stage 1 — Model
|
|
8
|
+
|
|
9
|
+
- **Owner:** `threat-modeler` · **Applies:** `owasp-top10`
|
|
10
|
+
- Draw the trust boundaries (client, server, third parties, data store). For each boundary, walk
|
|
11
|
+
STRIDE — Spoofing, Tampering, Repudiation, Information disclosure, Denial of service, Elevation of
|
|
12
|
+
privilege — and write down the concrete abuse cases for *this* feature.
|
|
13
|
+
- Map each threat to its OWASP Top 10 category so the audit later has a checklist.
|
|
14
|
+
- **Exit criterion:** a written list of threats, each tagged with an OWASP category and a planned
|
|
15
|
+
mitigation, that the build and audit stages can reference.
|
|
16
|
+
|
|
17
|
+
## Stage 2 — Build
|
|
18
|
+
|
|
19
|
+
- **Owners:** `secure-auth`, `secrets-management` (as the feature demands)
|
|
20
|
+
- Implement the risk-bearing parts against a vetted construction, not from memory. Auth: correct
|
|
21
|
+
hashing, single-use tokens, constant-time comparison, no user enumeration. Secrets: loaded from a
|
|
22
|
+
store or environment, never committed, never shipped to the client.
|
|
23
|
+
- Write the abuse-case tests from stage 1 alongside the implementation.
|
|
24
|
+
- **Exit criterion:** the risky paths are implemented per the skills' guidance and the abuse-case
|
|
25
|
+
tests pass.
|
|
26
|
+
|
|
27
|
+
## Stage 3 — Audit
|
|
28
|
+
|
|
29
|
+
- **Owner:** `security-auditor` · **Applies:** `owasp-top10`
|
|
30
|
+
- Review the diff security-first, using the stage-1 threat list as the agenda: broken access control
|
|
31
|
+
(IDOR/authorization), injection, auth failures, unsafe crypto, sensitive-data exposure. Every
|
|
32
|
+
finding cites `file:line`, a severity, and a concrete fix.
|
|
33
|
+
- **Exit criterion:** no CRITICAL or HIGH finding remains open.
|
|
34
|
+
|
|
35
|
+
## Stage 4 — Scan
|
|
36
|
+
|
|
37
|
+
- **Owner:** `vuln-scanner`
|
|
38
|
+
- Scan dependencies (including transitive) for known CVEs, and scan the working tree **and git
|
|
39
|
+
history** for secrets. A secret found in history is treated as exposed even if HEAD is clean.
|
|
40
|
+
- **Exit criterion:** no unaddressed CVE at High/Critical severity; no secret in tree or history
|
|
41
|
+
(removal alone is insufficient for a real leak — see below).
|
|
42
|
+
|
|
43
|
+
## Stage 5 — Ship
|
|
44
|
+
|
|
45
|
+
- **Gate:** no open CRITICAL/HIGH audit finding, clean dependency scan, no committed secret.
|
|
46
|
+
- If a secret was ever exposed, it is **rotated**, not merely deleted. If a CVE has no patch,
|
|
47
|
+
document the risk acceptance or apply a mitigating control before shipping.
|
|
48
|
+
- **Exit criterion:** all gates green; any accepted risk is written down with an owner.
|
|
49
|
+
|
|
50
|
+
## Why order matters
|
|
51
|
+
|
|
52
|
+
The threat model is the shared dependency of every later stage — it sets what to build carefully in
|
|
53
|
+
stage 2 and what to look for in stage 3. Building before modeling means you protect the wrong things.
|
|
54
|
+
Auditing before scanning means you can pass human review while a vulnerable dependency or a leaked key
|
|
55
|
+
sits in the tree. Skipping or reordering stages is the single biggest source of shipped
|
|
56
|
+
vulnerabilities.
|
|
@@ -0,0 +1,83 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: api-pagination
|
|
3
|
+
description: Implement correct, fast API pagination — cursor vs offset trade-offs, opaque cursor encoding, stable sort keys, page-size limits, total-count costs, and consumer-side iteration that survives inserts, deletes, and retries. With runnable cursor checks.
|
|
4
|
+
type: skill
|
|
5
|
+
version: 1.0.0
|
|
6
|
+
updated: 2026-07-10
|
|
7
|
+
---
|
|
8
|
+
# API Pagination
|
|
9
|
+
|
|
10
|
+
Pagination looks trivial until data changes underneath the reader. Offset pagination silently
|
|
11
|
+
**skips or duplicates rows** when inserts land mid-iteration; cursor pagination survives churn but
|
|
12
|
+
demands a stable sort key and an opaque token contract. This skill covers both sides: designing the
|
|
13
|
+
API and consuming one correctly. Deep detail lives in `references/`; copy-paste material in
|
|
14
|
+
`examples/`; a runnable cursor round-trip check in `scripts/`.
|
|
15
|
+
|
|
16
|
+
## Decision: offset vs cursor
|
|
17
|
+
|
|
18
|
+
| Situation | Use |
|
|
19
|
+
|---|---|
|
|
20
|
+
| Admin tables, small datasets, "jump to page 7" required | Offset (`LIMIT/OFFSET`) |
|
|
21
|
+
| Feeds, sync endpoints, anything users scroll | Cursor (keyset) |
|
|
22
|
+
| Data changes while clients iterate | Cursor — offset will skip/duplicate |
|
|
23
|
+
| Deep pages (offset > ~10k rows) | Cursor — `OFFSET n` scans and discards n rows |
|
|
24
|
+
|
|
25
|
+
Offset's failure mode is correctness, not just speed: a row inserted before your current position
|
|
26
|
+
shifts everything, so page 3 re-shows an item from page 2 (duplicate) or swallows one (skip).
|
|
27
|
+
|
|
28
|
+
## Keyset mechanics (the part people get wrong)
|
|
29
|
+
|
|
30
|
+
The sort key must be **unique and immutable**. `created_at` alone is not unique — two rows with the
|
|
31
|
+
same timestamp make the cursor ambiguous and rows vanish. Always add a tiebreaker:
|
|
32
|
+
|
|
33
|
+
```sql
|
|
34
|
+
-- WHERE clause for "next page after (2026-07-01T12:00:00Z, id 4711)" descending
|
|
35
|
+
SELECT * FROM items
|
|
36
|
+
WHERE (created_at, id) < ('2026-07-01T12:00:00Z', 4711)
|
|
37
|
+
ORDER BY created_at DESC, id DESC
|
|
38
|
+
LIMIT 51; -- page_size + 1 to detect has_more without COUNT(*)
|
|
39
|
+
```
|
|
40
|
+
|
|
41
|
+
Fetch `page_size + 1` rows: if you get 51, there is a next page and the 51st row's key becomes the
|
|
42
|
+
next cursor. This avoids `COUNT(*)`, which on large tables costs more than the page itself
|
|
43
|
+
(see `references/counting-and-totals.md`).
|
|
44
|
+
|
|
45
|
+
## Cursor token contract
|
|
46
|
+
|
|
47
|
+
Cursors are **opaque to clients** — never let them parse or build one. Encode the keyset values plus
|
|
48
|
+
the sort direction, base64url it, and (for public APIs) HMAC it so tampering is detectable. Include a
|
|
49
|
+
version byte so you can change the encoding later. Full recipe: `references/cursor-encoding.md`;
|
|
50
|
+
runnable round-trip: `scripts/check-cursor.mjs`.
|
|
51
|
+
|
|
52
|
+
## Response envelope
|
|
53
|
+
|
|
54
|
+
```json
|
|
55
|
+
{
|
|
56
|
+
"data": [ ... ],
|
|
57
|
+
"page_info": { "has_more": true, "next_cursor": "djEuMj..." }
|
|
58
|
+
}
|
|
59
|
+
```
|
|
60
|
+
|
|
61
|
+
Rules: `next_cursor` is absent (not empty-string) on the last page; page size is capped server-side
|
|
62
|
+
(a `limit=10000` request gets the cap, documented, not an error); the default page size appears in
|
|
63
|
+
the docs and never changes silently. Consumer-side iteration rules — retries, cursor expiry,
|
|
64
|
+
resumption — live in `references/consuming-pages.md`.
|
|
65
|
+
|
|
66
|
+
## Pitfalls
|
|
67
|
+
|
|
68
|
+
- **Mutable sort keys** (`updated_at`) — rows teleport across pages as they update. Sort on immutable
|
|
69
|
+
columns or accept re-delivery and dedupe client-side.
|
|
70
|
+
- **Cursor built from row position** instead of row values — it is offset with extra steps and
|
|
71
|
+
inherits every offset bug.
|
|
72
|
+
- **`COUNT(*)` on every page** — the count is stale by the time the client renders it; provide totals
|
|
73
|
+
on a separate, cacheable endpoint if the product truly needs them.
|
|
74
|
+
- **Breaking cursor encoding on deploy** — clients hold cursors across releases; version the token.
|
|
75
|
+
- **Missing composite index** — keyset needs an index on exactly `(created_at, id)` in sort order, or
|
|
76
|
+
every page is a scan.
|
|
77
|
+
|
|
78
|
+
## Verification
|
|
79
|
+
|
|
80
|
+
Run `node scripts/check-cursor.mjs` — encodes, decodes, tampers, and versions a cursor and asserts
|
|
81
|
+
every property above. For the SQL side, `examples/keyset-queries.sql` includes an `EXPLAIN` you can
|
|
82
|
+
run to confirm the index is used. Pairs with `rest-api-design` (envelopes, limits) and
|
|
83
|
+
`sql-index-tuning` (composite indexes).
|
|
@@ -0,0 +1,38 @@
|
|
|
1
|
+
-- Keyset pagination: canonical queries (Postgres syntax).
|
|
2
|
+
|
|
3
|
+
-- The supporting index. Column order and direction must match the ORDER BY exactly.
|
|
4
|
+
CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_items_created_id
|
|
5
|
+
ON items (created_at DESC, id DESC);
|
|
6
|
+
|
|
7
|
+
-- Page 1 (no cursor): newest first, page_size 50, +1 row to detect has_more.
|
|
8
|
+
SELECT id, created_at, title
|
|
9
|
+
FROM items
|
|
10
|
+
WHERE tenant_id = $1
|
|
11
|
+
ORDER BY created_at DESC, id DESC
|
|
12
|
+
LIMIT 51;
|
|
13
|
+
|
|
14
|
+
-- Page N (cursor decoded to ($2 = created_at, $3 = id) of the last row seen).
|
|
15
|
+
-- Row-value comparison keeps the composite ordering correct in one clause.
|
|
16
|
+
SELECT id, created_at, title
|
|
17
|
+
FROM items
|
|
18
|
+
WHERE tenant_id = $1
|
|
19
|
+
AND (created_at, id) < ($2, $3)
|
|
20
|
+
ORDER BY created_at DESC, id DESC
|
|
21
|
+
LIMIT 51;
|
|
22
|
+
|
|
23
|
+
-- Verify the index is used (look for "Index Scan ... idx_items_created_id",
|
|
24
|
+
-- no Sort node, and rows examined ≈ LIMIT):
|
|
25
|
+
EXPLAIN (ANALYZE, BUFFERS)
|
|
26
|
+
SELECT id, created_at, title
|
|
27
|
+
FROM items
|
|
28
|
+
WHERE tenant_id = $1 AND (created_at, id) < (now(), 2147483647)
|
|
29
|
+
ORDER BY created_at DESC, id DESC
|
|
30
|
+
LIMIT 51;
|
|
31
|
+
|
|
32
|
+
-- Ascending variant (jump-to-oldest without deep offsets): flip both the
|
|
33
|
+
-- comparison and the sort, same index scanned backwards.
|
|
34
|
+
SELECT id, created_at, title
|
|
35
|
+
FROM items
|
|
36
|
+
WHERE tenant_id = $1 AND (created_at, id) > ($2, $3)
|
|
37
|
+
ORDER BY created_at ASC, id ASC
|
|
38
|
+
LIMIT 51;
|
|
@@ -0,0 +1,51 @@
|
|
|
1
|
+
# Response Shapes
|
|
2
|
+
|
|
3
|
+
## Cursor-paginated list (recommended default)
|
|
4
|
+
|
|
5
|
+
```json
|
|
6
|
+
{
|
|
7
|
+
"data": [
|
|
8
|
+
{ "id": "itm_9x2", "created_at": "2026-07-01T12:00:00.412Z", "title": "..." }
|
|
9
|
+
],
|
|
10
|
+
"page_info": {
|
|
11
|
+
"has_more": true,
|
|
12
|
+
"next_cursor": "v1.eyJrIjpbIjIwMjYt...IsNDcxMV0sImQiOiJkZXNjIn0.3f9a1c22b0"
|
|
13
|
+
}
|
|
14
|
+
}
|
|
15
|
+
```
|
|
16
|
+
|
|
17
|
+
Last page: `has_more: false` and **no** `next_cursor` key. Clients must treat absence — not
|
|
18
|
+
empty string, not null — as the terminal signal (document exactly one convention and test it).
|
|
19
|
+
|
|
20
|
+
## Offset-paginated list (admin/small data only)
|
|
21
|
+
|
|
22
|
+
```json
|
|
23
|
+
{
|
|
24
|
+
"data": [ ... ],
|
|
25
|
+
"page_info": { "page": 3, "per_page": 50, "total": 1204, "total_pages": 25 }
|
|
26
|
+
}
|
|
27
|
+
```
|
|
28
|
+
|
|
29
|
+
Only ship `total`/`total_pages` when the dataset is small enough that COUNT is cheap and the
|
|
30
|
+
product genuinely offers page jumping.
|
|
31
|
+
|
|
32
|
+
## Request parameters
|
|
33
|
+
|
|
34
|
+
```
|
|
35
|
+
GET /v1/items?limit=50&cursor=v1.eyJr... # cursor flow
|
|
36
|
+
GET /v1/admin/items?page=3&per_page=50 # offset flow
|
|
37
|
+
```
|
|
38
|
+
|
|
39
|
+
- `limit` capped server-side (e.g., max 200); over-cap requests get the cap, not an error.
|
|
40
|
+
- Filters/sort are request parameters, never encoded inside the cursor — but changing them
|
|
41
|
+
mid-iteration invalidates the cursor (return `CURSOR_INVALID`, client restarts).
|
|
42
|
+
|
|
43
|
+
## Error shapes
|
|
44
|
+
|
|
45
|
+
```json
|
|
46
|
+
{ "error": { "code": "CURSOR_EXPIRED", "message": "Restart iteration from the beginning." } }
|
|
47
|
+
{ "error": { "code": "CURSOR_INVALID", "message": "Cursor is malformed or filters changed." } }
|
|
48
|
+
```
|
|
49
|
+
|
|
50
|
+
Both are 400s. Clients restart from their last durable checkpoint; idempotent processing absorbs
|
|
51
|
+
the overlap.
|
|
@@ -0,0 +1,39 @@
|
|
|
1
|
+
# Consuming Paginated APIs Correctly
|
|
2
|
+
|
|
3
|
+
The consumer side has its own failure modes: partial iteration on crash, duplicate processing on
|
|
4
|
+
retry, and silent stalls when a provider changes page semantics.
|
|
5
|
+
|
|
6
|
+
## The iteration loop
|
|
7
|
+
|
|
8
|
+
```js
|
|
9
|
+
let cursor = savedCheckpoint ?? null;
|
|
10
|
+
do {
|
|
11
|
+
const page = await get('/items', { cursor, limit: 100 });
|
|
12
|
+
for (const item of page.data) await processIdempotently(item);
|
|
13
|
+
cursor = page.page_info.next_cursor ?? null;
|
|
14
|
+
await saveCheckpoint(cursor); // AFTER processing the page
|
|
15
|
+
} while (cursor);
|
|
16
|
+
```
|
|
17
|
+
|
|
18
|
+
Rules encoded above:
|
|
19
|
+
|
|
20
|
+
- **Checkpoint the cursor, not the item index.** Persist `next_cursor` after each fully processed
|
|
21
|
+
page; on restart, resume from the checkpoint instead of page one.
|
|
22
|
+
- **Process idempotently.** At-least-once is the reality: a crash between processing and
|
|
23
|
+
checkpointing replays the page. Dedupe on item ID exactly like a webhook consumer.
|
|
24
|
+
- **Loop on `next_cursor` presence, not on `data.length > 0`.** Some providers return short or
|
|
25
|
+
empty pages mid-stream (filtered rows); an empty page with a cursor is NOT the end.
|
|
26
|
+
- **Bound the loop.** A provider bug that returns the same cursor forever becomes your infinite
|
|
27
|
+
loop; track `pages_fetched` against a sane ceiling and alert past it.
|
|
28
|
+
|
|
29
|
+
## Retry semantics
|
|
30
|
+
|
|
31
|
+
Page fetches are GETs — safe to retry with backoff. But a `CURSOR_EXPIRED` or `CURSOR_INVALID`
|
|
32
|
+
response is not retryable: restart the iteration from your last durable checkpoint (or from
|
|
33
|
+
scratch) and rely on idempotent processing to absorb the overlap.
|
|
34
|
+
|
|
35
|
+
## Rate limits and burst shape
|
|
36
|
+
|
|
37
|
+
Full-table syncs hammer providers. Respect `Retry-After` on 429s, spread scheduled syncs with
|
|
38
|
+
jitter, and prefer provider-side incremental endpoints (`updated_since=`) over re-walking
|
|
39
|
+
everything — pagination is for the backfill, deltas are for steady state.
|