@vextlabs/theron-cli 0.3.0 → 0.4.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/dist/api.d.ts +8 -0
- package/dist/api.js +3 -0
- package/dist/api.js.map +1 -1
- package/dist/auth.js +51 -1
- package/dist/auth.js.map +1 -1
- package/dist/banner.js +3 -2
- package/dist/banner.js.map +1 -1
- package/dist/checkpoints.d.ts +32 -0
- package/dist/checkpoints.js +61 -0
- package/dist/checkpoints.js.map +1 -0
- package/dist/index.js +59 -4
- package/dist/index.js.map +1 -1
- package/dist/input.d.ts +61 -0
- package/dist/input.js +574 -0
- package/dist/input.js.map +1 -0
- package/dist/profiles/index.js +5 -0
- package/dist/profiles/index.js.map +1 -1
- package/dist/profiles/methodologies/operate_domains.d.ts +8 -0
- package/dist/profiles/methodologies/operate_domains.js +1239 -0
- package/dist/profiles/methodologies/operate_domains.js.map +1 -0
- package/dist/profiles/seeds.js +57 -36
- package/dist/profiles/seeds.js.map +1 -1
- package/dist/receipt.d.ts +17 -0
- package/dist/receipt.js +46 -0
- package/dist/receipt.js.map +1 -0
- package/dist/render.d.ts +4 -1
- package/dist/render.js +95 -28
- package/dist/render.js.map +1 -1
- package/dist/repl.d.ts +8 -1
- package/dist/repl.js +420 -62
- package/dist/repl.js.map +1 -1
- package/dist/sessions.d.ts +14 -0
- package/dist/sessions.js +100 -0
- package/dist/sessions.js.map +1 -1
- package/dist/ship.d.ts +2 -0
- package/dist/ship.js +62 -0
- package/dist/ship.js.map +1 -0
- package/dist/skills/catalog.d.ts +13 -0
- package/dist/skills/catalog.js +86 -0
- package/dist/skills/catalog.js.map +1 -0
- package/dist/tools/bash.js +81 -14
- package/dist/tools/bash.js.map +1 -1
- package/dist/tools/edit.js +21 -1
- package/dist/tools/edit.js.map +1 -1
- package/dist/tools/glob.js +4 -1
- package/dist/tools/glob.js.map +1 -1
- package/dist/tools/grep.d.ts +5 -0
- package/dist/tools/grep.js +101 -2
- package/dist/tools/grep.js.map +1 -1
- package/dist/tools/index.d.ts +22 -0
- package/dist/tools/index.js +177 -41
- package/dist/tools/index.js.map +1 -1
- package/dist/tools/ls.d.ts +3 -0
- package/dist/tools/ls.js +23 -12
- package/dist/tools/ls.js.map +1 -1
- package/dist/tools/multiedit.d.ts +12 -0
- package/dist/tools/multiedit.js +79 -0
- package/dist/tools/multiedit.js.map +1 -0
- package/dist/tools/stoa.d.ts +1 -1
- package/dist/tools/stoa.js +7 -3
- package/dist/tools/stoa.js.map +1 -1
- package/dist/tools/task.d.ts +9 -0
- package/dist/tools/task.js +166 -0
- package/dist/tools/task.js.map +1 -0
- package/dist/tools/todowrite.d.ts +12 -0
- package/dist/tools/todowrite.js +38 -0
- package/dist/tools/todowrite.js.map +1 -0
- package/dist/tools/webfetch.d.ts +6 -0
- package/dist/tools/webfetch.js +98 -0
- package/dist/tools/webfetch.js.map +1 -0
- package/dist/tools/websearch.d.ts +7 -0
- package/dist/tools/websearch.js +83 -0
- package/dist/tools/websearch.js.map +1 -0
- package/dist/tools/write.js +17 -1
- package/dist/tools/write.js.map +1 -1
- package/dist/verifiers/confidence_marked.d.ts +2 -0
- package/dist/verifiers/confidence_marked.js +49 -0
- package/dist/verifiers/confidence_marked.js.map +1 -0
- package/dist/verifiers/disclaimer_gate.d.ts +2 -0
- package/dist/verifiers/disclaimer_gate.js +57 -0
- package/dist/verifiers/disclaimer_gate.js.map +1 -0
- package/dist/verifiers/index.d.ts +5 -0
- package/dist/verifiers/index.js +20 -7
- package/dist/verifiers/index.js.map +1 -1
- package/dist/verifiers/lint.js +4 -3
- package/dist/verifiers/lint.js.map +1 -1
- package/dist/verifiers/promoted_kernels.d.ts +8 -0
- package/dist/verifiers/promoted_kernels.js +190 -0
- package/dist/verifiers/promoted_kernels.js.map +1 -0
- package/dist/verifiers/source_gate.js +2 -3
- package/dist/verifiers/source_gate.js.map +1 -1
- package/dist/verifiers/test_smoke.js +30 -0
- package/dist/verifiers/test_smoke.js.map +1 -1
- package/dist/verifiers/types.d.ts +3 -0
- package/package.json +4 -2
- package/skills/README.md +123 -0
- package/skills/ab-test.md +89 -0
- package/skills/api-design.md +175 -0
- package/skills/architecture-design.md +185 -0
- package/skills/business-case.md +77 -0
- package/skills/causal-inference.md +77 -0
- package/skills/clinical-guideline.md +98 -0
- package/skills/code-review.md +98 -0
- package/skills/cold-outreach.md +268 -0
- package/skills/competitive-teardown.md +223 -0
- package/skills/component-spec.md +121 -0
- package/skills/content-calendar.md +280 -0
- package/skills/contract-review.md +155 -0
- package/skills/data-analysis.md +187 -0
- package/skills/debug.md +91 -0
- package/skills/design-audit.md +121 -0
- package/skills/differential-diagnosis.md +79 -0
- package/skills/discovery-call.md +206 -0
- package/skills/edit-pass.md +80 -0
- package/skills/engineering-calc.md +101 -0
- package/skills/estimate.md +70 -0
- package/skills/experiment-design.md +105 -0
- package/skills/fact-check.md +82 -0
- package/skills/financial-model.md +104 -0
- package/skills/grant-proposal.md +93 -0
- package/skills/harmony-analysis.md +93 -0
- package/skills/hypothesis-generation.md +99 -0
- package/skills/incident-response.md +134 -0
- package/skills/interview-loop.md +62 -0
- package/skills/job-scorecard.md +92 -0
- package/skills/kb-article.md +174 -0
- package/skills/launch-plan.md +85 -0
- package/skills/lease-review.md +93 -0
- package/skills/lesson-plan.md +198 -0
- package/skills/literature-review.md +69 -0
- package/skills/market-entry.md +137 -0
- package/skills/market-sizing.md +159 -0
- package/skills/meta-analysis.md +140 -0
- package/skills/migrate.md +117 -0
- package/skills/optimize.md +88 -0
- package/skills/options-strategy.md +166 -0
- package/skills/peer-review.md +96 -0
- package/skills/pentest-plan.md +193 -0
- package/skills/pitch-review.md +132 -0
- package/skills/plan.md +88 -0
- package/skills/policy-brief.md +124 -0
- package/skills/positioning.md +192 -0
- package/skills/postmortem.md +168 -0
- package/skills/prd.md +105 -0
- package/skills/prioritize.md +162 -0
- package/skills/proof.md +91 -0
- package/skills/property-underwrite.md +159 -0
- package/skills/recipe-develop.md +109 -0
- package/skills/red-team.md +142 -0
- package/skills/refactor.md +58 -0
- package/skills/reflection-session.md +115 -0
- package/skills/regulatory-compliance.md +136 -0
- package/skills/reproduce.md +87 -0
- package/skills/runbook.md +344 -0
- package/skills/security-audit.md +154 -0
- package/skills/seo-brief.md +201 -0
- package/skills/sql-query.md +161 -0
- package/skills/story-craft.md +163 -0
- package/skills/tdd.md +59 -0
- package/skills/term-sheet.md +298 -0
- package/skills/theory-of-change.md +88 -0
- package/skills/threat-model.md +104 -0
- package/skills/ticket-triage.md +200 -0
- package/skills/tolerance-analysis.md +149 -0
- package/skills/training-program.md +151 -0
- package/skills/translate.md +64 -0
- package/skills/unit-economics.md +238 -0
- package/skills/valuation.md +112 -0
- package/skills/write-tests.md +77 -0
|
@@ -0,0 +1,159 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: market-sizing
|
|
3
|
+
description: Size a market TAM/SAM/SOM via dual top-down and bottom-up triangulation, reconcile the gap, stress-test on the dominant driver, and produce a realistic obtainable-share view — invoke whenever a user asks to size a market, estimate addressable revenue, or validate a market opportunity for a product, segment, or geography.
|
|
4
|
+
allowed-tools: Read, Bash, Write
|
|
5
|
+
---
|
|
6
|
+
|
|
7
|
+
KEY PRINCIPLE (read first): A market size is an argument, not a fact. Own every assumption, label every guess, and let the sensitivity table do the honest work. A point estimate without a derivation chain is not analysis — it is a number you cannot defend.
|
|
8
|
+
|
|
9
|
+
═══ HARD RULES ═══
|
|
10
|
+
|
|
11
|
+
1. NEVER present a number without its derivation chain — source or explicit "ASSUMED" label required.
|
|
12
|
+
2. NEVER average the top-down and bottom-up figures to get a final number; reconcile by explaining the GAP.
|
|
13
|
+
3. NEVER use a market report number as a fact — it is a data point with a methodology; state what that methodology was or label it "vendor estimate, methodology unverified."
|
|
14
|
+
4. NEVER conflate TAM (universe) with SAM (serviceable) with SOM (obtainable) — use all three; collapsing them is a red flag.
|
|
15
|
+
5. NEVER skip the sensitivity table — the dominant driver must be isolated and shocked at ±25% and ±50%.
|
|
16
|
+
6. NEVER project SOM without a stated go-to-market constraint (sales capacity, channel reach, or budget ceiling). Wishful percentages of SAM are not SOM estimates.
|
|
17
|
+
7. NEVER fabricate a CAGR or growth rate — use a range, state the source, or label it "ASSUMED — no source."
|
|
18
|
+
8. NEVER cite a named company benchmark in Phase F without a source. If no comparable is cited, state "no comparable cited."
|
|
19
|
+
9. Every stated assumption must be tagged with a unique label (e.g., [ACV-1], [SEG-2], [PRICE-3]). All calculations must cite the tag they depend on.
|
|
20
|
+
10. If the market is supply-constrained (GPU compute, licensed spectrum, regulatory quotas), flag it before Phase B — demand-side TAM math is structurally misleading for supply-constrained markets.
|
|
21
|
+
|
|
22
|
+
═══ PHASE A — FRAME THE MARKET ═══
|
|
23
|
+
|
|
24
|
+
A1. STATE the product or service in one sentence: what it does, who buys it, and what it displaces or competes with.
|
|
25
|
+
|
|
26
|
+
A2. CLASSIFY the category maturity — this determines which sizing method to trust more:
|
|
27
|
+
- Existing category (clear substitutes, active competitors): market reports exist; use them as sanity checks only, not as the answer.
|
|
28
|
+
- Nascent category (no direct comps): build entirely from bottom-up; any top-down number is a proxy from the closest analog category, labeled as such.
|
|
29
|
+
- Supply-constrained category (GPU clusters, licensed slots, regulated capacity): stop and note the supply ceiling before continuing — TAM is bounded by supply, not demand.
|
|
30
|
+
|
|
31
|
+
A3. COMMIT to a market boundary — geography, customer segment, and channel — in writing before any calculation. If the product serves multiple segments, size each separately and aggregate at the end; never blend segments before sizing.
|
|
32
|
+
|
|
33
|
+
A4. SET a sizing year explicitly ("2026 current state" or "2028 at scale"). All inputs must be consistent with that year.
|
|
34
|
+
|
|
35
|
+
A5. LOG every assumption before you calculate. Use descriptive tags, not sequential letters that collide with phase labels:
|
|
36
|
+
- [PRICE-n] for price and ACV assumptions
|
|
37
|
+
- [SEG-n] for segment count and qualifier assumptions
|
|
38
|
+
- [PENET-n] for penetration and conversion assumptions
|
|
39
|
+
- [BENCH-n] for benchmark comparables
|
|
40
|
+
- [MACRO-n] for macro pool figures pulled from external sources
|
|
41
|
+
Every tag must appear at least once in a downstream calculation.
|
|
42
|
+
|
|
43
|
+
═══ PHASE B — TOP-DOWN SIZING ═══
|
|
44
|
+
|
|
45
|
+
B1. IDENTIFY the widest credible macro pool for your category.
|
|
46
|
+
- Prefer public data: census/labor statistics, government trade data, industry association filings.
|
|
47
|
+
- Label any private analyst figure (Gartner, IDC, Forrester, Statista) as [MACRO-n: vendor estimate, methodology unverified].
|
|
48
|
+
- Do not use a market report's TAM number as your macro pool starting point — that number is the answer someone else computed, not raw data.
|
|
49
|
+
|
|
50
|
+
B2. APPLY segment filters from the macro pool down to your target customer, one filter per step:
|
|
51
|
+
- Geography filter: apply population share, GDP share, or penetration ratio with source. [MACRO-n]
|
|
52
|
+
- Vertical/industry filter: use SIC/NAICS employment counts or revenue share as the denominator. [SEG-n]
|
|
53
|
+
- Buying unit filter: count the qualified buying units (companies of the right headcount band, households meeting income/behavior criteria, etc.). State how you counted them and whether sources overlap.
|
|
54
|
+
|
|
55
|
+
B3. COMPUTE: (# of qualified buying units) × (average annual spend per unit) = TOP-DOWN TAM.
|
|
56
|
+
- If spend-per-unit is unknown, use a price proxy from a comparable product category. Label it [PRICE-n: ASSUMED — derived from {analog product} pricing in {year}].
|
|
57
|
+
- Do not use the analyst report's implied ARPU to back-calculate TAM; that is circular.
|
|
58
|
+
|
|
59
|
+
B4. DERIVE TOP-DOWN SAM: apply one constraint that reflects what your product can actually reach today — the geography you can serve, or the customer segment your current feature set fits.
|
|
60
|
+
|
|
61
|
+
B5. RECORD the top-down TAM and SAM with every assumption tag referenced inline.
|
|
62
|
+
|
|
63
|
+
═══ PHASE C — BOTTOM-UP SIZING ═══
|
|
64
|
+
|
|
65
|
+
C1. DEFINE the atomic purchase unit — what is one transaction? (seat/month, transaction fee, one-time license, API call batch.) This is the irreducible unit your revenue model rests on.
|
|
66
|
+
|
|
67
|
+
C2. COUNT the universe of buyers from the ground up using at least two independent sources (e.g., LinkedIn company counts cross-checked against a government business register or Dun & Bradstreet). State explicitly whether you deduplicated across sources and what the overlap risk is.
|
|
68
|
+
|
|
69
|
+
C3. APPLY a two-stage qualification filter — these are multiplicative, applied in sequence, not blended:
|
|
70
|
+
- Stage 1: What fraction of the counted universe has the problem your product solves? [PENET-1: ASSUMED or cited]
|
|
71
|
+
- Stage 2: Of those with the problem, what fraction would pay for a solution vs. DIY or use a free alternative? [PENET-2: ASSUMED or cited]
|
|
72
|
+
- Qualified buyers = (universe) × [PENET-1] × [PENET-2]. Collapsing into a single percentage obscures which filter is the binding one.
|
|
73
|
+
|
|
74
|
+
C4. COMPUTE: (qualified buyers) × (ACV per year) = BOTTOM-UP TAM.
|
|
75
|
+
- Decompose ACV: (units per buyer per year) × (unit price). [PRICE-n] [SEG-n]
|
|
76
|
+
- Do not use a blended industry ACV from a report — build it from your pricing model or a named comparable with a source.
|
|
77
|
+
|
|
78
|
+
C5. DERIVE BOTTOM-UP SAM by constraining to the segment your product currently fits: feature completeness, required integrations, supported languages, applicable compliance certifications.
|
|
79
|
+
|
|
80
|
+
C6. RECORD the bottom-up TAM and SAM with every assumption tag referenced inline.
|
|
81
|
+
|
|
82
|
+
═══ PHASE D — RECONCILE THE GAP ═══
|
|
83
|
+
|
|
84
|
+
D1. COMPUTE the ratio: TOP-DOWN TAM ÷ BOTTOM-UP TAM. These are heuristic thresholds, not industry standards:
|
|
85
|
+
- Ratio < 1.5×: methods broadly agree; minor assumption differences; proceed.
|
|
86
|
+
- Ratio 1.5×–5×: material gap; must be diagnosed and explained before the output is credible.
|
|
87
|
+
- Ratio > 5×: at least one method has a structural flaw; diagnose and rerun that method before proceeding.
|
|
88
|
+
|
|
89
|
+
D2. DIAGNOSE the gap source — it is almost always one of the following:
|
|
90
|
+
- Double-counting in top-down: the macro pool included adjacent categories or non-customers.
|
|
91
|
+
- Under-counting in bottom-up: a buyer segment was missed, or ACV was set too low relative to actual deal sizes.
|
|
92
|
+
- Pricing mismatch: top-down used an industry blended average while bottom-up used your specific price point.
|
|
93
|
+
- Definition mismatch: the TAM boundary (geography, segment) was not identical between the two methods.
|
|
94
|
+
- Competitive displacement lag: bottom-up counted only available share; top-down counted the full market before competitive lock-in.
|
|
95
|
+
|
|
96
|
+
D3. PRODUCE a reconciled TAM and reconciled SAM: state which number you adopt as the working estimate, and give the specific reason why (e.g., "bottom-up is binding because the macro pool double-counts indirect spend").
|
|
97
|
+
|
|
98
|
+
D4. If the gap cannot be explained, present a RANGE — (low = bottom-up, high = top-down) — and state it explicitly as unresolved uncertainty. Do not pick a midpoint without a justification grounded in a specific assumption difference.
|
|
99
|
+
|
|
100
|
+
═══ PHASE E — SENSITIVITY ANALYSIS ═══
|
|
101
|
+
|
|
102
|
+
E1. IDENTIFY the dominant driver: the single assumption whose realistic shock (±25%) produces the largest absolute change in TAM or SAM. This is a tornado-chart ranking, not a mathematical ±1% elasticity — rank by impact magnitude at realistic shock magnitudes, not by mathematical slope.
|
|
103
|
+
Common dominant drivers: ACV/unit price, number of qualifying buyers, penetration rate, market growth rate.
|
|
104
|
+
|
|
105
|
+
E2. IDENTIFY the second-order driver: the next-highest-impact assumption after the dominant one.
|
|
106
|
+
|
|
107
|
+
E3. BUILD the sensitivity table — shock the dominant driver at −50%, −25%, base, +25%, +50%:
|
|
108
|
+
|
|
109
|
+
| Dominant Driver Value | −50% | −25% | Base | +25% | +50% |
|
|
110
|
+
|-----------------------|------|------|------|------|------|
|
|
111
|
+
| TAM ($) | | | | | |
|
|
112
|
+
| SAM ($) | | | | | |
|
|
113
|
+
|
|
114
|
+
E4. STATE the "plausible bear case": dominant driver at −25% AND second-order driver at −25%.
|
|
115
|
+
STATE the "plausible bull case": dominant driver at +25% AND second-order driver at +25%.
|
|
116
|
+
Do not compound both at ±50% — that is a stress test, not a planning scenario.
|
|
117
|
+
|
|
118
|
+
E5. The sensitivity table is the output. A point estimate without it is inadmissible. If the range between bear and bull cases is less than 2×, the model is likely over-constrained — check for circular assumptions.
|
|
119
|
+
|
|
120
|
+
═══ PHASE F — OBTAINABLE SHARE (SOM) ═══
|
|
121
|
+
|
|
122
|
+
F1. CHOOSE a time horizon: Year 1, Year 3, Year 5. Size SOM for each separately. Do not interpolate Year 3 from Year 5 — the binding constraint changes as the company scales.
|
|
123
|
+
|
|
124
|
+
F2. GROUND the SOM estimate in a go-to-market constraint, not a share percentage of SAM:
|
|
125
|
+
- Sales capacity path: (# of reps or founders) × (quota per rep) × (close rate) = max deals/year → SOM ceiling via this path. [BENCH-n if close rate is from a comparable]
|
|
126
|
+
- Channel path: (channel partner reach) × (historical conversion rate for this channel type) = max customers → SOM ceiling via this path.
|
|
127
|
+
- Budget path: (marketing + sales budget) ÷ (blended CAC estimate) = max customers → SOM ceiling via this path.
|
|
128
|
+
Use the LOWEST output across all paths as the binding SOM ceiling. If only one path is available, state that.
|
|
129
|
+
|
|
130
|
+
F3. BENCHMARK against comparable go-to-market motions ONLY IF you have a named, sourced comparable:
|
|
131
|
+
[BENCH-n: {Company}, {Year}, {GTM motion}, {ACV range}, {Y1/Y3 share achieved}, {source}]
|
|
132
|
+
If no comparable is available, state "no comparable cited" — do not invent one.
|
|
133
|
+
Flag as aggressive if Year 3 SOM/SAM > 10% in a market with two or more established competitors.
|
|
134
|
+
|
|
135
|
+
F4. COMPUTE SOM: (binding constraint output) × (average deal size). Express as both a dollar figure and a percentage of SAM.
|
|
136
|
+
|
|
137
|
+
F5. STATE the key risk that would compress SOM below the binding constraint — at minimum: competitive displacement rate (what is the switching cost for customers locked to an incumbent?), churn rate in Year 1, and any regulatory or procurement-cycle delay specific to this market.
|
|
138
|
+
|
|
139
|
+
═══ PHASE G — DELIVER THE OUTPUT ═══
|
|
140
|
+
|
|
141
|
+
G1. PRODUCE a one-page summary table:
|
|
142
|
+
|
|
143
|
+
| Metric | Method | Figure | Assumption Tags |
|
|
144
|
+
|----------------------|-------------------|----------|------------------|
|
|
145
|
+
| TAM (top-down) | Macro filter | $X B | MACRO-1, SEG-2 |
|
|
146
|
+
| TAM (bottom-up) | Unit build | $X B | PENET-1, PRICE-1 |
|
|
147
|
+
| TAM (reconciled) | Working estimate | $X B | D3 rationale |
|
|
148
|
+
| SAM | Serviceable slice | $X B | SEG-n boundary |
|
|
149
|
+
| SOM — Year 1 | GTM constraint | $X M | F2 binding path |
|
|
150
|
+
| SOM — Year 3 | GTM constraint | $X M | F2 binding path |
|
|
151
|
+
| Bear case TAM | Sensitivity | $X B | E4 |
|
|
152
|
+
| Bull case TAM | Sensitivity | $X B | E4 |
|
|
153
|
+
|
|
154
|
+
G2. PRODUCE an ASSUMPTION LOG block listing every tagged assumption in full:
|
|
155
|
+
[TAG]: {assumption text} | {source or "ASSUMED"} | {used in: phase references}
|
|
156
|
+
|
|
157
|
+
G3. LIST the top three assumptions you would revise with one additional data point, and state exactly what data point would change each one. This is the research priority list — be specific about what to look up, not generic about "more data."
|
|
158
|
+
|
|
159
|
+
G4. Present the output as a structured estimate with explicit uncertainty bounds. Never present it as a forecast or a projection. The correct framing is: "Under these assumptions, the market is approximately $X B (bear: $Y B, bull: $Z B)."
|
|
@@ -0,0 +1,140 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: meta-analysis
|
|
3
|
+
description: Pool studies rigorously — common effect-size metric, fixed/random-effects choice, heterogeneity (I²/τ²), publication-bias checks, sensitivity, and GRADE certainty.
|
|
4
|
+
allowed-tools: Read, Bash, WebSearch, WebFetch, Write
|
|
5
|
+
---
|
|
6
|
+
|
|
7
|
+
## PHASE 0 — SCOPE AND REGISTRATION
|
|
8
|
+
|
|
9
|
+
1. Frame the PICO: Population, Intervention/Exposure, Comparator, Outcome. One primary outcome; pre-specify secondary outcomes and subgroups BEFORE searching.
|
|
10
|
+
2. Write an eligibility table: inclusion criteria (design, population, comparator, minimum sample size, language, date range) and exclusion criteria. Lock it. Never revise post-hoc.
|
|
11
|
+
3. Register the protocol (PROSPERO, OSF, or a timestamped commit) before screening begins. If post-hoc, state it prominently as a limitation.
|
|
12
|
+
|
|
13
|
+
## PHASE 1 — SYSTEMATIC SEARCH + PRISMA FLOW
|
|
14
|
+
|
|
15
|
+
4. Search at minimum: PubMed/MEDLINE, Embase, CENTRAL (Cochrane), and one domain-specific database. Add grey literature (ClinicalTrials.gov, WHO ICTRP, dissertations) to reduce publication bias.
|
|
16
|
+
5. Build a reproducible query string per database (Boolean operators, MeSH/Emtree terms, free text). Log exact query + run date. Rerun if >12 months elapses.
|
|
17
|
+
6. Export all records to a deduplication tool (Rayyan, Covidence, or a dedupe script). Remove duplicates before screening.
|
|
18
|
+
7. Two reviewers screen titles/abstracts independently; resolve conflicts by consensus or third reviewer. Repeat for full-text.
|
|
19
|
+
8. Produce a PRISMA 2020 flow diagram: records identified → duplicates removed → screened → excluded (with reasons) → included. Report counts at each node.
|
|
20
|
+
|
|
21
|
+
## PHASE 2 — DATA EXTRACTION
|
|
22
|
+
|
|
23
|
+
9. Use a pre-piloted extraction form: study ID, design, N, population descriptors, intervention/exposure, comparator, outcome definition, measurement instrument, time point, reported statistic, and all data needed for effect-size conversion.
|
|
24
|
+
10. Extract in duplicate. Calculate inter-rater agreement (Cohen's κ ≥ 0.80 is acceptable; resolve all discrepancies before pooling).
|
|
25
|
+
11. When reported statistics differ, convert to one common metric (see Phase 3). Never pool raw means with ORs.
|
|
26
|
+
12. Extract variance (SE, SD, CI width, or p-value + N). If not reported: impute SD from similar studies only as a last resort and flag as sensitivity analysis.
|
|
27
|
+
|
|
28
|
+
## PHASE 3 — EFFECT-SIZE CONVERSION (common metric required)
|
|
29
|
+
|
|
30
|
+
13. Choose ONE metric for the primary analysis. Canonical options:
|
|
31
|
+
- Continuous outcomes: Hedges' g (SMD corrected for small-sample bias, preferred) or Cohen's d.
|
|
32
|
+
- Binary outcomes: log(OR) or log(RR) — log scale for pooling, exponentiate for reporting.
|
|
33
|
+
- Correlational: Fisher's z (r → z = 0.5·ln((1+r)/(1-r))) for pooling; back-transform for reporting.
|
|
34
|
+
14. Conversion formulas (Bash-compute when numbers are present):
|
|
35
|
+
- t-statistic → d: d = 2t/√(df)
|
|
36
|
+
- F(1,df) → d: d = √(F/((N1·N2)/(N1+N2)))
|
|
37
|
+
- OR → d (log-OR/π×√3): d = log(OR)·(√3/π)
|
|
38
|
+
- r → d: d = 2r/√(1−r²)
|
|
39
|
+
- Hedges' g: g = d·J where J = 1 − 3/(4·df−1)
|
|
40
|
+
- Variance of g: v_g = (n1+n2)/(n1·n2) + g²/(2·(n1+n2))
|
|
41
|
+
15. Compute in Bash when study data are supplied. Example:
|
|
42
|
+
```bash
|
|
43
|
+
python3 - <<'EOF'
|
|
44
|
+
import math
|
|
45
|
+
studies = [
|
|
46
|
+
{"id":"S1","g":0.42,"v":0.05},
|
|
47
|
+
{"id":"S2","g":0.61,"v":0.03},
|
|
48
|
+
{"id":"S3","g":0.28,"v":0.08},
|
|
49
|
+
]
|
|
50
|
+
# Inverse-variance weights
|
|
51
|
+
for s in studies:
|
|
52
|
+
s["w"] = 1/s["v"]
|
|
53
|
+
W = sum(s["w"] for s in studies)
|
|
54
|
+
pooled = sum(s["w"]*s["g"] for s in studies)/W
|
|
55
|
+
pooled_var = 1/W
|
|
56
|
+
pooled_se = math.sqrt(pooled_var)
|
|
57
|
+
z = pooled/pooled_se
|
|
58
|
+
import scipy.stats as st
|
|
59
|
+
ci_lo = pooled - 1.96*pooled_se
|
|
60
|
+
ci_hi = pooled + 1.96*pooled_se
|
|
61
|
+
# Cochran Q
|
|
62
|
+
Q = sum(s["w"]*(s["g"]-pooled)**2 for s in studies)
|
|
63
|
+
k = len(studies)
|
|
64
|
+
p_Q = 1 - st.chi2.cdf(Q, df=k-1)
|
|
65
|
+
I2 = max(0,(Q-(k-1))/Q)*100
|
|
66
|
+
# tau^2 (DerSimonian-Laird)
|
|
67
|
+
C = W - sum(s["w"]**2 for s in studies)/W
|
|
68
|
+
tau2 = max(0,(Q-(k-1))/C)
|
|
69
|
+
print(f"Pooled g = {pooled:.3f} [{ci_lo:.3f}, {ci_hi:.3f}]")
|
|
70
|
+
print(f"Q={Q:.2f} (p={p_Q:.3f}), I²={I2:.1f}%, τ²={tau2:.4f}")
|
|
71
|
+
EOF
|
|
72
|
+
```
|
|
73
|
+
16. HARD RULE: never pool studies with incommensurable outcomes (e.g., mortality vs symptom score) even if the same label is used. Verify instrument equivalence first.
|
|
74
|
+
|
|
75
|
+
## PHASE 4 — FIXED VS RANDOM EFFECTS (justify the choice)
|
|
76
|
+
|
|
77
|
+
17. Fixed-effects (common-effect) model: assumes all studies estimate the same true effect; appropriate only when studies are near-identical in population, intervention, and outcome AND k is small. Rare in practice.
|
|
78
|
+
18. Random-effects (DerSimonian-Laird or REML): assume a distribution of true effects; appropriate when studies differ in design, population, dose, or setting — i.e., almost always.
|
|
79
|
+
19. Decision rule: If Q p < 0.10 OR I² > 50% OR prior clinical knowledge predicts heterogeneity → random-effects. Document the rationale.
|
|
80
|
+
20. For small k (< 5 studies), DL underestimates τ²; prefer REML or Paule-Mandel. Report which estimator was used.
|
|
81
|
+
|
|
82
|
+
## PHASE 5 — POOLING AND FOREST PLOT
|
|
83
|
+
|
|
84
|
+
21. Pool with inverse-variance weighting: w_i = 1/(v_i + τ²) for random-effects; w_i = 1/v_i for fixed.
|
|
85
|
+
22. Forest plot requirements: one row per study (ID, N, effect size + 95% CI, weight %), diamond for pooled estimate, vertical null line, heterogeneity stats (Q, I², τ²), and the model type. Sort by year or effect size — be consistent.
|
|
86
|
+
23. Report: pooled estimate, 95% CI, prediction interval (95% PI = pooled ± 1.96·√(τ²+pooled_var)) for random-effects. PI answers "where will the next study's effect likely fall" — often wider than CI.
|
|
87
|
+
24. If k ≥ 10, also report 95% PI. If PI crosses the null, the evidence is weaker than the CI alone implies — say so.
|
|
88
|
+
|
|
89
|
+
## PHASE 6 — HETEROGENEITY QUANTIFICATION AND EXPLORATION
|
|
90
|
+
|
|
91
|
+
25. Report Q (Cochran), p(Q), I², and τ² (with 95% CI via Q-profile or bootstrap). I² benchmarks: 25% low, 50% moderate, 75% high — but they are context-dependent, not thresholds.
|
|
92
|
+
26. Pre-specified subgroup analysis: split studies by a categorical moderator (e.g., RCT vs observational, age group, dose level). Test between-subgroup heterogeneity with Q_between. Do NOT use subgroups to explain away heterogeneity post-hoc.
|
|
93
|
+
27. Meta-regression: for continuous moderators (year, mean age, dose). Use weighted least squares (WLS) with w_i as above. Report slope, 95% CI, R²_analog (proportion of τ² explained). Require k ≥ 10 per covariate; more is better.
|
|
94
|
+
28. Limit exploratory moderators to ≤ k/10 to avoid over-fitting. Flag any post-hoc moderator clearly.
|
|
95
|
+
|
|
96
|
+
## PHASE 7 — PUBLICATION BIAS
|
|
97
|
+
|
|
98
|
+
29. Funnel plot: SE on y-axis (inverted), effect size on x-axis. Asymmetry suggests small-study effects / publication bias — but asymmetry has many causes (report all plausible ones).
|
|
99
|
+
30. Egger's test (linear regression of effect/SE on 1/SE): significant intercept (p < 0.10) flags asymmetry. Requires k ≥ 10; underpowered below that — state the limitation.
|
|
100
|
+
31. Trim-and-fill (Duval-Tweedie): impute missing studies to restore funnel symmetry; re-pool with imputed studies. Report the adjusted estimate. This is a sensitivity check, not a correction.
|
|
101
|
+
32. Contour-enhanced funnel plot: overlay significance contours (p = 0.05, 0.01) to distinguish bias from heterogeneity.
|
|
102
|
+
33. If k < 10, skip formal bias tests (insufficient power); note this explicitly. Use clinical judgment and grey-literature search as bias controls instead.
|
|
103
|
+
|
|
104
|
+
## PHASE 8 — SENSITIVITY AND ROBUSTNESS
|
|
105
|
+
|
|
106
|
+
34. Leave-one-out analysis: re-pool omitting each study in turn. If removing one study flips the conclusion, the result is fragile — report it.
|
|
107
|
+
35. Influence diagnostics: Cook's distance, DFFITS, or Baujat plot to identify outlying / high-leverage studies.
|
|
108
|
+
36. Compare fixed vs random-effects estimates. Large divergence signals heterogeneity driving the result.
|
|
109
|
+
37. Restrict to high-quality studies (e.g., low risk of bias only). If the estimate changes substantially, quality is a moderator — discuss.
|
|
110
|
+
38. If imputed SDs or converted effect sizes exist, run the analysis with and without those studies.
|
|
111
|
+
|
|
112
|
+
## PHASE 9 — RISK OF BIAS WITHIN STUDIES
|
|
113
|
+
|
|
114
|
+
39. Use the appropriate tool for each design: RoB 2 (RCTs), ROBINS-I (non-randomized), QUADAS-2 (diagnostic), NOS (observational). Apply in duplicate.
|
|
115
|
+
40. Produce a risk-of-bias summary table and figure. Do not average or collapse domains into a single score — report domain-level judgments.
|
|
116
|
+
41. Incorporate RoB into GRADE (see Phase 10): high RoB across studies downgrades certainty.
|
|
117
|
+
|
|
118
|
+
## PHASE 10 — GRADE CERTAINTY OF EVIDENCE
|
|
119
|
+
|
|
120
|
+
42. Assign a starting certainty level: RCTs = High; observational = Low.
|
|
121
|
+
43. Downgrade for: risk of bias, inconsistency (high I²/unexplained heterogeneity), indirectness (population/intervention/outcome mismatch), imprecision (wide CI, small k or N), publication bias (likely present).
|
|
122
|
+
44. Upgrade for: large magnitude (OR > 5), dose-response gradient, all plausible confounders would attenuate the effect.
|
|
123
|
+
45. Final certainty: High / Moderate / Low / Very Low. One upgrade or downgrade per domain; document each decision.
|
|
124
|
+
46. Summarize in a GRADE Summary of Findings table: outcome, N studies, N participants, pooled estimate + CI, and certainty rating.
|
|
125
|
+
|
|
126
|
+
## PHASE 11 — REPORTING STANDARDS
|
|
127
|
+
|
|
128
|
+
47. Follow PRISMA 2020 checklist (27 items). Confirm each item before submission.
|
|
129
|
+
48. Report: protocol registration, full search strings, PRISMA flow, extraction agreement, all effect sizes and variances, model justification, heterogeneity statistics, bias assessment, sensitivity results, GRADE table, forest plot, funnel plot (if k ≥ 10), and prediction interval.
|
|
130
|
+
49. Honest limitations section must address: k and total N, risk of bias distribution, language/database restrictions, inability to obtain individual participant data (IPD), and any post-hoc analyses.
|
|
131
|
+
|
|
132
|
+
## HARD RULES (never violate)
|
|
133
|
+
|
|
134
|
+
- NEVER pool incommensurable outcomes. Verify metric equivalence before computing line 1.
|
|
135
|
+
- NEVER report only I² without τ² — I² is scale-dependent and misleading alone.
|
|
136
|
+
- NEVER interpret a non-significant Egger's test as "no publication bias" — it is underpowered.
|
|
137
|
+
- NEVER perform subgroup analyses not pre-specified without labeling them exploratory.
|
|
138
|
+
- NEVER upgrade GRADE certainty without a documented, domain-specific rationale.
|
|
139
|
+
- NEVER omit the prediction interval when k ≥ 5 and random-effects are used.
|
|
140
|
+
- ALL Bash computations must print study IDs alongside estimates for auditability.
|
|
@@ -0,0 +1,117 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: migrate
|
|
3
|
+
description: Migrate a codebase safely — inventory all call sites first, use expand-migrate-contract, change in small reversible batches keeping the build green, verify semantics preserved.
|
|
4
|
+
allowed-tools: Read, Edit, MultiEdit, Bash, Grep, Glob
|
|
5
|
+
---
|
|
6
|
+
|
|
7
|
+
## PHASE 0 — DEFINE (non-negotiable before any edit)
|
|
8
|
+
|
|
9
|
+
1. State the migration contract in one sentence: "FROM `old.doThing(x, y)` TO `newLib.thing({ x, y })` — same semantics, new import path + call shape." Write it down. If you cannot write it in one sentence, the scope is unclear — clarify first.
|
|
10
|
+
2. State the SUCCESS CRITERION explicitly: what must be true when done? (e.g., "zero imports of `old-lib`, all tests green, no TypeScript errors, bundle size ≤ before".) This is the acceptance gate.
|
|
11
|
+
3. Define what is OUT OF SCOPE. Migrations attract opportunistic cleanup. Any unrelated change goes into a separate commit or a follow-up ticket — never mixed in.
|
|
12
|
+
|
|
13
|
+
## PHASE 1 — INVENTORY (know the blast radius before touching anything)
|
|
14
|
+
|
|
15
|
+
4. Find every import/require of the old API:
|
|
16
|
+
```
|
|
17
|
+
grep -rn "from ['\"]old-lib['\"]" --include="*.ts" --include="*.tsx" --include="*.js" .
|
|
18
|
+
grep -rn "require(['\"]old-lib" --include="*.ts" --include="*.js" .
|
|
19
|
+
```
|
|
20
|
+
5. Find every call site of the specific symbol(s) being replaced:
|
|
21
|
+
```
|
|
22
|
+
grep -rn "oldFunction\|OldClass\|OLD_CONSTANT" --include="*.ts" --include="*.tsx" .
|
|
23
|
+
```
|
|
24
|
+
6. Check indirect references — re-exports, barrel files, dynamic access:
|
|
25
|
+
```
|
|
26
|
+
grep -rn "export.*oldFunction\|oldFunction['\"]" --include="*.ts" .
|
|
27
|
+
grep -rn "\[.*'oldFunction'.*\]\|\['.*'\]" --include="*.ts" .
|
|
28
|
+
```
|
|
29
|
+
7. Check non-TS surfaces: config files, shell scripts, CI YAML, Dockerfiles, docs:
|
|
30
|
+
```
|
|
31
|
+
grep -rn "old-lib\|oldFunction" --include="*.json" --include="*.yaml" --include="*.sh" --include="*.md" .
|
|
32
|
+
```
|
|
33
|
+
8. Check generated code and lockfiles that embed the old API (codegen output dirs, `*.generated.ts`).
|
|
34
|
+
9. Record the full inventory. Format: `path/to/file.ts — N call sites — [mechanical|tricky]`. Label each file: **mechanical** (pure find-replace, no logic change) or **tricky** (conditional branching on the API, overloaded usage, dynamic dispatch, tests that mock internals).
|
|
35
|
+
10. Count totals. If blast radius > 50 files, plan explicit batches now (group by layer: utils → services → routes → tests).
|
|
36
|
+
|
|
37
|
+
## PHASE 2 — PREPARE
|
|
38
|
+
|
|
39
|
+
11. Capture baseline: run the full build + tests, record pass/fail counts and any pre-existing failures. Do not let pre-existing failures mask new breakage.
|
|
40
|
+
```
|
|
41
|
+
npx tsc --noEmit 2>&1 | tail -5
|
|
42
|
+
npm test 2>&1 | tail -20
|
|
43
|
+
```
|
|
44
|
+
12. Install / configure the new dependency if needed. Commit this step alone: `chore(deps): add new-lib@x.y.z`. Do not mix dependency changes with code changes.
|
|
45
|
+
13. Write or identify tests that exercise the behavior being migrated — these are your semantic anchors. If tests are thin, write characterization tests now against the OLD API, commit them, and confirm they are green. They must stay green throughout (modulo expected import-path changes after the contract step).
|
|
46
|
+
14. Set up the codemod for mechanical changes (jscodeshift / sed / ts-morph script). Dry-run it first — pipe to a temp file and diff:
|
|
47
|
+
```
|
|
48
|
+
npx jscodeshift -t codemod.ts src/ --dry --print 2>&1 | head -60
|
|
49
|
+
```
|
|
50
|
+
Never run a codemod on the whole tree in one shot without a dry-run diff review.
|
|
51
|
+
|
|
52
|
+
## PHASE 3 — EXPAND-MIGRATE-CONTRACT
|
|
53
|
+
|
|
54
|
+
The cardinal rule: **the build must stay green at every commit.**
|
|
55
|
+
|
|
56
|
+
### 3a — EXPAND (add the new path alongside the old)
|
|
57
|
+
|
|
58
|
+
15. Introduce the new API/adapter/shim without touching any callers. If it is a new library, add a thin compatibility wrapper that makes both old and new signatures coexist. Commit: `feat(migrate): introduce new-lib adapter alongside old-lib`.
|
|
59
|
+
16. Typecheck. Tests green. If anything breaks here, fix before proceeding — you have not changed any caller yet.
|
|
60
|
+
|
|
61
|
+
### 3b — MIGRATE (move callers batch by batch)
|
|
62
|
+
|
|
63
|
+
17. Migrate mechanical files first using the codemod — one logical group (e.g., one directory or layer) per commit. Never the whole tree in one commit.
|
|
64
|
+
18. After each batch:
|
|
65
|
+
a. `npx tsc --noEmit 2>&1 | head -30` — zero new errors.
|
|
66
|
+
b. `npm test -- --testPathPattern=<changed-dir> 2>&1 | tail -20` — green.
|
|
67
|
+
c. Commit: `refactor(migrate): migrate <dir> from old-lib to new-lib (N files)`.
|
|
68
|
+
19. Migrate tricky files by hand. Read the old call site, understand the semantics, rewrite — do not blindly apply the codemod. Add an inline comment if the new call is non-obvious. One file or one cluster per commit.
|
|
69
|
+
20. Maintain a checklist: after each batch, mark files DONE. Keep remaining files visible. Example tracking comment at top of your scratch list:
|
|
70
|
+
```
|
|
71
|
+
[x] src/utils/http.ts
|
|
72
|
+
[x] src/services/auth.ts
|
|
73
|
+
[ ] src/routes/api.ts ← tricky: mocks old internals
|
|
74
|
+
[ ] tests/integration/ ← update last
|
|
75
|
+
```
|
|
76
|
+
21. Migrate tests last — only after all production code is migrated and green. Updating tests simultaneously with source obscures which side introduced a failure.
|
|
77
|
+
|
|
78
|
+
### 3c — CONTRACT (remove the old)
|
|
79
|
+
|
|
80
|
+
22. Only after 100% of callers are migrated and all tests green: delete the old import, old shim, old adapter, old re-exports.
|
|
81
|
+
23. Remove the old dependency from package.json if it has no other consumers.
|
|
82
|
+
24. Typecheck with zero tolerance. Any remaining reference to the old API is a miss — go back to inventory.
|
|
83
|
+
25. Commit: `chore(migrate): remove old-lib — migration complete`.
|
|
84
|
+
|
|
85
|
+
## PHASE 4 — VERIFICATION
|
|
86
|
+
|
|
87
|
+
26. Run the FULL test suite (not the targeted subset): `npm test 2>&1 | tail -30`. Pass count must be >= baseline; no new failures allowed.
|
|
88
|
+
27. Full typecheck: `npx tsc --noEmit 2>&1` — zero errors.
|
|
89
|
+
28. Bundle / build check if applicable: compare sizes, check for accidental dual-bundling of old + new lib.
|
|
90
|
+
29. Semantic diff review — read the aggregate diff across the migration:
|
|
91
|
+
```
|
|
92
|
+
git diff <pre-migration-sha>..HEAD -- src/
|
|
93
|
+
```
|
|
94
|
+
For every changed call site ask: "Did any default value, error path, argument order, return shape, or side effect change?" Flag any divergence as a separate `fix:` commit — never silently inside the migration.
|
|
95
|
+
30. Re-run the grep from Phase 1 to confirm zero remaining references to the old API:
|
|
96
|
+
```
|
|
97
|
+
grep -rn "old-lib\|oldFunction" --include="*.ts" --include="*.tsx" --include="*.js" .
|
|
98
|
+
```
|
|
99
|
+
Expected result: zero matches (or only the codemod/migration script itself, if kept).
|
|
100
|
+
31. Check non-TS surfaces (configs, docs, CI) one more time — same grep as step 7.
|
|
101
|
+
|
|
102
|
+
## PHASE 5 — ROLLBACK PLAN
|
|
103
|
+
|
|
104
|
+
32. Every batch commit is independently revertible: `git revert <sha>`. The expand-migrate-contract pattern means you can revert the contract step and re-expose the old API with a single commit revert.
|
|
105
|
+
33. If a tricky call site reveals a semantic mismatch mid-migration, stop the batch, revert that file, document the blocker, and continue with other callers. Do not paper over a semantic difference with a shim that silently changes behavior.
|
|
106
|
+
34. If the migration must be abandoned mid-flight, the expand step ensures the old API is still present — callers already migrated are safe on the new API; un-migrated callers still use the old. The codebase is always in a buildable state.
|
|
107
|
+
|
|
108
|
+
## HARD RULES
|
|
109
|
+
|
|
110
|
+
- **Inventory before first edit.** Never touch a call site before you have the full list. Surprises mid-migration cause half-applied states.
|
|
111
|
+
- **Build green at every commit.** A red commit is never acceptable, even as "WIP." Red history makes bisect useless.
|
|
112
|
+
- **One concern per commit.** Dependency bump, adapter introduction, mechanical batch, tricky hand-edit, old API removal — each is a separate commit. Never mix migration with unrelated refactors or feature changes.
|
|
113
|
+
- **Semantic anchor tests must stay green.** If a test breaks, that is a signal of a semantic divergence — treat it as a blocker, not a test to update.
|
|
114
|
+
- **Mechanical vs tricky are different risk levels.** Run the codemod on mechanical files; hand-edit tricky ones. Misclassification is the #1 source of silent semantic drift.
|
|
115
|
+
- **Never batch the entire tree in one commit.** No matter how mechanical the change, keep batches small enough to read the diff in one sitting (~10-20 files max per commit).
|
|
116
|
+
- **Do not let the migration creep.** If you notice a related smell while migrating a file, note it, finish the migration, then address the smell in a follow-up commit.
|
|
117
|
+
- **Document the migration in the PR.** State: what changed, what was automated vs hand-edited, how semantics were verified, what the rollback path is.
|
|
@@ -0,0 +1,88 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: optimize
|
|
3
|
+
description: Performance optimization — measure and profile first, fix the real bottleneck (algorithmic before micro), re-measure after each change, keep results correct.
|
|
4
|
+
allowed-tools: Read, Edit, Bash, Grep, Glob
|
|
5
|
+
---
|
|
6
|
+
|
|
7
|
+
## Phase 0 — Define the Target (non-negotiable before touching code)
|
|
8
|
+
|
|
9
|
+
1. State the workload precisely: which code path, which inputs, what load (RPS, payload size, concurrency).
|
|
10
|
+
2. Name one primary metric and a hard target: e.g. "p99 < 50 ms at 500 RPS" or "< 200 MB RSS under 10k items" or "> 10k inserts/s".
|
|
11
|
+
3. Name secondary guard-rails: correctness (tests must stay green), p50, error rate, memory ceiling.
|
|
12
|
+
4. Write the target down before running a single benchmark. Optimization without a target is refactoring.
|
|
13
|
+
|
|
14
|
+
## Phase 1 — Establish a Reproducible Baseline
|
|
15
|
+
|
|
16
|
+
5. Pin the environment: lock CPU frequency if possible, kill background noise, use the same machine/container for every run.
|
|
17
|
+
6. Write a benchmark that drives the exact hot path — not a micro-benchmark of a helper, the real call stack.
|
|
18
|
+
- Node/TS: use `hyperfine`, `autocannon`, or `vitest bench`; for in-process: `perf_hooks.performance.now()` around a tight loop (≥ 1 000 iterations, discard first 10% as warm-up).
|
|
19
|
+
- Python: `pytest-benchmark`, `timeit`, or `locust` for HTTP.
|
|
20
|
+
- Shell timing: `hyperfine --warmup 5 --runs 20 '<cmd>'`.
|
|
21
|
+
7. Record the baseline number with units, timestamp, and git SHA. Example:
|
|
22
|
+
```
|
|
23
|
+
baseline: p50=142ms p99=380ms @500RPS sha=abc1234 2026-06-27
|
|
24
|
+
```
|
|
25
|
+
8. Run the full test suite now. All green = correctness baseline locked.
|
|
26
|
+
|
|
27
|
+
## Phase 2 — Profile to Find the Real Hot Path (never guess)
|
|
28
|
+
|
|
29
|
+
9. Attach a profiler to the benchmark workload — NOT to a toy script.
|
|
30
|
+
- Node: `node --prof server.js` → `node --prof-process isolate-*.log > profile.txt`; or `clinic flame -- node server.js`.
|
|
31
|
+
- Python: `python -m cProfile -o out.prof script.py && python -m pstats out.prof`.
|
|
32
|
+
- Native/Go: `perf record -g ./bin && perf report`.
|
|
33
|
+
10. Read the flame graph top-down. The widest frame is the bottleneck. Ignore everything < 5% of wall time.
|
|
34
|
+
11. Identify the bottleneck category before writing any fix:
|
|
35
|
+
- **Algorithmic** (O(n²), full-table scan, redundant traversal) — fix this first, always.
|
|
36
|
+
- **I/O bound** (serial awaits, N+1 queries, missing index) — batch or parallelize.
|
|
37
|
+
- **Memory/GC** (high allocation rate, large retained objects) — reduce allocations or pool.
|
|
38
|
+
- **CPU bound after algo fix** (tight loop, serialization) — then micro-optimize or parallelize.
|
|
39
|
+
|
|
40
|
+
## Phase 3 — Attack the Biggest Bottleneck First
|
|
41
|
+
|
|
42
|
+
12. **Algorithmic wins first.** O(n log n) sort beats O(n²) loop by 1000x at n=10k. No constant-factor trick competes.
|
|
43
|
+
- Replace nested loops over the same collection with a Map/Set lookup.
|
|
44
|
+
- Replace repeated `.filter().find()` chains with a single indexed Map built once.
|
|
45
|
+
- Replace synchronous sequential I/O with `Promise.all` or a connection pool.
|
|
46
|
+
13. **Caching / memoization.** Cache the result of pure or slow functions keyed on inputs. Use `Map` for in-process; Redis for cross-process. Always set a TTL or max-size — unbounded caches become memory leaks.
|
|
47
|
+
14. **Batching.** Replace N individual DB/network calls with one batched call. Use DataLoader pattern for per-request batching. Batch size sweet spot: profile it (often 50–500 rows).
|
|
48
|
+
15. **N+1 elimination.** In ORMs/query layers: eager-load relations in one query; never query inside a loop.
|
|
49
|
+
16. **Better data structures.** Use a `Set` for membership tests (O(1) vs O(n) array). Use a sorted array + binary search for range queries. Use a `Map` keyed by ID instead of `.find()` on an array.
|
|
50
|
+
17. **Lazy / streaming.** Don't load 100k rows into memory to process 10. Use cursors, generators, or streaming transforms. In Node: `Readable.from()`, async generators, stream pipelines.
|
|
51
|
+
18. **Parallelism.** CPU-bound work: worker_threads (Node) or multiprocessing (Python). I/O-bound: concurrency limit with `p-limit` or semaphore (avoid unlimited `Promise.all` — it causes thundering herd).
|
|
52
|
+
19. **Reduce allocations.** In hot loops: reuse buffers, avoid string concatenation (use arrays + `.join`), avoid object spread in tight paths. Profile GC pause time separately (`--expose-gc` + `gc()` + `process.memoryUsage()`).
|
|
53
|
+
20. **Serialization.** JSON.parse/stringify on large payloads is expensive. Consider `fast-json-stringify`, MessagePack, or schema-aware codecs. Profile before switching — overhead is payload-size dependent.
|
|
54
|
+
|
|
55
|
+
## Phase 4 — Re-Measure After EVERY Change
|
|
56
|
+
|
|
57
|
+
21. After each fix: re-run the exact same benchmark. Record the new number with the git SHA of the fix.
|
|
58
|
+
22. Re-run the full test suite. A failing test means the optimization changed behavior — revert or fix correctness first.
|
|
59
|
+
23. Compare against baseline AND the previous measurement to isolate the contribution of each change.
|
|
60
|
+
24. If the metric did not improve by at least 10%, consider reverting — complexity has a cost.
|
|
61
|
+
|
|
62
|
+
## Phase 5 — Watch for Regressions
|
|
63
|
+
|
|
64
|
+
25. Check all guard-rail metrics (p50, memory, error rate) after each change — a p99 win that doubles memory is not a win.
|
|
65
|
+
26. Run the benchmark at 2x the target load to check for cliff edges (sudden latency spikes, OOM).
|
|
66
|
+
27. If optimization shifts the hot path to a new bottleneck, profile again (Phase 2) before touching anything else.
|
|
67
|
+
|
|
68
|
+
## Phase 6 — Stop When the Target Is Hit
|
|
69
|
+
|
|
70
|
+
28. When the primary metric clears the target defined in Phase 0, stop. Over-optimizing past the target trades maintainability for no user benefit.
|
|
71
|
+
29. Document the win in a comment or commit message with before/after numbers, the bottleneck type, and the fix applied. Example:
|
|
72
|
+
```
|
|
73
|
+
perf: replace O(n²) agent lookup with Map index
|
|
74
|
+
Before: p99=380ms After: p99=31ms (-92%) sha range abc1234..def5678
|
|
75
|
+
Bottleneck: agents.find() inside per-event loop (N×M = 500k iterations)
|
|
76
|
+
Fix: build Map<id,agent> once before loop, O(1) lookup per event
|
|
77
|
+
```
|
|
78
|
+
30. Add the benchmark to CI as a regression guard with a threshold (e.g. fail if p99 > 60ms).
|
|
79
|
+
|
|
80
|
+
## Hard Rules
|
|
81
|
+
|
|
82
|
+
- NEVER optimize without a profile showing the line is hot. Gut-feel micro-opts waste time and add bugs.
|
|
83
|
+
- ALWAYS keep tests green. Correctness is not negotiable.
|
|
84
|
+
- ONE change per measurement cycle. Mixing two changes makes it impossible to know which helped.
|
|
85
|
+
- Algorithmic complexity before constant-factor tricks, always.
|
|
86
|
+
- Caches MUST have eviction. Unbounded = memory leak.
|
|
87
|
+
- Parallelism MUST have a concurrency cap. Unlimited = thundering herd or OOM.
|
|
88
|
+
- Document every win with numbers. "It felt faster" is not a result.
|