heimdall-agent 1.0.0 → 1.1.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (231) hide show
  1. package/README.md +57 -26
  2. package/README.pt-BR.md +429 -0
  3. package/agents/cloud-engineer.md +77 -0
  4. package/agents/code-sentinel.md +66 -35
  5. package/agents/database-architect.md +31 -25
  6. package/agents/frontend-designer.md +30 -28
  7. package/agents/mobile-designer.md +25 -24
  8. package/agents/modernizer.md +66 -0
  9. package/agents/orchestrator.md +94 -93
  10. package/agents/reviewer.md +100 -71
  11. package/agents/security-auditor.md +27 -27
  12. package/agents/tester.md +97 -90
  13. package/commands/grill-me.md +20 -4
  14. package/commands/orchestrate.md +16 -0
  15. package/commands/orchestrator.md +16 -0
  16. package/commands/plan.md +17 -1
  17. package/config/agents.json +21 -3
  18. package/config/mcp.json +72 -0
  19. package/config/models.json +26 -16
  20. package/config/profiles.json +2 -4
  21. package/config/skills.json +210 -100
  22. package/docs/SKILLS.md +341 -245
  23. package/docs/designs/old/2026-10-31 plan-lifecycle-backlog-and-clean-code-skill.md +87 -0
  24. package/docs/designs/old/2026-11-02 skill-upstream-audit-and-update.md +105 -0
  25. package/docs/designs/old/2026-11-05 communication-doctrine-bluf-ste.md +94 -0
  26. package/docs/designs/old/2026-11-06-cloud-engineer-and-resilience-synthesis.md +67 -0
  27. package/docs/designs/old/2026-11-06-herdr-native-agent-orchestration.md +45 -0
  28. package/docs/runs/skill-audit/add_progressive_disclosure.cjs +30 -0
  29. package/docs/runs/skill-audit/check_heimdall_skills.ps1 +9 -0
  30. package/docs/runs/skill-audit/compare_classics.ps1 +26 -0
  31. package/docs/runs/skill-audit/compare_taste.ps1 +12 -0
  32. package/docs/runs/skill-audit/customization-report.json +380 -0
  33. package/docs/runs/skill-audit/diff_pending_skills.cjs +94 -0
  34. package/docs/runs/skill-audit/fetch_and_apply_skill.cjs +58 -0
  35. package/docs/runs/skill-audit/fetch_and_apply_taste_v2.cjs +65 -0
  36. package/docs/runs/skill-audit/inspect_pending_skills.cjs +71 -0
  37. package/docs/runs/skill-audit/inventory.json +308 -0
  38. package/docs/runs/skill-audit/list_local_taste.ps1 +1 -0
  39. package/docs/runs/skill-audit/phase5-update-plan.json +595 -0
  40. package/docs/runs/skill-audit/phase5-update-plan.md +138 -0
  41. package/docs/runs/skill-audit/probe.ps1 +16 -0
  42. package/docs/runs/skill-audit/probe_classics_branches.ps1 +3 -0
  43. package/docs/runs/skill-audit/probe_commits.ps1 +15 -0
  44. package/docs/runs/skill-audit/probe_repos.ps1 +17 -0
  45. package/docs/runs/skill-audit/probe_sentry_sec.ps1 +3 -0
  46. package/docs/runs/skill-audit/probe_sentry_sweep.ps1 +5 -0
  47. package/docs/runs/skill-audit/probe_snyk.ps1 +3 -0
  48. package/docs/runs/skill-audit/probe_snyk_recipes.ps1 +5 -0
  49. package/docs/runs/skill-audit/probe_taste_branches.ps1 +8 -0
  50. package/docs/runs/skill-audit/probe_taste_v19.ps1 +3 -0
  51. package/docs/runs/skill-audit/probe_user_repos.ps1 +19 -0
  52. package/docs/runs/skill-audit/prune_taste_skills.cjs +22 -0
  53. package/docs/runs/skill-audit/run_phase1_2.cjs +142 -0
  54. package/docs/runs/skill-audit/run_phase6_verification.cjs +107 -0
  55. package/docs/runs/skill-audit/show_16_details.ps1 +25 -0
  56. package/docs/runs/skill-audit/show_20_skills.ps1 +28 -0
  57. package/docs/runs/skill-audit/show_classics_diff.cjs +37 -0
  58. package/docs/runs/skill-audit/sync_agent_markdowns.cjs +30 -0
  59. package/docs/runs/skill-audit/taste-skill-upstream.md +542 -0
  60. package/docs/runs/skill-audit/update-manifest.md +88 -0
  61. package/docs/runs/skill-audit/update_config_skills.cjs +52 -0
  62. package/docs/runs/skill-audit/update_registry_phase7.cjs +83 -0
  63. package/docs/runs/skill-audit/upstream-status.json +563 -0
  64. package/docs/runs/skill-audit/verification-results.json +252 -0
  65. package/doctrine/AGENTS.md +39 -77
  66. package/hooks/git/pre-commit +11 -0
  67. package/hooks/heimdall-hook.cjs +81 -2
  68. package/hooks/herdr-agent-state.ps1 +55 -0
  69. package/hooks/hooks.json +0 -11
  70. package/lib/catalog-sync.cjs +162 -0
  71. package/lib/hermes-context.cjs +57 -0
  72. package/lib/model-context.cjs +123 -0
  73. package/lib/path-discovery.cjs +94 -0
  74. package/package.json +7 -3
  75. package/playbooks/01-interactive-grilling-gate.md +14 -1
  76. package/playbooks/02-fast-code-deep-verification.md +3 -2
  77. package/playbooks/03-adversarial-code-review.md +3 -3
  78. package/playbooks/05-canonical-plan-template.md +32 -18
  79. package/playbooks/06-model-registry-matrix.md +24 -13
  80. package/playbooks/07-legacy-code-modernization.md +74 -0
  81. package/playbooks/08-backlog-and-change-request-flow.md +122 -0
  82. package/setup.mjs +592 -172
  83. package/skills/antislop/SKILL.md +76 -56
  84. package/skills/antislop/VERSION +1 -0
  85. package/skills/antislop-copywriting/SKILL.md +372 -372
  86. package/skills/antislop-human/LICENSE-web-quality-skills +21 -0
  87. package/skills/antislop-human/SKILL.md +204 -147
  88. package/skills/antislop-human/contrast-check.py +168 -165
  89. package/skills/antislop-human/contrast-mcp.py +6 -3
  90. package/skills/antislop-human/references/A11Y-PATTERNS.md +249 -0
  91. package/skills/antislop-human/references/WCAG.md +127 -0
  92. package/skills/antislop-ui/LICENSE-web-quality-skills +21 -0
  93. package/skills/antislop-ui/SKILL.md +344 -304
  94. package/skills/antislop-ui/references/CORE-WEB-VITALS.md +181 -0
  95. package/skills/antislop-ui/references/WEB-BASELINE.md +28 -0
  96. package/skills/brainstorming/SKILL.md +274 -265
  97. package/skills/brainstorming/builder-check-prompt.md +23 -0
  98. package/skills/brainstorming/scripts/frame-template.html +213 -213
  99. package/skills/brainstorming/scripts/server.cjs +723 -723
  100. package/skills/brandkit/SKILL.md +798 -798
  101. package/skills/brutalist-skill/SKILL.md +93 -92
  102. package/skills/caveman/README.md +68 -0
  103. package/skills/caveman/SKILL.md +73 -38
  104. package/skills/claude-collaborator/SKILL.md +24 -44
  105. package/skills/clean-code/SKILL.md +152 -0
  106. package/skills/cloudflare-security-audit/AI-AND-LLM.md +83 -83
  107. package/skills/cloudflare-security-audit/ATTACK-CLASSES.md +130 -130
  108. package/skills/cloudflare-security-audit/CLIENT-SIDE.md +83 -83
  109. package/skills/cloudflare-security-audit/CLOUD-AND-DEPLOYMENT.md +86 -86
  110. package/skills/cloudflare-security-audit/DATA-ISOLATION-AND-LIFECYCLE.md +84 -84
  111. package/skills/cloudflare-security-audit/DESKTOP-MOBILE-AND-LOCAL-IPC.md +89 -89
  112. package/skills/cloudflare-security-audit/HUNTING.md +251 -251
  113. package/skills/cloudflare-security-audit/MEMORY-SAFETY-AND-BINARY.md +101 -101
  114. package/skills/cloudflare-security-audit/PROTOCOLS-RPC-AND-MESSAGING.md +81 -81
  115. package/skills/cloudflare-security-audit/RECONNAISSANCE.md +156 -156
  116. package/skills/cloudflare-security-audit/RESOURCE-EXHAUSTION-AND-AVAILABILITY.md +78 -78
  117. package/skills/cloudflare-security-audit/SKILL.md +192 -192
  118. package/skills/cloudflare-security-audit/SUPPLY-CHAIN-AND-RELEASE.md +73 -73
  119. package/skills/cloudflare-security-audit/VALIDATION-AND-REPORTING.md +186 -186
  120. package/skills/cloudflare-security-audit/WEB-PROTOCOL-AND-AUTH.md +105 -105
  121. package/skills/cloudflare-security-audit/report-schema.json +461 -461
  122. package/skills/cloudflare-security-audit/validate-coverage-ledger.cjs +872 -872
  123. package/skills/cloudflare-security-audit/validate-coverage-ledger.test.cjs +740 -740
  124. package/skills/cloudflare-security-audit/validate-findings.cjs +773 -773
  125. package/skills/cloudflare-security-audit/validate-findings.test.cjs +652 -652
  126. package/skills/codex-collaborator/SKILL.md +24 -45
  127. package/skills/communication-doctrine/SKILL.md +89 -0
  128. package/skills/diagnosing-superpowers/SKILL.md +1 -1
  129. package/skills/diagnosing-superpowers/references/github-issues.md +3 -3
  130. package/skills/diagnosing-superpowers/templates/issue.md +23 -4
  131. package/skills/executing-plans/SKILL.md +0 -52
  132. package/skills/executing-plans/scripts/task-done +52 -52
  133. package/skills/executing-plans/scripts/task-start +28 -28
  134. package/skills/gemini-collaborator/SKILL.md +24 -56
  135. package/skills/herdr/SKILL.md +243 -0
  136. package/skills/image-to-code-skill/SKILL.md +1228 -1228
  137. package/skills/imagegen-frontend-mobile/SKILL.md +1465 -1465
  138. package/skills/imagegen-frontend-web/SKILL.md +987 -987
  139. package/skills/karpathy-guidelines/SKILL.md +67 -67
  140. package/skills/mcp-engineering/SKILL.md +74 -0
  141. package/skills/minimalist-skill/SKILL.md +87 -85
  142. package/skills/output-skill/SKILL.md +49 -49
  143. package/skills/ponytail/SKILL.md +36 -52
  144. package/skills/postgres-best-practices/SKILL.md +30 -52
  145. package/skills/postgres-best-practices/references/backup-restore.md +372 -0
  146. package/skills/postgres-best-practices/references/bulk-loading.md +315 -0
  147. package/skills/postgres-best-practices/references/connection-pooling.md +310 -0
  148. package/skills/postgres-best-practices/references/hot-standby.md +221 -0
  149. package/skills/postgres-best-practices/references/indexing.md +318 -0
  150. package/skills/postgres-best-practices/references/logical-replication.md +610 -0
  151. package/skills/postgres-best-practices/references/major-version-upgrades.md +334 -0
  152. package/skills/postgres-best-practices/references/performance-diagnostics.md +553 -0
  153. package/skills/postgres-best-practices/references/query-optimization.md +294 -0
  154. package/skills/postgres-best-practices/references/query-patterns.md +566 -0
  155. package/skills/postgres-best-practices/references/schema-design.md +362 -0
  156. package/skills/postgres-best-practices/references/security-roles.md +489 -0
  157. package/skills/postgres-best-practices/references/transaction-isolation.md +344 -0
  158. package/skills/progressive-disclosure/SKILL.md +68 -0
  159. package/skills/pstack-discipline/SKILL.md +40 -0
  160. package/skills/receiving-code-review/SKILL.md +2 -23
  161. package/skills/requesting-code-review/SKILL.md +7 -36
  162. package/skills/soft-skill/SKILL.md +99 -98
  163. package/skills/stitch-skill/DESIGN.md +121 -121
  164. package/skills/stitch-skill/SKILL.md +185 -184
  165. package/skills/strix-security/SKILL.md +54 -53
  166. package/skills/subagent-driven-development/SKILL.md +28 -260
  167. package/skills/subagent-driven-development/references/batching.md +11 -0
  168. package/skills/subagent-driven-development/references/cannot-verify.md +10 -0
  169. package/skills/subagent-driven-development/references/final-review.md +47 -0
  170. package/skills/subagent-driven-development/references/fix-loop.md +77 -0
  171. package/skills/subagent-driven-development/references/handling-reports.md +15 -0
  172. package/skills/subagent-driven-development/references/preflight.md +26 -0
  173. package/skills/subagent-driven-development/references/resuming.md +12 -0
  174. package/skills/subagent-driven-development/scripts/review-package +53 -53
  175. package/skills/subagent-driven-development/scripts/sdd-workspace +87 -82
  176. package/skills/subagent-driven-development/scripts/task-brief +43 -43
  177. package/skills/systematic-debugging/SKILL.md +283 -306
  178. package/skills/taste-skill/SKILL.md +542 -1206
  179. package/skills/taste-skill/references/app-screens.md +1199 -0
  180. package/skills/taste-skill/references/colour.md +141 -0
  181. package/skills/taste-skill/references/copy.md +177 -0
  182. package/skills/taste-skill/references/critique.md +182 -0
  183. package/skills/taste-skill/references/imagery.md +791 -0
  184. package/skills/taste-skill/references/layout.md +292 -0
  185. package/skills/taste-skill/references/motion.md +980 -0
  186. package/skills/taste-skill/references/research.md +356 -0
  187. package/skills/taste-skill/references/scenes.md +174 -0
  188. package/skills/taste-skill/references/tells.md +115 -0
  189. package/skills/taste-skill/references/type.md +170 -0
  190. package/skills/test-driven-development/SKILL.md +330 -337
  191. package/skills/thermo/SKILL.md +67 -0
  192. package/skills/threat-modeling/SKILL.md +55 -0
  193. package/skills/trailofbits-audit-context/SKILL.md +72 -72
  194. package/skills/trailofbits-insecure-defaults/commands/audit.md +35 -0
  195. package/skills/trailofbits-insecure-defaults/references/debug-features.json +13 -13
  196. package/skills/trailofbits-insecure-defaults/references/debug-features.md +75 -75
  197. package/skills/trailofbits-insecure-defaults/references/default-credentials.json +12 -12
  198. package/skills/trailofbits-insecure-defaults/references/default-credentials.md +76 -76
  199. package/skills/trailofbits-insecure-defaults/references/fail-open-security.json +11 -11
  200. package/skills/trailofbits-insecure-defaults/references/fail-open-security.md +70 -70
  201. package/skills/trailofbits-insecure-defaults/references/fallback-secrets.json +11 -11
  202. package/skills/trailofbits-insecure-defaults/references/fallback-secrets.md +70 -70
  203. package/skills/trailofbits-insecure-defaults/references/permissive-access.json +13 -13
  204. package/skills/trailofbits-insecure-defaults/references/permissive-access.md +62 -62
  205. package/skills/trailofbits-insecure-defaults/references/weak-crypto.json +13 -13
  206. package/skills/trailofbits-insecure-defaults/references/weak-crypto.md +71 -71
  207. package/skills/trailofbits-triage-brocards/SKILL.md +208 -208
  208. package/skills/verification-before-completion/SKILL.md +0 -14
  209. package/skills/writing-plans/SKILL.md +11 -15
  210. package/skills/writing-skills/SKILL.md +21 -303
  211. package/skills/writing-skills/references/CLAUDE_MD_TESTING.md +189 -0
  212. package/skills/writing-skills/references/anthropic-best-practices.md +1150 -0
  213. package/skills/writing-skills/references/bulletproofing.md +80 -0
  214. package/skills/writing-skills/references/discovery.md +201 -0
  215. package/skills/writing-skills/references/flowcharts.md +37 -0
  216. package/skills/writing-skills/references/graphviz-conventions.dot +172 -0
  217. package/skills/writing-skills/references/persuasion-principles.md +187 -0
  218. package/skills/writing-skills/references/testing-skills-with-subagents.md +384 -0
  219. package/tools/{claude-bridge.ps1 → archive/claude-bridge.ps1} +10 -7
  220. package/tools/{codex-bridge.ps1 → archive/codex-bridge.ps1} +1 -1
  221. package/tools/{gemini-bridge.ps1 → archive/gemini-bridge.ps1} +12 -19
  222. package/skills/gpt-tasteskill/SKILL.md +0 -74
  223. package/skills/taste-skill-v1/SKILL.md +0 -226
  224. /package/tools/{claude-bridge.cmd → archive/claude-bridge.cmd} +0 -0
  225. /package/tools/{claude-bridge.sh → archive/claude-bridge.sh} +0 -0
  226. /package/tools/{codex-bridge.cmd → archive/codex-bridge.cmd} +0 -0
  227. /package/tools/{codex-bridge.sh → archive/codex-bridge.sh} +0 -0
  228. /package/tools/{codexcli-bridge.cmd → archive/codexcli-bridge.cmd} +0 -0
  229. /package/tools/{codexcli-bridge.ps1 → archive/codexcli-bridge.ps1} +0 -0
  230. /package/tools/{gemini-bridge.cmd → archive/gemini-bridge.cmd} +0 -0
  231. /package/tools/{gemini-bridge.sh → archive/gemini-bridge.sh} +0 -0
@@ -0,0 +1,221 @@
1
+ # Hot Standby & Read Replicas Reference
2
+
3
+ ## Contents
4
+ - Streaming replication overview
5
+ - Hot standby configuration
6
+ - Monitoring replication lag
7
+ - Replication conflicts
8
+ - Synchronous vs asynchronous replication
9
+ - Promoting a standby
10
+
11
+ ## Streaming Replication Overview
12
+
13
+ Physical (streaming) replication creates an exact copy of the primary database on one or more standby servers. Unlike logical replication, it replicates the entire cluster (all databases, all objects) at the WAL level.
14
+
15
+ **Hot standby** allows read-only queries on the standby while it continuously replays WAL from the primary. This is the standard mechanism for PostgreSQL read replicas.
16
+
17
+ ### Key Differences from Logical Replication
18
+
19
+ | Aspect | Physical (streaming) | Logical |
20
+ |--------|---------------------|---------|
21
+ | Scope | Entire cluster | Selected tables |
22
+ | PG versions | Must match major version | Can differ |
23
+ | Standby writable? | No (read-only) | Yes (for non-replicated tables) |
24
+ | DDL replicated? | Yes (via WAL) | No |
25
+ | Use case | HA, read replicas | CDC, selective sync, cross-version migration |
26
+
27
+ ## Hot Standby Configuration
28
+
29
+ ### On the Primary
30
+
31
+ ```sql
32
+ -- Check current settings
33
+ SHOW wal_level; -- must be 'replica' or 'logical'
34
+ SHOW max_wal_senders; -- must have available sender slots
35
+ SHOW max_replication_slots; -- one per standby for slot-based replication
36
+ ```
37
+
38
+ Key settings (in `postgresql.conf`):
39
+ - `wal_level = replica` (default)
40
+ - `max_wal_senders = 10` (enough for all standbys)
41
+ - `max_replication_slots = 10` (optional but recommended — prevents WAL removal before standby catches up)
42
+
43
+ ### On the Standby
44
+
45
+ Key settings:
46
+ - `hot_standby = on` (allows read queries during recovery — default)
47
+ - `primary_conninfo = 'host=primary_host user=repl_user ...'` (connection to primary)
48
+ - `primary_slot_name = 'standby_slot'` (optional — use a replication slot)
49
+
50
+ ### Replication Slots (Recommended)
51
+
52
+ Slots prevent the primary from removing WAL segments before the standby has replayed them:
53
+
54
+ ```sql
55
+ -- On the primary: create a physical replication slot
56
+ SELECT pg_create_physical_replication_slot('standby1_slot');
57
+
58
+ -- List slots
59
+ SELECT slot_name, slot_type, active, restart_lsn
60
+ FROM pg_replication_slots;
61
+
62
+ -- Drop a slot (if standby is permanently removed)
63
+ SELECT pg_drop_replication_slot('standby1_slot');
64
+ ```
65
+
66
+ **Warning**: An inactive slot causes WAL to accumulate indefinitely on the primary, potentially filling the disk.
67
+
68
+ ## Monitoring Replication Lag
69
+
70
+ ### On the Primary: pg_stat_replication
71
+
72
+ ```sql
73
+ SELECT
74
+ pid,
75
+ application_name,
76
+ client_addr,
77
+ state,
78
+ sent_lsn,
79
+ write_lsn,
80
+ flush_lsn,
81
+ replay_lsn,
82
+ pg_size_pretty(pg_wal_lsn_diff(sent_lsn, replay_lsn)) AS replay_lag,
83
+ pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), sent_lsn)) AS send_lag,
84
+ write_lag,
85
+ flush_lag,
86
+ replay_lag AS replay_lag_time
87
+ FROM pg_stat_replication;
88
+ ```
89
+
90
+ **LSN progression**: `pg_current_wal_lsn()` → `sent_lsn` → `write_lsn` → `flush_lsn` → `replay_lsn`. The gap between any two is a measure of lag at that stage.
91
+
92
+ ### On the Standby: pg_stat_wal_receiver
93
+
94
+ ```sql
95
+ SELECT
96
+ status,
97
+ written_lsn,
98
+ flushed_lsn,
99
+ latest_end_lsn,
100
+ latest_end_time,
101
+ slot_name,
102
+ conninfo
103
+ FROM pg_stat_wal_receiver;
104
+ ```
105
+
106
+ ### On the Standby: Lag in Seconds
107
+
108
+ ```sql
109
+ -- How far behind is the standby?
110
+ SELECT
111
+ now() - pg_last_xact_replay_timestamp() AS replay_lag,
112
+ pg_last_wal_receive_lsn() AS received_lsn,
113
+ pg_last_wal_replay_lsn() AS replayed_lsn,
114
+ pg_wal_lsn_diff(pg_last_wal_receive_lsn(), pg_last_wal_replay_lsn()) AS replay_lag_bytes;
115
+ ```
116
+
117
+ **Caveat**: `pg_last_xact_replay_timestamp()` only updates when transactions are replayed. On an idle primary, it may show a large lag even though the standby is fully caught up. Check `received_lsn = replayed_lsn` for actual status.
118
+
119
+ ### Is the Standby in Recovery?
120
+
121
+ ```sql
122
+ -- Returns true on a standby, false on a primary
123
+ SELECT pg_is_in_recovery();
124
+ ```
125
+
126
+ ## Replication Conflicts
127
+
128
+ On a hot standby, long-running read queries can conflict with WAL replay. When replay needs to apply changes that conflict with an active query (e.g., dropping a table, vacuuming rows the query is reading), PostgreSQL must choose: wait for the query or cancel it.
129
+
130
+ ### max_standby_streaming_delay
131
+
132
+ Controls how long the standby waits for conflicting queries before cancelling them:
133
+
134
+ ```sql
135
+ SHOW max_standby_streaming_delay; -- default: 30s
136
+ ```
137
+
138
+ - `30s` (default): Standby waits up to 30 seconds, then cancels conflicting queries
139
+ - `-1`: Wait forever (replay pauses until the query finishes — can cause unbounded lag)
140
+ - `0`: Cancel conflicting queries immediately (minimal lag, but queries may fail)
141
+
142
+ ### max_standby_archive_delay
143
+
144
+ Same concept but for WAL segments being replayed from archive (rather than streaming):
145
+
146
+ ```sql
147
+ SHOW max_standby_archive_delay; -- default: 30s
148
+ ```
149
+
150
+ ### The Common Error
151
+
152
+ When a query on the standby is cancelled due to a conflict:
153
+
154
+ ```
155
+ ERROR: canceling statement due to conflict with recovery
156
+ DETAIL: User was holding shared buffer pin for too long.
157
+ ```
158
+
159
+ **Solutions (in order of preference):**
160
+ 1. Keep queries on the standby short
161
+ 2. Increase `max_standby_streaming_delay` (trades lag for query stability)
162
+ 3. Enable `hot_standby_feedback` (see below)
163
+ 4. Use a logical replica instead of hot standby for long-running analytics
164
+
165
+ ### hot_standby_feedback
166
+
167
+ When enabled, the standby informs the primary about which rows it still needs, preventing the primary's VACUUM from removing them:
168
+
169
+ ```sql
170
+ SHOW hot_standby_feedback; -- default: off
171
+ ```
172
+
173
+ **Pros**: Eliminates most replication conflicts — long queries on the standby won't be cancelled.
174
+
175
+ **Cons**: Can cause table bloat on the primary because VACUUM can't remove dead rows that the standby still references.
176
+
177
+ **Recommendation**: Enable only if you have long-running analytical queries on the standby and can tolerate some extra bloat on the primary.
178
+
179
+ ## Synchronous vs Asynchronous Replication
180
+
181
+ ### Asynchronous (Default)
182
+
183
+ The primary doesn't wait for standby acknowledgment before committing. Fastest, but a primary failure can lose recently committed transactions not yet replicated.
184
+
185
+ ### Synchronous
186
+
187
+ The primary waits for at least one standby to confirm before returning commit success:
188
+
189
+ ```sql
190
+ -- On the primary
191
+ SHOW synchronous_standby_names; -- e.g., 'standby1'
192
+ SHOW synchronous_commit; -- 'on', 'remote_write', 'remote_apply', etc.
193
+ ```
194
+
195
+ Synchronous commit levels:
196
+
197
+ | Level | Primary waits for | Durability | Latency impact |
198
+ |-------|-------------------|------------|----------------|
199
+ | `on` (default with sync standbys) | Standby WAL flush | Strong | Moderate |
200
+ | `remote_write` | Standby WAL write (not fsync) | Good | Lower |
201
+ | `remote_apply` | Standby WAL replay | Strongest (read-your-writes on standby) | Highest |
202
+ | `local` | Local WAL flush only | Primary only | None |
203
+ | `off` | Nothing | Weakest | None |
204
+
205
+ ## Promoting a Standby
206
+
207
+ To promote a standby to become the new primary (failover):
208
+
209
+ ```sql
210
+ -- On the standby:
211
+ SELECT pg_promote();
212
+
213
+ -- Or from the command line:
214
+ -- pg_ctl promote -D /path/to/data
215
+ ```
216
+
217
+ After promotion:
218
+ - The standby stops replay and opens for writes
219
+ - Applications must be redirected to the new primary
220
+ - Other standbys must be reconfigured to follow the new primary
221
+ - The old primary must not be restarted without reconfiguring (risk of split-brain)
@@ -0,0 +1,318 @@
1
+ # Indexing Reference
2
+
3
+ ## Contents
4
+ - Index types and when to use each
5
+ - Composite index column ordering
6
+ - Partial indexes
7
+ - Covering indexes (INCLUDE)
8
+ - Expression indexes
9
+ - Index maintenance and bloat
10
+ - Finding unused and duplicate indexes
11
+
12
+ ## Index Types
13
+
14
+ ### B-tree (Default)
15
+
16
+ Supports: `=`, `<`, `>`, `<=`, `>=`, `BETWEEN`, `IN`, `IS NULL`, `IS NOT NULL`
17
+
18
+ ```sql
19
+ CREATE INDEX idx_orders_created ON orders(created_at);
20
+ ```
21
+
22
+ Best for: equality and range queries on scalar types. The default and most common choice.
23
+
24
+ ### GIN (Generalized Inverted Index)
25
+
26
+ Supports: containment operators on composite values.
27
+
28
+ ```sql
29
+ -- JSONB containment
30
+ CREATE INDEX idx_data_gin ON events USING gin(payload);
31
+ -- Matches: WHERE payload @> '{"status": "active"}'
32
+
33
+ -- Array containment
34
+ CREATE INDEX idx_tags_gin ON articles USING gin(tags);
35
+ -- Matches: WHERE tags @> ARRAY['postgres']
36
+
37
+ -- Full-text search
38
+ CREATE INDEX idx_fts ON articles USING gin(to_tsvector('english', body));
39
+ -- Matches: WHERE to_tsvector('english', body) @@ to_tsquery('postgres & index')
40
+
41
+ -- Trigram (fuzzy text search)
42
+ CREATE EXTENSION IF NOT EXISTS pg_trgm;
43
+ CREATE INDEX idx_name_trgm ON users USING gin(name gin_trgm_ops);
44
+ -- Matches: WHERE name ILIKE '%pattern%'
45
+ ```
46
+
47
+ **Operator class tip**: For JSONB, `jsonb_path_ops` is 2-3x smaller than the default `jsonb_ops` but only supports `@>` (containment). Use it when you only need containment queries:
48
+
49
+ ```sql
50
+ CREATE INDEX idx_events_gin_path ON events USING gin(payload jsonb_path_ops);
51
+ -- Only supports: WHERE payload @> '{"type": "click"}'
52
+ -- Does NOT support: WHERE payload ? 'type'
53
+ ```
54
+
55
+ **Stored generated column for FTS**:
56
+
57
+ ```sql
58
+ ALTER TABLE articles ADD COLUMN search_vector tsvector
59
+ GENERATED ALWAYS AS (to_tsvector('english', title || ' ' || body)) STORED;
60
+ CREATE INDEX idx_articles_search ON articles USING gin(search_vector);
61
+ -- Query: WHERE search_vector @@ to_tsquery('english', 'postgres & optimization')
62
+ ```
63
+
64
+ GIN indexes are larger and slower to update than B-tree but excel at multi-valued containment queries.
65
+
66
+ **Parallel GIN builds (PG18+)**: GIN indexes can now be built in parallel, significantly speeding up index creation on large tables.
67
+
68
+ ### GiST (Generalized Search Tree)
69
+
70
+ Supports: overlap, containment, nearest-neighbor on geometric, range, and full-text types.
71
+
72
+ ```sql
73
+ -- Range overlap
74
+ CREATE INDEX idx_booking_range ON bookings USING gist(during);
75
+ -- Matches: WHERE during && '[2024-01-01, 2024-02-01)'
76
+
77
+ -- Used in exclusion constraints
78
+ CREATE EXTENSION IF NOT EXISTS btree_gist;
79
+ ALTER TABLE bookings
80
+ ADD CONSTRAINT bookings_no_overlap
81
+ EXCLUDE USING gist (room_id WITH =, during WITH &&);
82
+ ```
83
+
84
+ ### BRIN (Block Range Index)
85
+
86
+ Tiny index for naturally ordered data (e.g., append-only timestamp columns).
87
+
88
+ ```sql
89
+ CREATE INDEX idx_events_ts_brin ON events USING brin(occurred_at);
90
+ ```
91
+
92
+ BRIN stores min/max per block range. Effective when physical row order correlates with column value. Very small (~0.1% of B-tree size) but less precise — may read extra blocks.
93
+
94
+ ### Hash
95
+
96
+ Only supports `=`. Rarely better than B-tree.
97
+
98
+ ```sql
99
+ CREATE INDEX idx_session_hash ON sessions USING hash(session_token);
100
+ ```
101
+
102
+ ## Composite Index Column Ordering
103
+
104
+ Order matters. The index is a sorted tree, leftmost column first.
105
+
106
+ **Rule of thumb**: equality columns first, then range/sort columns.
107
+
108
+ ```sql
109
+ -- Query: WHERE tenant_id = 1 AND created_at > '2024-01-01' ORDER BY created_at
110
+ CREATE INDEX idx_tenant_created ON orders(tenant_id, created_at);
111
+ ```
112
+
113
+ The index skips to `tenant_id = 1` (equality), then range-scans `created_at` in order.
114
+
115
+ **Leading column rule**: A composite index on `(a, b, c)` can serve queries on:
116
+ - `a` alone
117
+ - `a, b`
118
+ - `a, b, c`
119
+
120
+ But NOT `b` alone or `c` alone (pre-PG18).
121
+
122
+ **B-tree skip scan (PG18+)**: PG18 can skip through distinct values of leading columns, so an index on `(a, b, c)` can now serve queries on `b` or `c` alone — by scanning each distinct `a` value. This works best when the leading column has low cardinality (few distinct values). It eliminates many cases where you previously needed a separate single-column index.
123
+
124
+ ## Partial Indexes
125
+
126
+ Index only the rows that matter. Smaller index = faster lookups, less maintenance.
127
+
128
+ ```sql
129
+ -- Only index active orders (95% of queries filter on active)
130
+ CREATE INDEX idx_orders_active ON orders(customer_id)
131
+ WHERE status = 'active';
132
+
133
+ -- Only index non-null values
134
+ CREATE INDEX idx_orders_shipped ON orders(shipped_at)
135
+ WHERE shipped_at IS NOT NULL;
136
+ ```
137
+
138
+ The query's WHERE clause must match (or imply) the index predicate for the planner to use it.
139
+
140
+ ## Covering Indexes (INCLUDE)
141
+
142
+ Add non-key columns to enable index-only scans without bloating the B-tree structure.
143
+
144
+ ```sql
145
+ -- Query: SELECT email, name FROM users WHERE email = ?
146
+ CREATE UNIQUE INDEX idx_users_email ON users(email) INCLUDE (name);
147
+ ```
148
+
149
+ The `name` column is stored in the index leaf pages but not in the B-tree structure. This enables an index-only scan (no heap fetch) without affecting index ordering or uniqueness.
150
+
151
+ ## Expression Indexes
152
+
153
+ Index the result of an expression or function.
154
+
155
+ ```sql
156
+ -- Case-insensitive email lookup
157
+ CREATE UNIQUE INDEX idx_users_email_lower ON users(lower(email));
158
+ -- Query must match: WHERE lower(email) = lower($1)
159
+
160
+ -- JSONB field extraction
161
+ CREATE INDEX idx_events_type ON events((payload->>'type'));
162
+ -- Query: WHERE payload->>'type' = 'click'
163
+
164
+ -- Date truncation
165
+ -- timestamptz must be made timezone-independent for an immutable expression
166
+ CREATE INDEX idx_orders_month
167
+ ON orders(date_trunc('month', created_at AT TIME ZONE 'UTC'));
168
+ ```
169
+
170
+ The query must use the same expression for the planner to match the index.
171
+
172
+ ## Indexes on Partitioned Tables
173
+
174
+ Indexes defined on a partitioned parent table are **automatically created on all existing and future child partitions**.
175
+
176
+ ```sql
177
+ -- Create index on the partitioned parent — propagates to all partitions
178
+ CREATE INDEX idx_events_type ON events(occurred_at, (payload->>'type'));
179
+ ```
180
+
181
+ ### Key Rules
182
+
183
+ - **Partition key in unique indexes**: Any UNIQUE or PRIMARY KEY index on a partitioned table must include all partition key columns.
184
+
185
+ ```sql
186
+ -- Partitioned by occurred_at — PK must include it
187
+ CREATE TABLE events (
188
+ id bigint GENERATED ALWAYS AS IDENTITY,
189
+ occurred_at timestamptz NOT NULL,
190
+ payload jsonb
191
+ ) PARTITION BY RANGE (occurred_at);
192
+
193
+ -- This works:
194
+ ALTER TABLE events ADD PRIMARY KEY (id, occurred_at);
195
+
196
+ -- This fails: partition key 'occurred_at' not in the constraint
197
+ -- ALTER TABLE events ADD PRIMARY KEY (id);
198
+ ```
199
+
200
+ - **Per-partition indexes**: You can also create indexes on individual partitions for partition-specific optimizations. These won't propagate to other partitions.
201
+
202
+ ```sql
203
+ -- Extra index only on the hot partition
204
+ CREATE INDEX idx_events_q1_status ON events_2024_q1((payload->>'status'));
205
+ ```
206
+
207
+ - **CONCURRENTLY on partitioned tables**: PostgreSQL does not support `CREATE INDEX CONCURRENTLY` directly on a partitioned parent. Create an invalid parent index with `ON ONLY`, build matching indexes concurrently on each partition, then attach them:
208
+
209
+ ```sql
210
+ CREATE INDEX idx_events_customer
211
+ ON ONLY events ((payload->>'customer_id'));
212
+
213
+ CREATE INDEX CONCURRENTLY idx_events_q1_customer
214
+ ON events_2024_q1 ((payload->>'customer_id'));
215
+
216
+ ALTER INDEX idx_events_customer
217
+ ATTACH PARTITION idx_events_q1_customer;
218
+ ```
219
+
220
+ Repeat the concurrent build and attach steps for every partition. The parent index becomes valid after all partition indexes are attached. If a child build fails, drop or rebuild that invalid child index before attaching it.
221
+
222
+ - **Index-only scans**: Work across partitions. The planner prunes irrelevant partitions first, then uses index-only scans on the remaining ones.
223
+
224
+ - **REINDEX on partitioned tables**: `REINDEX TABLE` on a partitioned table reindexes all partitions.
225
+
226
+ ```sql
227
+ REINDEX TABLE CONCURRENTLY events;
228
+ ```
229
+
230
+ ## Concurrent Index Creation
231
+
232
+ For production use, always create indexes concurrently to avoid locking writes:
233
+
234
+ ```sql
235
+ CREATE INDEX CONCURRENTLY idx_orders_customer ON orders(customer_id);
236
+ ```
237
+
238
+ Caveats:
239
+ - Takes longer (two table scans instead of one)
240
+ - Cannot run inside a transaction block
241
+ - If it fails, leaves an `INVALID` index — drop and retry
242
+ - Check for invalid indexes: `SELECT * FROM pg_index WHERE NOT indisvalid;`
243
+
244
+ ## Index Maintenance
245
+
246
+ ### Reindexing Bloated Indexes
247
+
248
+ ```sql
249
+ -- Concurrent reindex
250
+ REINDEX INDEX CONCURRENTLY idx_orders_customer;
251
+
252
+ -- Or reindex all indexes on a table
253
+ REINDEX TABLE CONCURRENTLY orders;
254
+ ```
255
+
256
+ ### Monitoring Index Size
257
+
258
+ ```sql
259
+ SELECT
260
+ indexrelid::regclass AS index_name,
261
+ pg_size_pretty(pg_relation_size(indexrelid)) AS size,
262
+ idx_scan AS scans,
263
+ idx_tup_read AS tuples_read
264
+ FROM pg_stat_user_indexes
265
+ ORDER BY pg_relation_size(indexrelid) DESC;
266
+ ```
267
+
268
+ ## Finding Unused Indexes
269
+
270
+ ```sql
271
+ SELECT
272
+ schemaname,
273
+ relname AS table_name,
274
+ indexrelname AS index_name,
275
+ pg_size_pretty(pg_relation_size(indexrelid)) AS index_size,
276
+ idx_scan AS times_used
277
+ FROM pg_stat_user_indexes
278
+ WHERE idx_scan = 0
279
+ AND indexrelname NOT LIKE '%_pkey' -- exclude primary keys
280
+ AND indexrelname NOT LIKE '%unique%' -- exclude unique constraints
281
+ ORDER BY pg_relation_size(indexrelid) DESC;
282
+ ```
283
+
284
+ Reset statistics after a representative period: `SELECT pg_stat_reset();`
285
+
286
+ ## Finding Duplicate Indexes
287
+
288
+ ```sql
289
+ SELECT
290
+ array_agg(indexrelid::regclass) AS indexes,
291
+ indrelid::regclass AS table_name,
292
+ indkey AS column_numbers
293
+ FROM pg_index
294
+ GROUP BY indrelid, indkey
295
+ HAVING count(*) > 1;
296
+ ```
297
+
298
+ Also check for indexes that are a prefix of another:
299
+ - `(a)` is redundant if `(a, b)` exists
300
+ - `(a, b)` is NOT redundant if `(a, b, c)` exists and you need index-only scans on just `(a, b)`
301
+
302
+ ## Index Selection Decision Tree
303
+
304
+ 1. **What operator?**
305
+ - `=`, `<`, `>`, `BETWEEN` → B-tree
306
+ - `@>`, `?`, `&&` on jsonb/array → GIN
307
+ - `@@` full-text → GIN
308
+ - `ILIKE '%x%'` → GIN + pg_trgm
309
+ - Range/geometric overlap → GiST
310
+ - Naturally ordered append-only → BRIN
311
+
312
+ 2. **How many rows match?** If > 10-20% of table, index may not help (seq scan is cheaper).
313
+
314
+ 3. **Can you narrow the index?** Use a partial index if most queries filter on a subset.
315
+
316
+ 4. **Do you need index-only scans?** Add `INCLUDE` columns.
317
+
318
+ 5. **Is the column an expression?** Use an expression index.