heimdall-agent 1.0.0 → 1.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +57 -26
- package/README.pt-BR.md +429 -0
- package/agents/cloud-engineer.md +77 -0
- package/agents/code-sentinel.md +66 -35
- package/agents/database-architect.md +31 -25
- package/agents/frontend-designer.md +30 -28
- package/agents/mobile-designer.md +25 -24
- package/agents/modernizer.md +66 -0
- package/agents/orchestrator.md +94 -93
- package/agents/reviewer.md +100 -71
- package/agents/security-auditor.md +27 -27
- package/agents/tester.md +97 -90
- package/commands/grill-me.md +20 -4
- package/commands/orchestrate.md +16 -0
- package/commands/orchestrator.md +16 -0
- package/commands/plan.md +17 -1
- package/config/agents.json +21 -3
- package/config/mcp.json +72 -0
- package/config/models.json +26 -16
- package/config/profiles.json +2 -4
- package/config/skills.json +210 -100
- package/docs/SKILLS.md +341 -245
- package/docs/designs/old/2026-10-31 plan-lifecycle-backlog-and-clean-code-skill.md +87 -0
- package/docs/designs/old/2026-11-02 skill-upstream-audit-and-update.md +105 -0
- package/docs/designs/old/2026-11-05 communication-doctrine-bluf-ste.md +94 -0
- package/docs/designs/old/2026-11-06-cloud-engineer-and-resilience-synthesis.md +67 -0
- package/docs/designs/old/2026-11-06-herdr-native-agent-orchestration.md +45 -0
- package/docs/runs/skill-audit/add_progressive_disclosure.cjs +30 -0
- package/docs/runs/skill-audit/check_heimdall_skills.ps1 +9 -0
- package/docs/runs/skill-audit/compare_classics.ps1 +26 -0
- package/docs/runs/skill-audit/compare_taste.ps1 +12 -0
- package/docs/runs/skill-audit/customization-report.json +380 -0
- package/docs/runs/skill-audit/diff_pending_skills.cjs +94 -0
- package/docs/runs/skill-audit/fetch_and_apply_skill.cjs +58 -0
- package/docs/runs/skill-audit/fetch_and_apply_taste_v2.cjs +65 -0
- package/docs/runs/skill-audit/inspect_pending_skills.cjs +71 -0
- package/docs/runs/skill-audit/inventory.json +308 -0
- package/docs/runs/skill-audit/list_local_taste.ps1 +1 -0
- package/docs/runs/skill-audit/phase5-update-plan.json +595 -0
- package/docs/runs/skill-audit/phase5-update-plan.md +138 -0
- package/docs/runs/skill-audit/probe.ps1 +16 -0
- package/docs/runs/skill-audit/probe_classics_branches.ps1 +3 -0
- package/docs/runs/skill-audit/probe_commits.ps1 +15 -0
- package/docs/runs/skill-audit/probe_repos.ps1 +17 -0
- package/docs/runs/skill-audit/probe_sentry_sec.ps1 +3 -0
- package/docs/runs/skill-audit/probe_sentry_sweep.ps1 +5 -0
- package/docs/runs/skill-audit/probe_snyk.ps1 +3 -0
- package/docs/runs/skill-audit/probe_snyk_recipes.ps1 +5 -0
- package/docs/runs/skill-audit/probe_taste_branches.ps1 +8 -0
- package/docs/runs/skill-audit/probe_taste_v19.ps1 +3 -0
- package/docs/runs/skill-audit/probe_user_repos.ps1 +19 -0
- package/docs/runs/skill-audit/prune_taste_skills.cjs +22 -0
- package/docs/runs/skill-audit/run_phase1_2.cjs +142 -0
- package/docs/runs/skill-audit/run_phase6_verification.cjs +107 -0
- package/docs/runs/skill-audit/show_16_details.ps1 +25 -0
- package/docs/runs/skill-audit/show_20_skills.ps1 +28 -0
- package/docs/runs/skill-audit/show_classics_diff.cjs +37 -0
- package/docs/runs/skill-audit/sync_agent_markdowns.cjs +30 -0
- package/docs/runs/skill-audit/taste-skill-upstream.md +542 -0
- package/docs/runs/skill-audit/update-manifest.md +88 -0
- package/docs/runs/skill-audit/update_config_skills.cjs +52 -0
- package/docs/runs/skill-audit/update_registry_phase7.cjs +83 -0
- package/docs/runs/skill-audit/upstream-status.json +563 -0
- package/docs/runs/skill-audit/verification-results.json +252 -0
- package/doctrine/AGENTS.md +39 -77
- package/hooks/git/pre-commit +11 -0
- package/hooks/heimdall-hook.cjs +81 -2
- package/hooks/herdr-agent-state.ps1 +55 -0
- package/hooks/hooks.json +0 -11
- package/lib/catalog-sync.cjs +162 -0
- package/lib/hermes-context.cjs +57 -0
- package/lib/model-context.cjs +123 -0
- package/lib/path-discovery.cjs +94 -0
- package/package.json +7 -3
- package/playbooks/01-interactive-grilling-gate.md +14 -1
- package/playbooks/02-fast-code-deep-verification.md +3 -2
- package/playbooks/03-adversarial-code-review.md +3 -3
- package/playbooks/05-canonical-plan-template.md +32 -18
- package/playbooks/06-model-registry-matrix.md +24 -13
- package/playbooks/07-legacy-code-modernization.md +74 -0
- package/playbooks/08-backlog-and-change-request-flow.md +122 -0
- package/setup.mjs +592 -172
- package/skills/antislop/SKILL.md +76 -56
- package/skills/antislop/VERSION +1 -0
- package/skills/antislop-copywriting/SKILL.md +372 -372
- package/skills/antislop-human/LICENSE-web-quality-skills +21 -0
- package/skills/antislop-human/SKILL.md +204 -147
- package/skills/antislop-human/contrast-check.py +168 -165
- package/skills/antislop-human/contrast-mcp.py +6 -3
- package/skills/antislop-human/references/A11Y-PATTERNS.md +249 -0
- package/skills/antislop-human/references/WCAG.md +127 -0
- package/skills/antislop-ui/LICENSE-web-quality-skills +21 -0
- package/skills/antislop-ui/SKILL.md +344 -304
- package/skills/antislop-ui/references/CORE-WEB-VITALS.md +181 -0
- package/skills/antislop-ui/references/WEB-BASELINE.md +28 -0
- package/skills/brainstorming/SKILL.md +274 -265
- package/skills/brainstorming/builder-check-prompt.md +23 -0
- package/skills/brainstorming/scripts/frame-template.html +213 -213
- package/skills/brainstorming/scripts/server.cjs +723 -723
- package/skills/brandkit/SKILL.md +798 -798
- package/skills/brutalist-skill/SKILL.md +93 -92
- package/skills/caveman/README.md +68 -0
- package/skills/caveman/SKILL.md +73 -38
- package/skills/claude-collaborator/SKILL.md +24 -44
- package/skills/clean-code/SKILL.md +152 -0
- package/skills/cloudflare-security-audit/AI-AND-LLM.md +83 -83
- package/skills/cloudflare-security-audit/ATTACK-CLASSES.md +130 -130
- package/skills/cloudflare-security-audit/CLIENT-SIDE.md +83 -83
- package/skills/cloudflare-security-audit/CLOUD-AND-DEPLOYMENT.md +86 -86
- package/skills/cloudflare-security-audit/DATA-ISOLATION-AND-LIFECYCLE.md +84 -84
- package/skills/cloudflare-security-audit/DESKTOP-MOBILE-AND-LOCAL-IPC.md +89 -89
- package/skills/cloudflare-security-audit/HUNTING.md +251 -251
- package/skills/cloudflare-security-audit/MEMORY-SAFETY-AND-BINARY.md +101 -101
- package/skills/cloudflare-security-audit/PROTOCOLS-RPC-AND-MESSAGING.md +81 -81
- package/skills/cloudflare-security-audit/RECONNAISSANCE.md +156 -156
- package/skills/cloudflare-security-audit/RESOURCE-EXHAUSTION-AND-AVAILABILITY.md +78 -78
- package/skills/cloudflare-security-audit/SKILL.md +192 -192
- package/skills/cloudflare-security-audit/SUPPLY-CHAIN-AND-RELEASE.md +73 -73
- package/skills/cloudflare-security-audit/VALIDATION-AND-REPORTING.md +186 -186
- package/skills/cloudflare-security-audit/WEB-PROTOCOL-AND-AUTH.md +105 -105
- package/skills/cloudflare-security-audit/report-schema.json +461 -461
- package/skills/cloudflare-security-audit/validate-coverage-ledger.cjs +872 -872
- package/skills/cloudflare-security-audit/validate-coverage-ledger.test.cjs +740 -740
- package/skills/cloudflare-security-audit/validate-findings.cjs +773 -773
- package/skills/cloudflare-security-audit/validate-findings.test.cjs +652 -652
- package/skills/codex-collaborator/SKILL.md +24 -45
- package/skills/communication-doctrine/SKILL.md +89 -0
- package/skills/diagnosing-superpowers/SKILL.md +1 -1
- package/skills/diagnosing-superpowers/references/github-issues.md +3 -3
- package/skills/diagnosing-superpowers/templates/issue.md +23 -4
- package/skills/executing-plans/SKILL.md +0 -52
- package/skills/executing-plans/scripts/task-done +52 -52
- package/skills/executing-plans/scripts/task-start +28 -28
- package/skills/gemini-collaborator/SKILL.md +24 -56
- package/skills/herdr/SKILL.md +243 -0
- package/skills/image-to-code-skill/SKILL.md +1228 -1228
- package/skills/imagegen-frontend-mobile/SKILL.md +1465 -1465
- package/skills/imagegen-frontend-web/SKILL.md +987 -987
- package/skills/karpathy-guidelines/SKILL.md +67 -67
- package/skills/mcp-engineering/SKILL.md +74 -0
- package/skills/minimalist-skill/SKILL.md +87 -85
- package/skills/output-skill/SKILL.md +49 -49
- package/skills/ponytail/SKILL.md +36 -52
- package/skills/postgres-best-practices/SKILL.md +30 -52
- package/skills/postgres-best-practices/references/backup-restore.md +372 -0
- package/skills/postgres-best-practices/references/bulk-loading.md +315 -0
- package/skills/postgres-best-practices/references/connection-pooling.md +310 -0
- package/skills/postgres-best-practices/references/hot-standby.md +221 -0
- package/skills/postgres-best-practices/references/indexing.md +318 -0
- package/skills/postgres-best-practices/references/logical-replication.md +610 -0
- package/skills/postgres-best-practices/references/major-version-upgrades.md +334 -0
- package/skills/postgres-best-practices/references/performance-diagnostics.md +553 -0
- package/skills/postgres-best-practices/references/query-optimization.md +294 -0
- package/skills/postgres-best-practices/references/query-patterns.md +566 -0
- package/skills/postgres-best-practices/references/schema-design.md +362 -0
- package/skills/postgres-best-practices/references/security-roles.md +489 -0
- package/skills/postgres-best-practices/references/transaction-isolation.md +344 -0
- package/skills/progressive-disclosure/SKILL.md +68 -0
- package/skills/pstack-discipline/SKILL.md +40 -0
- package/skills/receiving-code-review/SKILL.md +2 -23
- package/skills/requesting-code-review/SKILL.md +7 -36
- package/skills/soft-skill/SKILL.md +99 -98
- package/skills/stitch-skill/DESIGN.md +121 -121
- package/skills/stitch-skill/SKILL.md +185 -184
- package/skills/strix-security/SKILL.md +54 -53
- package/skills/subagent-driven-development/SKILL.md +28 -260
- package/skills/subagent-driven-development/references/batching.md +11 -0
- package/skills/subagent-driven-development/references/cannot-verify.md +10 -0
- package/skills/subagent-driven-development/references/final-review.md +47 -0
- package/skills/subagent-driven-development/references/fix-loop.md +77 -0
- package/skills/subagent-driven-development/references/handling-reports.md +15 -0
- package/skills/subagent-driven-development/references/preflight.md +26 -0
- package/skills/subagent-driven-development/references/resuming.md +12 -0
- package/skills/subagent-driven-development/scripts/review-package +53 -53
- package/skills/subagent-driven-development/scripts/sdd-workspace +87 -82
- package/skills/subagent-driven-development/scripts/task-brief +43 -43
- package/skills/systematic-debugging/SKILL.md +283 -306
- package/skills/taste-skill/SKILL.md +542 -1206
- package/skills/taste-skill/references/app-screens.md +1199 -0
- package/skills/taste-skill/references/colour.md +141 -0
- package/skills/taste-skill/references/copy.md +177 -0
- package/skills/taste-skill/references/critique.md +182 -0
- package/skills/taste-skill/references/imagery.md +791 -0
- package/skills/taste-skill/references/layout.md +292 -0
- package/skills/taste-skill/references/motion.md +980 -0
- package/skills/taste-skill/references/research.md +356 -0
- package/skills/taste-skill/references/scenes.md +174 -0
- package/skills/taste-skill/references/tells.md +115 -0
- package/skills/taste-skill/references/type.md +170 -0
- package/skills/test-driven-development/SKILL.md +330 -337
- package/skills/thermo/SKILL.md +67 -0
- package/skills/threat-modeling/SKILL.md +55 -0
- package/skills/trailofbits-audit-context/SKILL.md +72 -72
- package/skills/trailofbits-insecure-defaults/commands/audit.md +35 -0
- package/skills/trailofbits-insecure-defaults/references/debug-features.json +13 -13
- package/skills/trailofbits-insecure-defaults/references/debug-features.md +75 -75
- package/skills/trailofbits-insecure-defaults/references/default-credentials.json +12 -12
- package/skills/trailofbits-insecure-defaults/references/default-credentials.md +76 -76
- package/skills/trailofbits-insecure-defaults/references/fail-open-security.json +11 -11
- package/skills/trailofbits-insecure-defaults/references/fail-open-security.md +70 -70
- package/skills/trailofbits-insecure-defaults/references/fallback-secrets.json +11 -11
- package/skills/trailofbits-insecure-defaults/references/fallback-secrets.md +70 -70
- package/skills/trailofbits-insecure-defaults/references/permissive-access.json +13 -13
- package/skills/trailofbits-insecure-defaults/references/permissive-access.md +62 -62
- package/skills/trailofbits-insecure-defaults/references/weak-crypto.json +13 -13
- package/skills/trailofbits-insecure-defaults/references/weak-crypto.md +71 -71
- package/skills/trailofbits-triage-brocards/SKILL.md +208 -208
- package/skills/verification-before-completion/SKILL.md +0 -14
- package/skills/writing-plans/SKILL.md +11 -15
- package/skills/writing-skills/SKILL.md +21 -303
- package/skills/writing-skills/references/CLAUDE_MD_TESTING.md +189 -0
- package/skills/writing-skills/references/anthropic-best-practices.md +1150 -0
- package/skills/writing-skills/references/bulletproofing.md +80 -0
- package/skills/writing-skills/references/discovery.md +201 -0
- package/skills/writing-skills/references/flowcharts.md +37 -0
- package/skills/writing-skills/references/graphviz-conventions.dot +172 -0
- package/skills/writing-skills/references/persuasion-principles.md +187 -0
- package/skills/writing-skills/references/testing-skills-with-subagents.md +384 -0
- package/tools/{claude-bridge.ps1 → archive/claude-bridge.ps1} +10 -7
- package/tools/{codex-bridge.ps1 → archive/codex-bridge.ps1} +1 -1
- package/tools/{gemini-bridge.ps1 → archive/gemini-bridge.ps1} +12 -19
- package/skills/gpt-tasteskill/SKILL.md +0 -74
- package/skills/taste-skill-v1/SKILL.md +0 -226
- /package/tools/{claude-bridge.cmd → archive/claude-bridge.cmd} +0 -0
- /package/tools/{claude-bridge.sh → archive/claude-bridge.sh} +0 -0
- /package/tools/{codex-bridge.cmd → archive/codex-bridge.cmd} +0 -0
- /package/tools/{codex-bridge.sh → archive/codex-bridge.sh} +0 -0
- /package/tools/{codexcli-bridge.cmd → archive/codexcli-bridge.cmd} +0 -0
- /package/tools/{codexcli-bridge.ps1 → archive/codexcli-bridge.ps1} +0 -0
- /package/tools/{gemini-bridge.cmd → archive/gemini-bridge.cmd} +0 -0
- /package/tools/{gemini-bridge.sh → archive/gemini-bridge.sh} +0 -0
|
@@ -0,0 +1,221 @@
|
|
|
1
|
+
# Hot Standby & Read Replicas Reference
|
|
2
|
+
|
|
3
|
+
## Contents
|
|
4
|
+
- Streaming replication overview
|
|
5
|
+
- Hot standby configuration
|
|
6
|
+
- Monitoring replication lag
|
|
7
|
+
- Replication conflicts
|
|
8
|
+
- Synchronous vs asynchronous replication
|
|
9
|
+
- Promoting a standby
|
|
10
|
+
|
|
11
|
+
## Streaming Replication Overview
|
|
12
|
+
|
|
13
|
+
Physical (streaming) replication creates an exact copy of the primary database on one or more standby servers. Unlike logical replication, it replicates the entire cluster (all databases, all objects) at the WAL level.
|
|
14
|
+
|
|
15
|
+
**Hot standby** allows read-only queries on the standby while it continuously replays WAL from the primary. This is the standard mechanism for PostgreSQL read replicas.
|
|
16
|
+
|
|
17
|
+
### Key Differences from Logical Replication
|
|
18
|
+
|
|
19
|
+
| Aspect | Physical (streaming) | Logical |
|
|
20
|
+
|--------|---------------------|---------|
|
|
21
|
+
| Scope | Entire cluster | Selected tables |
|
|
22
|
+
| PG versions | Must match major version | Can differ |
|
|
23
|
+
| Standby writable? | No (read-only) | Yes (for non-replicated tables) |
|
|
24
|
+
| DDL replicated? | Yes (via WAL) | No |
|
|
25
|
+
| Use case | HA, read replicas | CDC, selective sync, cross-version migration |
|
|
26
|
+
|
|
27
|
+
## Hot Standby Configuration
|
|
28
|
+
|
|
29
|
+
### On the Primary
|
|
30
|
+
|
|
31
|
+
```sql
|
|
32
|
+
-- Check current settings
|
|
33
|
+
SHOW wal_level; -- must be 'replica' or 'logical'
|
|
34
|
+
SHOW max_wal_senders; -- must have available sender slots
|
|
35
|
+
SHOW max_replication_slots; -- one per standby for slot-based replication
|
|
36
|
+
```
|
|
37
|
+
|
|
38
|
+
Key settings (in `postgresql.conf`):
|
|
39
|
+
- `wal_level = replica` (default)
|
|
40
|
+
- `max_wal_senders = 10` (enough for all standbys)
|
|
41
|
+
- `max_replication_slots = 10` (optional but recommended — prevents WAL removal before standby catches up)
|
|
42
|
+
|
|
43
|
+
### On the Standby
|
|
44
|
+
|
|
45
|
+
Key settings:
|
|
46
|
+
- `hot_standby = on` (allows read queries during recovery — default)
|
|
47
|
+
- `primary_conninfo = 'host=primary_host user=repl_user ...'` (connection to primary)
|
|
48
|
+
- `primary_slot_name = 'standby_slot'` (optional — use a replication slot)
|
|
49
|
+
|
|
50
|
+
### Replication Slots (Recommended)
|
|
51
|
+
|
|
52
|
+
Slots prevent the primary from removing WAL segments before the standby has replayed them:
|
|
53
|
+
|
|
54
|
+
```sql
|
|
55
|
+
-- On the primary: create a physical replication slot
|
|
56
|
+
SELECT pg_create_physical_replication_slot('standby1_slot');
|
|
57
|
+
|
|
58
|
+
-- List slots
|
|
59
|
+
SELECT slot_name, slot_type, active, restart_lsn
|
|
60
|
+
FROM pg_replication_slots;
|
|
61
|
+
|
|
62
|
+
-- Drop a slot (if standby is permanently removed)
|
|
63
|
+
SELECT pg_drop_replication_slot('standby1_slot');
|
|
64
|
+
```
|
|
65
|
+
|
|
66
|
+
**Warning**: An inactive slot causes WAL to accumulate indefinitely on the primary, potentially filling the disk.
|
|
67
|
+
|
|
68
|
+
## Monitoring Replication Lag
|
|
69
|
+
|
|
70
|
+
### On the Primary: pg_stat_replication
|
|
71
|
+
|
|
72
|
+
```sql
|
|
73
|
+
SELECT
|
|
74
|
+
pid,
|
|
75
|
+
application_name,
|
|
76
|
+
client_addr,
|
|
77
|
+
state,
|
|
78
|
+
sent_lsn,
|
|
79
|
+
write_lsn,
|
|
80
|
+
flush_lsn,
|
|
81
|
+
replay_lsn,
|
|
82
|
+
pg_size_pretty(pg_wal_lsn_diff(sent_lsn, replay_lsn)) AS replay_lag,
|
|
83
|
+
pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), sent_lsn)) AS send_lag,
|
|
84
|
+
write_lag,
|
|
85
|
+
flush_lag,
|
|
86
|
+
replay_lag AS replay_lag_time
|
|
87
|
+
FROM pg_stat_replication;
|
|
88
|
+
```
|
|
89
|
+
|
|
90
|
+
**LSN progression**: `pg_current_wal_lsn()` → `sent_lsn` → `write_lsn` → `flush_lsn` → `replay_lsn`. The gap between any two is a measure of lag at that stage.
|
|
91
|
+
|
|
92
|
+
### On the Standby: pg_stat_wal_receiver
|
|
93
|
+
|
|
94
|
+
```sql
|
|
95
|
+
SELECT
|
|
96
|
+
status,
|
|
97
|
+
written_lsn,
|
|
98
|
+
flushed_lsn,
|
|
99
|
+
latest_end_lsn,
|
|
100
|
+
latest_end_time,
|
|
101
|
+
slot_name,
|
|
102
|
+
conninfo
|
|
103
|
+
FROM pg_stat_wal_receiver;
|
|
104
|
+
```
|
|
105
|
+
|
|
106
|
+
### On the Standby: Lag in Seconds
|
|
107
|
+
|
|
108
|
+
```sql
|
|
109
|
+
-- How far behind is the standby?
|
|
110
|
+
SELECT
|
|
111
|
+
now() - pg_last_xact_replay_timestamp() AS replay_lag,
|
|
112
|
+
pg_last_wal_receive_lsn() AS received_lsn,
|
|
113
|
+
pg_last_wal_replay_lsn() AS replayed_lsn,
|
|
114
|
+
pg_wal_lsn_diff(pg_last_wal_receive_lsn(), pg_last_wal_replay_lsn()) AS replay_lag_bytes;
|
|
115
|
+
```
|
|
116
|
+
|
|
117
|
+
**Caveat**: `pg_last_xact_replay_timestamp()` only updates when transactions are replayed. On an idle primary, it may show a large lag even though the standby is fully caught up. Check `received_lsn = replayed_lsn` for actual status.
|
|
118
|
+
|
|
119
|
+
### Is the Standby in Recovery?
|
|
120
|
+
|
|
121
|
+
```sql
|
|
122
|
+
-- Returns true on a standby, false on a primary
|
|
123
|
+
SELECT pg_is_in_recovery();
|
|
124
|
+
```
|
|
125
|
+
|
|
126
|
+
## Replication Conflicts
|
|
127
|
+
|
|
128
|
+
On a hot standby, long-running read queries can conflict with WAL replay. When replay needs to apply changes that conflict with an active query (e.g., dropping a table, vacuuming rows the query is reading), PostgreSQL must choose: wait for the query or cancel it.
|
|
129
|
+
|
|
130
|
+
### max_standby_streaming_delay
|
|
131
|
+
|
|
132
|
+
Controls how long the standby waits for conflicting queries before cancelling them:
|
|
133
|
+
|
|
134
|
+
```sql
|
|
135
|
+
SHOW max_standby_streaming_delay; -- default: 30s
|
|
136
|
+
```
|
|
137
|
+
|
|
138
|
+
- `30s` (default): Standby waits up to 30 seconds, then cancels conflicting queries
|
|
139
|
+
- `-1`: Wait forever (replay pauses until the query finishes — can cause unbounded lag)
|
|
140
|
+
- `0`: Cancel conflicting queries immediately (minimal lag, but queries may fail)
|
|
141
|
+
|
|
142
|
+
### max_standby_archive_delay
|
|
143
|
+
|
|
144
|
+
Same concept but for WAL segments being replayed from archive (rather than streaming):
|
|
145
|
+
|
|
146
|
+
```sql
|
|
147
|
+
SHOW max_standby_archive_delay; -- default: 30s
|
|
148
|
+
```
|
|
149
|
+
|
|
150
|
+
### The Common Error
|
|
151
|
+
|
|
152
|
+
When a query on the standby is cancelled due to a conflict:
|
|
153
|
+
|
|
154
|
+
```
|
|
155
|
+
ERROR: canceling statement due to conflict with recovery
|
|
156
|
+
DETAIL: User was holding shared buffer pin for too long.
|
|
157
|
+
```
|
|
158
|
+
|
|
159
|
+
**Solutions (in order of preference):**
|
|
160
|
+
1. Keep queries on the standby short
|
|
161
|
+
2. Increase `max_standby_streaming_delay` (trades lag for query stability)
|
|
162
|
+
3. Enable `hot_standby_feedback` (see below)
|
|
163
|
+
4. Use a logical replica instead of hot standby for long-running analytics
|
|
164
|
+
|
|
165
|
+
### hot_standby_feedback
|
|
166
|
+
|
|
167
|
+
When enabled, the standby informs the primary about which rows it still needs, preventing the primary's VACUUM from removing them:
|
|
168
|
+
|
|
169
|
+
```sql
|
|
170
|
+
SHOW hot_standby_feedback; -- default: off
|
|
171
|
+
```
|
|
172
|
+
|
|
173
|
+
**Pros**: Eliminates most replication conflicts — long queries on the standby won't be cancelled.
|
|
174
|
+
|
|
175
|
+
**Cons**: Can cause table bloat on the primary because VACUUM can't remove dead rows that the standby still references.
|
|
176
|
+
|
|
177
|
+
**Recommendation**: Enable only if you have long-running analytical queries on the standby and can tolerate some extra bloat on the primary.
|
|
178
|
+
|
|
179
|
+
## Synchronous vs Asynchronous Replication
|
|
180
|
+
|
|
181
|
+
### Asynchronous (Default)
|
|
182
|
+
|
|
183
|
+
The primary doesn't wait for standby acknowledgment before committing. Fastest, but a primary failure can lose recently committed transactions not yet replicated.
|
|
184
|
+
|
|
185
|
+
### Synchronous
|
|
186
|
+
|
|
187
|
+
The primary waits for at least one standby to confirm before returning commit success:
|
|
188
|
+
|
|
189
|
+
```sql
|
|
190
|
+
-- On the primary
|
|
191
|
+
SHOW synchronous_standby_names; -- e.g., 'standby1'
|
|
192
|
+
SHOW synchronous_commit; -- 'on', 'remote_write', 'remote_apply', etc.
|
|
193
|
+
```
|
|
194
|
+
|
|
195
|
+
Synchronous commit levels:
|
|
196
|
+
|
|
197
|
+
| Level | Primary waits for | Durability | Latency impact |
|
|
198
|
+
|-------|-------------------|------------|----------------|
|
|
199
|
+
| `on` (default with sync standbys) | Standby WAL flush | Strong | Moderate |
|
|
200
|
+
| `remote_write` | Standby WAL write (not fsync) | Good | Lower |
|
|
201
|
+
| `remote_apply` | Standby WAL replay | Strongest (read-your-writes on standby) | Highest |
|
|
202
|
+
| `local` | Local WAL flush only | Primary only | None |
|
|
203
|
+
| `off` | Nothing | Weakest | None |
|
|
204
|
+
|
|
205
|
+
## Promoting a Standby
|
|
206
|
+
|
|
207
|
+
To promote a standby to become the new primary (failover):
|
|
208
|
+
|
|
209
|
+
```sql
|
|
210
|
+
-- On the standby:
|
|
211
|
+
SELECT pg_promote();
|
|
212
|
+
|
|
213
|
+
-- Or from the command line:
|
|
214
|
+
-- pg_ctl promote -D /path/to/data
|
|
215
|
+
```
|
|
216
|
+
|
|
217
|
+
After promotion:
|
|
218
|
+
- The standby stops replay and opens for writes
|
|
219
|
+
- Applications must be redirected to the new primary
|
|
220
|
+
- Other standbys must be reconfigured to follow the new primary
|
|
221
|
+
- The old primary must not be restarted without reconfiguring (risk of split-brain)
|
|
@@ -0,0 +1,318 @@
|
|
|
1
|
+
# Indexing Reference
|
|
2
|
+
|
|
3
|
+
## Contents
|
|
4
|
+
- Index types and when to use each
|
|
5
|
+
- Composite index column ordering
|
|
6
|
+
- Partial indexes
|
|
7
|
+
- Covering indexes (INCLUDE)
|
|
8
|
+
- Expression indexes
|
|
9
|
+
- Index maintenance and bloat
|
|
10
|
+
- Finding unused and duplicate indexes
|
|
11
|
+
|
|
12
|
+
## Index Types
|
|
13
|
+
|
|
14
|
+
### B-tree (Default)
|
|
15
|
+
|
|
16
|
+
Supports: `=`, `<`, `>`, `<=`, `>=`, `BETWEEN`, `IN`, `IS NULL`, `IS NOT NULL`
|
|
17
|
+
|
|
18
|
+
```sql
|
|
19
|
+
CREATE INDEX idx_orders_created ON orders(created_at);
|
|
20
|
+
```
|
|
21
|
+
|
|
22
|
+
Best for: equality and range queries on scalar types. The default and most common choice.
|
|
23
|
+
|
|
24
|
+
### GIN (Generalized Inverted Index)
|
|
25
|
+
|
|
26
|
+
Supports: containment operators on composite values.
|
|
27
|
+
|
|
28
|
+
```sql
|
|
29
|
+
-- JSONB containment
|
|
30
|
+
CREATE INDEX idx_data_gin ON events USING gin(payload);
|
|
31
|
+
-- Matches: WHERE payload @> '{"status": "active"}'
|
|
32
|
+
|
|
33
|
+
-- Array containment
|
|
34
|
+
CREATE INDEX idx_tags_gin ON articles USING gin(tags);
|
|
35
|
+
-- Matches: WHERE tags @> ARRAY['postgres']
|
|
36
|
+
|
|
37
|
+
-- Full-text search
|
|
38
|
+
CREATE INDEX idx_fts ON articles USING gin(to_tsvector('english', body));
|
|
39
|
+
-- Matches: WHERE to_tsvector('english', body) @@ to_tsquery('postgres & index')
|
|
40
|
+
|
|
41
|
+
-- Trigram (fuzzy text search)
|
|
42
|
+
CREATE EXTENSION IF NOT EXISTS pg_trgm;
|
|
43
|
+
CREATE INDEX idx_name_trgm ON users USING gin(name gin_trgm_ops);
|
|
44
|
+
-- Matches: WHERE name ILIKE '%pattern%'
|
|
45
|
+
```
|
|
46
|
+
|
|
47
|
+
**Operator class tip**: For JSONB, `jsonb_path_ops` is 2-3x smaller than the default `jsonb_ops` but only supports `@>` (containment). Use it when you only need containment queries:
|
|
48
|
+
|
|
49
|
+
```sql
|
|
50
|
+
CREATE INDEX idx_events_gin_path ON events USING gin(payload jsonb_path_ops);
|
|
51
|
+
-- Only supports: WHERE payload @> '{"type": "click"}'
|
|
52
|
+
-- Does NOT support: WHERE payload ? 'type'
|
|
53
|
+
```
|
|
54
|
+
|
|
55
|
+
**Stored generated column for FTS**:
|
|
56
|
+
|
|
57
|
+
```sql
|
|
58
|
+
ALTER TABLE articles ADD COLUMN search_vector tsvector
|
|
59
|
+
GENERATED ALWAYS AS (to_tsvector('english', title || ' ' || body)) STORED;
|
|
60
|
+
CREATE INDEX idx_articles_search ON articles USING gin(search_vector);
|
|
61
|
+
-- Query: WHERE search_vector @@ to_tsquery('english', 'postgres & optimization')
|
|
62
|
+
```
|
|
63
|
+
|
|
64
|
+
GIN indexes are larger and slower to update than B-tree but excel at multi-valued containment queries.
|
|
65
|
+
|
|
66
|
+
**Parallel GIN builds (PG18+)**: GIN indexes can now be built in parallel, significantly speeding up index creation on large tables.
|
|
67
|
+
|
|
68
|
+
### GiST (Generalized Search Tree)
|
|
69
|
+
|
|
70
|
+
Supports: overlap, containment, nearest-neighbor on geometric, range, and full-text types.
|
|
71
|
+
|
|
72
|
+
```sql
|
|
73
|
+
-- Range overlap
|
|
74
|
+
CREATE INDEX idx_booking_range ON bookings USING gist(during);
|
|
75
|
+
-- Matches: WHERE during && '[2024-01-01, 2024-02-01)'
|
|
76
|
+
|
|
77
|
+
-- Used in exclusion constraints
|
|
78
|
+
CREATE EXTENSION IF NOT EXISTS btree_gist;
|
|
79
|
+
ALTER TABLE bookings
|
|
80
|
+
ADD CONSTRAINT bookings_no_overlap
|
|
81
|
+
EXCLUDE USING gist (room_id WITH =, during WITH &&);
|
|
82
|
+
```
|
|
83
|
+
|
|
84
|
+
### BRIN (Block Range Index)
|
|
85
|
+
|
|
86
|
+
Tiny index for naturally ordered data (e.g., append-only timestamp columns).
|
|
87
|
+
|
|
88
|
+
```sql
|
|
89
|
+
CREATE INDEX idx_events_ts_brin ON events USING brin(occurred_at);
|
|
90
|
+
```
|
|
91
|
+
|
|
92
|
+
BRIN stores min/max per block range. Effective when physical row order correlates with column value. Very small (~0.1% of B-tree size) but less precise — may read extra blocks.
|
|
93
|
+
|
|
94
|
+
### Hash
|
|
95
|
+
|
|
96
|
+
Only supports `=`. Rarely better than B-tree.
|
|
97
|
+
|
|
98
|
+
```sql
|
|
99
|
+
CREATE INDEX idx_session_hash ON sessions USING hash(session_token);
|
|
100
|
+
```
|
|
101
|
+
|
|
102
|
+
## Composite Index Column Ordering
|
|
103
|
+
|
|
104
|
+
Order matters. The index is a sorted tree, leftmost column first.
|
|
105
|
+
|
|
106
|
+
**Rule of thumb**: equality columns first, then range/sort columns.
|
|
107
|
+
|
|
108
|
+
```sql
|
|
109
|
+
-- Query: WHERE tenant_id = 1 AND created_at > '2024-01-01' ORDER BY created_at
|
|
110
|
+
CREATE INDEX idx_tenant_created ON orders(tenant_id, created_at);
|
|
111
|
+
```
|
|
112
|
+
|
|
113
|
+
The index skips to `tenant_id = 1` (equality), then range-scans `created_at` in order.
|
|
114
|
+
|
|
115
|
+
**Leading column rule**: A composite index on `(a, b, c)` can serve queries on:
|
|
116
|
+
- `a` alone
|
|
117
|
+
- `a, b`
|
|
118
|
+
- `a, b, c`
|
|
119
|
+
|
|
120
|
+
But NOT `b` alone or `c` alone (pre-PG18).
|
|
121
|
+
|
|
122
|
+
**B-tree skip scan (PG18+)**: PG18 can skip through distinct values of leading columns, so an index on `(a, b, c)` can now serve queries on `b` or `c` alone — by scanning each distinct `a` value. This works best when the leading column has low cardinality (few distinct values). It eliminates many cases where you previously needed a separate single-column index.
|
|
123
|
+
|
|
124
|
+
## Partial Indexes
|
|
125
|
+
|
|
126
|
+
Index only the rows that matter. Smaller index = faster lookups, less maintenance.
|
|
127
|
+
|
|
128
|
+
```sql
|
|
129
|
+
-- Only index active orders (95% of queries filter on active)
|
|
130
|
+
CREATE INDEX idx_orders_active ON orders(customer_id)
|
|
131
|
+
WHERE status = 'active';
|
|
132
|
+
|
|
133
|
+
-- Only index non-null values
|
|
134
|
+
CREATE INDEX idx_orders_shipped ON orders(shipped_at)
|
|
135
|
+
WHERE shipped_at IS NOT NULL;
|
|
136
|
+
```
|
|
137
|
+
|
|
138
|
+
The query's WHERE clause must match (or imply) the index predicate for the planner to use it.
|
|
139
|
+
|
|
140
|
+
## Covering Indexes (INCLUDE)
|
|
141
|
+
|
|
142
|
+
Add non-key columns to enable index-only scans without bloating the B-tree structure.
|
|
143
|
+
|
|
144
|
+
```sql
|
|
145
|
+
-- Query: SELECT email, name FROM users WHERE email = ?
|
|
146
|
+
CREATE UNIQUE INDEX idx_users_email ON users(email) INCLUDE (name);
|
|
147
|
+
```
|
|
148
|
+
|
|
149
|
+
The `name` column is stored in the index leaf pages but not in the B-tree structure. This enables an index-only scan (no heap fetch) without affecting index ordering or uniqueness.
|
|
150
|
+
|
|
151
|
+
## Expression Indexes
|
|
152
|
+
|
|
153
|
+
Index the result of an expression or function.
|
|
154
|
+
|
|
155
|
+
```sql
|
|
156
|
+
-- Case-insensitive email lookup
|
|
157
|
+
CREATE UNIQUE INDEX idx_users_email_lower ON users(lower(email));
|
|
158
|
+
-- Query must match: WHERE lower(email) = lower($1)
|
|
159
|
+
|
|
160
|
+
-- JSONB field extraction
|
|
161
|
+
CREATE INDEX idx_events_type ON events((payload->>'type'));
|
|
162
|
+
-- Query: WHERE payload->>'type' = 'click'
|
|
163
|
+
|
|
164
|
+
-- Date truncation
|
|
165
|
+
-- timestamptz must be made timezone-independent for an immutable expression
|
|
166
|
+
CREATE INDEX idx_orders_month
|
|
167
|
+
ON orders(date_trunc('month', created_at AT TIME ZONE 'UTC'));
|
|
168
|
+
```
|
|
169
|
+
|
|
170
|
+
The query must use the same expression for the planner to match the index.
|
|
171
|
+
|
|
172
|
+
## Indexes on Partitioned Tables
|
|
173
|
+
|
|
174
|
+
Indexes defined on a partitioned parent table are **automatically created on all existing and future child partitions**.
|
|
175
|
+
|
|
176
|
+
```sql
|
|
177
|
+
-- Create index on the partitioned parent — propagates to all partitions
|
|
178
|
+
CREATE INDEX idx_events_type ON events(occurred_at, (payload->>'type'));
|
|
179
|
+
```
|
|
180
|
+
|
|
181
|
+
### Key Rules
|
|
182
|
+
|
|
183
|
+
- **Partition key in unique indexes**: Any UNIQUE or PRIMARY KEY index on a partitioned table must include all partition key columns.
|
|
184
|
+
|
|
185
|
+
```sql
|
|
186
|
+
-- Partitioned by occurred_at — PK must include it
|
|
187
|
+
CREATE TABLE events (
|
|
188
|
+
id bigint GENERATED ALWAYS AS IDENTITY,
|
|
189
|
+
occurred_at timestamptz NOT NULL,
|
|
190
|
+
payload jsonb
|
|
191
|
+
) PARTITION BY RANGE (occurred_at);
|
|
192
|
+
|
|
193
|
+
-- This works:
|
|
194
|
+
ALTER TABLE events ADD PRIMARY KEY (id, occurred_at);
|
|
195
|
+
|
|
196
|
+
-- This fails: partition key 'occurred_at' not in the constraint
|
|
197
|
+
-- ALTER TABLE events ADD PRIMARY KEY (id);
|
|
198
|
+
```
|
|
199
|
+
|
|
200
|
+
- **Per-partition indexes**: You can also create indexes on individual partitions for partition-specific optimizations. These won't propagate to other partitions.
|
|
201
|
+
|
|
202
|
+
```sql
|
|
203
|
+
-- Extra index only on the hot partition
|
|
204
|
+
CREATE INDEX idx_events_q1_status ON events_2024_q1((payload->>'status'));
|
|
205
|
+
```
|
|
206
|
+
|
|
207
|
+
- **CONCURRENTLY on partitioned tables**: PostgreSQL does not support `CREATE INDEX CONCURRENTLY` directly on a partitioned parent. Create an invalid parent index with `ON ONLY`, build matching indexes concurrently on each partition, then attach them:
|
|
208
|
+
|
|
209
|
+
```sql
|
|
210
|
+
CREATE INDEX idx_events_customer
|
|
211
|
+
ON ONLY events ((payload->>'customer_id'));
|
|
212
|
+
|
|
213
|
+
CREATE INDEX CONCURRENTLY idx_events_q1_customer
|
|
214
|
+
ON events_2024_q1 ((payload->>'customer_id'));
|
|
215
|
+
|
|
216
|
+
ALTER INDEX idx_events_customer
|
|
217
|
+
ATTACH PARTITION idx_events_q1_customer;
|
|
218
|
+
```
|
|
219
|
+
|
|
220
|
+
Repeat the concurrent build and attach steps for every partition. The parent index becomes valid after all partition indexes are attached. If a child build fails, drop or rebuild that invalid child index before attaching it.
|
|
221
|
+
|
|
222
|
+
- **Index-only scans**: Work across partitions. The planner prunes irrelevant partitions first, then uses index-only scans on the remaining ones.
|
|
223
|
+
|
|
224
|
+
- **REINDEX on partitioned tables**: `REINDEX TABLE` on a partitioned table reindexes all partitions.
|
|
225
|
+
|
|
226
|
+
```sql
|
|
227
|
+
REINDEX TABLE CONCURRENTLY events;
|
|
228
|
+
```
|
|
229
|
+
|
|
230
|
+
## Concurrent Index Creation
|
|
231
|
+
|
|
232
|
+
For production use, always create indexes concurrently to avoid locking writes:
|
|
233
|
+
|
|
234
|
+
```sql
|
|
235
|
+
CREATE INDEX CONCURRENTLY idx_orders_customer ON orders(customer_id);
|
|
236
|
+
```
|
|
237
|
+
|
|
238
|
+
Caveats:
|
|
239
|
+
- Takes longer (two table scans instead of one)
|
|
240
|
+
- Cannot run inside a transaction block
|
|
241
|
+
- If it fails, leaves an `INVALID` index — drop and retry
|
|
242
|
+
- Check for invalid indexes: `SELECT * FROM pg_index WHERE NOT indisvalid;`
|
|
243
|
+
|
|
244
|
+
## Index Maintenance
|
|
245
|
+
|
|
246
|
+
### Reindexing Bloated Indexes
|
|
247
|
+
|
|
248
|
+
```sql
|
|
249
|
+
-- Concurrent reindex
|
|
250
|
+
REINDEX INDEX CONCURRENTLY idx_orders_customer;
|
|
251
|
+
|
|
252
|
+
-- Or reindex all indexes on a table
|
|
253
|
+
REINDEX TABLE CONCURRENTLY orders;
|
|
254
|
+
```
|
|
255
|
+
|
|
256
|
+
### Monitoring Index Size
|
|
257
|
+
|
|
258
|
+
```sql
|
|
259
|
+
SELECT
|
|
260
|
+
indexrelid::regclass AS index_name,
|
|
261
|
+
pg_size_pretty(pg_relation_size(indexrelid)) AS size,
|
|
262
|
+
idx_scan AS scans,
|
|
263
|
+
idx_tup_read AS tuples_read
|
|
264
|
+
FROM pg_stat_user_indexes
|
|
265
|
+
ORDER BY pg_relation_size(indexrelid) DESC;
|
|
266
|
+
```
|
|
267
|
+
|
|
268
|
+
## Finding Unused Indexes
|
|
269
|
+
|
|
270
|
+
```sql
|
|
271
|
+
SELECT
|
|
272
|
+
schemaname,
|
|
273
|
+
relname AS table_name,
|
|
274
|
+
indexrelname AS index_name,
|
|
275
|
+
pg_size_pretty(pg_relation_size(indexrelid)) AS index_size,
|
|
276
|
+
idx_scan AS times_used
|
|
277
|
+
FROM pg_stat_user_indexes
|
|
278
|
+
WHERE idx_scan = 0
|
|
279
|
+
AND indexrelname NOT LIKE '%_pkey' -- exclude primary keys
|
|
280
|
+
AND indexrelname NOT LIKE '%unique%' -- exclude unique constraints
|
|
281
|
+
ORDER BY pg_relation_size(indexrelid) DESC;
|
|
282
|
+
```
|
|
283
|
+
|
|
284
|
+
Reset statistics after a representative period: `SELECT pg_stat_reset();`
|
|
285
|
+
|
|
286
|
+
## Finding Duplicate Indexes
|
|
287
|
+
|
|
288
|
+
```sql
|
|
289
|
+
SELECT
|
|
290
|
+
array_agg(indexrelid::regclass) AS indexes,
|
|
291
|
+
indrelid::regclass AS table_name,
|
|
292
|
+
indkey AS column_numbers
|
|
293
|
+
FROM pg_index
|
|
294
|
+
GROUP BY indrelid, indkey
|
|
295
|
+
HAVING count(*) > 1;
|
|
296
|
+
```
|
|
297
|
+
|
|
298
|
+
Also check for indexes that are a prefix of another:
|
|
299
|
+
- `(a)` is redundant if `(a, b)` exists
|
|
300
|
+
- `(a, b)` is NOT redundant if `(a, b, c)` exists and you need index-only scans on just `(a, b)`
|
|
301
|
+
|
|
302
|
+
## Index Selection Decision Tree
|
|
303
|
+
|
|
304
|
+
1. **What operator?**
|
|
305
|
+
- `=`, `<`, `>`, `BETWEEN` → B-tree
|
|
306
|
+
- `@>`, `?`, `&&` on jsonb/array → GIN
|
|
307
|
+
- `@@` full-text → GIN
|
|
308
|
+
- `ILIKE '%x%'` → GIN + pg_trgm
|
|
309
|
+
- Range/geometric overlap → GiST
|
|
310
|
+
- Naturally ordered append-only → BRIN
|
|
311
|
+
|
|
312
|
+
2. **How many rows match?** If > 10-20% of table, index may not help (seq scan is cheaper).
|
|
313
|
+
|
|
314
|
+
3. **Can you narrow the index?** Use a partial index if most queries filter on a subset.
|
|
315
|
+
|
|
316
|
+
4. **Do you need index-only scans?** Add `INCLUDE` columns.
|
|
317
|
+
|
|
318
|
+
5. **Is the column an expression?** Use an expression index.
|