@coralai/sps-cli 0.42.0 → 0.43.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +34 -3
- package/dist/commands/projectInit.d.ts.map +1 -1
- package/dist/commands/projectInit.js +40 -53
- package/dist/commands/projectInit.js.map +1 -1
- package/dist/commands/skillCommand.d.ts +2 -0
- package/dist/commands/skillCommand.d.ts.map +1 -0
- package/dist/commands/skillCommand.js +235 -0
- package/dist/commands/skillCommand.js.map +1 -0
- package/dist/core/skillStore.d.ts +46 -0
- package/dist/core/skillStore.d.ts.map +1 -0
- package/dist/core/skillStore.js +197 -0
- package/dist/core/skillStore.js.map +1 -0
- package/dist/core/skillStore.test.d.ts +2 -0
- package/dist/core/skillStore.test.d.ts.map +1 -0
- package/dist/core/skillStore.test.js +190 -0
- package/dist/core/skillStore.test.js.map +1 -0
- package/dist/main.js +19 -17
- package/dist/main.js.map +1 -1
- package/package.json +1 -1
- package/skills/architecture-decision-records/SKILL.md +207 -0
- package/skills/backend/SKILL.md +62 -0
- package/skills/backend/references/api-design.md +168 -0
- package/skills/backend/references/caching.md +181 -0
- package/skills/backend/references/data-access.md +173 -0
- package/skills/backend/references/layering.md +181 -0
- package/skills/backend/references/observability.md +190 -0
- package/skills/backend/references/resilience.md +201 -0
- package/skills/backend/references/security.md +186 -0
- package/skills/backend-architect/SKILL.md +119 -0
- package/skills/code-reviewer/SKILL.md +143 -0
- package/skills/coding-standards/SKILL.md +60 -0
- package/skills/coding-standards/references/clean-code.md +258 -0
- package/skills/coding-standards/references/code-review.md +192 -0
- package/skills/coding-standards/references/commits-and-prs.md +226 -0
- package/skills/coding-standards/references/error-strategy.md +193 -0
- package/skills/coding-standards/references/naming.md +185 -0
- package/skills/coding-standards/references/tdd.md +171 -0
- package/skills/database/SKILL.md +53 -0
- package/skills/database/references/indexing.md +190 -0
- package/skills/database/references/migrations.md +199 -0
- package/skills/database/references/nosql.md +185 -0
- package/skills/database/references/queries.md +295 -0
- package/skills/database/references/scaling.md +203 -0
- package/skills/database/references/schema.md +191 -0
- package/skills/database-optimizer/SKILL.md +168 -0
- package/skills/debugging-workflow/SKILL.md +244 -0
- package/skills/devops/SKILL.md +55 -0
- package/skills/devops/references/ci-cd.md +204 -0
- package/skills/devops/references/containers.md +272 -0
- package/skills/devops/references/deploy.md +201 -0
- package/skills/devops/references/iac.md +252 -0
- package/skills/devops/references/observability.md +228 -0
- package/skills/devops/references/secrets.md +178 -0
- package/skills/devops-automator/SKILL.md +164 -0
- package/skills/frontend/SKILL.md +52 -0
- package/skills/frontend/references/accessibility.md +222 -0
- package/skills/frontend/references/components.md +206 -0
- package/skills/frontend/references/performance.md +219 -0
- package/skills/frontend/references/routing.md +209 -0
- package/skills/frontend/references/state.md +190 -0
- package/skills/frontend/references/testing.md +216 -0
- package/skills/frontend-developer/SKILL.md +115 -0
- package/skills/git-workflow/SKILL.md +355 -0
- package/skills/golang/SKILL.md +49 -0
- package/skills/golang/references/concurrency.md +284 -0
- package/skills/golang/references/errors.md +241 -0
- package/skills/golang/references/idioms.md +285 -0
- package/skills/golang/references/testing.md +238 -0
- package/skills/java/SKILL.md +50 -0
- package/skills/java/references/concurrency.md +194 -0
- package/skills/java/references/idioms.md +283 -0
- package/skills/java/references/testing.md +228 -0
- package/skills/kotlin/SKILL.md +47 -0
- package/skills/kotlin/references/coroutines.md +240 -0
- package/skills/kotlin/references/idioms.md +268 -0
- package/skills/kotlin/references/testing.md +219 -0
- package/skills/mobile/SKILL.md +50 -0
- package/skills/mobile/references/architecture.md +204 -0
- package/skills/mobile/references/navigation.md +158 -0
- package/skills/mobile/references/performance.md +152 -0
- package/skills/mobile/references/platform.md +166 -0
- package/skills/mobile/references/state-and-data.md +174 -0
- package/skills/python/SKILL.md +51 -0
- package/skills/python/THIRD_PARTY.md +14 -0
- package/skills/python/references/async.md +218 -0
- package/skills/python/references/error-handling.md +254 -0
- package/skills/python/references/idioms.md +279 -0
- package/skills/python/references/packaging.md +233 -0
- package/skills/python/references/testing.md +269 -0
- package/skills/python/references/typing.md +292 -0
- package/skills/qa-tester/SKILL.md +186 -0
- package/skills/rust/SKILL.md +50 -0
- package/skills/rust/references/async.md +224 -0
- package/skills/rust/references/errors.md +240 -0
- package/skills/rust/references/ownership.md +263 -0
- package/skills/rust/references/testing.md +274 -0
- package/skills/rust/references/traits.md +250 -0
- package/skills/security-engineer/SKILL.md +157 -0
- package/skills/swift/SKILL.md +48 -0
- package/skills/swift/references/concurrency.md +280 -0
- package/skills/swift/references/idioms.md +334 -0
- package/skills/swift/references/testing.md +229 -0
- package/skills/typescript/SKILL.md +51 -0
- package/skills/typescript/references/async.md +241 -0
- package/skills/typescript/references/errors.md +208 -0
- package/skills/typescript/references/idioms.md +246 -0
- package/skills/typescript/references/testing.md +225 -0
- package/skills/typescript/references/tooling.md +208 -0
- package/skills/typescript/references/types.md +259 -0
|
@@ -0,0 +1,185 @@
|
|
|
1
|
+
# NoSQL
|
|
2
|
+
|
|
3
|
+
Document, key-value, time-series, graph — when each fits.
|
|
4
|
+
|
|
5
|
+
## Picking the right engine
|
|
6
|
+
|
|
7
|
+
```
|
|
8
|
+
Pick by access pattern, not by popularity.
|
|
9
|
+
```
|
|
10
|
+
|
|
11
|
+
| Engine | Shape | Good for |
|
|
12
|
+
|---|---|---|
|
|
13
|
+
| **Postgres / MySQL** | Relational + JSON | Almost everything; strong default |
|
|
14
|
+
| **MongoDB / Couchbase** | Document | Shape varies per record; embed-heavy reads |
|
|
15
|
+
| **DynamoDB / Cassandra** | Key-value / wide column | Massive scale; predictable access patterns |
|
|
16
|
+
| **Redis** | Key-value (in-memory) | Cache, leaderboard, session, pub-sub |
|
|
17
|
+
| **ClickHouse / BigQuery / Redshift** | Columnar / OLAP | Analytics; big scans; aggregations |
|
|
18
|
+
| **TimescaleDB / InfluxDB** | Time-series | Metrics, events, sensor data |
|
|
19
|
+
| **Neo4j / DGraph** | Graph | Traversals (relationships matter more than rows) |
|
|
20
|
+
| **Elasticsearch / OpenSearch** | Inverted index | Full text, log search |
|
|
21
|
+
|
|
22
|
+
Rule: start with Postgres. Add a specialized store **when the access pattern justifies operational cost**. A lonely Elasticsearch cluster is a bug farm.
|
|
23
|
+
|
|
24
|
+
## Document stores (MongoDB, Couchbase)
|
|
25
|
+
|
|
26
|
+
Store JSON-ish documents. No joins; embed or reference.
|
|
27
|
+
|
|
28
|
+
```json
|
|
29
|
+
{
|
|
30
|
+
"_id": "u_01H...",
|
|
31
|
+
"email": "a@x.com",
|
|
32
|
+
"orders": [
|
|
33
|
+
{ "id": "ord_01H...", "items": [...], "total_cents": 2599 }
|
|
34
|
+
]
|
|
35
|
+
}
|
|
36
|
+
```
|
|
37
|
+
|
|
38
|
+
### Embed vs. reference
|
|
39
|
+
|
|
40
|
+
- **Embed** what you always read together and that doesn't grow unboundedly.
|
|
41
|
+
- **Reference** what's read separately or grows.
|
|
42
|
+
|
|
43
|
+
Rules:
|
|
44
|
+
- One-to-few (user ↔ addresses) → embed.
|
|
45
|
+
- One-to-many (user ↔ orders, bounded) → embed with a soft limit.
|
|
46
|
+
- One-to-many unbounded (user ↔ events, could be millions) → reference.
|
|
47
|
+
- Many-to-many → reference on both sides.
|
|
48
|
+
|
|
49
|
+
### Indexes
|
|
50
|
+
|
|
51
|
+
Every MongoDB query benefits from an index; without, it scans the collection. Index on the field you query, and compound on `(filter, sort)`.
|
|
52
|
+
|
|
53
|
+
### Schema is still a thing
|
|
54
|
+
|
|
55
|
+
Schemaless doesn't mean unschematic. Use a library (Zod, Joi, Pydantic, Mongoose) to define the shape in code and validate at write time. Otherwise two weeks later you have five variants of the same "user" shape.
|
|
56
|
+
|
|
57
|
+
### Transactions
|
|
58
|
+
|
|
59
|
+
Modern MongoDB supports multi-document transactions. Use them when you need them; don't hand-roll two-phase commits in app code.
|
|
60
|
+
|
|
61
|
+
## Key-value (DynamoDB, Cassandra)
|
|
62
|
+
|
|
63
|
+
Scale: massive. Flexibility: low.
|
|
64
|
+
|
|
65
|
+
Access patterns **must** be known at design time. You design the key to match the queries, not the other way around.
|
|
66
|
+
|
|
67
|
+
### Single-table design (DynamoDB)
|
|
68
|
+
|
|
69
|
+
```
|
|
70
|
+
pk sk attrs
|
|
71
|
+
USER#u1 PROFILE { name, email }
|
|
72
|
+
USER#u1 ORDER#o1 { total, status }
|
|
73
|
+
USER#u1 ORDER#o2 { total, status }
|
|
74
|
+
ORDER#o1 META { user_id, items }
|
|
75
|
+
```
|
|
76
|
+
|
|
77
|
+
One table, careful composite keys. Supports:
|
|
78
|
+
- Get user profile: PK=USER#u1, SK=PROFILE
|
|
79
|
+
- List user orders: PK=USER#u1, SK begins_with ORDER#
|
|
80
|
+
- Get order: PK=ORDER#o1
|
|
81
|
+
|
|
82
|
+
Queries that don't fit the key structure need a global secondary index (expensive) or an adapter (client-side fan-out).
|
|
83
|
+
|
|
84
|
+
If access patterns aren't stable, use Postgres.
|
|
85
|
+
|
|
86
|
+
### Eventual consistency
|
|
87
|
+
|
|
88
|
+
Most KV stores offer strong consistency on single-key ops and eventual on cross-key. Design around it:
|
|
89
|
+
- Read-your-writes: query the primary (DynamoDB: `ConsistentRead=true`).
|
|
90
|
+
- Listing results may lag recent writes briefly.
|
|
91
|
+
|
|
92
|
+
## Redis
|
|
93
|
+
|
|
94
|
+
Primarily in-memory; data structures (strings, lists, sets, sorted sets, hashes, streams, geo).
|
|
95
|
+
|
|
96
|
+
### Patterns
|
|
97
|
+
|
|
98
|
+
- **Cache-aside** — most common.
|
|
99
|
+
- **Session store** — cookie → Redis key → session blob.
|
|
100
|
+
- **Rate limiter** — `INCR` with TTL on window keys.
|
|
101
|
+
- **Leaderboard** — sorted set (`ZADD` / `ZREVRANGE`).
|
|
102
|
+
- **Job queue** — lists (`LPUSH` / `BRPOP`) or Streams (better for ack-and-retry).
|
|
103
|
+
- **Pub/sub** — fan out events to subscribers.
|
|
104
|
+
- **Distributed lock** — Redlock or `SET NX PX`.
|
|
105
|
+
|
|
106
|
+
### Persistence
|
|
107
|
+
|
|
108
|
+
- **RDB** snapshots: point-in-time, fast recovery, up to minutes of data loss.
|
|
109
|
+
- **AOF** append-only log: near-zero loss, slower restart.
|
|
110
|
+
- **AOF + RDB** together: recommended.
|
|
111
|
+
|
|
112
|
+
Even "in-memory" Redis should have persistence. Otherwise a restart vacuums state.
|
|
113
|
+
|
|
114
|
+
### Memory limits
|
|
115
|
+
|
|
116
|
+
Set `maxmemory` + eviction policy (`allkeys-lru`, `volatile-ttl`). Otherwise Redis OOMs and crashes.
|
|
117
|
+
|
|
118
|
+
## Time-series (TimescaleDB, InfluxDB)
|
|
119
|
+
|
|
120
|
+
Purpose-built for write-heavy append-mostly workloads with time-range reads.
|
|
121
|
+
|
|
122
|
+
- High ingest rate (100K+ points / sec).
|
|
123
|
+
- Downsampling / rollups built in.
|
|
124
|
+
- Retention policies: drop data older than N.
|
|
125
|
+
- Optimized compression.
|
|
126
|
+
|
|
127
|
+
If your workload is "mostly append, rarely update, queried by time", this is the right tool. Forcing a generic DB to do it (insert rate, index bloat, storage cost) is misery.
|
|
128
|
+
|
|
129
|
+
TimescaleDB sits on top of Postgres — you get SQL, joins to normal tables, familiar ops.
|
|
130
|
+
|
|
131
|
+
## Graph databases (Neo4j, DGraph)
|
|
132
|
+
|
|
133
|
+
When the data is relationships and queries are traversals: "friends of friends with a common interest", "shortest path", "who did this user influence".
|
|
134
|
+
|
|
135
|
+
In a relational DB, that's a multi-step recursive CTE. In a graph DB, it's a one-liner in Cypher / GraphQL+ / Gremlin.
|
|
136
|
+
|
|
137
|
+
Cost: operational, learning curve, ecosystem. Use only when traversals dominate.
|
|
138
|
+
|
|
139
|
+
## Search (Elasticsearch / OpenSearch / Typesense / MeiliSearch)
|
|
140
|
+
|
|
141
|
+
Full-text with relevance, filters, aggregations. Postgres tsvector handles basic needs; search engines handle:
|
|
142
|
+
|
|
143
|
+
- Fuzzy matching, typo tolerance.
|
|
144
|
+
- Multi-language stemming.
|
|
145
|
+
- Custom relevance tuning (boosts per field).
|
|
146
|
+
- Faceted filtering with aggregates.
|
|
147
|
+
- Log / event search at scale.
|
|
148
|
+
|
|
149
|
+
**Never** use a search engine as the source of truth. Index from the authoritative DB; reindex is a feature, not a crisis.
|
|
150
|
+
|
|
151
|
+
## Analytics stores
|
|
152
|
+
|
|
153
|
+
OLAP vs. OLTP is the single biggest architectural fork.
|
|
154
|
+
|
|
155
|
+
- **OLTP** (Postgres, MySQL): many small reads/writes, low latency, strong consistency.
|
|
156
|
+
- **OLAP** (ClickHouse, BigQuery, Snowflake, Redshift, DuckDB): few big scans, column-oriented, optimized for aggregations.
|
|
157
|
+
|
|
158
|
+
Running analytics on the prod DB:
|
|
159
|
+
- OK for small teams, small data.
|
|
160
|
+
- Hits limits fast: locks the primary, slows the site, bursts workload.
|
|
161
|
+
|
|
162
|
+
Standard path:
|
|
163
|
+
1. Start with OLTP + read replica for reports.
|
|
164
|
+
2. Add a warehouse (BigQuery / Snowflake) when replica reads aren't enough.
|
|
165
|
+
3. Build an ELT pipeline (Fivetran, Airbyte, or home-grown) to move OLTP → warehouse.
|
|
166
|
+
|
|
167
|
+
## Polyglot persistence
|
|
168
|
+
|
|
169
|
+
Using 5 different stores because each is "best at one thing" sounds great, adds up in ops cost. Rule: every new store doubles what your oncall rotation needs to know.
|
|
170
|
+
|
|
171
|
+
Add stores sparingly. Prefer a generalist (Postgres) doing a specialist's job badly over two specialists that must be kept in sync.
|
|
172
|
+
|
|
173
|
+
## Anti-patterns
|
|
174
|
+
|
|
175
|
+
| Anti-pattern | Fix |
|
|
176
|
+
|---|---|
|
|
177
|
+
| MongoDB for a heavily relational domain | Postgres |
|
|
178
|
+
| DynamoDB single-table design with unknown future queries | Postgres until you know what you're querying |
|
|
179
|
+
| Redis as the source of truth | Cache only; durable store behind it |
|
|
180
|
+
| Elasticsearch as primary data store | Secondary index; reindex from the primary |
|
|
181
|
+
| Graph DB because "everything is a graph" | Only if traversals dominate |
|
|
182
|
+
| Analytics on the OLTP primary | Replica → warehouse |
|
|
183
|
+
| One schemaless collection per "microservice" | Validate shape; otherwise chaos in six months |
|
|
184
|
+
| Eventual-consistency reads presented to the user as "final" | Surface "just a moment" UX; or read from primary |
|
|
185
|
+
| No persistence on Redis in production | Always configure AOF + RDB |
|
|
@@ -0,0 +1,295 @@
|
|
|
1
|
+
# Queries
|
|
2
|
+
|
|
3
|
+
JOINs, subqueries, CTEs, window functions, EXPLAIN.
|
|
4
|
+
|
|
5
|
+
## Read the plan
|
|
6
|
+
|
|
7
|
+
`EXPLAIN (ANALYZE, BUFFERS)` shows what the DB actually did.
|
|
8
|
+
|
|
9
|
+
```sql
|
|
10
|
+
EXPLAIN (ANALYZE, BUFFERS)
|
|
11
|
+
SELECT u.email, count(*) AS n
|
|
12
|
+
FROM users u JOIN orders o ON o.user_id = u.id
|
|
13
|
+
WHERE u.active AND o.created_at > now() - interval '30 days'
|
|
14
|
+
GROUP BY u.email
|
|
15
|
+
HAVING count(*) > 5;
|
|
16
|
+
```
|
|
17
|
+
|
|
18
|
+
Red flags:
|
|
19
|
+
- `Seq Scan` on large tables where a filter expects few rows.
|
|
20
|
+
- Actual rows hugely diverge from the estimate (stats are stale → `ANALYZE`).
|
|
21
|
+
- `Hash Join` spilling to disk (`Disk: ...`).
|
|
22
|
+
- Nested Loop where Hash/Merge would be cheaper (low cardinality estimate misled the planner).
|
|
23
|
+
|
|
24
|
+
## Join types
|
|
25
|
+
|
|
26
|
+
| Join | Meaning |
|
|
27
|
+
|---|---|
|
|
28
|
+
| `INNER JOIN` | Rows where both sides match |
|
|
29
|
+
| `LEFT JOIN` | All left rows; NULLs for unmatched right |
|
|
30
|
+
| `RIGHT JOIN` | Rare; usually rewrite as LEFT with swapped sides |
|
|
31
|
+
| `FULL OUTER JOIN` | Both sides; NULLs where unmatched |
|
|
32
|
+
| `CROSS JOIN` | Cartesian product — use deliberately |
|
|
33
|
+
| `LATERAL` | Right side can reference left — per-row subquery |
|
|
34
|
+
|
|
35
|
+
## `EXISTS` vs. `IN` vs. `JOIN`
|
|
36
|
+
|
|
37
|
+
For "users who have at least one paid order":
|
|
38
|
+
|
|
39
|
+
```sql
|
|
40
|
+
-- ✅ Standard, optimizer handles well in modern DBs
|
|
41
|
+
SELECT u.* FROM users u
|
|
42
|
+
WHERE EXISTS (SELECT 1 FROM orders o WHERE o.user_id = u.id AND o.status = 'paid');
|
|
43
|
+
|
|
44
|
+
-- Equivalent, often same plan
|
|
45
|
+
SELECT DISTINCT u.* FROM users u JOIN orders o ON o.user_id = u.id
|
|
46
|
+
WHERE o.status = 'paid';
|
|
47
|
+
|
|
48
|
+
-- Avoid — duplicates without DISTINCT; subtle
|
|
49
|
+
SELECT u.* FROM users u
|
|
50
|
+
WHERE u.id IN (SELECT o.user_id FROM orders o WHERE o.status = 'paid');
|
|
51
|
+
```
|
|
52
|
+
|
|
53
|
+
`EXISTS` is usually the clearest for "does any matching row exist". `JOIN + DISTINCT` can double-count on many-to-many relationships.
|
|
54
|
+
|
|
55
|
+
## CTEs (`WITH`)
|
|
56
|
+
|
|
57
|
+
Named subqueries for readability. In Postgres < 12, CTEs were optimization fences; 12+ inlines unless `MATERIALIZED` is specified.
|
|
58
|
+
|
|
59
|
+
```sql
|
|
60
|
+
WITH active_users AS (
|
|
61
|
+
SELECT id FROM users WHERE active
|
|
62
|
+
),
|
|
63
|
+
recent_orders AS (
|
|
64
|
+
SELECT user_id, sum(total_cents) AS total
|
|
65
|
+
FROM orders
|
|
66
|
+
WHERE created_at > now() - interval '30 days'
|
|
67
|
+
GROUP BY user_id
|
|
68
|
+
)
|
|
69
|
+
SELECT u.id, r.total
|
|
70
|
+
FROM active_users u
|
|
71
|
+
JOIN recent_orders r ON r.user_id = u.id;
|
|
72
|
+
```
|
|
73
|
+
|
|
74
|
+
Recursive CTEs for trees:
|
|
75
|
+
|
|
76
|
+
```sql
|
|
77
|
+
WITH RECURSIVE subordinates AS (
|
|
78
|
+
SELECT id, manager_id, 0 AS depth FROM employees WHERE id = ?
|
|
79
|
+
UNION ALL
|
|
80
|
+
SELECT e.id, e.manager_id, s.depth + 1
|
|
81
|
+
FROM employees e JOIN subordinates s ON e.manager_id = s.id
|
|
82
|
+
)
|
|
83
|
+
SELECT * FROM subordinates;
|
|
84
|
+
```
|
|
85
|
+
|
|
86
|
+
## Window functions
|
|
87
|
+
|
|
88
|
+
Computations over rows without collapsing them.
|
|
89
|
+
|
|
90
|
+
```sql
|
|
91
|
+
-- Rank orders by total within each user
|
|
92
|
+
SELECT user_id, id, total_cents,
|
|
93
|
+
row_number() OVER (PARTITION BY user_id ORDER BY total_cents DESC) AS rn
|
|
94
|
+
FROM orders;
|
|
95
|
+
|
|
96
|
+
-- Running total
|
|
97
|
+
SELECT created_at, total_cents,
|
|
98
|
+
sum(total_cents) OVER (ORDER BY created_at) AS running_total
|
|
99
|
+
FROM orders;
|
|
100
|
+
|
|
101
|
+
-- Day-over-day
|
|
102
|
+
SELECT day, revenue,
|
|
103
|
+
revenue - lag(revenue) OVER (ORDER BY day) AS delta
|
|
104
|
+
FROM daily_revenue;
|
|
105
|
+
```
|
|
106
|
+
|
|
107
|
+
Often replaces a self-join or a complex subquery.
|
|
108
|
+
|
|
109
|
+
## Aggregations
|
|
110
|
+
|
|
111
|
+
```sql
|
|
112
|
+
-- Count
|
|
113
|
+
SELECT count(*) FROM orders; -- counts rows
|
|
114
|
+
SELECT count(email) FROM users; -- counts non-NULL emails
|
|
115
|
+
SELECT count(DISTINCT user_id) FROM orders;
|
|
116
|
+
|
|
117
|
+
-- Conditional aggregates
|
|
118
|
+
SELECT
|
|
119
|
+
count(*) FILTER (WHERE status = 'paid') AS paid,
|
|
120
|
+
count(*) FILTER (WHERE status = 'pending') AS pending
|
|
121
|
+
FROM orders;
|
|
122
|
+
|
|
123
|
+
-- Array / string aggregation
|
|
124
|
+
SELECT user_id, string_agg(tag, ',' ORDER BY tag) FROM user_tags GROUP BY user_id;
|
|
125
|
+
SELECT user_id, array_agg(tag) FROM user_tags GROUP BY user_id;
|
|
126
|
+
```
|
|
127
|
+
|
|
128
|
+
`FILTER` is cleaner than `CASE WHEN ... ELSE NULL END` in an aggregate.
|
|
129
|
+
|
|
130
|
+
## Upsert
|
|
131
|
+
|
|
132
|
+
Insert-or-update in one statement, atomically.
|
|
133
|
+
|
|
134
|
+
```sql
|
|
135
|
+
-- Postgres
|
|
136
|
+
INSERT INTO users (id, email, name)
|
|
137
|
+
VALUES (?, ?, ?)
|
|
138
|
+
ON CONFLICT (email) DO UPDATE
|
|
139
|
+
SET name = EXCLUDED.name, updated_at = now();
|
|
140
|
+
|
|
141
|
+
-- MySQL
|
|
142
|
+
INSERT INTO users (id, email, name)
|
|
143
|
+
VALUES (?, ?, ?)
|
|
144
|
+
ON DUPLICATE KEY UPDATE
|
|
145
|
+
name = VALUES(name), updated_at = now();
|
|
146
|
+
```
|
|
147
|
+
|
|
148
|
+
Don't SELECT-then-INSERT-or-UPDATE in app code. Race condition → duplicate rows.
|
|
149
|
+
|
|
150
|
+
## Batch operations
|
|
151
|
+
|
|
152
|
+
Process many rows in one statement.
|
|
153
|
+
|
|
154
|
+
```sql
|
|
155
|
+
-- Bulk insert with VALUES
|
|
156
|
+
INSERT INTO events (id, payload) VALUES (?, ?), (?, ?), (?, ?), ...;
|
|
157
|
+
|
|
158
|
+
-- Bulk delete with IN
|
|
159
|
+
DELETE FROM stale_events WHERE id IN (?, ?, ?);
|
|
160
|
+
|
|
161
|
+
-- Update with CTE + JOIN
|
|
162
|
+
WITH bad_ids AS (SELECT id FROM events WHERE created_at < now() - interval '1 year')
|
|
163
|
+
DELETE FROM events USING bad_ids WHERE events.id = bad_ids.id;
|
|
164
|
+
```
|
|
165
|
+
|
|
166
|
+
## Pagination — keyset over offset
|
|
167
|
+
|
|
168
|
+
```sql
|
|
169
|
+
-- ❌ Slow on page 1000
|
|
170
|
+
SELECT * FROM events ORDER BY id LIMIT 50 OFFSET 50000;
|
|
171
|
+
|
|
172
|
+
-- ✅ Keyset pagination
|
|
173
|
+
SELECT * FROM events WHERE id > ? ORDER BY id LIMIT 50;
|
|
174
|
+
```
|
|
175
|
+
|
|
176
|
+
Offset pagination's cost grows with offset. Keyset is O(log n) regardless.
|
|
177
|
+
|
|
178
|
+
For reverse / filtered queries, the cursor is a tuple: `(created_at, id)`.
|
|
179
|
+
|
|
180
|
+
```sql
|
|
181
|
+
SELECT * FROM events
|
|
182
|
+
WHERE (created_at, id) < (?, ?)
|
|
183
|
+
ORDER BY created_at DESC, id DESC
|
|
184
|
+
LIMIT 50;
|
|
185
|
+
```
|
|
186
|
+
|
|
187
|
+
## `LIMIT` with `ORDER BY`
|
|
188
|
+
|
|
189
|
+
Always use `ORDER BY` when using `LIMIT`. Without it, the order is undefined — results can change between runs even on the same data.
|
|
190
|
+
|
|
191
|
+
## Avoid SELECT *
|
|
192
|
+
|
|
193
|
+
```sql
|
|
194
|
+
-- ❌ Sends every column over the wire; brittle when schema evolves
|
|
195
|
+
SELECT * FROM users WHERE id = ?;
|
|
196
|
+
|
|
197
|
+
-- ✅
|
|
198
|
+
SELECT id, email, active FROM users WHERE id = ?;
|
|
199
|
+
```
|
|
200
|
+
|
|
201
|
+
Explicit column lists:
|
|
202
|
+
- Ship less bandwidth.
|
|
203
|
+
- Enable covering indexes.
|
|
204
|
+
- Don't break when a column is added / renamed.
|
|
205
|
+
|
|
206
|
+
## Locking
|
|
207
|
+
|
|
208
|
+
Explicit locks for concurrency control:
|
|
209
|
+
|
|
210
|
+
```sql
|
|
211
|
+
-- Read the row, block concurrent writers
|
|
212
|
+
BEGIN;
|
|
213
|
+
SELECT * FROM accounts WHERE id = 1 FOR UPDATE;
|
|
214
|
+
-- decide, modify
|
|
215
|
+
UPDATE accounts SET balance = balance - 100 WHERE id = 1;
|
|
216
|
+
COMMIT;
|
|
217
|
+
```
|
|
218
|
+
|
|
219
|
+
`FOR UPDATE SKIP LOCKED` — for job-queue workers to take different rows:
|
|
220
|
+
|
|
221
|
+
```sql
|
|
222
|
+
SELECT * FROM jobs WHERE status = 'ready'
|
|
223
|
+
ORDER BY created_at
|
|
224
|
+
FOR UPDATE SKIP LOCKED
|
|
225
|
+
LIMIT 1;
|
|
226
|
+
```
|
|
227
|
+
|
|
228
|
+
Advisory locks for non-row locking (Postgres):
|
|
229
|
+
|
|
230
|
+
```sql
|
|
231
|
+
SELECT pg_try_advisory_xact_lock(hashtext('reindex-job'));
|
|
232
|
+
```
|
|
233
|
+
|
|
234
|
+
Great for cross-process leader election / run-once jobs.
|
|
235
|
+
|
|
236
|
+
## Avoid N+1 at SQL level
|
|
237
|
+
|
|
238
|
+
```sql
|
|
239
|
+
-- ❌ In app code:
|
|
240
|
+
for order in orders:
|
|
241
|
+
user = SELECT * FROM users WHERE id = order.user_id
|
|
242
|
+
-- N+1 queries
|
|
243
|
+
|
|
244
|
+
-- ✅ Single JOIN
|
|
245
|
+
SELECT o.*, u.email FROM orders o JOIN users u ON u.id = o.user_id;
|
|
246
|
+
|
|
247
|
+
-- ✅ Or a single IN:
|
|
248
|
+
SELECT * FROM users WHERE id IN (?, ?, ?, ...);
|
|
249
|
+
```
|
|
250
|
+
|
|
251
|
+
## NULL semantics — three-valued logic
|
|
252
|
+
|
|
253
|
+
`NULL = NULL` is `UNKNOWN`, not `TRUE`. Catches people out:
|
|
254
|
+
|
|
255
|
+
```sql
|
|
256
|
+
SELECT * FROM users WHERE email = 'a@x.com' OR email != 'a@x.com';
|
|
257
|
+
-- Rows where email IS NULL are NOT returned (NULL doesn't match either)
|
|
258
|
+
|
|
259
|
+
-- To include NULL:
|
|
260
|
+
WHERE email = 'a@x.com' OR email IS NULL
|
|
261
|
+
```
|
|
262
|
+
|
|
263
|
+
Use `COALESCE(x, default)` when comparing nullable columns.
|
|
264
|
+
|
|
265
|
+
## String / date functions — DB-specific
|
|
266
|
+
|
|
267
|
+
Every engine has its own. Postgres examples:
|
|
268
|
+
|
|
269
|
+
```sql
|
|
270
|
+
date_trunc('day', created_at) -- 2026-04-20 00:00:00
|
|
271
|
+
extract(epoch from (end_at - start_at)) -- seconds between
|
|
272
|
+
now() - interval '1 day' -- 24h ago
|
|
273
|
+
age(now(), created_at) -- human-friendly interval
|
|
274
|
+
|
|
275
|
+
lower(email), upper(email), length(text), substring(text, 1, 10)
|
|
276
|
+
split_part('a,b,c', ',', 2) -- 'b'
|
|
277
|
+
regexp_replace(s, '[0-9]+', '#', 'g')
|
|
278
|
+
```
|
|
279
|
+
|
|
280
|
+
Check the manual before inventing portable helpers.
|
|
281
|
+
|
|
282
|
+
## Anti-patterns
|
|
283
|
+
|
|
284
|
+
| Anti-pattern | Fix |
|
|
285
|
+
|---|---|
|
|
286
|
+
| `SELECT *` in app code | Name the columns |
|
|
287
|
+
| Queries built by string-concatenating user input | Parameterize |
|
|
288
|
+
| Offset pagination on large tables | Keyset |
|
|
289
|
+
| `WHERE function(col) = ?` without expression index | Rewrite or index |
|
|
290
|
+
| N+1 queries in app code | JOIN / IN / batch |
|
|
291
|
+
| Multiple round-trips where one query suffices | CTE / subquery |
|
|
292
|
+
| Guessing with ORDER BY RANDOM() on big tables | Pre-shuffle, reservoir sample, or `TABLESAMPLE` |
|
|
293
|
+
| SELECT-then-INSERT for upsert | Native `ON CONFLICT` / `MERGE` |
|
|
294
|
+
| Long transactions reading large data | Split, or use a cursor / pagination |
|
|
295
|
+
| `LIKE '%x%'` on a big table | Full-text index |
|
|
@@ -0,0 +1,203 @@
|
|
|
1
|
+
# Scaling
|
|
2
|
+
|
|
3
|
+
Replication, read replicas, sharding, partitioning, pooling.
|
|
4
|
+
|
|
5
|
+
## Scale vertical before horizontal
|
|
6
|
+
|
|
7
|
+
A single Postgres / MySQL instance on modern hardware handles a LOT:
|
|
8
|
+
- Postgres on a large machine: tens of thousands of QPS, terabytes of data.
|
|
9
|
+
- Most teams that "need" sharding actually have a missing index.
|
|
10
|
+
|
|
11
|
+
Scaling checklist before scaling out:
|
|
12
|
+
1. Are queries using indexes? (EXPLAIN)
|
|
13
|
+
2. Is the instance CPU- or I/O-bound? (top / iostat)
|
|
14
|
+
3. Connection pool configured? (see below)
|
|
15
|
+
4. Any long-held locks / long transactions?
|
|
16
|
+
5. Table bloat? (Postgres VACUUM health)
|
|
17
|
+
|
|
18
|
+
## Read replicas
|
|
19
|
+
|
|
20
|
+
Streaming replication (Postgres, MySQL) sends WAL / binlog to followers. Route reads to replicas, writes to primary.
|
|
21
|
+
|
|
22
|
+
```
|
|
23
|
+
app ──writes──▶ primary
|
|
24
|
+
──reads──▶ replica 1, replica 2, ...
|
|
25
|
+
```
|
|
26
|
+
|
|
27
|
+
Benefits:
|
|
28
|
+
- Horizontal read scaling.
|
|
29
|
+
- Failover target.
|
|
30
|
+
- Analytics offload.
|
|
31
|
+
|
|
32
|
+
Gotchas:
|
|
33
|
+
- **Replication lag**. Milliseconds usually, seconds under load. Reads immediately after a write may see stale data.
|
|
34
|
+
- **Read-your-writes**: route the current user's reads to primary for N seconds after they write, or always read from primary for critical paths.
|
|
35
|
+
|
|
36
|
+
## Connection pooling
|
|
37
|
+
|
|
38
|
+
Every real backend uses a pool. DBs limit max connections (Postgres default ~100); without pooling, a traffic spike hits the ceiling.
|
|
39
|
+
|
|
40
|
+
Pool parameters:
|
|
41
|
+
|
|
42
|
+
| Parameter | Starting value |
|
|
43
|
+
|---|---|
|
|
44
|
+
| min idle | 2–5 |
|
|
45
|
+
| max size | (DB max ÷ app replicas) − safety margin |
|
|
46
|
+
| connect timeout | 2–5 s |
|
|
47
|
+
| idle timeout | 30 s – 5 min |
|
|
48
|
+
| max lifetime | 30 min |
|
|
49
|
+
|
|
50
|
+
Serverless + traditional DB: use a pooler (PgBouncer, RDS Proxy, Supabase Pooler, Hyperdrive) — each cold lambda can't open its own pool.
|
|
51
|
+
|
|
52
|
+
PgBouncer modes:
|
|
53
|
+
- **Session pooling** — safest, but holds one backend connection per client session.
|
|
54
|
+
- **Transaction pooling** — best efficiency; some features (session-level variables, prepared statements in certain drivers) don't work.
|
|
55
|
+
- **Statement pooling** — rare; most strict restrictions.
|
|
56
|
+
|
|
57
|
+
## Partitioning
|
|
58
|
+
|
|
59
|
+
Split a large table into pieces based on a key (time, tenant, region). Each partition is its own physical table.
|
|
60
|
+
|
|
61
|
+
```sql
|
|
62
|
+
CREATE TABLE events (
|
|
63
|
+
id UUID, tenant_id UUID, created_at TIMESTAMPTZ, payload JSONB
|
|
64
|
+
) PARTITION BY RANGE (created_at);
|
|
65
|
+
|
|
66
|
+
CREATE TABLE events_2026_04 PARTITION OF events
|
|
67
|
+
FOR VALUES FROM ('2026-04-01') TO ('2026-05-01');
|
|
68
|
+
```
|
|
69
|
+
|
|
70
|
+
Benefits:
|
|
71
|
+
- Queries with `WHERE created_at BETWEEN ...` scan only the relevant partitions.
|
|
72
|
+
- Drop old data via `DROP PARTITION` (no delete scan, no VACUUM).
|
|
73
|
+
- Smaller indexes per partition.
|
|
74
|
+
|
|
75
|
+
Costs:
|
|
76
|
+
- Schema evolution is more work.
|
|
77
|
+
- Queries that don't include the partition key scan all partitions.
|
|
78
|
+
- Some ORMs / tools handle partitioning poorly.
|
|
79
|
+
|
|
80
|
+
Partition when:
|
|
81
|
+
- A single table is approaching 100M+ rows and queries typically touch a subset.
|
|
82
|
+
- You need to age out old data regularly.
|
|
83
|
+
- Insert rate is pushing one partition's size limits.
|
|
84
|
+
|
|
85
|
+
## Sharding
|
|
86
|
+
|
|
87
|
+
Distribute a dataset across multiple databases by a shard key.
|
|
88
|
+
|
|
89
|
+
```
|
|
90
|
+
tenant_id % N → which shard holds this tenant's data
|
|
91
|
+
```
|
|
92
|
+
|
|
93
|
+
Each shard is a self-contained DB. Cross-shard queries require the app to fan out and merge.
|
|
94
|
+
|
|
95
|
+
Before sharding, try:
|
|
96
|
+
- Vertical scale + read replicas.
|
|
97
|
+
- Partitioning.
|
|
98
|
+
- Service-level split (separate the auth DB from the orders DB).
|
|
99
|
+
|
|
100
|
+
When sharding is justified:
|
|
101
|
+
- Write throughput > what one instance can take.
|
|
102
|
+
- Data size > one instance's disk.
|
|
103
|
+
- Strong per-tenant isolation required.
|
|
104
|
+
|
|
105
|
+
Hard parts:
|
|
106
|
+
- **Cross-shard joins**: don't. Or use a small read replica that aggregates.
|
|
107
|
+
- **Cross-shard transactions**: don't. Use eventual consistency patterns.
|
|
108
|
+
- **Rebalancing**: moving data between shards is painful; pick a shard key that rarely grows imbalanced.
|
|
109
|
+
|
|
110
|
+
Managed services (Vitess, Citus, Spanner, CockroachDB, Yugabyte) do a lot of this for you. Self-building a sharding layer is a project unto itself.
|
|
111
|
+
|
|
112
|
+
## Caching layer
|
|
113
|
+
|
|
114
|
+
Offload read traffic from the DB.
|
|
115
|
+
|
|
116
|
+
| Layer | Cost | Gain |
|
|
117
|
+
|---|---|---|
|
|
118
|
+
| In-process cache | Free | Limited to pod memory |
|
|
119
|
+
| Redis / Memcached | Ops + infra | Cross-pod shared |
|
|
120
|
+
| CDN / HTTP cache | Free on public GETs | Offloads to edge |
|
|
121
|
+
|
|
122
|
+
Cache-aside is the default pattern. See `backend/references/caching.md`.
|
|
123
|
+
|
|
124
|
+
Guardrail: a cache layer is another thing to monitor. Measure its hit rate — a low hit rate means you're paying the complexity without the benefit.
|
|
125
|
+
|
|
126
|
+
## Write amplification
|
|
127
|
+
|
|
128
|
+
Every index is a write tax. A row written to a table with 5 indexes is actually 6 writes. Review indexes periodically:
|
|
129
|
+
|
|
130
|
+
```sql
|
|
131
|
+
-- Postgres: unused indexes in recent weeks
|
|
132
|
+
SELECT relname, indexrelname, idx_scan
|
|
133
|
+
FROM pg_stat_user_indexes
|
|
134
|
+
WHERE idx_scan = 0 AND indexrelname NOT LIKE 'pg_%'
|
|
135
|
+
ORDER BY pg_relation_size(indexrelid) DESC;
|
|
136
|
+
```
|
|
137
|
+
|
|
138
|
+
Drop what isn't used.
|
|
139
|
+
|
|
140
|
+
## Hot rows
|
|
141
|
+
|
|
142
|
+
A single row updated by many clients (a counter, a shared config, a leaderboard entry) becomes a contention point.
|
|
143
|
+
|
|
144
|
+
Mitigations:
|
|
145
|
+
- **Increment via atomic SQL**: `UPDATE stats SET count = count + 1 WHERE id = ?`.
|
|
146
|
+
- **Shard the counter**: N sub-counters, summed when read.
|
|
147
|
+
- **Move to a cache**: increment in Redis, flush periodically.
|
|
148
|
+
- **Denormalize**: pre-aggregate at write time into a stream that downstream consumers read.
|
|
149
|
+
|
|
150
|
+
## Archival
|
|
151
|
+
|
|
152
|
+
Active data and historical data usually have different access patterns. Move old data out of the hot path.
|
|
153
|
+
|
|
154
|
+
- **Partition by time** and drop old partitions.
|
|
155
|
+
- **Archive to a data warehouse** (Snowflake, BigQuery, Redshift, ClickHouse) for analytics.
|
|
156
|
+
- **Cold storage** (S3 + Athena / Parquet) for years-old data accessed rarely.
|
|
157
|
+
|
|
158
|
+
Keeps the hot DB small, fast, cheap.
|
|
159
|
+
|
|
160
|
+
## Replication topologies
|
|
161
|
+
|
|
162
|
+
- **Single primary, multiple replicas** — read scaling, simple.
|
|
163
|
+
- **Cascading replication** — replica of a replica, for geographic distribution.
|
|
164
|
+
- **Logical replication** (Postgres) — table-level; supports zero-downtime upgrades and data migration between major versions.
|
|
165
|
+
- **Multi-primary** — rare, complex, use managed services (Spanner, CockroachDB) that built for it.
|
|
166
|
+
|
|
167
|
+
Be skeptical of "multi-master" in traditional engines. Conflict resolution is user-facing work.
|
|
168
|
+
|
|
169
|
+
## Backups
|
|
170
|
+
|
|
171
|
+
- **Automated daily backups + point-in-time recovery** (PITR). Not optional.
|
|
172
|
+
- **Test restore** quarterly. A backup you've never restored is a hope, not a backup.
|
|
173
|
+
- **Off-region storage** for disaster recovery.
|
|
174
|
+
- **Encryption at rest**.
|
|
175
|
+
|
|
176
|
+
Many incidents are resolved by restoring a table / row from a backup. Make sure that's possible without a day of pain.
|
|
177
|
+
|
|
178
|
+
## Monitoring — the must-haves
|
|
179
|
+
|
|
180
|
+
- **Connection count** vs. max.
|
|
181
|
+
- **Replication lag** per replica.
|
|
182
|
+
- **Query p95 / p99 latency**.
|
|
183
|
+
- **Slow query log** (threshold: 500 ms+).
|
|
184
|
+
- **Lock waits and deadlocks**.
|
|
185
|
+
- **Cache hit ratio** (Postgres `pg_statio_*`, MySQL InnoDB buffer pool).
|
|
186
|
+
- **Disk space** and growth rate.
|
|
187
|
+
- **Autovacuum / maintenance activity** (Postgres).
|
|
188
|
+
|
|
189
|
+
Dashboards + alerts on all of the above.
|
|
190
|
+
|
|
191
|
+
## Anti-patterns
|
|
192
|
+
|
|
193
|
+
| Anti-pattern | Fix |
|
|
194
|
+
|---|---|
|
|
195
|
+
| Sharding before trying indexes / replicas / partitioning | Exhaust simpler options first |
|
|
196
|
+
| One connection per request without a pool | Use a pool; use a pooler for serverless |
|
|
197
|
+
| Read from a replica for "write-then-read" flows | Read from primary for stickiness window |
|
|
198
|
+
| No automated backups | Set up PITR yesterday |
|
|
199
|
+
| 50 indexes on a write-heavy table | Prune; serve reports from a replica / warehouse |
|
|
200
|
+
| Ignoring replication lag in the app | Observe and degrade |
|
|
201
|
+
| Cross-shard queries in the app | Architect around the shard key |
|
|
202
|
+
| Running the DB on the same host as the app | One process's CPU spike kills the DB |
|
|
203
|
+
| Unrestricted `pg_dump` over the wire during business hours | Replicate to a snapshot host and dump there |
|