@earth-app/collegedb 1.0.1 → 1.0.3
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +1614 -35
- package/dist/durable.d.ts +12 -2
- package/dist/durable.d.ts.map +1 -1
- package/dist/errors.d.ts +52 -0
- package/dist/errors.d.ts.map +1 -0
- package/dist/index.d.ts +4 -3
- package/dist/index.d.ts.map +1 -1
- package/dist/index.js +7 -7
- package/dist/index.js.map +9 -8
- package/dist/kvmap.d.ts +80 -26
- package/dist/kvmap.d.ts.map +1 -1
- package/dist/migrations.d.ts +14 -0
- package/dist/migrations.d.ts.map +1 -1
- package/dist/router.d.ts +38 -3
- package/dist/router.d.ts.map +1 -1
- package/dist/types.d.ts +72 -4
- package/dist/types.d.ts.map +1 -1
- package/package.json +2 -2
package/README.md
CHANGED
|
@@ -1,27 +1,60 @@
|
|
|
1
1
|
# CollegeDB
|
|
2
2
|
|
|
3
|
-
> Cloudflare D1 Sharding Router
|
|
3
|
+
> Cloudflare D1 Horizontal Sharding Router
|
|
4
4
|
|
|
5
5
|
[](https://www.typescriptlang.org/)
|
|
6
|
+
[](https://github.com/earth-app/CollegeDB/issues)
|
|
6
7
|
[](https://workers.cloudflare.com/)
|
|
7
|
-
[](LICENSE)
|
|
9
|
+

|
|
8
10
|
|
|
9
|
-
A TypeScript library for horizontal scaling of SQLite-style databases on Cloudflare using D1 and KV. CollegeDB
|
|
11
|
+
A TypeScript library for **true horizontal scaling** of SQLite-style databases on Cloudflare using D1 and KV. CollegeDB distributes your data across multiple D1 databases, with each table's records split by primary key across different database instances.
|
|
10
12
|
|
|
11
|
-
|
|
13
|
+
CollegeDB implements **data distribution** where a single logical table is physically stored across multiple D1 databases:
|
|
14
|
+
|
|
15
|
+
```txt
|
|
16
|
+
env.db-east (Shard 1)
|
|
17
|
+
┌────────────────────────────────────────────┐
|
|
18
|
+
│ table users: [user-1, user-3, user-5, ...] │
|
|
19
|
+
│ table posts: [post-2, post-7, post-9, ...] │
|
|
20
|
+
└────────────────────────────────────────────┘
|
|
21
|
+
|
|
22
|
+
env.db-west (Shard 2)
|
|
23
|
+
┌────────────────────────────────────────────┐
|
|
24
|
+
│ table users: [user-2, user-4, user-6, ...] │
|
|
25
|
+
│ table posts: [post-1, post-3, post-8, ...] │
|
|
26
|
+
└────────────────────────────────────────────┘
|
|
27
|
+
|
|
28
|
+
env.db-central (Shard 3)
|
|
29
|
+
┌────────────────────────────────────────────┐
|
|
30
|
+
│ table users: [user-7, user-8, user-9, ...] │
|
|
31
|
+
│ table posts: [post-4, post-5, post-6, ...] │
|
|
32
|
+
└────────────────────────────────────────────┘
|
|
33
|
+
```
|
|
34
|
+
|
|
35
|
+
This allows you to:
|
|
36
|
+
|
|
37
|
+
- **Break through D1's single database limits** by spreading data across many databases
|
|
38
|
+
- **Improve query performance** by reducing data per database instance
|
|
39
|
+
- **Scale geographically** by placing shards in different regions
|
|
40
|
+
- **Increase write throughput** by parallelizing across multiple database instances
|
|
41
|
+
|
|
42
|
+
## 📈 Overview
|
|
12
43
|
|
|
13
44
|
CollegeDB provides a sharding layer on top of Cloudflare D1 databases, enabling you to:
|
|
14
45
|
|
|
15
|
-
- **Scale horizontally** across multiple D1 instances
|
|
16
|
-
- **Route queries automatically** based on primary
|
|
17
|
-
- **Maintain consistency** with KV-based mapping
|
|
46
|
+
- **Scale horizontally** by distributing table data across multiple D1 instances
|
|
47
|
+
- **Route queries automatically** based on primary key mappings
|
|
48
|
+
- **Maintain consistency** with KV-based shard mapping
|
|
49
|
+
- **Optimize for geography** with location-aware shard allocation
|
|
18
50
|
- **Monitor and rebalance** shard distribution
|
|
19
51
|
- **Handle migrations** between shards seamlessly
|
|
20
52
|
|
|
21
53
|
## 📦 Features
|
|
22
54
|
|
|
23
55
|
- **🔀 Automatic Query Routing**: Primary key → shard mapping using Cloudflare KV
|
|
24
|
-
- **🎯 Multiple Allocation Strategies**: Round-robin, random,
|
|
56
|
+
- **🎯 Multiple Allocation Strategies**: Round-robin, random, hash-based, and location-aware distribution
|
|
57
|
+
- **🔄 Mixed Strategy Support**: Different strategies for reads vs writes (e.g., location for writes, hash for reads)
|
|
25
58
|
- **📊 Shard Coordination**: Durable Objects for allocation and statistics
|
|
26
59
|
- **🛠 Migration Support**: Move data between shards with zero downtime
|
|
27
60
|
- **🔄 Automatic Drop-in Replacement**: Zero-config integration with existing databases
|
|
@@ -51,7 +84,7 @@ collegedb(
|
|
|
51
84
|
'db-east': env['db-east'], // Can be existing DB with data
|
|
52
85
|
'db-west': env['db-west'] // Can be existing DB with data
|
|
53
86
|
},
|
|
54
|
-
strategy: 'hash'
|
|
87
|
+
strategy: 'hash'
|
|
55
88
|
},
|
|
56
89
|
async () => {
|
|
57
90
|
// Create schema on new shards only (existing shards auto-detected)
|
|
@@ -68,6 +101,184 @@ collegedb(
|
|
|
68
101
|
);
|
|
69
102
|
```
|
|
70
103
|
|
|
104
|
+
### Geographic Distribution Example
|
|
105
|
+
|
|
106
|
+
```typescript
|
|
107
|
+
import { collegedb, first, run } from 'collegedb';
|
|
108
|
+
|
|
109
|
+
// Optimize for North American users with geographic sharding
|
|
110
|
+
collegedb(
|
|
111
|
+
{
|
|
112
|
+
kv: env.KV,
|
|
113
|
+
strategy: 'location',
|
|
114
|
+
targetRegion: 'wnam', // Western North America
|
|
115
|
+
shardLocations: {
|
|
116
|
+
'db-west': { region: 'wnam', priority: 2 }, // SF - Preferred for target region
|
|
117
|
+
'db-east': { region: 'enam', priority: 1 }, // NYC - Secondary
|
|
118
|
+
'db-europe': { region: 'weur', priority: 0.5 } // London - Fallback
|
|
119
|
+
},
|
|
120
|
+
shards: {
|
|
121
|
+
'db-west': env.DB_WEST,
|
|
122
|
+
'db-east': env.DB_EAST,
|
|
123
|
+
'db-europe': env.DB_EUROPE
|
|
124
|
+
}
|
|
125
|
+
},
|
|
126
|
+
async () => {
|
|
127
|
+
// New users will be allocated to db-west (closest to target region)
|
|
128
|
+
await run('user-west-123', 'INSERT INTO users (id, name, location) VALUES (?, ?, ?)', [
|
|
129
|
+
'user-west-123',
|
|
130
|
+
'West Coast User',
|
|
131
|
+
'California'
|
|
132
|
+
]);
|
|
133
|
+
|
|
134
|
+
// Queries are routed to the correct geographic shard
|
|
135
|
+
const user = await first<User>('user-west-123', 'SELECT * FROM users WHERE id = ?', ['user-west-123']);
|
|
136
|
+
console.log(`User found in optimal shard: ${user?.name}`);
|
|
137
|
+
}
|
|
138
|
+
);
|
|
139
|
+
```
|
|
140
|
+
|
|
141
|
+
### Mixed Strategy Example
|
|
142
|
+
|
|
143
|
+
```typescript
|
|
144
|
+
import { collegedb, first, run, type MixedShardingStrategy } from 'collegedb';
|
|
145
|
+
|
|
146
|
+
// Use location strategy for writes (optimal data placement) and hash for reads (optimal performance)
|
|
147
|
+
const mixedStrategy: MixedShardingStrategy = {
|
|
148
|
+
write: 'location', // New data goes to geographically optimal shards
|
|
149
|
+
read: 'hash' // Reads use consistent hashing for best performance
|
|
150
|
+
};
|
|
151
|
+
|
|
152
|
+
collegedb(
|
|
153
|
+
{
|
|
154
|
+
kv: env.KV,
|
|
155
|
+
strategy: mixedStrategy,
|
|
156
|
+
targetRegion: 'wnam', // Western North America for writes
|
|
157
|
+
shardLocations: {
|
|
158
|
+
'db-west': { region: 'wnam', priority: 2 },
|
|
159
|
+
'db-east': { region: 'enam', priority: 1 },
|
|
160
|
+
'db-central': { region: 'enam', priority: 1 }
|
|
161
|
+
},
|
|
162
|
+
shards: {
|
|
163
|
+
'db-west': env.DB_WEST,
|
|
164
|
+
'db-east': env.DB_EAST,
|
|
165
|
+
'db-central': env.DB_CENTRAL
|
|
166
|
+
}
|
|
167
|
+
},
|
|
168
|
+
async () => {
|
|
169
|
+
// Write operations use location strategy - new users placed optimally
|
|
170
|
+
await run('user-california-456', 'INSERT INTO users (id, name, location) VALUES (?, ?, ?)', [
|
|
171
|
+
'user-california-456',
|
|
172
|
+
'California User',
|
|
173
|
+
'Los Angeles'
|
|
174
|
+
]);
|
|
175
|
+
|
|
176
|
+
// Read operations use hash strategy - consistent and fast routing
|
|
177
|
+
const user = await first<User>('user-california-456', 'SELECT * FROM users WHERE id = ?', ['user-california-456']);
|
|
178
|
+
|
|
179
|
+
// Different operations can route to different shards based on strategy
|
|
180
|
+
// This optimizes both data placement (writes) and query performance (reads)
|
|
181
|
+
console.log(`User: ${user?.name}, Location: ${user?.location}`);
|
|
182
|
+
}
|
|
183
|
+
);
|
|
184
|
+
```
|
|
185
|
+
|
|
186
|
+
This approach provides:
|
|
187
|
+
|
|
188
|
+
- **Optimal data placement**: New records are written to geographically optimal shards using `location` strategy
|
|
189
|
+
- **Optimal read performance**: Queries use `hash` strategy for consistent, high-performance routing
|
|
190
|
+
- **Flexibility**: Each operation type can use the most appropriate routing strategy
|
|
191
|
+
|
|
192
|
+
## Multi-Key Shard Mappings
|
|
193
|
+
|
|
194
|
+
CollegeDB supports **multiple lookup keys** for the same record, allowing you to query by username, email, ID, or any unique identifier. Keys are automatically hashed with SHA-256 for security and privacy.
|
|
195
|
+
|
|
196
|
+
```typescript
|
|
197
|
+
import { collegedb, first, run, KVShardMapper } from 'collegedb';
|
|
198
|
+
|
|
199
|
+
collegedb(
|
|
200
|
+
{
|
|
201
|
+
kv: env.KV,
|
|
202
|
+
shards: { 'db-east': env.DB_EAST, 'db-west': env.DB_WEST },
|
|
203
|
+
hashShardMappings: true, // Default: enabled for security
|
|
204
|
+
strategy: 'hash'
|
|
205
|
+
},
|
|
206
|
+
async () => {
|
|
207
|
+
// Create a user with multiple lookup keys
|
|
208
|
+
const mapper = new KVShardMapper(env.KV, { hashShardMappings: true });
|
|
209
|
+
|
|
210
|
+
await mapper.setShardMapping('user-123', 'db-east', ['username:john_doe', 'email:john@example.com', 'id:123']);
|
|
211
|
+
|
|
212
|
+
// Now you can query by ANY of these keys
|
|
213
|
+
const byId = await first('user-123', 'SELECT * FROM users WHERE id = ?', ['user-123']);
|
|
214
|
+
const byUsername = await first('username:john_doe', 'SELECT * FROM users WHERE username = ?', ['john_doe']);
|
|
215
|
+
const byEmail = await first('email:john@example.com', 'SELECT * FROM users WHERE email = ?', ['john@example.com']);
|
|
216
|
+
|
|
217
|
+
// All queries route to the same shard (db-east)
|
|
218
|
+
console.log('All queries find the same user:', byId?.name);
|
|
219
|
+
}
|
|
220
|
+
);
|
|
221
|
+
```
|
|
222
|
+
|
|
223
|
+
### Adding Lookup Keys to Existing Mappings
|
|
224
|
+
|
|
225
|
+
s
|
|
226
|
+
|
|
227
|
+
```typescript
|
|
228
|
+
const mapper = new KVShardMapper(env.KV);
|
|
229
|
+
|
|
230
|
+
// User initially created with just ID
|
|
231
|
+
await mapper.setShardMapping('user-456', 'db-west');
|
|
232
|
+
|
|
233
|
+
// Later, add additional lookup methods
|
|
234
|
+
await mapper.addLookupKeys('user-456', ['email:jane@example.com', 'username:jane']);
|
|
235
|
+
|
|
236
|
+
// Now works with any key
|
|
237
|
+
const user = await first('email:jane@example.com', 'SELECT * FROM users WHERE email = ?', ['jane@example.com']);
|
|
238
|
+
```
|
|
239
|
+
|
|
240
|
+
### Security and Privacy
|
|
241
|
+
|
|
242
|
+
**SHA-256 Hashing (Enabled by Default)**: Sensitive data like emails are hashed before being stored as KV keys, protecting user privacy:
|
|
243
|
+
|
|
244
|
+
```typescript
|
|
245
|
+
// With hashShardMappings: true (default)
|
|
246
|
+
// KV stores: "shard:a1b2c3d4..." instead of "shard:email:user@example.com"
|
|
247
|
+
|
|
248
|
+
const config = {
|
|
249
|
+
kv: env.KV,
|
|
250
|
+
shards: {
|
|
251
|
+
/* ... */
|
|
252
|
+
},
|
|
253
|
+
hashShardMappings: true, // Hashes keys with SHA-256
|
|
254
|
+
strategy: 'hash'
|
|
255
|
+
};
|
|
256
|
+
```
|
|
257
|
+
|
|
258
|
+
**⚠️ Performance Trade-off**: When hashing is enabled, operations like `getKeysForShard()` cannot return original key names, only hashed versions. For full key recovery, disable hashing:
|
|
259
|
+
|
|
260
|
+
```typescript
|
|
261
|
+
const config = {
|
|
262
|
+
hashShardMappings: false // Disables hashing - keys stored in plain text
|
|
263
|
+
};
|
|
264
|
+
```
|
|
265
|
+
|
|
266
|
+
### Multi-Key Management
|
|
267
|
+
|
|
268
|
+
```typescript
|
|
269
|
+
const mapper = new KVShardMapper(env.KV);
|
|
270
|
+
|
|
271
|
+
// Get all lookup keys for a mapping
|
|
272
|
+
const allKeys = await mapper.getAllLookupKeys('email:user@example.com');
|
|
273
|
+
console.log(allKeys); // ['user-123', 'username:john', 'email:user@example.com']
|
|
274
|
+
|
|
275
|
+
// Update shard assignment (updates all keys)
|
|
276
|
+
await mapper.updateShardMapping('username:john', 'db-central');
|
|
277
|
+
|
|
278
|
+
// Delete mapping (removes all associated keys)
|
|
279
|
+
await mapper.deleteShardMapping('user-123');
|
|
280
|
+
```
|
|
281
|
+
|
|
71
282
|
## Drop-in Replacement for Existing Databases
|
|
72
283
|
|
|
73
284
|
CollegeDB supports **seamless, automatic integration** with existing D1 databases that already contain data. Simply add your existing databases as shards in the configuration. CollegeDB will automatically detect existing data and create the necessary shard mappings **without requiring any manual migration steps**.
|
|
@@ -320,18 +531,22 @@ for (const [table, pkColumn] of Object.entries(customIntegration)) {
|
|
|
320
531
|
|
|
321
532
|
## 📚 API Reference
|
|
322
533
|
|
|
323
|
-
| Function
|
|
324
|
-
|
|
|
325
|
-
| `collegedb(config, callback)`
|
|
326
|
-
| `initialize(config)`
|
|
327
|
-
| `createSchema(d1)`
|
|
328
|
-
| `prepare(key, sql)`
|
|
329
|
-
| `run(key, sql, bindings)`
|
|
330
|
-
| `first(key, sql, bindings)`
|
|
331
|
-
| `all(key, sql, bindings)`
|
|
332
|
-
| `
|
|
333
|
-
| `
|
|
334
|
-
| `
|
|
534
|
+
| Function | Description | Parameters |
|
|
535
|
+
| ---------------------------------- | ---------------------------------------------------- | ------------------------ |
|
|
536
|
+
| `collegedb(config, callback)` | Initialize CollegeDB, then run a callback | `CollegeDBConfig, ()=>T` |
|
|
537
|
+
| `initialize(config)` | Initialize CollegeDB with configuration | `CollegeDBConfig` |
|
|
538
|
+
| `createSchema(d1)` | Create database schema on a D1 instance | `D1Database` |
|
|
539
|
+
| `prepare(key, sql)` | Prepare a SQL statement for execution | `string, string` |
|
|
540
|
+
| `run(key, sql, bindings)` | Execute a SQL query with primary key routing | `string, string, any[]` |
|
|
541
|
+
| `first(key, sql, bindings)` | Execute a SQL query and return first result | `string, string, any[]` |
|
|
542
|
+
| `all(key, sql, bindings)` | Execute a SQL query and return all results | `string, string, any[]` |
|
|
543
|
+
| `runShard(shard, sql, bindings)` | Execute a SQL query directly on a specific shard | `string, string, any[]` |
|
|
544
|
+
| `allShard(shard, sql, bindings)` | Execute query on specific shard, return all results | `string, string, any[]` |
|
|
545
|
+
| `firstShard(shard, sql, bindings)` | Execute query on specific shard, return first result | `string, string, any[]` |
|
|
546
|
+
| `reassignShard(key, newShard)` | Move primary key to different shard | `string, string` |
|
|
547
|
+
| `listKnownShards()` | Get list of available shards | `void` |
|
|
548
|
+
| `getShardStats()` | Get statistics for all shards | `void` |
|
|
549
|
+
| `flush()` | Clear all shard mappings (development only) | `void` |
|
|
335
550
|
|
|
336
551
|
### Drop-in Replacement Functions
|
|
337
552
|
|
|
@@ -346,6 +561,184 @@ for (const [table, pkColumn] of Object.entries(customIntegration)) {
|
|
|
346
561
|
| `listTables(d1)` | Get list of tables in database | `D1Database` |
|
|
347
562
|
| `clearMigrationCache()` | Clear automatic migration cache | `void` |
|
|
348
563
|
|
|
564
|
+
### Error Handling
|
|
565
|
+
|
|
566
|
+
| Class | Description | Usage |
|
|
567
|
+
| ---------------- | ------------------------------------------- | ------------------------------------- |
|
|
568
|
+
| `CollegeDBError` | Custom error class for CollegeDB operations | `throw new CollegeDBError(msg, code)` |
|
|
569
|
+
|
|
570
|
+
The `CollegeDBError` class extends the native `Error` class and includes an optional error code for better error categorization:
|
|
571
|
+
|
|
572
|
+
```typescript
|
|
573
|
+
try {
|
|
574
|
+
await run('invalid-key', 'SELECT * FROM users WHERE id = ?', ['invalid-key']);
|
|
575
|
+
} catch (error) {
|
|
576
|
+
if (error instanceof CollegeDBError) {
|
|
577
|
+
console.error(`CollegeDB Error (${error.code}): ${error.message}`);
|
|
578
|
+
}
|
|
579
|
+
}
|
|
580
|
+
```
|
|
581
|
+
|
|
582
|
+
### ShardCoordinator (Durable Object) API
|
|
583
|
+
|
|
584
|
+
The `ShardCoordinator` is an optional Durable Object that provides centralized shard allocation and statistics management. All endpoints return JSON responses.
|
|
585
|
+
|
|
586
|
+
#### HTTP API Endpoints
|
|
587
|
+
|
|
588
|
+
| Endpoint | Method | Description | Request Body | Response |
|
|
589
|
+
| ----------- | ------ | ---------------------------------- | ------------------------------------------------ | -------------------------------------- |
|
|
590
|
+
| `/shards` | GET | List all registered shards | None | `["db-east", "db-west"]` |
|
|
591
|
+
| `/shards` | POST | Register a new shard | `{"shard": "db-new"}` | `{"success": true}` |
|
|
592
|
+
| `/shards` | DELETE | Unregister a shard | `{"shard": "db-old"}` | `{"success": true}` |
|
|
593
|
+
| `/stats` | GET | Get shard statistics | None | `[{"binding":"db-east","count":1542}]` |
|
|
594
|
+
| `/stats` | POST | Update shard statistics | `{"shard": "db-east", "count": 1600}` | `{"success": true}` |
|
|
595
|
+
| `/allocate` | POST | Allocate shard for primary key | `{"primaryKey": "user-123"}` | `{"shard": "db-west"}` |
|
|
596
|
+
| `/allocate` | POST | Allocate with specific strategy | `{"primaryKey": "user-123", "strategy": "hash"}` | `{"shard": "db-west"}` |
|
|
597
|
+
| `/flush` | POST | Clear all state (development only) | None | `{"success": true}` |
|
|
598
|
+
| `/health` | GET | Health check | None | `"OK"` |
|
|
599
|
+
|
|
600
|
+
#### Programmatic Methods
|
|
601
|
+
|
|
602
|
+
| Method | Description | Parameters | Returns |
|
|
603
|
+
| ----------------------------- | ----------------------------- | -------------------- | ------------------- |
|
|
604
|
+
| `new ShardCoordinator(state)` | Create coordinator instance | `DurableObjectState` | `ShardCoordinator` |
|
|
605
|
+
| `fetch(request)` | Handle HTTP requests | `Request` | `Promise<Response>` |
|
|
606
|
+
| `incrementShardCount(shard)` | Increment key count for shard | `string` | `Promise<void>` |
|
|
607
|
+
| `decrementShardCount(shard)` | Decrement key count for shard | `string` | `Promise<void>` |
|
|
608
|
+
|
|
609
|
+
#### Usage Example
|
|
610
|
+
|
|
611
|
+
```typescript
|
|
612
|
+
import { ShardCoordinator } from 'collegedb';
|
|
613
|
+
|
|
614
|
+
// Export for Cloudflare Workers runtime
|
|
615
|
+
export { ShardCoordinator };
|
|
616
|
+
|
|
617
|
+
// Use in your worker
|
|
618
|
+
export default {
|
|
619
|
+
async fetch(request: Request, env: Env) {
|
|
620
|
+
const coordinatorId = env.ShardCoordinator.idFromName('default');
|
|
621
|
+
const coordinator = env.ShardCoordinator.get(coordinatorId);
|
|
622
|
+
|
|
623
|
+
// Allocate shard for user
|
|
624
|
+
const response = await coordinator.fetch('http://coordinator/allocate', {
|
|
625
|
+
method: 'POST',
|
|
626
|
+
headers: { 'Content-Type': 'application/json' },
|
|
627
|
+
body: JSON.stringify({ primaryKey: 'user-123', strategy: 'hash' })
|
|
628
|
+
});
|
|
629
|
+
|
|
630
|
+
const { shard } = await response.json();
|
|
631
|
+
// Use allocated shard for database operations...
|
|
632
|
+
}
|
|
633
|
+
};
|
|
634
|
+
```
|
|
635
|
+
|
|
636
|
+
### Configuration Types
|
|
637
|
+
|
|
638
|
+
#### CollegeDBConfig
|
|
639
|
+
|
|
640
|
+
The main configuration interface supports both single strategies and mixed strategies:
|
|
641
|
+
|
|
642
|
+
```typescript
|
|
643
|
+
interface CollegeDBConfig {
|
|
644
|
+
kv: KVNamespace;
|
|
645
|
+
coordinator?: DurableObjectNamespace;
|
|
646
|
+
shards: Record<string, D1Database>;
|
|
647
|
+
strategy?: ShardingStrategy | MixedShardingStrategy;
|
|
648
|
+
targetRegion?: D1Region;
|
|
649
|
+
shardLocations?: Record<string, ShardLocation>;
|
|
650
|
+
disableAutoMigration?: boolean; // Default: false
|
|
651
|
+
hashShardMappings?: boolean; // Default: true
|
|
652
|
+
}
|
|
653
|
+
```
|
|
654
|
+
|
|
655
|
+
When `hashShardMappings` is enabled (default), original keys cannot be recovered during shard operations like `getKeysForShard()`. This is intentional for privacy but means you'll get fewer results from such operations. For full key recovery, set `hashShardMappings: false`, but be aware this may expose sensitive data in KV keys.
|
|
656
|
+
|
|
657
|
+
#### Strategy Types
|
|
658
|
+
|
|
659
|
+
```typescript
|
|
660
|
+
// Single strategy for all operations
|
|
661
|
+
type ShardingStrategy = 'round-robin' | 'random' | 'hash' | 'location';
|
|
662
|
+
|
|
663
|
+
// Mixed strategy for different operation types
|
|
664
|
+
interface MixedShardingStrategy {
|
|
665
|
+
read: ShardingStrategy; // Strategy for SELECT operations
|
|
666
|
+
write: ShardingStrategy; // Strategy for INSERT/UPDATE/DELETE operations
|
|
667
|
+
}
|
|
668
|
+
|
|
669
|
+
// Operation types for internal routing
|
|
670
|
+
type OperationType = 'read' | 'write';
|
|
671
|
+
```
|
|
672
|
+
|
|
673
|
+
#### Example Configurations
|
|
674
|
+
|
|
675
|
+
```typescript
|
|
676
|
+
// Single strategy configuration (traditional)
|
|
677
|
+
const singleStrategyConfig: CollegeDBConfig = {
|
|
678
|
+
kv: env.KV,
|
|
679
|
+
strategy: 'hash', // All operations use hash strategy
|
|
680
|
+
shards: {
|
|
681
|
+
/* ... */
|
|
682
|
+
}
|
|
683
|
+
};
|
|
684
|
+
|
|
685
|
+
// Mixed strategy configuration (new feature)
|
|
686
|
+
const mixedStrategyConfig: CollegeDBConfig = {
|
|
687
|
+
kv: env.KV,
|
|
688
|
+
strategy: {
|
|
689
|
+
read: 'hash', // Fast, consistent reads
|
|
690
|
+
write: 'location' // Optimal data placement
|
|
691
|
+
},
|
|
692
|
+
targetRegion: 'wnam',
|
|
693
|
+
shardLocations: {
|
|
694
|
+
/* ... */
|
|
695
|
+
},
|
|
696
|
+
shards: {
|
|
697
|
+
/* ... */
|
|
698
|
+
}
|
|
699
|
+
};
|
|
700
|
+
```
|
|
701
|
+
|
|
702
|
+
### Types
|
|
703
|
+
|
|
704
|
+
CollegeDB exports TypeScript types for better development experience and type safety:
|
|
705
|
+
|
|
706
|
+
| Type | Description | Example |
|
|
707
|
+
| ----------------------- | ------------------------------ | --------------------------------------------------- |
|
|
708
|
+
| `CollegeDBConfig` | Main configuration object | `{ kv, shards, strategy }` |
|
|
709
|
+
| `ShardingStrategy` | Single strategy options | `'hash' \| 'location' \| 'round-robin' \| 'random'` |
|
|
710
|
+
| `MixedShardingStrategy` | Mixed strategy configuration | `{ read: 'hash', write: 'location' }` |
|
|
711
|
+
| `OperationType` | Database operation types | `'read' \| 'write'` |
|
|
712
|
+
| `D1Region` | Cloudflare D1 regions | `'wnam' \| 'enam' \| 'weur' \| ...` |
|
|
713
|
+
| `ShardLocation` | Geographic shard configuration | `{ region: 'wnam', priority: 2 }` |
|
|
714
|
+
| `ShardStats` | Shard usage statistics | `{ binding: 'db-east', count: 1542 }` |
|
|
715
|
+
|
|
716
|
+
#### Mixed Strategy Configuration
|
|
717
|
+
|
|
718
|
+
```typescript
|
|
719
|
+
import type { MixedShardingStrategy, CollegeDBConfig } from 'collegedb';
|
|
720
|
+
|
|
721
|
+
// Type-safe mixed strategy configuration
|
|
722
|
+
const mixedStrategy: MixedShardingStrategy = {
|
|
723
|
+
read: 'hash', // Fast, deterministic reads
|
|
724
|
+
write: 'location' // Geographically optimized writes
|
|
725
|
+
};
|
|
726
|
+
|
|
727
|
+
const config: CollegeDBConfig = {
|
|
728
|
+
kv: env.KV,
|
|
729
|
+
strategy: mixedStrategy, // Type-checked
|
|
730
|
+
targetRegion: 'wnam',
|
|
731
|
+
shardLocations: {
|
|
732
|
+
'db-west': { region: 'wnam', priority: 2 },
|
|
733
|
+
'db-east': { region: 'enam', priority: 1 }
|
|
734
|
+
},
|
|
735
|
+
shards: {
|
|
736
|
+
'db-west': env.DB_WEST,
|
|
737
|
+
'db-east': env.DB_EAST
|
|
738
|
+
}
|
|
739
|
+
};
|
|
740
|
+
```
|
|
741
|
+
|
|
349
742
|
## 🏗 Architecture
|
|
350
743
|
|
|
351
744
|
```txt
|
|
@@ -369,16 +762,53 @@ for (const [table, pkColumn] of Object.entries(customIntegration)) {
|
|
|
369
762
|
|
|
370
763
|
### Data Flow
|
|
371
764
|
|
|
765
|
+
#### Without ShardCoordinator (Hash/Random/Location strategies)
|
|
766
|
+
|
|
767
|
+
1. **Query Received**: Application sends query with primary key
|
|
768
|
+
2. **Shard Resolution**: CollegeDB checks KV for existing mapping or calculates shard using strategy
|
|
769
|
+
3. **Direct Allocation**: For new keys, shard selected using hash/random/location algorithm
|
|
770
|
+
4. **Query Execution**: SQL executed on appropriate D1 database
|
|
771
|
+
5. **Response**: Results returned to application
|
|
772
|
+
|
|
773
|
+
#### With ShardCoordinator (Round-Robin strategy)
|
|
774
|
+
|
|
372
775
|
1. **Query Received**: Application sends query with primary key
|
|
373
|
-
2. **Shard Resolution**: CollegeDB checks KV for existing mapping
|
|
374
|
-
3. **
|
|
375
|
-
4. **
|
|
776
|
+
2. **Shard Resolution**: CollegeDB checks KV for existing mapping
|
|
777
|
+
3. **Coordinator Allocation**: For new keys, coordinator allocates shard using round-robin
|
|
778
|
+
4. **State Update**: Coordinator updates round-robin index and shard statistics
|
|
779
|
+
5. **Query Execution**: SQL executed on appropriate D1 database
|
|
780
|
+
6. **Response**: Results returned to application
|
|
781
|
+
|
|
782
|
+
#### ShardCoordinator Internal Flow
|
|
783
|
+
|
|
784
|
+
```txt
|
|
785
|
+
┌─────────────────────────────────────────────────────────────┐
|
|
786
|
+
│ ShardCoordinator (Durable Object) │
|
|
787
|
+
├─────────────────────────────────────────────────────────────┤
|
|
788
|
+
│ ┌─────────────────┐ ┌─────────────────────────────────┐ │
|
|
789
|
+
│ │ HTTP API │ │ Persistent Storage │ │
|
|
790
|
+
│ │ - /allocate │ │ - knownShards: string[] │ │
|
|
791
|
+
│ │ - /shards │ │ - shardStats: ShardStats{} │ │
|
|
792
|
+
│ │ - /stats │ │ - strategy: ShardingStrategy │ │
|
|
793
|
+
│ │ - /health │ │ - roundRobinIndex: number │ │
|
|
794
|
+
│ └─────────────────┘ └─────────────────────────────────┘ │
|
|
795
|
+
│ │ │
|
|
796
|
+
│ ┌─────────────────────────────────────────────────────┐ │
|
|
797
|
+
│ │ Allocation Algorithms │ │
|
|
798
|
+
│ │ - Round-Robin: state.roundRobinIndex │ │
|
|
799
|
+
│ │ - Hash: consistent hash(primaryKey) │ │
|
|
800
|
+
│ │ - Random: Math.random() * shards.length │ │
|
|
801
|
+
│ │ - Location: region proximity + priority │ │
|
|
802
|
+
│ └─────────────────────────────────────────────────────┘ │
|
|
803
|
+
└─────────────────────────────────────────────────────────────┘
|
|
804
|
+
```
|
|
376
805
|
|
|
377
806
|
### Shard Allocation Strategies
|
|
378
807
|
|
|
379
808
|
- **Hash**: Consistent hashing for deterministic shard selection
|
|
380
809
|
- **Round-Robin**: Evenly distribute new keys across shards
|
|
381
810
|
- **Random**: Random shard selection for load balancing
|
|
811
|
+
- **Location**: Geographic proximity-based allocation for optimal latency
|
|
382
812
|
|
|
383
813
|
## 🌐 Cloudflare Setup
|
|
384
814
|
|
|
@@ -420,6 +850,114 @@ name = "ShardCoordinator"
|
|
|
420
850
|
class_name = "ShardCoordinator"
|
|
421
851
|
```
|
|
422
852
|
|
|
853
|
+
#### Complete wrangler.toml with ShardCoordinator
|
|
854
|
+
|
|
855
|
+
```toml
|
|
856
|
+
name = "collegedb-app"
|
|
857
|
+
main = "src/index.ts"
|
|
858
|
+
compatibility_date = "2024-08-10"
|
|
859
|
+
|
|
860
|
+
# D1 Database bindings
|
|
861
|
+
[[d1_databases]]
|
|
862
|
+
binding = "db-east"
|
|
863
|
+
database_name = "collegedb-east"
|
|
864
|
+
database_id = "your-east-database-id"
|
|
865
|
+
|
|
866
|
+
[[d1_databases]]
|
|
867
|
+
binding = "db-west"
|
|
868
|
+
database_name = "collegedb-west"
|
|
869
|
+
database_id = "your-west-database-id"
|
|
870
|
+
|
|
871
|
+
[[d1_databases]]
|
|
872
|
+
binding = "db-central"
|
|
873
|
+
database_name = "collegedb-central"
|
|
874
|
+
database_id = "your-central-database-id"
|
|
875
|
+
|
|
876
|
+
# KV namespace for shard mappings
|
|
877
|
+
[[kv_namespaces]]
|
|
878
|
+
binding = "KV"
|
|
879
|
+
id = "your-kv-namespace-id"
|
|
880
|
+
preview_id = "your-kv-preview-id" # For local development
|
|
881
|
+
|
|
882
|
+
# Durable Object for shard coordination
|
|
883
|
+
[[durable_objects.bindings]]
|
|
884
|
+
name = "ShardCoordinator"
|
|
885
|
+
class_name = "ShardCoordinator"
|
|
886
|
+
|
|
887
|
+
# Environment-specific configurations
|
|
888
|
+
[env.production]
|
|
889
|
+
[[env.production.d1_databases]]
|
|
890
|
+
binding = "db-east"
|
|
891
|
+
database_name = "collegedb-prod-east"
|
|
892
|
+
database_id = "your-prod-east-id"
|
|
893
|
+
|
|
894
|
+
[[env.production.d1_databases]]
|
|
895
|
+
binding = "db-west"
|
|
896
|
+
database_name = "collegedb-prod-west"
|
|
897
|
+
database_id = "your-prod-west-id"
|
|
898
|
+
|
|
899
|
+
[[env.production.kv_namespaces]]
|
|
900
|
+
binding = "KV"
|
|
901
|
+
id = "your-prod-kv-namespace-id"
|
|
902
|
+
|
|
903
|
+
[[env.production.durable_objects.bindings]]
|
|
904
|
+
name = "ShardCoordinator"
|
|
905
|
+
class_name = "ShardCoordinator"
|
|
906
|
+
```
|
|
907
|
+
|
|
908
|
+
### 3.1. Worker Script Setup (Required for ShardCoordinator)
|
|
909
|
+
|
|
910
|
+
Create your main worker file with ShardCoordinator export:
|
|
911
|
+
|
|
912
|
+
```typescript
|
|
913
|
+
// src/index.ts
|
|
914
|
+
import { collegedb, ShardCoordinator, first, run } from 'collegedb';
|
|
915
|
+
|
|
916
|
+
// IMPORTANT: Export ShardCoordinator for Cloudflare Workers runtime
|
|
917
|
+
export { ShardCoordinator };
|
|
918
|
+
|
|
919
|
+
interface Env {
|
|
920
|
+
KV: KVNamespace;
|
|
921
|
+
ShardCoordinator: DurableObjectNamespace;
|
|
922
|
+
'db-east': D1Database;
|
|
923
|
+
'db-west': D1Database;
|
|
924
|
+
'db-central': D1Database;
|
|
925
|
+
}
|
|
926
|
+
|
|
927
|
+
export default {
|
|
928
|
+
async fetch(request: Request, env: Env): Promise<Response> {
|
|
929
|
+
return await collegedb(
|
|
930
|
+
{
|
|
931
|
+
kv: env.KV,
|
|
932
|
+
coordinator: env.ShardCoordinator, // Optional: only needed for round-robin
|
|
933
|
+
strategy: 'hash', // or 'round-robin', 'random', 'location'
|
|
934
|
+
shards: {
|
|
935
|
+
'db-east': env['db-east'],
|
|
936
|
+
'db-west': env['db-west'],
|
|
937
|
+
'db-central': env['db-central']
|
|
938
|
+
}
|
|
939
|
+
},
|
|
940
|
+
async () => {
|
|
941
|
+
// Your application logic here
|
|
942
|
+
const url = new URL(request.url);
|
|
943
|
+
|
|
944
|
+
if (url.pathname === '/user') {
|
|
945
|
+
const userId = url.searchParams.get('id');
|
|
946
|
+
if (!userId) {
|
|
947
|
+
return new Response('Missing user ID', { status: 400 });
|
|
948
|
+
}
|
|
949
|
+
|
|
950
|
+
const user = await first(userId, 'SELECT * FROM users WHERE id = ?', [userId]);
|
|
951
|
+
return Response.json(user);
|
|
952
|
+
}
|
|
953
|
+
|
|
954
|
+
return new Response('CollegeDB API', { status: 200 });
|
|
955
|
+
}
|
|
956
|
+
);
|
|
957
|
+
}
|
|
958
|
+
};
|
|
959
|
+
```
|
|
960
|
+
|
|
423
961
|
### 4. Deploy
|
|
424
962
|
|
|
425
963
|
```bash
|
|
@@ -434,6 +972,8 @@ wrangler deploy --env production
|
|
|
434
972
|
|
|
435
973
|
### Shard Statistics
|
|
436
974
|
|
|
975
|
+
#### Using CollegeDB Functions
|
|
976
|
+
|
|
437
977
|
```typescript
|
|
438
978
|
import { getShardStats, listKnownShards } from 'collegedb';
|
|
439
979
|
|
|
@@ -450,6 +990,96 @@ const shards = await listKnownShards();
|
|
|
450
990
|
console.log(shards); // ['db-east', 'db-west']
|
|
451
991
|
```
|
|
452
992
|
|
|
993
|
+
#### Using ShardCoordinator Directly
|
|
994
|
+
|
|
995
|
+
```typescript
|
|
996
|
+
// Get coordinator instance
|
|
997
|
+
const coordinatorId = env.ShardCoordinator.idFromName('default');
|
|
998
|
+
const coordinator = env.ShardCoordinator.get(coordinatorId);
|
|
999
|
+
|
|
1000
|
+
// Get real-time shard statistics
|
|
1001
|
+
const statsResponse = await coordinator.fetch('http://coordinator/stats');
|
|
1002
|
+
const detailedStats = await statsResponse.json();
|
|
1003
|
+
console.log(detailedStats);
|
|
1004
|
+
/* Returns:
|
|
1005
|
+
[
|
|
1006
|
+
{
|
|
1007
|
+
"binding": "db-east",
|
|
1008
|
+
"count": 1542,
|
|
1009
|
+
"lastUpdated": 1672531200000
|
|
1010
|
+
},
|
|
1011
|
+
{
|
|
1012
|
+
"binding": "db-west",
|
|
1013
|
+
"count": 1458,
|
|
1014
|
+
"lastUpdated": 1672531205000
|
|
1015
|
+
}
|
|
1016
|
+
]
|
|
1017
|
+
*/
|
|
1018
|
+
|
|
1019
|
+
// List registered shards
|
|
1020
|
+
const shardsResponse = await coordinator.fetch('http://coordinator/shards');
|
|
1021
|
+
const allShards = await shardsResponse.json();
|
|
1022
|
+
console.log(allShards); // ['db-east', 'db-west', 'db-central']
|
|
1023
|
+
```
|
|
1024
|
+
|
|
1025
|
+
#### Advanced Monitoring Dashboard
|
|
1026
|
+
|
|
1027
|
+
```typescript
|
|
1028
|
+
async function createMonitoringDashboard(env: Env) {
|
|
1029
|
+
const coordinatorId = env.ShardCoordinator.idFromName('default');
|
|
1030
|
+
const coordinator = env.ShardCoordinator.get(coordinatorId);
|
|
1031
|
+
|
|
1032
|
+
// Get comprehensive metrics
|
|
1033
|
+
const [shardsResponse, statsResponse, healthResponse] = await Promise.all([
|
|
1034
|
+
coordinator.fetch('http://coordinator/shards'),
|
|
1035
|
+
coordinator.fetch('http://coordinator/stats'),
|
|
1036
|
+
coordinator.fetch('http://coordinator/health')
|
|
1037
|
+
]);
|
|
1038
|
+
|
|
1039
|
+
const shards = await shardsResponse.json();
|
|
1040
|
+
const stats = await statsResponse.json();
|
|
1041
|
+
const isHealthy = healthResponse.ok;
|
|
1042
|
+
|
|
1043
|
+
// Calculate distribution metrics
|
|
1044
|
+
const totalKeys = stats.reduce((sum: number, shard: any) => sum + shard.count, 0);
|
|
1045
|
+
const avgKeysPerShard = totalKeys / stats.length;
|
|
1046
|
+
const maxKeys = Math.max(...stats.map((s: any) => s.count));
|
|
1047
|
+
const minKeys = Math.min(...stats.map((s: any) => s.count));
|
|
1048
|
+
const distributionRatio = maxKeys / (minKeys || 1);
|
|
1049
|
+
|
|
1050
|
+
// Check for stale statistics (>5 minutes)
|
|
1051
|
+
const now = Date.now();
|
|
1052
|
+
const staleThreshold = 5 * 60 * 1000; // 5 minutes
|
|
1053
|
+
const staleShards = stats.filter((shard: any) => now - shard.lastUpdated > staleThreshold);
|
|
1054
|
+
|
|
1055
|
+
return {
|
|
1056
|
+
healthy: isHealthy,
|
|
1057
|
+
totalShards: shards.length,
|
|
1058
|
+
totalKeys,
|
|
1059
|
+
avgKeysPerShard: Math.round(avgKeysPerShard),
|
|
1060
|
+
distributionRatio: Math.round(distributionRatio * 100) / 100,
|
|
1061
|
+
isBalanced: distributionRatio < 1.5, // Less than 50% difference
|
|
1062
|
+
staleShards: staleShards.length,
|
|
1063
|
+
shardDetails: stats.map((shard: any) => ({
|
|
1064
|
+
...shard,
|
|
1065
|
+
loadPercentage: Math.round((shard.count / totalKeys) * 100),
|
|
1066
|
+
isStale: now - shard.lastUpdated > staleThreshold
|
|
1067
|
+
}))
|
|
1068
|
+
};
|
|
1069
|
+
}
|
|
1070
|
+
|
|
1071
|
+
// Usage in monitoring endpoint
|
|
1072
|
+
export default {
|
|
1073
|
+
async fetch(request: Request, env: Env) {
|
|
1074
|
+
if (new URL(request.url).pathname === '/monitor') {
|
|
1075
|
+
const dashboard = await createMonitoringDashboard(env);
|
|
1076
|
+
return Response.json(dashboard);
|
|
1077
|
+
}
|
|
1078
|
+
// ... rest of your app
|
|
1079
|
+
}
|
|
1080
|
+
};
|
|
1081
|
+
```
|
|
1082
|
+
|
|
453
1083
|
### Shard Rebalancing
|
|
454
1084
|
|
|
455
1085
|
```typescript
|
|
@@ -467,24 +1097,568 @@ Monitor your CollegeDB deployment by tracking:
|
|
|
467
1097
|
- **Query latency per shard**
|
|
468
1098
|
- **Error rates and failed queries**
|
|
469
1099
|
- **KV operation metrics**
|
|
1100
|
+
- **ShardCoordinator health and availability**
|
|
470
1101
|
|
|
471
|
-
|
|
472
|
-
|
|
473
|
-
### Custom Allocation Strategy
|
|
1102
|
+
#### Automated Health Checks
|
|
474
1103
|
|
|
475
1104
|
```typescript
|
|
476
|
-
|
|
477
|
-
|
|
478
|
-
|
|
479
|
-
|
|
480
|
-
}
|
|
1105
|
+
async function performHealthChecks(env: Env): Promise<HealthReport> {
|
|
1106
|
+
const results: HealthReport = {
|
|
1107
|
+
overall: 'healthy',
|
|
1108
|
+
timestamp: new Date().toISOString(),
|
|
1109
|
+
checks: {}
|
|
1110
|
+
};
|
|
1111
|
+
|
|
1112
|
+
// 1. Test KV availability
|
|
1113
|
+
try {
|
|
1114
|
+
await env.KV.put('health-check', 'ok', { expirationTtl: 60 });
|
|
1115
|
+
const kvTest = await env.KV.get('health-check');
|
|
1116
|
+
results.checks.kv = kvTest === 'ok' ? 'healthy' : 'degraded';
|
|
1117
|
+
} catch (error) {
|
|
1118
|
+
results.checks.kv = 'unhealthy';
|
|
1119
|
+
results.overall = 'unhealthy';
|
|
1120
|
+
}
|
|
1121
|
+
|
|
1122
|
+
// 2. Test ShardCoordinator availability
|
|
1123
|
+
if (env.ShardCoordinator) {
|
|
1124
|
+
try {
|
|
1125
|
+
const coordinatorId = env.ShardCoordinator.idFromName('default');
|
|
1126
|
+
const coordinator = env.ShardCoordinator.get(coordinatorId);
|
|
1127
|
+
const healthResponse = await coordinator.fetch('http://coordinator/health');
|
|
1128
|
+
results.checks.coordinator = healthResponse.ok ? 'healthy' : 'unhealthy';
|
|
1129
|
+
|
|
1130
|
+
if (!healthResponse.ok) {
|
|
1131
|
+
results.overall = 'degraded';
|
|
1132
|
+
}
|
|
1133
|
+
} catch (error) {
|
|
1134
|
+
results.checks.coordinator = 'unhealthy';
|
|
1135
|
+
results.overall = 'degraded'; // Can fallback to hash allocation
|
|
1136
|
+
}
|
|
1137
|
+
}
|
|
1138
|
+
|
|
1139
|
+
// 3. Test each D1 shard
|
|
1140
|
+
const shardTests = Object.entries(env)
|
|
1141
|
+
.filter(([key]) => key.startsWith('db-'))
|
|
1142
|
+
.map(async ([shardName, db]: [string, any]) => {
|
|
1143
|
+
try {
|
|
1144
|
+
// Simple query to test connectivity
|
|
1145
|
+
await db.prepare('SELECT 1 as test').first();
|
|
1146
|
+
results.checks[shardName] = 'healthy';
|
|
1147
|
+
} catch (error) {
|
|
1148
|
+
results.checks[shardName] = 'unhealthy';
|
|
1149
|
+
results.overall = 'unhealthy';
|
|
1150
|
+
}
|
|
1151
|
+
});
|
|
1152
|
+
|
|
1153
|
+
await Promise.all(shardTests);
|
|
1154
|
+
|
|
1155
|
+
// 4. Check shard distribution balance
|
|
1156
|
+
if (results.checks.coordinator === 'healthy') {
|
|
1157
|
+
try {
|
|
1158
|
+
const coordinatorId = env.ShardCoordinator.idFromName('default');
|
|
1159
|
+
const coordinator = env.ShardCoordinator.get(coordinatorId);
|
|
1160
|
+
const statsResponse = await coordinator.fetch('http://coordinator/stats');
|
|
1161
|
+
const stats = await statsResponse.json();
|
|
1162
|
+
|
|
1163
|
+
const totalKeys = stats.reduce((sum: number, shard: any) => sum + shard.count, 0);
|
|
1164
|
+
if (totalKeys > 0) {
|
|
1165
|
+
const avgKeys = totalKeys / stats.length;
|
|
1166
|
+
const maxKeys = Math.max(...stats.map((s: any) => s.count));
|
|
1167
|
+
const distributionRatio = maxKeys / avgKeys;
|
|
1168
|
+
|
|
1169
|
+
results.checks.distribution = distributionRatio < 2 ? 'healthy' : 'degraded';
|
|
1170
|
+
results.distributionRatio = distributionRatio;
|
|
1171
|
+
|
|
1172
|
+
if (distributionRatio >= 3 && results.overall === 'healthy') {
|
|
1173
|
+
results.overall = 'degraded';
|
|
1174
|
+
}
|
|
1175
|
+
}
|
|
1176
|
+
} catch (error) {
|
|
1177
|
+
results.checks.distribution = 'unknown';
|
|
1178
|
+
}
|
|
1179
|
+
}
|
|
1180
|
+
|
|
1181
|
+
return results;
|
|
1182
|
+
}
|
|
1183
|
+
|
|
1184
|
+
interface HealthReport {
|
|
1185
|
+
overall: 'healthy' | 'degraded' | 'unhealthy';
|
|
1186
|
+
timestamp: string;
|
|
1187
|
+
checks: Record<string, 'healthy' | 'degraded' | 'unhealthy' | 'unknown'>;
|
|
1188
|
+
distributionRatio?: number;
|
|
1189
|
+
}
|
|
1190
|
+
|
|
1191
|
+
// Health endpoint example
|
|
1192
|
+
export default {
|
|
1193
|
+
async fetch(request: Request, env: Env) {
|
|
1194
|
+
if (new URL(request.url).pathname === '/health') {
|
|
1195
|
+
const health = await performHealthChecks(env);
|
|
1196
|
+
const statusCode = health.overall === 'healthy' ? 200 : health.overall === 'degraded' ? 206 : 503;
|
|
1197
|
+
return Response.json(health, { status: statusCode });
|
|
1198
|
+
}
|
|
1199
|
+
// ... rest of your app
|
|
1200
|
+
}
|
|
1201
|
+
};
|
|
481
1202
|
```
|
|
482
1203
|
|
|
483
|
-
|
|
1204
|
+
#### Alerting and Monitoring Integration
|
|
484
1205
|
|
|
485
1206
|
```typescript
|
|
486
|
-
|
|
487
|
-
|
|
1207
|
+
// Integration with external monitoring services
|
|
1208
|
+
async function sendAlert(severity: 'warning' | 'critical', message: string, env: Env) {
|
|
1209
|
+
// Example: Slack webhook
|
|
1210
|
+
if (env.SLACK_WEBHOOK_URL) {
|
|
1211
|
+
await fetch(env.SLACK_WEBHOOK_URL, {
|
|
1212
|
+
method: 'POST',
|
|
1213
|
+
headers: { 'Content-Type': 'application/json' },
|
|
1214
|
+
body: JSON.stringify({
|
|
1215
|
+
text: `🚨 CollegeDB ${severity.toUpperCase()}: ${message}`,
|
|
1216
|
+
username: 'CollegeDB Monitor'
|
|
1217
|
+
})
|
|
1218
|
+
});
|
|
1219
|
+
}
|
|
1220
|
+
|
|
1221
|
+
// Example: Custom webhook
|
|
1222
|
+
if (env.MONITORING_WEBHOOK_URL) {
|
|
1223
|
+
await fetch(env.MONITORING_WEBHOOK_URL, {
|
|
1224
|
+
method: 'POST',
|
|
1225
|
+
headers: { 'Content-Type': 'application/json' },
|
|
1226
|
+
body: JSON.stringify({
|
|
1227
|
+
service: 'collegedb',
|
|
1228
|
+
severity,
|
|
1229
|
+
message,
|
|
1230
|
+
timestamp: new Date().toISOString()
|
|
1231
|
+
})
|
|
1232
|
+
});
|
|
1233
|
+
}
|
|
1234
|
+
}
|
|
1235
|
+
|
|
1236
|
+
// Scheduled monitoring (using Cron Triggers)
|
|
1237
|
+
export default {
|
|
1238
|
+
async scheduled(event: ScheduledEvent, env: Env, ctx: ExecutionContext): Promise<void> {
|
|
1239
|
+
const health = await performHealthChecks(env);
|
|
1240
|
+
|
|
1241
|
+
if (health.overall === 'unhealthy') {
|
|
1242
|
+
await sendAlert('critical', `System unhealthy: ${JSON.stringify(health.checks)}`, env);
|
|
1243
|
+
} else if (health.overall === 'degraded') {
|
|
1244
|
+
await sendAlert('warning', `System degraded: ${JSON.stringify(health.checks)}`, env);
|
|
1245
|
+
}
|
|
1246
|
+
|
|
1247
|
+
// Check for severe shard imbalance
|
|
1248
|
+
if (health.distributionRatio && health.distributionRatio > 5) {
|
|
1249
|
+
await sendAlert('warning', `Severe shard imbalance detected: ${health.distributionRatio}x difference`, env);
|
|
1250
|
+
}
|
|
1251
|
+
}
|
|
1252
|
+
};
|
|
1253
|
+
```
|
|
1254
|
+
|
|
1255
|
+
## ⚙️ Performance Analysis
|
|
1256
|
+
|
|
1257
|
+
### Scaling Performance Comparison
|
|
1258
|
+
|
|
1259
|
+
CollegeDB provides significant performance improvements through horizontal scaling. Here are mathematical estimates comparing single D1 database vs CollegeDB with different shard counts:
|
|
1260
|
+
|
|
1261
|
+
#### Query Performance
|
|
1262
|
+
|
|
1263
|
+
_SELECT, VALUES, TABLE, PRAGMA, ..._
|
|
1264
|
+
|
|
1265
|
+
| Configuration | Query Latency\* | Concurrent Queries | Throughput Gain |
|
|
1266
|
+
| ----------------------- | --------------- | ----------------------- | --------------- |
|
|
1267
|
+
| Single D1 | ~50-80ms | Limited by D1 limits | 1x (baseline) |
|
|
1268
|
+
| CollegeDB (10 shards) | ~55-85ms | 10x parallel capacity | ~8-9x |
|
|
1269
|
+
| CollegeDB (100 shards) | ~60-90ms | 100x parallel capacity | ~75-80x |
|
|
1270
|
+
| CollegeDB (1000 shards) | ~65-95ms | 1000x parallel capacity | ~650-700x |
|
|
1271
|
+
|
|
1272
|
+
\*Includes KV lookup overhead (~5-15ms) and SHA-256 hashing overhead (~1-3ms when `hashShardMappings: true`)
|
|
1273
|
+
|
|
1274
|
+
#### Write Performance
|
|
1275
|
+
|
|
1276
|
+
_INSERT, UPDATE, DELETE, ..._
|
|
1277
|
+
|
|
1278
|
+
| Configuration | Write Latency\* | Concurrent Writes | Throughput Gain |
|
|
1279
|
+
| ----------------------- | --------------- | ------------------ | --------------- |
|
|
1280
|
+
| Single D1 | ~80-120ms | ~50 writes/sec | 1x (baseline) |
|
|
1281
|
+
| CollegeDB (10 shards) | ~90-135ms | ~450 writes/sec | ~9x |
|
|
1282
|
+
| CollegeDB (100 shards) | ~95-145ms | ~4,200 writes/sec | ~84x |
|
|
1283
|
+
| CollegeDB (1000 shards) | ~105-160ms | ~35,000 writes/sec | ~700x |
|
|
1284
|
+
|
|
1285
|
+
\*Includes KV mapping creation/update overhead (~10-25ms) and SHA-256 hashing overhead (~1-3ms when `hashShardMappings: true`)
|
|
1286
|
+
|
|
1287
|
+
### Strategy-Specific Performance
|
|
1288
|
+
|
|
1289
|
+
#### Hash Strategy
|
|
1290
|
+
|
|
1291
|
+
- **Best for**: Consistent performance, even data distribution
|
|
1292
|
+
- **Latency**: Lowest overhead (no coordinator calls, ~1-3ms SHA-256 hashing when enabled)
|
|
1293
|
+
- **Throughput**: Optimal for high-volume scenarios
|
|
1294
|
+
|
|
1295
|
+
| Shards | Avg Latency | Distribution Quality | Coordinator Dependency |
|
|
1296
|
+
| ------ | ----------- | -------------------- | ---------------------- |
|
|
1297
|
+
| 10 | +5ms | Excellent | None |
|
|
1298
|
+
| 100 | +5ms | Excellent | None |
|
|
1299
|
+
| 1000 | +5ms | Excellent | None |
|
|
1300
|
+
|
|
1301
|
+
#### Round-Robin Strategy
|
|
1302
|
+
|
|
1303
|
+
- **Best for**: Guaranteed even distribution
|
|
1304
|
+
- **Latency**: Requires coordinator communication
|
|
1305
|
+
- **Throughput**: Good, limited by coordinator
|
|
1306
|
+
|
|
1307
|
+
| Shards | Avg Latency | Distribution Quality | Coordinator Dependency |
|
|
1308
|
+
| ------ | ----------- | -------------------- | ---------------------- |
|
|
1309
|
+
| 10 | +15ms | Perfect | High |
|
|
1310
|
+
| 100 | +20ms | Perfect | High |
|
|
1311
|
+
| 1000 | +25ms | Perfect | High |
|
|
1312
|
+
|
|
1313
|
+
#### Random Strategy
|
|
1314
|
+
|
|
1315
|
+
- **Best for**: Simple setup, good distribution over time
|
|
1316
|
+
- **Latency**: Low overhead
|
|
1317
|
+
- **Throughput**: Good for medium-scale deployments
|
|
1318
|
+
|
|
1319
|
+
| Shards | Avg Latency | Distribution Quality | Coordinator Dependency |
|
|
1320
|
+
| ------ | ----------- | -------------------- | ---------------------- |
|
|
1321
|
+
| 10 | +3ms | Good | None |
|
|
1322
|
+
| 100 | +3ms | Good | None |
|
|
1323
|
+
| 1000 | +3ms | Fair | None |
|
|
1324
|
+
|
|
1325
|
+
#### Location Strategy
|
|
1326
|
+
|
|
1327
|
+
- **Best for**: Geographic optimization, reduced latency
|
|
1328
|
+
- **Latency**: Optimized by region proximity
|
|
1329
|
+
- **Throughput**: Regional performance benefits
|
|
1330
|
+
|
|
1331
|
+
| Shards | Avg Latency | Geographic Benefit | Coordinator Dependency |
|
|
1332
|
+
| ------ | ----------- | -------------------- | ---------------------- |
|
|
1333
|
+
| 10 | +8ms | Excellent (-20-40ms) | Optional |
|
|
1334
|
+
| 100 | +10ms | Excellent (-20-40ms) | Optional |
|
|
1335
|
+
| 1000 | +12ms | Excellent (-20-40ms) | Optional |
|
|
1336
|
+
|
|
1337
|
+
#### Mixed Strategy
|
|
1338
|
+
|
|
1339
|
+
- **Best for**: Optimizing both read and write performance independently
|
|
1340
|
+
- **Latency**: Best of both strategies combined
|
|
1341
|
+
- **Throughput**: Optimal for workloads with different read/write patterns
|
|
1342
|
+
|
|
1343
|
+
**High-Performance Mix**: `{ read: 'hash', write: 'location' }`
|
|
1344
|
+
|
|
1345
|
+
| Operation | Strategy | Latency Impact | Throughput Benefit | Geographic Benefit |
|
|
1346
|
+
| --------- | -------- | ------------------------ | ------------------ | -------------------- |
|
|
1347
|
+
| Reads | Hash | +5ms | Excellent | None |
|
|
1348
|
+
| Writes | Location | +8ms (-20-40ms regional) | Good | Excellent (-20-40ms) |
|
|
1349
|
+
|
|
1350
|
+
**Balanced Mix**: `{ read: 'location', write: 'hash' }`
|
|
1351
|
+
|
|
1352
|
+
| Operation | Strategy | Latency Impact | Throughput Benefit | Geographic Benefit |
|
|
1353
|
+
| --------- | -------- | ------------------------ | ------------------ | -------------------- |
|
|
1354
|
+
| Reads | Location | +8ms (-20-40ms regional) | Good | Excellent (-20-40ms) |
|
|
1355
|
+
| Writes | Hash | +5ms | Excellent | None |
|
|
1356
|
+
|
|
1357
|
+
**Enterprise Mix**: `{ read: 'hash', write: 'round-robin' }`
|
|
1358
|
+
|
|
1359
|
+
| Operation | Strategy | Latency Impact | Distribution Quality | Coordinator Dependency |
|
|
1360
|
+
| --------- | ----------- | -------------- | -------------------- | ---------------------- |
|
|
1361
|
+
| Reads | Hash | +5ms | Excellent | None |
|
|
1362
|
+
| Writes | Round-Robin | +15-25ms | Perfect | High |
|
|
1363
|
+
|
|
1364
|
+
##### By Shard Count
|
|
1365
|
+
|
|
1366
|
+
**Hash + Location Mix** (`{ read: 'hash', write: 'location' }`)
|
|
1367
|
+
|
|
1368
|
+
| Shards | Read Latency | Write Latency | Combined Benefit | Best Use Case |
|
|
1369
|
+
| ------ | ------------ | ---------------------- | --------------------- | ---------------- |
|
|
1370
|
+
| 10 | +5ms | +8ms (-30ms regional) | ~22ms net improvement | Global apps |
|
|
1371
|
+
| 100 | +5ms | +10ms (-30ms regional) | ~20ms net improvement | Enterprise scale |
|
|
1372
|
+
| 1000 | +5ms | +12ms (-30ms regional) | ~18ms net improvement | Massive scale |
|
|
1373
|
+
|
|
1374
|
+
**Location + Hash Mix** (`{ read: 'location', write: 'hash' }`)
|
|
1375
|
+
|
|
1376
|
+
| Shards | Read Latency | Write Latency | Combined Benefit | Best Use Case |
|
|
1377
|
+
| ------ | ---------------------- | ------------- | --------------------- | --------------------- |
|
|
1378
|
+
| 10 | +8ms (-30ms regional) | +5ms | ~17ms net improvement | Read-heavy regional |
|
|
1379
|
+
| 100 | +10ms (-30ms regional) | +5ms | ~15ms net improvement | Analytics workloads |
|
|
1380
|
+
| 1000 | +12ms (-30ms regional) | +5ms | ~13ms net improvement | Large-scale reporting |
|
|
1381
|
+
|
|
1382
|
+
**Hash + Round-Robin Mix** (`{ read: 'hash', write: 'round-robin' }`)
|
|
1383
|
+
|
|
1384
|
+
| Shards | Read Latency | Write Latency | Distribution Quality | Best Use Case |
|
|
1385
|
+
| ------ | ------------ | ------------- | ------------------------------- | ------------------ |
|
|
1386
|
+
| 10 | +5ms | +15ms | Perfect writes, Excellent reads | Balanced workloads |
|
|
1387
|
+
| 100 | +5ms | +20ms | Perfect writes, Excellent reads | Large databases |
|
|
1388
|
+
| 1000 | +5ms | +25ms | Perfect writes, Excellent reads | Enterprise scale |
|
|
1389
|
+
|
|
1390
|
+
### Mixed Strategy Scenarios & Recommendations
|
|
1391
|
+
|
|
1392
|
+
#### Large Database Scenarios (>10M records)
|
|
1393
|
+
|
|
1394
|
+
**Scenario**: Massive datasets requiring optimal query performance and balanced growth
|
|
1395
|
+
|
|
1396
|
+
```typescript
|
|
1397
|
+
// Recommended: Hash reads + Round-Robin writes
|
|
1398
|
+
{
|
|
1399
|
+
strategy: { read: 'hash', write: 'round-robin' },
|
|
1400
|
+
coordinator: env.ShardCoordinator // Required for round-robin
|
|
1401
|
+
}
|
|
1402
|
+
```
|
|
1403
|
+
|
|
1404
|
+
**Performance Profile**:
|
|
1405
|
+
|
|
1406
|
+
- Read latency: +5ms (fastest possible routing)
|
|
1407
|
+
- Write latency: +15-25ms (coordinator overhead)
|
|
1408
|
+
- Data distribution: Perfect balance over time
|
|
1409
|
+
- **Ideal for**: Analytics platforms, data warehouses, reporting systems
|
|
1410
|
+
|
|
1411
|
+
#### Vast Geographic Spread Scenarios
|
|
1412
|
+
|
|
1413
|
+
**Scenario**: Global applications with users across multiple continents
|
|
1414
|
+
|
|
1415
|
+
```typescript
|
|
1416
|
+
// Recommended: Hash reads + Location writes
|
|
1417
|
+
{
|
|
1418
|
+
strategy: { read: 'hash', write: 'location' },
|
|
1419
|
+
targetRegion: getClosestRegionFromIP(request), // Dynamic region targeting
|
|
1420
|
+
shardLocations: {
|
|
1421
|
+
'db-americas': { region: 'wnam', priority: 2 },
|
|
1422
|
+
'db-europe': { region: 'weur', priority: 2 },
|
|
1423
|
+
'db-asia': { region: 'apac', priority: 2 }
|
|
1424
|
+
}
|
|
1425
|
+
}
|
|
1426
|
+
```
|
|
1427
|
+
|
|
1428
|
+
**Performance Profile**:
|
|
1429
|
+
|
|
1430
|
+
- Read latency: +5ms (consistent global performance)
|
|
1431
|
+
- Write latency: +8ms baseline (-20-40ms regional benefit)
|
|
1432
|
+
- **Net improvement**: 15-35ms for geographically distributed users
|
|
1433
|
+
- **Ideal for**: Social networks, e-commerce, content platforms
|
|
1434
|
+
|
|
1435
|
+
#### High-Volume Write Scenarios
|
|
1436
|
+
|
|
1437
|
+
**Scenario**: Applications with heavy write loads (IoT, logging, real-time data)
|
|
1438
|
+
|
|
1439
|
+
```typescript
|
|
1440
|
+
// Recommended: Location reads + Hash writes
|
|
1441
|
+
{
|
|
1442
|
+
strategy: { read: 'location', write: 'hash' },
|
|
1443
|
+
targetRegion: 'wnam',
|
|
1444
|
+
shardLocations: {
|
|
1445
|
+
'db-west': { region: 'wnam', priority: 3 },
|
|
1446
|
+
'db-central': { region: 'enam', priority: 2 },
|
|
1447
|
+
'db-east': { region: 'enam', priority: 1 }
|
|
1448
|
+
}
|
|
1449
|
+
}
|
|
1450
|
+
```
|
|
1451
|
+
|
|
1452
|
+
**Performance Profile**:
|
|
1453
|
+
|
|
1454
|
+
- Read latency: +8ms baseline (-20-40ms regional benefit)
|
|
1455
|
+
- Write latency: +5ms (fastest write routing)
|
|
1456
|
+
- Write throughput: Maximum possible for hash strategy
|
|
1457
|
+
- **Ideal for**: IoT data collection, real-time analytics, logging systems
|
|
1458
|
+
|
|
1459
|
+
#### Multi-Tenant SaaS Scenarios
|
|
1460
|
+
|
|
1461
|
+
**Scenario**: SaaS applications with predictable performance requirements
|
|
1462
|
+
|
|
1463
|
+
```typescript
|
|
1464
|
+
// Recommended: Hash reads + Hash writes (consistent performance)
|
|
1465
|
+
{
|
|
1466
|
+
strategy: { read: 'hash', write: 'hash' }
|
|
1467
|
+
// No coordinator needed, predictable routing for both operations
|
|
1468
|
+
}
|
|
1469
|
+
```
|
|
1470
|
+
|
|
1471
|
+
**Performance Profile**:
|
|
1472
|
+
|
|
1473
|
+
- Read latency: +5ms (most predictable)
|
|
1474
|
+
- Write latency: +5ms (most predictable)
|
|
1475
|
+
- Tenant isolation: Natural sharding by tenant ID
|
|
1476
|
+
- **Ideal for**: B2B SaaS, multi-tenant platforms, predictable workloads
|
|
1477
|
+
|
|
1478
|
+
#### Read-Heavy Analytics Scenarios
|
|
1479
|
+
|
|
1480
|
+
**Scenario**: Analytics and reporting workloads with occasional writes
|
|
1481
|
+
|
|
1482
|
+
```typescript
|
|
1483
|
+
// Recommended: Random reads + Location writes
|
|
1484
|
+
{
|
|
1485
|
+
strategy: { read: 'random', write: 'location' },
|
|
1486
|
+
targetRegion: 'wnam',
|
|
1487
|
+
shardLocations: { /* geographic configuration */ }
|
|
1488
|
+
}
|
|
1489
|
+
```
|
|
1490
|
+
|
|
1491
|
+
**Performance Profile**:
|
|
1492
|
+
|
|
1493
|
+
- Read latency: +3ms (lowest overhead, good load balancing)
|
|
1494
|
+
- Write latency: +8ms baseline (-20-40ms regional benefit)
|
|
1495
|
+
- Read load distribution: Excellent across all shards
|
|
1496
|
+
- **Ideal for**: Business intelligence, data analysis, reporting dashboards
|
|
1497
|
+
|
|
1498
|
+
### Mixed Strategy Performance Comparison
|
|
1499
|
+
|
|
1500
|
+
#### By Database Size
|
|
1501
|
+
|
|
1502
|
+
| Database Size | Best Mixed Strategy | Read Performance | Write Performance | Overall Benefit |
|
|
1503
|
+
| ----------------------------- | ------------------------------------------ | --------------------- | -------------------- | --------------------- |
|
|
1504
|
+
| **Small (1K-100K records)** | `{read: 'hash', write: 'hash'}` | Excellent | Excellent | Consistent, simple |
|
|
1505
|
+
| **Medium (100K-1M records)** | `{read: 'hash', write: 'location'}` | Excellent | Good + Regional | 15-35ms improvement |
|
|
1506
|
+
| **Large (1M-10M records)** | `{read: 'hash', write: 'round-robin'}` | Excellent | Perfect distribution | Optimal scaling |
|
|
1507
|
+
| **Very Large (10M+ records)** | `{read: 'location', write: 'round-robin'}` | Regional optimization | Perfect distribution | Best for global scale |
|
|
1508
|
+
|
|
1509
|
+
#### By Geographic Distribution
|
|
1510
|
+
|
|
1511
|
+
| Geographic Spread | Best Mixed Strategy | Latency Improvement | Use Case |
|
|
1512
|
+
| ----------------- | --------------------------------------- | ----------------------- | ------------------------------- |
|
|
1513
|
+
| **Single Region** | `{read: 'hash', write: 'hash'}` | +5ms both operations | Simple, fast |
|
|
1514
|
+
| **Multi-Region** | `{read: 'hash', write: 'location'}` | 15-35ms net improvement | Global apps |
|
|
1515
|
+
| **Global** | `{read: 'location', write: 'location'}` | 20-40ms both operations | Maximum geographic optimization |
|
|
1516
|
+
|
|
1517
|
+
#### By Workload Pattern
|
|
1518
|
+
|
|
1519
|
+
| Workload Type | Read/Write Ratio | Best Mixed Strategy | Primary Benefit |
|
|
1520
|
+
| --------------- | ---------------- | ------------------------------------------ | ------------------------------- |
|
|
1521
|
+
| **Read-Heavy** | 90% reads | `{read: 'random', write: 'location'}` | Load-balanced queries |
|
|
1522
|
+
| **Write-Heavy** | 70% writes | `{read: 'location', write: 'hash'}` | Fast write processing |
|
|
1523
|
+
| **Balanced** | 50/50 | `{read: 'hash', write: 'hash'}` | Consistent performance |
|
|
1524
|
+
| **Analytics** | 95% reads | `{read: 'location', write: 'round-robin'}` | Regional + perfect distribution |
|
|
1525
|
+
|
|
1526
|
+
### SHA-256 Hashing Performance Impact
|
|
1527
|
+
|
|
1528
|
+
CollegeDB uses SHA-256 hashing by default (`hashShardMappings: true`) to protect sensitive data in KV keys. This adds a small but measurable performance overhead:
|
|
1529
|
+
|
|
1530
|
+
#### Hashing Performance Characteristics
|
|
1531
|
+
|
|
1532
|
+
| Operation Type | SHA-256 Overhead | Total Latency Impact | Security Benefit |
|
|
1533
|
+
| ------------------ | ---------------- | -------------------- | ---------------------------- |
|
|
1534
|
+
| **Query (Read)** | ~1-2ms | 2-4% increase | Keys hashed in KV storage |
|
|
1535
|
+
| **Insert (Write)** | ~2-3ms | 2-3% increase | Multi-key mappings protected |
|
|
1536
|
+
| **Update Mapping** | ~1-3ms | 1-2% increase | Existing keys remain secure |
|
|
1537
|
+
|
|
1538
|
+
#### Performance by Key Length
|
|
1539
|
+
|
|
1540
|
+
| Key Type | Example | Hash Time | Recommendation |
|
|
1541
|
+
| ------------------------ | ------------------------------ | ------------ | ----------------------- |
|
|
1542
|
+
| **Short keys** | `user-123` | ~0.5-1ms | Minimal impact |
|
|
1543
|
+
| **Medium keys** | `email:user@example.com` | ~1-2ms | Good balance |
|
|
1544
|
+
| **Long keys** | `session:very-long-token-here` | ~2-3ms | Consider key shortening |
|
|
1545
|
+
| **Multi-key operations** | 3+ lookup keys | ~3-5ms total | Benefits outweigh cost |
|
|
1546
|
+
|
|
1547
|
+
#### Hashing vs No-Hashing Trade-offs
|
|
1548
|
+
|
|
1549
|
+
```typescript
|
|
1550
|
+
// With hashing (default - recommended for production)
|
|
1551
|
+
const secureConfig = {
|
|
1552
|
+
hashShardMappings: true // Default
|
|
1553
|
+
// + Privacy: Sensitive data not visible in KV
|
|
1554
|
+
// + Security: Keys cannot be enumerated
|
|
1555
|
+
// - Performance: +1-3ms per operation
|
|
1556
|
+
// - Debugging: Original keys not recoverable
|
|
1557
|
+
};
|
|
1558
|
+
|
|
1559
|
+
// Without hashing (development/debugging only)
|
|
1560
|
+
const developmentConfig = {
|
|
1561
|
+
hashShardMappings: false
|
|
1562
|
+
// + Performance: No hashing overhead
|
|
1563
|
+
// + Debugging: Original keys visible in KV
|
|
1564
|
+
// - Privacy: Sensitive data exposed in KV keys
|
|
1565
|
+
// - Security: Keys can be enumerated
|
|
1566
|
+
};
|
|
1567
|
+
```
|
|
1568
|
+
|
|
1569
|
+
#### Optimization Recommendations
|
|
1570
|
+
|
|
1571
|
+
1. **Keep keys reasonably short** - Hash time scales with key length
|
|
1572
|
+
2. **Use hashing in production** - Security benefits outweigh minimal performance cost
|
|
1573
|
+
3. **Disable hashing for development** - When debugging shard distribution
|
|
1574
|
+
4. **Monitor hash performance** - Track operation latencies in high-volume scenarios
|
|
1575
|
+
|
|
1576
|
+
**Bottom Line**: SHA-256 hashing adds 1-3ms overhead but provides essential privacy and security benefits. The performance impact is minimal compared to network latency and D1 query time.
|
|
1577
|
+
|
|
1578
|
+
### Real-World Scaling Benefits
|
|
1579
|
+
|
|
1580
|
+
#### Database Size Limits
|
|
1581
|
+
|
|
1582
|
+
- **Single D1**: Limited to D1's database size constraints
|
|
1583
|
+
- **CollegeDB**: Virtually unlimited through horizontal distribution
|
|
1584
|
+
- **Data per shard**: Scales inversely with shard count (1000 shards = 1/1000 data per shard)
|
|
1585
|
+
|
|
1586
|
+
#### Geographic Distribution
|
|
1587
|
+
|
|
1588
|
+
```typescript
|
|
1589
|
+
// Location-aware sharding reduces latency by 20-40ms
|
|
1590
|
+
initialize({
|
|
1591
|
+
kv: env.KV,
|
|
1592
|
+
strategy: 'location',
|
|
1593
|
+
targetRegion: 'wnam', // Western North America
|
|
1594
|
+
shardLocations: {
|
|
1595
|
+
'db-west': { region: 'wnam', priority: 2 }, // Preferred
|
|
1596
|
+
'db-east': { region: 'enam', priority: 1 }, // Secondary
|
|
1597
|
+
'db-europe': { region: 'weur', priority: 0.5 } // Fallback
|
|
1598
|
+
},
|
|
1599
|
+
shards: { ... }
|
|
1600
|
+
});
|
|
1601
|
+
```
|
|
1602
|
+
|
|
1603
|
+
#### Fault Tolerance
|
|
1604
|
+
|
|
1605
|
+
- **Single D1**: Single point of failure
|
|
1606
|
+
- **CollegeDB**: Distributed failure isolation (failure of 1 shard affects only 1/N of data)
|
|
1607
|
+
|
|
1608
|
+
### Cost-Performance Analysis
|
|
1609
|
+
|
|
1610
|
+
| Shards | D1 Costs\*\* | Performance Gain | Cost per Performance Unit |
|
|
1611
|
+
| ------ | ------------ | ---------------- | ------------------------- |
|
|
1612
|
+
| 1 | 1x | 1x | 1.00x |
|
|
1613
|
+
| 10 | 1.2x | ~9x | 0.13x |
|
|
1614
|
+
| 100 | 2.5x | ~80x | 0.03x |
|
|
1615
|
+
| 1000 | 15x | ~700x | 0.02x |
|
|
1616
|
+
|
|
1617
|
+
\*\*Estimated based on D1's pricing model including KV overhead
|
|
1618
|
+
|
|
1619
|
+
### When to Use CollegeDB
|
|
1620
|
+
|
|
1621
|
+
✅ **Recommended for:**
|
|
1622
|
+
|
|
1623
|
+
- High-traffic applications (>1000 QPS)
|
|
1624
|
+
- Large datasets approaching D1 limits
|
|
1625
|
+
- Geographic distribution requirements
|
|
1626
|
+
- Applications needing >50 concurrent operations
|
|
1627
|
+
- Systems requiring fault tolerance
|
|
1628
|
+
|
|
1629
|
+
✅ **Mixed Strategy specifically recommended for:**
|
|
1630
|
+
|
|
1631
|
+
- **Global applications** needing both fast queries and optimal data placement
|
|
1632
|
+
- **Large-scale databases** requiring different optimization for reads vs writes
|
|
1633
|
+
- **Multi-workload systems** with distinct read/write patterns
|
|
1634
|
+
- **Geographic optimization** while maintaining query performance
|
|
1635
|
+
- **Enterprise applications** needing fine-tuned performance control
|
|
1636
|
+
|
|
1637
|
+
❌ **Not recommended for:**
|
|
1638
|
+
|
|
1639
|
+
- Small applications (<100 QPS)
|
|
1640
|
+
- Simple CRUD operations with minimal scale
|
|
1641
|
+
- Applications without geographic spread
|
|
1642
|
+
- Cost-sensitive deployments at small scale
|
|
1643
|
+
- **Single-strategy applications** where reads and writes have identical performance needs
|
|
1644
|
+
|
|
1645
|
+
## �🔧 Advanced Configuration
|
|
1646
|
+
|
|
1647
|
+
### Custom Allocation Strategy
|
|
1648
|
+
|
|
1649
|
+
```typescript
|
|
1650
|
+
initialize({
|
|
1651
|
+
kv: env.KV,
|
|
1652
|
+
shards: { 'db-east': env['db-east'], 'db-west': env['db-west'] },
|
|
1653
|
+
strategy: 'hash' // Shard selection based on primary key hash
|
|
1654
|
+
});
|
|
1655
|
+
```
|
|
1656
|
+
|
|
1657
|
+
### Environment-Specific Setup
|
|
1658
|
+
|
|
1659
|
+
```typescript
|
|
1660
|
+
const config = {
|
|
1661
|
+
kv: env.KV,
|
|
488
1662
|
shards: env.NODE_ENV === 'production' ? { 'db-prod-1': env['db-prod-1'], 'db-prod-2': env['db-prod-2'] } : { 'db-dev': env['db-dev'] },
|
|
489
1663
|
strategy: 'round-robin' // Shard selection is evenly distributed, regardless of size
|
|
490
1664
|
};
|
|
@@ -492,6 +1666,411 @@ const config = {
|
|
|
492
1666
|
initialize(config);
|
|
493
1667
|
```
|
|
494
1668
|
|
|
1669
|
+
### Durable Objects Integration (ShardCoordinator)
|
|
1670
|
+
|
|
1671
|
+
CollegeDB includes an optional **ShardCoordinator** Durable Object that provides centralized shard allocation and statistics management. This is particularly useful for round-robin allocation strategies and monitoring shard utilization across your application.
|
|
1672
|
+
|
|
1673
|
+
#### Durable Object Setup
|
|
1674
|
+
|
|
1675
|
+
First, configure the Durable Object in your `wrangler.toml`:
|
|
1676
|
+
|
|
1677
|
+
```toml
|
|
1678
|
+
[[durable_objects.bindings]]
|
|
1679
|
+
name = "ShardCoordinator"
|
|
1680
|
+
class_name = "ShardCoordinator"
|
|
1681
|
+
|
|
1682
|
+
# Export the ShardCoordinator class
|
|
1683
|
+
[durable_objects.bindings.script_name]
|
|
1684
|
+
# If using modules format
|
|
1685
|
+
[[durable_objects.bindings]]
|
|
1686
|
+
name = "ShardCoordinator"
|
|
1687
|
+
class_name = "ShardCoordinator"
|
|
1688
|
+
```
|
|
1689
|
+
|
|
1690
|
+
#### Basic Usage with ShardCoordinator
|
|
1691
|
+
|
|
1692
|
+
```typescript
|
|
1693
|
+
import { collegedb, ShardCoordinator } from 'collegedb';
|
|
1694
|
+
|
|
1695
|
+
// Export the Durable Object class for Cloudflare Workers
|
|
1696
|
+
export { ShardCoordinator };
|
|
1697
|
+
|
|
1698
|
+
export default {
|
|
1699
|
+
async fetch(request: Request, env: Env): Promise<Response> {
|
|
1700
|
+
// Initialize CollegeDB with coordinator support
|
|
1701
|
+
await collegedb(
|
|
1702
|
+
{
|
|
1703
|
+
kv: env.KV,
|
|
1704
|
+
coordinator: env.ShardCoordinator, // Add coordinator binding
|
|
1705
|
+
strategy: 'round-robin',
|
|
1706
|
+
shards: {
|
|
1707
|
+
'db-east': env.DB_EAST,
|
|
1708
|
+
'db-west': env.DB_WEST,
|
|
1709
|
+
'db-central': env.DB_CENTRAL
|
|
1710
|
+
}
|
|
1711
|
+
},
|
|
1712
|
+
async () => {
|
|
1713
|
+
// Your application logic here
|
|
1714
|
+
const user = await first('user-123', 'SELECT * FROM users WHERE id = ?', ['user-123']);
|
|
1715
|
+
return Response.json(user);
|
|
1716
|
+
}
|
|
1717
|
+
);
|
|
1718
|
+
}
|
|
1719
|
+
};
|
|
1720
|
+
```
|
|
1721
|
+
|
|
1722
|
+
#### ShardCoordinator HTTP API
|
|
1723
|
+
|
|
1724
|
+
The ShardCoordinator exposes a comprehensive HTTP API for managing shards and allocation:
|
|
1725
|
+
|
|
1726
|
+
##### Shard Management
|
|
1727
|
+
|
|
1728
|
+
```typescript
|
|
1729
|
+
// Get coordinator instance
|
|
1730
|
+
const coordinatorId = env.ShardCoordinator.idFromName('default');
|
|
1731
|
+
const coordinator = env.ShardCoordinator.get(coordinatorId);
|
|
1732
|
+
|
|
1733
|
+
// List all registered shards
|
|
1734
|
+
const shardsResponse = await coordinator.fetch('http://coordinator/shards');
|
|
1735
|
+
const shards = await shardsResponse.json();
|
|
1736
|
+
// Returns: ["db-east", "db-west", "db-central"]
|
|
1737
|
+
|
|
1738
|
+
// Register a new shard
|
|
1739
|
+
await coordinator.fetch('http://coordinator/shards', {
|
|
1740
|
+
method: 'POST',
|
|
1741
|
+
headers: { 'Content-Type': 'application/json' },
|
|
1742
|
+
body: JSON.stringify({ shard: 'db-new-region' })
|
|
1743
|
+
});
|
|
1744
|
+
|
|
1745
|
+
// Remove a shard
|
|
1746
|
+
await coordinator.fetch('http://coordinator/shards', {
|
|
1747
|
+
method: 'DELETE',
|
|
1748
|
+
headers: { 'Content-Type': 'application/json' },
|
|
1749
|
+
body: JSON.stringify({ shard: 'db-old-region' })
|
|
1750
|
+
});
|
|
1751
|
+
```
|
|
1752
|
+
|
|
1753
|
+
##### Statistics and Monitoring
|
|
1754
|
+
|
|
1755
|
+
```typescript
|
|
1756
|
+
// Get shard statistics
|
|
1757
|
+
const statsResponse = await coordinator.fetch('http://coordinator/stats');
|
|
1758
|
+
const stats = await statsResponse.json();
|
|
1759
|
+
/* Returns:
|
|
1760
|
+
[
|
|
1761
|
+
{
|
|
1762
|
+
"binding": "db-east",
|
|
1763
|
+
"count": 1542,
|
|
1764
|
+
"lastUpdated": 1672531200000
|
|
1765
|
+
},
|
|
1766
|
+
{
|
|
1767
|
+
"binding": "db-west",
|
|
1768
|
+
"count": 1458,
|
|
1769
|
+
"lastUpdated": 1672531205000
|
|
1770
|
+
}
|
|
1771
|
+
]
|
|
1772
|
+
*/
|
|
1773
|
+
|
|
1774
|
+
// Update shard statistics manually
|
|
1775
|
+
await coordinator.fetch('http://coordinator/stats', {
|
|
1776
|
+
method: 'POST',
|
|
1777
|
+
headers: { 'Content-Type': 'application/json' },
|
|
1778
|
+
body: JSON.stringify({
|
|
1779
|
+
shard: 'db-east',
|
|
1780
|
+
count: 1600
|
|
1781
|
+
})
|
|
1782
|
+
});
|
|
1783
|
+
```
|
|
1784
|
+
|
|
1785
|
+
##### Shard Allocation
|
|
1786
|
+
|
|
1787
|
+
```typescript
|
|
1788
|
+
// Allocate a shard for a primary key
|
|
1789
|
+
const allocationResponse = await coordinator.fetch('http://coordinator/allocate', {
|
|
1790
|
+
method: 'POST',
|
|
1791
|
+
headers: { 'Content-Type': 'application/json' },
|
|
1792
|
+
body: JSON.stringify({
|
|
1793
|
+
primaryKey: 'user-123',
|
|
1794
|
+
strategy: 'round-robin' // Optional, uses coordinator default if not specified
|
|
1795
|
+
})
|
|
1796
|
+
});
|
|
1797
|
+
|
|
1798
|
+
const { shard } = await allocationResponse.json();
|
|
1799
|
+
// Returns: { "shard": "db-west" }
|
|
1800
|
+
|
|
1801
|
+
// Hash-based allocation (consistent for same key)
|
|
1802
|
+
const hashAllocation = await coordinator.fetch('http://coordinator/allocate', {
|
|
1803
|
+
method: 'POST',
|
|
1804
|
+
headers: { 'Content-Type': 'application/json' },
|
|
1805
|
+
body: JSON.stringify({
|
|
1806
|
+
primaryKey: 'user-456',
|
|
1807
|
+
strategy: 'hash'
|
|
1808
|
+
})
|
|
1809
|
+
});
|
|
1810
|
+
```
|
|
1811
|
+
|
|
1812
|
+
##### Health Check and Development
|
|
1813
|
+
|
|
1814
|
+
```typescript
|
|
1815
|
+
// Health check endpoint
|
|
1816
|
+
const healthResponse = await coordinator.fetch('http://coordinator/health');
|
|
1817
|
+
// Returns: "OK" with 200 status
|
|
1818
|
+
|
|
1819
|
+
// Clear all coordinator state (DEVELOPMENT ONLY!)
|
|
1820
|
+
await coordinator.fetch('http://coordinator/flush', {
|
|
1821
|
+
method: 'POST'
|
|
1822
|
+
});
|
|
1823
|
+
// WARNING: This removes all shard registrations and statistics
|
|
1824
|
+
```
|
|
1825
|
+
|
|
1826
|
+
#### Programmatic Shard Management
|
|
1827
|
+
|
|
1828
|
+
The ShardCoordinator also provides methods for direct programmatic access:
|
|
1829
|
+
|
|
1830
|
+
```typescript
|
|
1831
|
+
// Get coordinator instance
|
|
1832
|
+
const coordinatorId = env.ShardCoordinator.idFromName('default');
|
|
1833
|
+
const coordinator = env.ShardCoordinator.get(coordinatorId);
|
|
1834
|
+
|
|
1835
|
+
// Increment shard count (when adding new keys)
|
|
1836
|
+
await coordinator.incrementShardCount('db-east');
|
|
1837
|
+
|
|
1838
|
+
// Decrement shard count (when removing keys)
|
|
1839
|
+
await coordinator.decrementShardCount('db-west');
|
|
1840
|
+
```
|
|
1841
|
+
|
|
1842
|
+
#### Advanced Monitoring Setup
|
|
1843
|
+
|
|
1844
|
+
Set up comprehensive monitoring of your shard distribution:
|
|
1845
|
+
|
|
1846
|
+
```typescript
|
|
1847
|
+
async function monitorShardHealth(env: Env) {
|
|
1848
|
+
const coordinatorId = env.ShardCoordinator.idFromName('default');
|
|
1849
|
+
const coordinator = env.ShardCoordinator.get(coordinatorId);
|
|
1850
|
+
|
|
1851
|
+
// Get current statistics
|
|
1852
|
+
const statsResponse = await coordinator.fetch('http://coordinator/stats');
|
|
1853
|
+
const stats = await statsResponse.json();
|
|
1854
|
+
|
|
1855
|
+
// Calculate distribution balance
|
|
1856
|
+
const totalKeys = stats.reduce((sum: number, shard: any) => sum + shard.count, 0);
|
|
1857
|
+
const avgKeysPerShard = totalKeys / stats.length;
|
|
1858
|
+
|
|
1859
|
+
// Check for imbalanced shards (>20% deviation from average)
|
|
1860
|
+
const imbalancedShards = stats.filter((shard: any) => {
|
|
1861
|
+
const deviation = Math.abs(shard.count - avgKeysPerShard) / avgKeysPerShard;
|
|
1862
|
+
return deviation > 0.2;
|
|
1863
|
+
});
|
|
1864
|
+
|
|
1865
|
+
if (imbalancedShards.length > 0) {
|
|
1866
|
+
console.warn('Shard imbalance detected:', imbalancedShards);
|
|
1867
|
+
// Trigger rebalancing logic or alerts
|
|
1868
|
+
}
|
|
1869
|
+
|
|
1870
|
+
// Check for stale statistics (>1 hour old)
|
|
1871
|
+
const now = Date.now();
|
|
1872
|
+
const staleShards = stats.filter((shard: any) => {
|
|
1873
|
+
return now - shard.lastUpdated > 3600000; // 1 hour in ms
|
|
1874
|
+
});
|
|
1875
|
+
|
|
1876
|
+
if (staleShards.length > 0) {
|
|
1877
|
+
console.warn('Stale shard statistics detected:', staleShards);
|
|
1878
|
+
}
|
|
1879
|
+
|
|
1880
|
+
return {
|
|
1881
|
+
totalKeys,
|
|
1882
|
+
avgKeysPerShard,
|
|
1883
|
+
balance: imbalancedShards.length === 0,
|
|
1884
|
+
freshStats: staleShards.length === 0,
|
|
1885
|
+
shards: stats
|
|
1886
|
+
};
|
|
1887
|
+
}
|
|
1888
|
+
```
|
|
1889
|
+
|
|
1890
|
+
#### Error Handling with ShardCoordinator
|
|
1891
|
+
|
|
1892
|
+
When using the ShardCoordinator, ensure you handle potential errors gracefully:
|
|
1893
|
+
|
|
1894
|
+
```typescript
|
|
1895
|
+
try {
|
|
1896
|
+
const coordinator = env.ShardCoordinator.get(coordinatorId);
|
|
1897
|
+
const response = await coordinator.fetch('http://coordinator/allocate', {
|
|
1898
|
+
method: 'POST',
|
|
1899
|
+
headers: { 'Content-Type': 'application/json' },
|
|
1900
|
+
body: JSON.stringify({ primaryKey: 'user-123' })
|
|
1901
|
+
});
|
|
1902
|
+
|
|
1903
|
+
if (!response.ok) {
|
|
1904
|
+
const error = await response.json();
|
|
1905
|
+
throw new Error(`ShardCoordinator error: ${error.error}`);
|
|
1906
|
+
}
|
|
1907
|
+
|
|
1908
|
+
const { shard } = await response.json();
|
|
1909
|
+
return shard;
|
|
1910
|
+
} catch (error) {
|
|
1911
|
+
console.error('Failed to allocate shard:', error);
|
|
1912
|
+
// Fallback to hash-based allocation without coordinator
|
|
1913
|
+
return hashFunction('user-123', availableShards);
|
|
1914
|
+
}
|
|
1915
|
+
```
|
|
1916
|
+
|
|
1917
|
+
#### Performance Considerations
|
|
1918
|
+
|
|
1919
|
+
- **Coordinator Latency**: Round-robin allocation adds ~10-20ms latency due to coordinator communication
|
|
1920
|
+
- **Scalability**: Single coordinator instance can handle thousands of allocations per second
|
|
1921
|
+
- **Fault Tolerance**: Design fallback allocation strategies when coordinator is unavailable
|
|
1922
|
+
- **Caching**: Consider caching allocation results for frequently accessed keys
|
|
1923
|
+
|
|
1924
|
+
```typescript
|
|
1925
|
+
// Fallback allocation when coordinator is unavailable
|
|
1926
|
+
function fallbackAllocation(primaryKey: string, shards: string[]): string {
|
|
1927
|
+
// Use hash-based allocation as fallback
|
|
1928
|
+
const hash = simpleHash(primaryKey);
|
|
1929
|
+
return shards[hash % shards.length];
|
|
1930
|
+
}
|
|
1931
|
+
|
|
1932
|
+
async function allocateWithFallback(coordinator: DurableObjectNamespace, primaryKey: string, shards: string[]): Promise<string> {
|
|
1933
|
+
try {
|
|
1934
|
+
const coordinatorId = coordinator.idFromName('default');
|
|
1935
|
+
const instance = coordinator.get(coordinatorId);
|
|
1936
|
+
|
|
1937
|
+
const response = await instance.fetch('http://coordinator/allocate', {
|
|
1938
|
+
method: 'POST',
|
|
1939
|
+
headers: { 'Content-Type': 'application/json' },
|
|
1940
|
+
body: JSON.stringify({ primaryKey })
|
|
1941
|
+
});
|
|
1942
|
+
|
|
1943
|
+
if (response.ok) {
|
|
1944
|
+
const { shard } = await response.json();
|
|
1945
|
+
return shard;
|
|
1946
|
+
}
|
|
1947
|
+
} catch (error) {
|
|
1948
|
+
console.warn('Coordinator unavailable, using fallback allocation:', error);
|
|
1949
|
+
}
|
|
1950
|
+
|
|
1951
|
+
// Fallback to hash-based allocation
|
|
1952
|
+
return fallbackAllocation(primaryKey, shards);
|
|
1953
|
+
}
|
|
1954
|
+
```
|
|
1955
|
+
|
|
1956
|
+
## 🚀 Quick Reference
|
|
1957
|
+
|
|
1958
|
+
### Strategy Selection Guide
|
|
1959
|
+
|
|
1960
|
+
| Strategy | Use Case | Latency | Distribution | Coordinator Required |
|
|
1961
|
+
| ------------- | ---------------------------------------- | ------------------ | ------------ | -------------------- |
|
|
1962
|
+
| `hash` | High-volume apps, consistent performance | Lowest | Excellent | No |
|
|
1963
|
+
| `round-robin` | Guaranteed even distribution | Medium | Perfect | Yes |
|
|
1964
|
+
| `random` | Simple setup, good enough distribution | Low | Good | No |
|
|
1965
|
+
| `location` | Geographic optimization, reduced latency | Region-optimized | Good | No |
|
|
1966
|
+
| `mixed` | Optimized read/write performance | Strategy-dependent | Variable | Strategy-dependent |
|
|
1967
|
+
|
|
1968
|
+
#### Mixed Strategy Recommendations by Scenario
|
|
1969
|
+
|
|
1970
|
+
| Scenario | Recommended Mix | Read Strategy | Write Strategy | Benefits |
|
|
1971
|
+
| ---------------------------------- | -------------------------------------- | ------------- | -------------- | ---------------------------------------------- |
|
|
1972
|
+
| **Large Databases (>10M records)** | `{read: 'hash', write: 'round-robin'}` | Hash | Round-Robin | Fastest reads, even data distribution |
|
|
1973
|
+
| **Global Applications** | `{read: 'hash', write: 'location'}` | Hash | Location | Fast queries, optimal geographic placement |
|
|
1974
|
+
| **High Write Volume** | `{read: 'location', write: 'hash'}` | Location | Hash | Regional read optimization, fast write routing |
|
|
1975
|
+
| **Analytics Workloads** | `{read: 'random', write: 'location'}` | Random | Location | Load-balanced queries, optimal data placement |
|
|
1976
|
+
| **Multi-Tenant SaaS** | `{read: 'hash', write: 'hash'}` | Hash | Hash | Consistent performance, predictable routing |
|
|
1977
|
+
|
|
1978
|
+
### Configuration Templates
|
|
1979
|
+
|
|
1980
|
+
**Hash Strategy (Recommended for most apps):**
|
|
1981
|
+
|
|
1982
|
+
```typescript
|
|
1983
|
+
{
|
|
1984
|
+
kv: env.KV,
|
|
1985
|
+
strategy: 'hash',
|
|
1986
|
+
shards: { 'db-1': env.DB_1, 'db-2': env.DB_2 }
|
|
1987
|
+
}
|
|
1988
|
+
```
|
|
1989
|
+
|
|
1990
|
+
**Location Strategy (Geographic optimization):**
|
|
1991
|
+
|
|
1992
|
+
```typescript
|
|
1993
|
+
{
|
|
1994
|
+
kv: env.KV,
|
|
1995
|
+
strategy: 'location',
|
|
1996
|
+
targetRegion: 'wnam',
|
|
1997
|
+
shardLocations: {
|
|
1998
|
+
'db-west': { region: 'wnam', priority: 2 },
|
|
1999
|
+
'db-east': { region: 'enam', priority: 1 }
|
|
2000
|
+
},
|
|
2001
|
+
shards: { 'db-west': env.DB_WEST, 'db-east': env.DB_EAST }
|
|
2002
|
+
}
|
|
2003
|
+
```
|
|
2004
|
+
|
|
2005
|
+
**Round-Robin Strategy (Even distribution):**
|
|
2006
|
+
|
|
2007
|
+
```typescript
|
|
2008
|
+
{
|
|
2009
|
+
kv: env.KV,
|
|
2010
|
+
coordinator: env.ShardCoordinator,
|
|
2011
|
+
strategy: 'round-robin',
|
|
2012
|
+
shards: { 'db-1': env.DB_1, 'db-2': env.DB_2, 'db-3': env.DB_3 }
|
|
2013
|
+
}
|
|
2014
|
+
```
|
|
2015
|
+
|
|
2016
|
+
**Mixed Strategy (Global applications):**
|
|
2017
|
+
|
|
2018
|
+
```typescript
|
|
2019
|
+
{
|
|
2020
|
+
kv: env.KV,
|
|
2021
|
+
strategy: {
|
|
2022
|
+
read: 'hash', // Fast, consistent reads
|
|
2023
|
+
write: 'location' // Optimal geographic placement
|
|
2024
|
+
},
|
|
2025
|
+
targetRegion: 'wnam',
|
|
2026
|
+
shardLocations: {
|
|
2027
|
+
'db-west': { region: 'wnam', priority: 2 },
|
|
2028
|
+
'db-east': { region: 'enam', priority: 1 }
|
|
2029
|
+
},
|
|
2030
|
+
shards: { 'db-west': env.DB_WEST, 'db-east': env.DB_EAST }
|
|
2031
|
+
}
|
|
2032
|
+
```
|
|
2033
|
+
|
|
2034
|
+
**Mixed Strategy (Large databases):**
|
|
2035
|
+
|
|
2036
|
+
```typescript
|
|
2037
|
+
{
|
|
2038
|
+
kv: env.KV,
|
|
2039
|
+
coordinator: env.ShardCoordinator,
|
|
2040
|
+
strategy: {
|
|
2041
|
+
read: 'hash', // Fastest possible reads
|
|
2042
|
+
write: 'round-robin' // Perfect distribution
|
|
2043
|
+
},
|
|
2044
|
+
shards: { 'db-1': env.DB_1, 'db-2': env.DB_2, 'db-3': env.DB_3 }
|
|
2045
|
+
}
|
|
2046
|
+
```
|
|
2047
|
+
|
|
2048
|
+
**Mixed Strategy (High-performance consistent):**
|
|
2049
|
+
|
|
2050
|
+
```typescript
|
|
2051
|
+
{
|
|
2052
|
+
kv: env.KV,
|
|
2053
|
+
strategy: {
|
|
2054
|
+
read: 'hash', // Predictable read performance
|
|
2055
|
+
write: 'hash' // Predictable write performance
|
|
2056
|
+
},
|
|
2057
|
+
shards: { 'db-1': env.DB_1, 'db-2': env.DB_2 }
|
|
2058
|
+
}
|
|
2059
|
+
```
|
|
2060
|
+
|
|
2061
|
+
### Region Codes Reference
|
|
2062
|
+
|
|
2063
|
+
| Code | Region | Typical Location |
|
|
2064
|
+
| ------ | --------------------- | ---------------- |
|
|
2065
|
+
| `wnam` | Western North America | San Francisco |
|
|
2066
|
+
| `enam` | Eastern North America | New York |
|
|
2067
|
+
| `weur` | Western Europe | London |
|
|
2068
|
+
| `eeur` | Eastern Europe | Berlin |
|
|
2069
|
+
| `apac` | Asia Pacific | Tokyo |
|
|
2070
|
+
| `oc` | Oceania | Sydney |
|
|
2071
|
+
| `me` | Middle East | Dubai |
|
|
2072
|
+
| `af` | Africa | Johannesburg |
|
|
2073
|
+
|
|
495
2074
|
## 🤝 Contributing
|
|
496
2075
|
|
|
497
2076
|
1. Fork the repository
|