@earth-app/collegedb 1.0.8 → 1.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +349 -100
- package/dist/durable.d.ts +8 -4
- package/dist/durable.d.ts.map +1 -1
- package/dist/index.d.ts +2 -1
- package/dist/index.d.ts.map +1 -1
- package/dist/index.js +8 -7
- package/dist/index.js.map +9 -8
- package/dist/kvmap.d.ts +68 -5
- package/dist/kvmap.d.ts.map +1 -1
- package/dist/migrations.d.ts +18 -15
- package/dist/migrations.d.ts.map +1 -1
- package/dist/providers.d.ts +207 -0
- package/dist/providers.d.ts.map +1 -0
- package/dist/router.d.ts +10 -10
- package/dist/router.d.ts.map +1 -1
- package/dist/types.d.ts +142 -13
- package/dist/types.d.ts.map +1 -1
- package/package.json +39 -14
package/README.md
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# CollegeDB
|
|
2
2
|
|
|
3
|
-
|
|
3
|
+
Universal Database Horizontal Sharding Router
|
|
4
4
|
|
|
5
5
|
[](https://www.typescriptlang.org/)
|
|
6
6
|
[](https://github.com/earth-app/CollegeDB/issues)
|
|
@@ -8,7 +8,7 @@ _Cloudflare D1 Horizontal Sharding Router_
|
|
|
8
8
|
[](LICENSE)
|
|
9
9
|

|
|
10
10
|
|
|
11
|
-
A TypeScript library for **true horizontal scaling** of SQLite-style databases
|
|
11
|
+
A TypeScript library for **true horizontal scaling** of SQLite-style databases primarily for Cloudflare using D1 and KV, with additional provider adapters for Redis/Valkey KV and PostgreSQL/MySQL/SQLite SQL backends. CollegeDB distributes your data across multiple database shards, with each table's records split by primary key across different database instances.
|
|
12
12
|
|
|
13
13
|
CollegeDB implements **data distribution** where a single logical table is physically stored across multiple D1 databases:
|
|
14
14
|
|
|
@@ -46,6 +46,7 @@ CollegeDB provides a sharding layer on top of Cloudflare D1 databases, enabling
|
|
|
46
46
|
- **Scale horizontally** by distributing table data across multiple D1 instances
|
|
47
47
|
- **Route queries automatically** based on primary key mappings
|
|
48
48
|
- **Maintain consistency** with KV-based shard mapping
|
|
49
|
+
- **Run on multiple providers** through `KVStorage` and `SQLDatabase` contracts
|
|
49
50
|
- **Optimize for geography** with location-aware shard allocation
|
|
50
51
|
- **Monitor and rebalance** shard distribution
|
|
51
52
|
- **Handle migrations** between shards seamlessly
|
|
@@ -53,6 +54,7 @@ CollegeDB provides a sharding layer on top of Cloudflare D1 databases, enabling
|
|
|
53
54
|
## 📦 Features
|
|
54
55
|
|
|
55
56
|
- **🔀 Automatic Query Routing**: Primary key → shard mapping using Cloudflare KV
|
|
57
|
+
- **🧩 Provider Adapters (v1.1.0)**: Redis/Valkey KV + PostgreSQL/MySQL/SQLite SQL adapters while preserving Cloudflare compatibility
|
|
56
58
|
- **🎯 Multiple Allocation Strategies**: Round-robin, random, hash-based, and location-aware distribution
|
|
57
59
|
- **🔄 Mixed Strategy Support**: Different strategies for reads vs writes (e.g., location for writes, hash for reads)
|
|
58
60
|
- **📊 Shard Coordination**: Durable Objects for allocation and statistics
|
|
@@ -62,6 +64,58 @@ CollegeDB provides a sharding layer on top of Cloudflare D1 databases, enabling
|
|
|
62
64
|
- **⚡ High Performance**: Optimized for Cloudflare Workers runtime
|
|
63
65
|
- **🔧 TypeScript First**: Full type safety and excellent DX
|
|
64
66
|
|
|
67
|
+
## Benchmark Suite
|
|
68
|
+
|
|
69
|
+
CollegeDB includes a comprehensive benchmark suite covering real-world latency across provider combinations and Cloudflare Worker routing paths.
|
|
70
|
+
|
|
71
|
+
### Matrix
|
|
72
|
+
|
|
73
|
+
| Scenario Key | Scenario | What Happens | Workload Per Run |
|
|
74
|
+
| ----------------- | -------------------------------- | --------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------- |
|
|
75
|
+
| basic_crud | Basic CRUD round-trip | Insert, read, update, and delete a user via routed queries. | 20 iterations; 4 routed SQL ops per iteration |
|
|
76
|
+
| advanced_usage | Advanced lookup workflow | Writes user+post, adds lookup aliases, then validates join and alias-based lookup. | 15 iterations; ~5 routed SQL ops + KV lookup-key updates per iteration |
|
|
77
|
+
| migration_mapping | Migration-style mapping creation | Inserts legacy records on a fixed shard, then builds shard mappings in batch and validates routing. | 10 iterations; 20 legacy records mapped per iteration |
|
|
78
|
+
| bulk_crud | Bulk CRUD pressure | Performs bulk inserts, half updates, and full delete sweep, then validates shard-wide totals. | 7 iterations; 160 inserts + 80 updates + 160 deletes per iteration |
|
|
79
|
+
| indexing | Indexed query scan | Creates an index on posts(user_id) and repeatedly queries the indexed path. | 15 iterations after warmup dataset build |
|
|
80
|
+
|
|
81
|
+
Real-world latency benchmarks across provider combinations (`average / p95`):
|
|
82
|
+
|
|
83
|
+
| Combination | Basic CRUD | Advanced Operations | Migration | Bulk CRUD | Indexing | Overall Avg |
|
|
84
|
+
| --------------- | ------------------- | ------------------- | ------------------- | --------------------- | -------------------- | ----------- |
|
|
85
|
+
| cloudflare | 13.14 ms / 16.50 ms | 4.43 ms / 9.65 ms | 27.68 ms / 30.69 ms | 156.30 ms / 163.76 ms | 67.17 ms / 106.63 ms | 28.40 ms |
|
|
86
|
+
| postgres+redis | 2.77 ms / 3.90 ms | 3.11 ms / 4.64 ms | 6.55 ms / 8.07 ms | 42.33 ms / 80.67 ms | 0.34 ms / 0.61 ms | 5.87 ms |
|
|
87
|
+
| postgres+valkey | 1.65 ms / 2.23 ms | 2.10 ms / 2.82 ms | 5.60 ms / 6.05 ms | 33.13 ms / 43.69 ms | 0.30 ms / 0.46 ms | 4.64 ms |
|
|
88
|
+
| mysql+redis | 5.11 ms / 8.38 ms | 5.45 ms / 8.51 ms | 27.41 ms / 61.56 ms | 92.70 ms / 139.70 ms | 0.49 ms / 1.22 ms | 13.91 ms |
|
|
89
|
+
| mysql+valkey | 4.99 ms / 6.66 ms | 4.21 ms / 6.42 ms | 21.68 ms / 27.20 ms | 87.67 ms / 109.44 ms | 0.55 ms / 1.92 ms | 12.54 ms |
|
|
90
|
+
| mariadb+redis | 2.64 ms / 5.90 ms | 3.02 ms / 7.55 ms | 6.48 ms / 7.66 ms | 46.99 ms / 58.08 ms | 0.37 ms / 1.08 ms | 6.29 ms |
|
|
91
|
+
| mariadb+valkey | 2.34 ms / 4.58 ms | 2.91 ms / 5.69 ms | 5.73 ms / 7.35 ms | 45.04 ms / 61.42 ms | 0.36 ms / 0.79 ms | 5.96 ms |
|
|
92
|
+
| sqlite+redis | 2.21 ms / 3.84 ms | 2.43 ms / 3.14 ms | 10.49 ms / 17.31 ms | 140.85 ms / 184.48 ms | 0.07 ms / 0.14 ms | 15.87 ms |
|
|
93
|
+
| sqlite+valkey | 1.36 ms / 1.80 ms | 2.06 ms / 2.70 ms | 6.36 ms / 8.77 ms | 121.13 ms / 156.31 ms | 0.06 ms / 0.14 ms | 13.42 ms |
|
|
94
|
+
|
|
95
|
+
### Overview
|
|
96
|
+
|
|
97
|
+
CollegeDB includes an integration benchmark suite covering both local provider matrices and Cloudflare Worker routing paths.
|
|
98
|
+
|
|
99
|
+
Top-level benchmark scenarios:
|
|
100
|
+
|
|
101
|
+
- `basic_crud`: insert/read/update/delete round-trip routing
|
|
102
|
+
- `advanced_usage`: join + multi-key lookup behavior
|
|
103
|
+
- `migration_mapping`: batch mapping creation for existing keys
|
|
104
|
+
- `bulk_crud`: high-volume insert/update/delete flow
|
|
105
|
+
- `indexing`: indexed query latency under warm data
|
|
106
|
+
- `metadata_fetch`: schema/metadata query latency
|
|
107
|
+
- `pragma_or_info`: provider-specific pragma/info query latency
|
|
108
|
+
- `counting`: shard-wide aggregate counting
|
|
109
|
+
- `shard_fanout`: all-shards fanout query aggregation
|
|
110
|
+
- `reassignment`: shard reassignment and routed-read validation
|
|
111
|
+
|
|
112
|
+
Benchmark reports include:
|
|
113
|
+
|
|
114
|
+
- an interpretation guide (`How To Read This Report`)
|
|
115
|
+
- a benchmark catalog with per-run workload details
|
|
116
|
+
- a compact overall matrix (passed/failed/skipped + overall average)
|
|
117
|
+
- split scenario matrices for core workload latency and introspection/routing latency
|
|
118
|
+
|
|
65
119
|
## Installation
|
|
66
120
|
|
|
67
121
|
```bash
|
|
@@ -70,6 +124,129 @@ bun add @earth-app/collegedb
|
|
|
70
124
|
npm install @earth-app/collegedb
|
|
71
125
|
```
|
|
72
126
|
|
|
127
|
+
## Provider Adapters
|
|
128
|
+
|
|
129
|
+
CollegeDB can run with either native Cloudflare bindings or custom providers as long as they match the exported `KVStorage` and `SQLDatabase` interfaces.
|
|
130
|
+
|
|
131
|
+
Supported adapters:
|
|
132
|
+
|
|
133
|
+
- `createRedisKVProvider`
|
|
134
|
+
- `createValkeyKVProvider`
|
|
135
|
+
- `createPostgresSQLProvider`
|
|
136
|
+
- `createMySQLSQLProvider`
|
|
137
|
+
- `createSQLiteSQLProvider`
|
|
138
|
+
- `createHyperdrivePostgresProvider`
|
|
139
|
+
- `createHyperdriveMySQLProvider`
|
|
140
|
+
|
|
141
|
+
```typescript
|
|
142
|
+
import { createClient as createRedisClient } from 'redis';
|
|
143
|
+
import { Pool } from 'pg';
|
|
144
|
+
import { createPostgresSQLProvider, createRedisKVProvider, initialize, run, type CollegeDBConfig } from '@earth-app/collegedb';
|
|
145
|
+
|
|
146
|
+
const redisClient = createRedisClient({ url: process.env.REDIS_URL });
|
|
147
|
+
const pgPool = new Pool({ connectionString: process.env.POSTGRES_URL });
|
|
148
|
+
|
|
149
|
+
const config: CollegeDBConfig = {
|
|
150
|
+
kv: createRedisKVProvider(redisClient),
|
|
151
|
+
shards: {
|
|
152
|
+
'pg-east': createPostgresSQLProvider(pgPool)
|
|
153
|
+
},
|
|
154
|
+
strategy: 'hash',
|
|
155
|
+
disableAutoMigration: true
|
|
156
|
+
};
|
|
157
|
+
|
|
158
|
+
async function bootstrap() {
|
|
159
|
+
await redisClient.connect();
|
|
160
|
+
initialize(config);
|
|
161
|
+
await run('user-1', 'INSERT INTO users (id, name) VALUES (?, ?)', ['user-1', 'Taylor']);
|
|
162
|
+
}
|
|
163
|
+
|
|
164
|
+
bootstrap().catch(console.error);
|
|
165
|
+
```
|
|
166
|
+
|
|
167
|
+
For Hyperdrive-backed SQL connections, use `createHyperdrivePostgresProvider` or `createHyperdriveMySQLProvider` with your database client factory.
|
|
168
|
+
|
|
169
|
+
For a complete non-Cloudflare setup, see `examples/provider-sandbox.ts`.
|
|
170
|
+
|
|
171
|
+
## Sandbox Benchmarks (Docker Compose)
|
|
172
|
+
|
|
173
|
+
CollegeDB ships with an integration sandbox runner that benchmarks real latency across provider combinations.
|
|
174
|
+
|
|
175
|
+
Requirements:
|
|
176
|
+
|
|
177
|
+
- Docker + Docker Compose plugin
|
|
178
|
+
- Bun
|
|
179
|
+
- Wrangler (installed as a dev dependency and invoked by scripts)
|
|
180
|
+
|
|
181
|
+
The Cloudflare benchmark path runs against the dedicated sandbox worker:
|
|
182
|
+
|
|
183
|
+
- Worker entry: `sandbox/worker.ts`
|
|
184
|
+
- Wrangler config: `sandbox/wrangler.jsonc`
|
|
185
|
+
|
|
186
|
+
Main commands:
|
|
187
|
+
|
|
188
|
+
```bash
|
|
189
|
+
# Run full SQL x KV matrix plus Cloudflare local benchmark
|
|
190
|
+
bun run test:sandbox
|
|
191
|
+
|
|
192
|
+
# Run full SQL x KV matrix only
|
|
193
|
+
bun run test:sandbox:all
|
|
194
|
+
|
|
195
|
+
# Run Cloudflare local benchmark only (wrangler dev --local)
|
|
196
|
+
bun run test:sandbox:cloudflare
|
|
197
|
+
```
|
|
198
|
+
|
|
199
|
+
Provider filters:
|
|
200
|
+
|
|
201
|
+
```bash
|
|
202
|
+
# One SQL provider against both KV providers
|
|
203
|
+
bun run test:sandbox:mysql
|
|
204
|
+
bun run test:sandbox:postgres
|
|
205
|
+
bun run test:sandbox:mariadb
|
|
206
|
+
bun run test:sandbox:sqlite
|
|
207
|
+
|
|
208
|
+
# One KV provider against all SQL providers
|
|
209
|
+
bun run test:sandbox:redis
|
|
210
|
+
bun run test:sandbox:valkey
|
|
211
|
+
|
|
212
|
+
# Explicit pairwise combinations
|
|
213
|
+
bun run test:sandbox:postgres+redis
|
|
214
|
+
bun run test:sandbox:postgres+valkey
|
|
215
|
+
bun run test:sandbox:mysql+redis
|
|
216
|
+
bun run test:sandbox:mysql+valkey
|
|
217
|
+
bun run test:sandbox:mariadb+redis
|
|
218
|
+
bun run test:sandbox:mariadb+valkey
|
|
219
|
+
bun run test:sandbox:sqlite+redis
|
|
220
|
+
bun run test:sandbox:sqlite+valkey
|
|
221
|
+
```
|
|
222
|
+
|
|
223
|
+
Output behavior:
|
|
224
|
+
|
|
225
|
+
- Every run writes a timestamped Markdown report to `sandbox/results/`
|
|
226
|
+
- `sandbox/results/latest.md` is always updated to the newest report
|
|
227
|
+
- The runner prints the report in-terminal using Bun's Markdown renderer with ANSI formatting
|
|
228
|
+
- `test:sandbox` produces a matrix for all SQL x KV combinations; filtered commands produce matrix subsets
|
|
229
|
+
|
|
230
|
+
Benchmark coverage includes:
|
|
231
|
+
|
|
232
|
+
- basic CRUD
|
|
233
|
+
- advanced lookup/routing workflows
|
|
234
|
+
- migration-style mapping creation
|
|
235
|
+
- bulk CRUD
|
|
236
|
+
- indexing queries
|
|
237
|
+
- metadata fetch
|
|
238
|
+
- pragma/info queries (provider-specific)
|
|
239
|
+
- counting across shards
|
|
240
|
+
- shard fanout aggregation
|
|
241
|
+
- shard reassignment workflow
|
|
242
|
+
|
|
243
|
+
How to read benchmark rows:
|
|
244
|
+
|
|
245
|
+
- Latency cells are formatted as `average / p95` in milliseconds.
|
|
246
|
+
- `FAILED` means the scenario returned an error.
|
|
247
|
+
- `N/A` means the scenario was intentionally skipped in that environment.
|
|
248
|
+
- Use the detailed section for full `avg`, `p50`, `p95`, `min`, `max`, and sample count (`n`).
|
|
249
|
+
|
|
73
250
|
## Basic Usage
|
|
74
251
|
|
|
75
252
|
```typescript
|
|
@@ -88,7 +265,7 @@ collegedb(
|
|
|
88
265
|
},
|
|
89
266
|
async () => {
|
|
90
267
|
// Create schema on new shards only (existing shards auto-detected)
|
|
91
|
-
await createSchema(env['db-new-shard']);
|
|
268
|
+
await createSchema(env['db-new-shard'], 'CREATE TABLE IF NOT EXISTS users (id TEXT PRIMARY KEY, name TEXT, email TEXT)');
|
|
92
269
|
|
|
93
270
|
// Insert data (automatically routed to appropriate shard)
|
|
94
271
|
await run('user-123', 'INSERT INTO users (id, name, email) VALUES (?, ?, ?)', ['user-123', 'Johnson', 'alice@example.com']);
|
|
@@ -222,8 +399,6 @@ collegedb(
|
|
|
222
399
|
|
|
223
400
|
### Adding Lookup Keys to Existing Mappings
|
|
224
401
|
|
|
225
|
-
s
|
|
226
|
-
|
|
227
402
|
```typescript
|
|
228
403
|
const mapper = new KVShardMapper(env.KV);
|
|
229
404
|
|
|
@@ -535,7 +710,7 @@ for (const [table, pkColumn] of Object.entries(customIntegration)) {
|
|
|
535
710
|
| ------------------------------------------ | ---------------------------------------------------------------- | -------------------------- |
|
|
536
711
|
| `collegedb(config, callback)` | Initialize CollegeDB, then run a callback | `CollegeDBConfig, () => T` |
|
|
537
712
|
| `initialize(config)` | Initialize CollegeDB with configuration | `CollegeDBConfig` |
|
|
538
|
-
| `createSchema(
|
|
713
|
+
| `createSchema(db, schema)` | Create schema on a shard database | `SQLDatabase, string` |
|
|
539
714
|
| `prepare(key, sql)` | Prepare a SQL statement for execution | `string, string` |
|
|
540
715
|
| `run(key, sql, bindings)` | Execute a SQL query with primary key routing | `string, string, any[]` |
|
|
541
716
|
| `first(key, sql, bindings)` | Execute a SQL query and return first result | `string, string, any[]` |
|
|
@@ -552,17 +727,31 @@ for (const [table, pkColumn] of Object.entries(customIntegration)) {
|
|
|
552
727
|
| `getDatabaseSizeForShard(shard)` | Get size of a specific shard in bytes | `string` |
|
|
553
728
|
| `flush()` | Clear all shard mappings (development only) | `void` |
|
|
554
729
|
|
|
730
|
+
### Provider Adapter Functions
|
|
731
|
+
|
|
732
|
+
| Function | Description | Parameters |
|
|
733
|
+
| ---------------------------------------------------------- | -------------------------------------------------------------------- | -------------------------------------------------------- |
|
|
734
|
+
| `createRedisKVProvider(client, options?)` | Adapt a Redis client to CollegeDB's `KVStorage` contract | `RedisLikeClient, { scanCount?: number }` |
|
|
735
|
+
| `createValkeyKVProvider(client, options?)` | Adapt a Valkey client to CollegeDB's `KVStorage` contract | `RedisLikeClient, { scanCount?: number }` |
|
|
736
|
+
| `createPostgresSQLProvider(client)` | Adapt a PostgreSQL client/pool to `SQLDatabase` | `PostgresClientLike` |
|
|
737
|
+
| `createMySQLSQLProvider(client)` | Adapt a MySQL/MariaDB client to `SQLDatabase` | `MySQLClientLike` |
|
|
738
|
+
| `createSQLiteSQLProvider(client)` | Adapt a SQLite client to `SQLDatabase` | `SQLiteClientLike` |
|
|
739
|
+
| `createHyperdrivePostgresProvider(binding, clientFactory)` | Create a PostgreSQL `SQLDatabase` adapter using a Hyperdrive binding | `HyperdriveBindingLike, HyperdrivePostgresClientFactory` |
|
|
740
|
+
| `createHyperdriveMySQLProvider(binding, clientFactory)` | Create a MySQL `SQLDatabase` adapter using a Hyperdrive binding | `HyperdriveBindingLike, HyperdriveMySQLClientFactory` |
|
|
741
|
+
| `isKVStorage(value)` | Runtime guard for `KVStorage` | `unknown` |
|
|
742
|
+
| `isSQLDatabase(value)` | Runtime guard for `SQLDatabase` | `unknown` |
|
|
743
|
+
|
|
555
744
|
### Drop-in Replacement Functions
|
|
556
745
|
|
|
557
746
|
| Function | Description | Parameters |
|
|
558
747
|
| ----------------------------------------- | ---------------------------------------------- | ------------------------------ |
|
|
559
|
-
| `autoDetectAndMigrate(d1, shard, config)` | Automatically detect and migrate existing data | `
|
|
560
|
-
| `checkMigrationNeeded(d1, shard, config)` | Check if database needs migration | `
|
|
561
|
-
| `validateTableForSharding(d1, table)` | Check if table is suitable for sharding | `
|
|
562
|
-
| `discoverExistingPrimaryKeys(d1, table)` | Find all primary keys in existing table | `
|
|
563
|
-
| `integrateExistingDatabase(d1, shard)` | Complete drop-in integration of existing DB | `
|
|
748
|
+
| `autoDetectAndMigrate(d1, shard, config)` | Automatically detect and migrate existing data | `SQLDatabase, string, config` |
|
|
749
|
+
| `checkMigrationNeeded(d1, shard, config)` | Check if database needs migration | `SQLDatabase, string, config` |
|
|
750
|
+
| `validateTableForSharding(d1, table)` | Check if table is suitable for sharding | `SQLDatabase, string` |
|
|
751
|
+
| `discoverExistingPrimaryKeys(d1, table)` | Find all primary keys in existing table | `SQLDatabase, string` |
|
|
752
|
+
| `integrateExistingDatabase(d1, shard)` | Complete drop-in integration of existing DB | `SQLDatabase, string, mapper` |
|
|
564
753
|
| `createMappingsForExistingKeys(keys)` | Create shard mappings for existing keys | `string[], string[], strategy` |
|
|
565
|
-
| `listTables(d1)` | Get list of tables in database | `
|
|
754
|
+
| `listTables(d1)` | Get list of tables in database | `SQLDatabase` |
|
|
566
755
|
| `clearMigrationCache()` | Clear automatic migration cache | `void` |
|
|
567
756
|
|
|
568
757
|
### Error Handling
|
|
@@ -645,15 +834,19 @@ The main configuration interface supports both single strategies and mixed strat
|
|
|
645
834
|
|
|
646
835
|
```typescript
|
|
647
836
|
interface CollegeDBConfig {
|
|
648
|
-
kv:
|
|
837
|
+
kv: KVStorage;
|
|
649
838
|
coordinator?: DurableObjectNamespace;
|
|
650
|
-
shards: Record<string,
|
|
839
|
+
shards: Record<string, SQLDatabase>;
|
|
651
840
|
strategy?: ShardingStrategy | MixedShardingStrategy;
|
|
652
841
|
targetRegion?: D1Region;
|
|
653
842
|
shardLocations?: Record<string, ShardLocation>;
|
|
654
843
|
disableAutoMigration?: boolean; // Default: false
|
|
655
844
|
hashShardMappings?: boolean; // Default: true
|
|
656
845
|
maxDatabaseSize?: number; // Default: undefined (no limit)
|
|
846
|
+
mappingCacheTtlMs?: number; // Default: 30000
|
|
847
|
+
knownShardsCacheTtlMs?: number; // Default: 10000
|
|
848
|
+
sizeCacheTtlMs?: number; // Default: 30000
|
|
849
|
+
migrationConcurrency?: number; // Default: 25
|
|
657
850
|
}
|
|
658
851
|
```
|
|
659
852
|
|
|
@@ -706,7 +899,7 @@ const mixedStrategyConfig: CollegeDBConfig = {
|
|
|
706
899
|
|
|
707
900
|
### Database Size Management
|
|
708
901
|
|
|
709
|
-
CollegeDB supports automatic size-based shard exclusion to prevent individual shards from becoming too large. This feature helps maintain optimal performance and prevents hitting
|
|
902
|
+
CollegeDB supports automatic size-based shard exclusion to prevent individual shards from becoming too large. This feature helps maintain optimal performance and prevents hitting database storage limits.
|
|
710
903
|
|
|
711
904
|
#### Configuration
|
|
712
905
|
|
|
@@ -782,12 +975,12 @@ const config: CollegeDBConfig = {
|
|
|
782
975
|
// "Excluded 2 shards due to size limits: db-east, db-central"
|
|
783
976
|
```
|
|
784
977
|
|
|
785
|
-
#### Performance Impact
|
|
978
|
+
#### Size-Limit Performance Impact
|
|
786
979
|
|
|
787
980
|
- **Size Check Frequency**: Only performed during new allocations (not on reads)
|
|
788
981
|
- **Query Efficiency**: Uses fast SQLite pragmas (microsecond execution time)
|
|
789
982
|
- **Parallel Execution**: Size checks run concurrently across all shards
|
|
790
|
-
- **Caching**:
|
|
983
|
+
- **Caching**: Size checks are cached in-memory (controlled by `sizeCacheTtlMs`, default `30000`)
|
|
791
984
|
|
|
792
985
|
### Types
|
|
793
986
|
|
|
@@ -796,6 +989,10 @@ CollegeDB exports TypeScript types for better development experience and type sa
|
|
|
796
989
|
| Type | Description | Example |
|
|
797
990
|
| ----------------------- | ------------------------------ | --------------------------------------------------- |
|
|
798
991
|
| `CollegeDBConfig` | Main configuration object | `{ kv, shards, strategy }` |
|
|
992
|
+
| `KVStorage` | Provider-agnostic KV contract | `createRedisKVProvider(redisClient)` |
|
|
993
|
+
| `SQLDatabase` | Provider-agnostic SQL contract | `createPostgresSQLProvider(pgPool)` |
|
|
994
|
+
| `QueryResult` | Standard query response shape | `{ success, results, meta }` |
|
|
995
|
+
| `QueryResultMeta` | Query execution metadata | `{ duration, changes?, last_row_id? }` |
|
|
799
996
|
| `ShardingStrategy` | Single strategy options | `'hash' \| 'location' \| 'round-robin' \| 'random'` |
|
|
800
997
|
| `MixedShardingStrategy` | Mixed strategy configuration | `{ read: 'hash', write: 'location' }` |
|
|
801
998
|
| `OperationType` | Database operation types | `'read' \| 'write'` |
|
|
@@ -918,81 +1115,127 @@ wrangler d1 create collegedb-central
|
|
|
918
1115
|
wrangler kv namespace create "KV"
|
|
919
1116
|
```
|
|
920
1117
|
|
|
921
|
-
### 3. Configure wrangler.
|
|
922
|
-
|
|
923
|
-
```toml
|
|
924
|
-
[[d1_databases]]
|
|
925
|
-
binding = "db-east"
|
|
926
|
-
database_name = "collegedb-east"
|
|
927
|
-
database_id = "your-database-id"
|
|
1118
|
+
### 3. Configure wrangler.jsonc
|
|
928
1119
|
|
|
929
|
-
|
|
930
|
-
|
|
931
|
-
|
|
932
|
-
|
|
933
|
-
|
|
934
|
-
|
|
935
|
-
|
|
936
|
-
|
|
937
|
-
|
|
938
|
-
|
|
939
|
-
|
|
940
|
-
|
|
1120
|
+
```jsonc
|
|
1121
|
+
{
|
|
1122
|
+
"$schema": "./node_modules/wrangler/config-schema.json",
|
|
1123
|
+
"name": "collegedb-app",
|
|
1124
|
+
"main": "src/index.ts",
|
|
1125
|
+
"compatibility_date": "2026-04-15",
|
|
1126
|
+
"d1_databases": [
|
|
1127
|
+
{
|
|
1128
|
+
"binding": "db-east",
|
|
1129
|
+
"database_name": "collegedb-east",
|
|
1130
|
+
"database_id": "your-east-database-id"
|
|
1131
|
+
},
|
|
1132
|
+
{
|
|
1133
|
+
"binding": "db-west",
|
|
1134
|
+
"database_name": "collegedb-west",
|
|
1135
|
+
"database_id": "your-west-database-id"
|
|
1136
|
+
}
|
|
1137
|
+
],
|
|
1138
|
+
"kv_namespaces": [
|
|
1139
|
+
{
|
|
1140
|
+
"binding": "KV",
|
|
1141
|
+
"id": "your-kv-namespace-id",
|
|
1142
|
+
"preview_id": "your-kv-preview-id"
|
|
1143
|
+
}
|
|
1144
|
+
],
|
|
1145
|
+
"durable_objects": {
|
|
1146
|
+
"bindings": [
|
|
1147
|
+
{
|
|
1148
|
+
"name": "ShardCoordinator",
|
|
1149
|
+
"class_name": "ShardCoordinator"
|
|
1150
|
+
}
|
|
1151
|
+
]
|
|
1152
|
+
},
|
|
1153
|
+
"migrations": [
|
|
1154
|
+
{
|
|
1155
|
+
"tag": "v1",
|
|
1156
|
+
"new_sqlite_classes": ["ShardCoordinator"]
|
|
1157
|
+
}
|
|
1158
|
+
]
|
|
1159
|
+
}
|
|
941
1160
|
```
|
|
942
1161
|
|
|
943
|
-
#### Complete wrangler.
|
|
944
|
-
|
|
945
|
-
```toml
|
|
946
|
-
name = "collegedb-app"
|
|
947
|
-
main = "src/index.ts"
|
|
948
|
-
compatibility_date = "2024-08-10"
|
|
949
|
-
|
|
950
|
-
# D1 Database bindings
|
|
951
|
-
[[d1_databases]]
|
|
952
|
-
binding = "db-east"
|
|
953
|
-
database_name = "collegedb-east"
|
|
954
|
-
database_id = "your-east-database-id"
|
|
955
|
-
|
|
956
|
-
[[d1_databases]]
|
|
957
|
-
binding = "db-west"
|
|
958
|
-
database_name = "collegedb-west"
|
|
959
|
-
database_id = "your-west-database-id"
|
|
960
|
-
|
|
961
|
-
[[d1_databases]]
|
|
962
|
-
binding = "db-central"
|
|
963
|
-
database_name = "collegedb-central"
|
|
964
|
-
database_id = "your-central-database-id"
|
|
965
|
-
|
|
966
|
-
# KV namespace for shard mappings
|
|
967
|
-
[[kv_namespaces]]
|
|
968
|
-
binding = "KV"
|
|
969
|
-
id = "your-kv-namespace-id"
|
|
970
|
-
preview_id = "your-kv-preview-id" # For local development
|
|
971
|
-
|
|
972
|
-
# Durable Object for shard coordination
|
|
973
|
-
[[durable_objects.bindings]]
|
|
974
|
-
name = "ShardCoordinator"
|
|
975
|
-
class_name = "ShardCoordinator"
|
|
976
|
-
|
|
977
|
-
# Environment-specific configurations
|
|
978
|
-
[env.production]
|
|
979
|
-
[[env.production.d1_databases]]
|
|
980
|
-
binding = "db-east"
|
|
981
|
-
database_name = "collegedb-prod-east"
|
|
982
|
-
database_id = "your-prod-east-id"
|
|
983
|
-
|
|
984
|
-
[[env.production.d1_databases]]
|
|
985
|
-
binding = "db-west"
|
|
986
|
-
database_name = "collegedb-prod-west"
|
|
987
|
-
database_id = "your-prod-west-id"
|
|
988
|
-
|
|
989
|
-
[[env.production.kv_namespaces]]
|
|
990
|
-
binding = "KV"
|
|
991
|
-
id = "your-prod-kv-namespace-id"
|
|
1162
|
+
#### Complete wrangler.jsonc with ShardCoordinator
|
|
992
1163
|
|
|
993
|
-
|
|
994
|
-
|
|
995
|
-
|
|
1164
|
+
```jsonc
|
|
1165
|
+
{
|
|
1166
|
+
"$schema": "./node_modules/wrangler/config-schema.json",
|
|
1167
|
+
"name": "collegedb-app",
|
|
1168
|
+
"main": "src/index.ts",
|
|
1169
|
+
"compatibility_date": "2026-04-15",
|
|
1170
|
+
"d1_databases": [
|
|
1171
|
+
{
|
|
1172
|
+
"binding": "db-east",
|
|
1173
|
+
"database_name": "collegedb-east",
|
|
1174
|
+
"database_id": "your-east-database-id"
|
|
1175
|
+
},
|
|
1176
|
+
{
|
|
1177
|
+
"binding": "db-west",
|
|
1178
|
+
"database_name": "collegedb-west",
|
|
1179
|
+
"database_id": "your-west-database-id"
|
|
1180
|
+
},
|
|
1181
|
+
{
|
|
1182
|
+
"binding": "db-central",
|
|
1183
|
+
"database_name": "collegedb-central",
|
|
1184
|
+
"database_id": "your-central-database-id"
|
|
1185
|
+
}
|
|
1186
|
+
],
|
|
1187
|
+
"kv_namespaces": [
|
|
1188
|
+
{
|
|
1189
|
+
"binding": "KV",
|
|
1190
|
+
"id": "your-kv-namespace-id",
|
|
1191
|
+
"preview_id": "your-kv-preview-id"
|
|
1192
|
+
}
|
|
1193
|
+
],
|
|
1194
|
+
"durable_objects": {
|
|
1195
|
+
"bindings": [
|
|
1196
|
+
{
|
|
1197
|
+
"name": "ShardCoordinator",
|
|
1198
|
+
"class_name": "ShardCoordinator"
|
|
1199
|
+
}
|
|
1200
|
+
]
|
|
1201
|
+
},
|
|
1202
|
+
"migrations": [
|
|
1203
|
+
{
|
|
1204
|
+
"tag": "v1",
|
|
1205
|
+
"new_sqlite_classes": ["ShardCoordinator"]
|
|
1206
|
+
}
|
|
1207
|
+
],
|
|
1208
|
+
"env": {
|
|
1209
|
+
"production": {
|
|
1210
|
+
"d1_databases": [
|
|
1211
|
+
{
|
|
1212
|
+
"binding": "db-east",
|
|
1213
|
+
"database_name": "collegedb-prod-east",
|
|
1214
|
+
"database_id": "your-prod-east-id"
|
|
1215
|
+
},
|
|
1216
|
+
{
|
|
1217
|
+
"binding": "db-west",
|
|
1218
|
+
"database_name": "collegedb-prod-west",
|
|
1219
|
+
"database_id": "your-prod-west-id"
|
|
1220
|
+
}
|
|
1221
|
+
],
|
|
1222
|
+
"kv_namespaces": [
|
|
1223
|
+
{
|
|
1224
|
+
"binding": "KV",
|
|
1225
|
+
"id": "your-prod-kv-namespace-id"
|
|
1226
|
+
}
|
|
1227
|
+
],
|
|
1228
|
+
"durable_objects": {
|
|
1229
|
+
"bindings": [
|
|
1230
|
+
{
|
|
1231
|
+
"name": "ShardCoordinator",
|
|
1232
|
+
"class_name": "ShardCoordinator"
|
|
1233
|
+
}
|
|
1234
|
+
]
|
|
1235
|
+
}
|
|
1236
|
+
}
|
|
1237
|
+
}
|
|
1238
|
+
}
|
|
996
1239
|
```
|
|
997
1240
|
|
|
998
1241
|
### 3.1. Worker Script Setup (Required for ShardCoordinator)
|
|
@@ -1762,19 +2005,25 @@ CollegeDB includes an optional **ShardCoordinator** Durable Object that provides
|
|
|
1762
2005
|
|
|
1763
2006
|
#### Durable Object Setup
|
|
1764
2007
|
|
|
1765
|
-
First, configure the Durable Object in your `wrangler.
|
|
1766
|
-
|
|
1767
|
-
```toml
|
|
1768
|
-
[[durable_objects.bindings]]
|
|
1769
|
-
name = "ShardCoordinator"
|
|
1770
|
-
class_name = "ShardCoordinator"
|
|
2008
|
+
First, configure the Durable Object in your `wrangler.jsonc`:
|
|
1771
2009
|
|
|
1772
|
-
|
|
1773
|
-
|
|
1774
|
-
|
|
1775
|
-
[
|
|
1776
|
-
|
|
1777
|
-
|
|
2010
|
+
```jsonc
|
|
2011
|
+
{
|
|
2012
|
+
"durable_objects": {
|
|
2013
|
+
"bindings": [
|
|
2014
|
+
{
|
|
2015
|
+
"name": "ShardCoordinator",
|
|
2016
|
+
"class_name": "ShardCoordinator"
|
|
2017
|
+
}
|
|
2018
|
+
]
|
|
2019
|
+
},
|
|
2020
|
+
"migrations": [
|
|
2021
|
+
{
|
|
2022
|
+
"tag": "v1",
|
|
2023
|
+
"new_sqlite_classes": ["ShardCoordinator"]
|
|
2024
|
+
}
|
|
2025
|
+
]
|
|
2026
|
+
}
|
|
1778
2027
|
```
|
|
1779
2028
|
|
|
1780
2029
|
#### Basic Usage with ShardCoordinator
|
package/dist/durable.d.ts
CHANGED
|
@@ -7,10 +7,14 @@
|
|
|
7
7
|
*
|
|
8
8
|
* @example
|
|
9
9
|
* ```typescript
|
|
10
|
-
* // In wrangler.
|
|
11
|
-
*
|
|
12
|
-
*
|
|
13
|
-
*
|
|
10
|
+
* // In wrangler.jsonc:
|
|
11
|
+
* // {
|
|
12
|
+
* // "durable_objects": {
|
|
13
|
+
* // "bindings": [
|
|
14
|
+
* // { "name": "ShardCoordinator", "class_name": "ShardCoordinator" }
|
|
15
|
+
* // ]
|
|
16
|
+
* // }
|
|
17
|
+
* // }
|
|
14
18
|
*
|
|
15
19
|
* // Usage in a Worker:
|
|
16
20
|
* const coordinatorId = env.ShardCoordinator.idFromName('default');
|
package/dist/durable.d.ts.map
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"durable.d.ts","sourceRoot":"","sources":["../src/durable.ts"],"names":[],"mappings":"AAAA
|
|
1
|
+
{"version":3,"file":"durable.d.ts","sourceRoot":"","sources":["../src/durable.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;;;;;;;;;;;;;;;;;;;GA6BG;AAEH,OAAO,KAAK,EAAE,kBAAkB,EAAE,MAAM,2BAA2B,CAAC;AAIpE;;;;;;;;;;;;;;;;;;;;;;GAsBG;AACH,qBAAa,gBAAgB;IAC5B;;;OAGG;IACH,OAAO,CAAC,KAAK,CAAqB;IAElC;;;OAGG;gBACS,KAAK,EAAE,kBAAkB;IAIrC;;;;;;;;;;OAUG;YACW,QAAQ;IAYtB;;;;;;;;;;OAUG;YACW,SAAS;IAIvB;;;;;;;;;;;;;;;;;;;;;;;;;;;OA2BG;IACG,KAAK,CAAC,OAAO,EAAE,OAAO,GAAG,OAAO,CAAC,QAAQ,CAAC;IAgChD;;;;;;OAMG;YACW,gBAAgB;IAO9B;;;;;;;;;OASG;YACW,cAAc;IA4B5B;;;;;;;;;OASG;YACW,iBAAiB;IA6B/B;;;;;;OAMG;YACW,cAAc;IAQ5B;;;;;;;;OAQG;YACW,iBAAiB;IA+B/B;;;;;;;;;;;;;;;OAeG;YACW,mBAAmB;IAsCjC;;;;;;;;;;OAUG;YACW,WAAW;IAOzB;;;;;;;;OAQG;IACH,OAAO,CAAC,eAAe;IAmBvB;;;;;;;;;;;;;;;;;;;;;OAqBG;IACH,OAAO,CAAC,WAAW;IAgGnB;;;;;;;;;;;OAWG;IACG,mBAAmB,CAAC,KAAK,EAAE,MAAM,GAAG,OAAO,CAAC,IAAI,CAAC;IASvD;;;;;;;;;;;OAWG;IACG,mBAAmB,CAAC,KAAK,EAAE,MAAM,GAAG,OAAO,CAAC,IAAI,CAAC;CAQvD"}
|
package/dist/index.d.ts
CHANGED
|
@@ -12,6 +12,7 @@ export { all, allAllShards, allShard, collegedb, createSchema, first, firstAllSh
|
|
|
12
12
|
export { ShardCoordinator } from './durable.js';
|
|
13
13
|
export { CollegeDBError } from './errors.js';
|
|
14
14
|
export { KVShardMapper } from './kvmap.js';
|
|
15
|
+
export { createHyperdriveMySQLProvider, createHyperdrivePostgresProvider, createMySQLProvider as createMySQLSQLProvider, createPostgreSQLProvider as createPostgresSQLProvider, createRedisKVProvider, createSQLiteProvider as createSQLiteSQLProvider, createValkeyKVProvider, isKVStorage, isSQLDatabase, type HyperdriveBindingLike, type HyperdriveMySQLClientFactory, type HyperdrivePostgresClientFactory, type MySQLClientLike, type PostgresClientLike, type RedisLikeClient, type SQLiteClientLike } from './providers.js';
|
|
15
16
|
export { autoDetectAndMigrate, checkMigrationNeeded, clearMigrationCache, clearShardMigrationCache, createMappingsForExistingKeys, createSchemaAcrossShards, discoverExistingPrimaryKeys, discoverExistingRecordsWithColumns, dropSchema, integrateExistingDatabase, listTables, migrateRecord, schemaExists, validateTableForSharding, type IntegrationOptions, type IntegrationResult, type ValidationResult } from './migrations.js';
|
|
16
|
-
export type { CollegeDBConfig, D1Region, Env, MixedShardingStrategy, OperationType, ShardCoordinatorState, ShardLocation, ShardMapping, ShardStats, ShardingStrategy } from './types.js';
|
|
17
|
+
export type { CollegeDBConfig, D1Region, Env, KVListResult, KVStorage, MixedShardingStrategy, OperationType, PreparedStatement, QueryResult, QueryResultMeta, SQLDatabase, ShardCoordinatorState, ShardLocation, ShardMapping, ShardStats, ShardingStrategy } from './types.js';
|
|
17
18
|
//# sourceMappingURL=index.d.ts.map
|
package/dist/index.d.ts.map
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"index.d.ts","sourceRoot":"","sources":["../src/index.ts"],"names":[],"mappings":"AAAA;;;;;;;;;GASG;AAGH,OAAO,EACN,GAAG,EACH,YAAY,EACZ,QAAQ,EACR,SAAS,EACT,YAAY,EACZ,KAAK,EACL,cAAc,EACd,UAAU,EACV,KAAK,EACL,sBAAsB,EACtB,uBAAuB,EACvB,aAAa,EACb,UAAU,EACV,eAAe,EACf,eAAe,EACf,OAAO,EACP,aAAa,EACb,WAAW,EACX,GAAG,EACH,YAAY,EACZ,QAAQ,EACR,MAAM,aAAa,CAAC;AAGrB,OAAO,EAAE,gBAAgB,EAAE,MAAM,cAAc,CAAC;AAChD,OAAO,EAAE,cAAc,EAAE,MAAM,aAAa,CAAC;AAC7C,OAAO,EAAE,aAAa,EAAE,MAAM,YAAY,CAAC;AAG3C,OAAO,EACN,oBAAoB,EACpB,oBAAoB,EACpB,mBAAmB,EACnB,wBAAwB,EACxB,6BAA6B,EAC7B,wBAAwB,EACxB,2BAA2B,EAC3B,kCAAkC,EAClC,UAAU,EACV,yBAAyB,EACzB,UAAU,EACV,aAAa,EACb,YAAY,EACZ,wBAAwB,EACxB,KAAK,kBAAkB,EACvB,KAAK,iBAAiB,EACtB,KAAK,gBAAgB,EACrB,MAAM,iBAAiB,CAAC;AAGzB,YAAY,EACX,eAAe,EACf,QAAQ,EACR,GAAG,EACH,qBAAqB,EACrB,aAAa,EACb,qBAAqB,EACrB,aAAa,EACb,YAAY,EACZ,UAAU,EACV,gBAAgB,EAChB,MAAM,YAAY,CAAC"}
|
|
1
|
+
{"version":3,"file":"index.d.ts","sourceRoot":"","sources":["../src/index.ts"],"names":[],"mappings":"AAAA;;;;;;;;;GASG;AAGH,OAAO,EACN,GAAG,EACH,YAAY,EACZ,QAAQ,EACR,SAAS,EACT,YAAY,EACZ,KAAK,EACL,cAAc,EACd,UAAU,EACV,KAAK,EACL,sBAAsB,EACtB,uBAAuB,EACvB,aAAa,EACb,UAAU,EACV,eAAe,EACf,eAAe,EACf,OAAO,EACP,aAAa,EACb,WAAW,EACX,GAAG,EACH,YAAY,EACZ,QAAQ,EACR,MAAM,aAAa,CAAC;AAGrB,OAAO,EAAE,gBAAgB,EAAE,MAAM,cAAc,CAAC;AAChD,OAAO,EAAE,cAAc,EAAE,MAAM,aAAa,CAAC;AAC7C,OAAO,EAAE,aAAa,EAAE,MAAM,YAAY,CAAC;AAG3C,OAAO,EACN,6BAA6B,EAC7B,gCAAgC,EAChC,mBAAmB,IAAI,sBAAsB,EAC7C,wBAAwB,IAAI,yBAAyB,EACrD,qBAAqB,EACrB,oBAAoB,IAAI,uBAAuB,EAC/C,sBAAsB,EACtB,WAAW,EACX,aAAa,EACb,KAAK,qBAAqB,EAC1B,KAAK,4BAA4B,EACjC,KAAK,+BAA+B,EACpC,KAAK,eAAe,EACpB,KAAK,kBAAkB,EACvB,KAAK,eAAe,EACpB,KAAK,gBAAgB,EACrB,MAAM,gBAAgB,CAAC;AAGxB,OAAO,EACN,oBAAoB,EACpB,oBAAoB,EACpB,mBAAmB,EACnB,wBAAwB,EACxB,6BAA6B,EAC7B,wBAAwB,EACxB,2BAA2B,EAC3B,kCAAkC,EAClC,UAAU,EACV,yBAAyB,EACzB,UAAU,EACV,aAAa,EACb,YAAY,EACZ,wBAAwB,EACxB,KAAK,kBAAkB,EACvB,KAAK,iBAAiB,EACtB,KAAK,gBAAgB,EACrB,MAAM,iBAAiB,CAAC;AAGzB,YAAY,EACX,eAAe,EACf,QAAQ,EACR,GAAG,EACH,YAAY,EACZ,SAAS,EACT,qBAAqB,EACrB,aAAa,EACb,iBAAiB,EACjB,WAAW,EACX,eAAe,EACf,WAAW,EACX,qBAAqB,EACrB,aAAa,EACb,YAAY,EACZ,UAAU,EACV,gBAAgB,EAChB,MAAM,YAAY,CAAC"}
|