@byok-sdk/cloud-dataplane 0.4.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/LICENSE +21 -0
- package/README.md +224 -0
- package/dist/cleanup.d.ts +106 -0
- package/dist/index.d.ts +30 -0
- package/dist/index.js +4284 -0
- package/dist/index.js.map +1 -0
- package/dist/migrate.d.ts +54 -0
- package/dist/migrations-dir.d.ts +7 -0
- package/dist/pool.d.ts +39 -0
- package/dist/sql/0001_cloud_local.sql +184 -0
- package/dist/sql/0002_core_domain.sql +416 -0
- package/dist/sql/0003_cloud_cleanup.sql +97 -0
- package/dist/sql/0004_device_proof_truth.sql +39 -0
- package/dist/sql/0005_skill_packs.sql +78 -0
- package/dist/sql/0006_device_presence_toolsets.sql +18 -0
- package/dist/stores/core/board.d.ts +38 -0
- package/dist/stores/core/index.d.ts +37 -0
- package/dist/stores/core/mailbox-sequence.d.ts +10 -0
- package/dist/stores/core/mailbox.d.ts +32 -0
- package/dist/stores/core/objects.d.ts +49 -0
- package/dist/stores/core/presence.d.ts +34 -0
- package/dist/stores/core/quota.d.ts +56 -0
- package/dist/stores/core/skill-pack.d.ts +41 -0
- package/dist/stores/core/truth.d.ts +32 -0
- package/dist/stores/dedup.d.ts +22 -0
- package/dist/stores/devices.d.ts +23 -0
- package/dist/stores/index.d.ts +55 -0
- package/dist/stores/nonces.d.ts +26 -0
- package/dist/stores/pairing-codes.d.ts +26 -0
- package/dist/stores/proof-receipts.d.ts +12 -0
- package/dist/stores/r2-blobs.d.ts +249 -0
- package/dist/stores/receipts.d.ts +25 -0
- package/dist/stores/task-attempts.d.ts +36 -0
- package/dist/truth-committer.d.ts +15 -0
- package/package.json +59 -0
|
@@ -0,0 +1,416 @@
|
|
|
1
|
+
-- 0002_core_domain.sql — the eleven core-domain port tables.
|
|
2
|
+
--
|
|
3
|
+
-- Frozen at merge, like 0001. Migrations are forward-only (sprint S4A.6): a
|
|
4
|
+
-- correction is a NEW file, never an edit to this one, and the runner
|
|
5
|
+
-- checksums every applied file against its ledger row
|
|
6
|
+
-- (packages/cloud-postgres/src/migrate.ts). 0001 is not touched here, by
|
|
7
|
+
-- construction and by a machine-checked zero diff.
|
|
8
|
+
--
|
|
9
|
+
-- Key design follows docs/researches/s4a-dataplane-design.md §5 and the
|
|
10
|
+
-- tenant-first discipline in docs/architecture/sdk-architecture.md §12.6.2.
|
|
11
|
+
-- The rule 0001 states and this file inherits:
|
|
12
|
+
--
|
|
13
|
+
-- EVERY unique index or constraint on a tenant-owned table starts with
|
|
14
|
+
-- tenant_id.
|
|
15
|
+
--
|
|
16
|
+
-- This file adds no exception to it. The two whitelisted ones both live in
|
|
17
|
+
-- 0001 (device.device_id, pairing_code.code) and both are single-step
|
|
18
|
+
-- pre-tenant resolutions of a cloud-minted credential.
|
|
19
|
+
-- tests/sql/control_plane_invariants.sql turns that rule into an executable
|
|
20
|
+
-- catalog assertion with the same two-entry whitelist.
|
|
21
|
+
--
|
|
22
|
+
-- ---------------------------------------------------------------------------
|
|
23
|
+
-- Two type decisions that apply to every table below
|
|
24
|
+
-- ---------------------------------------------------------------------------
|
|
25
|
+
--
|
|
26
|
+
-- 1. Canonical instants are `text`, not `timestamptz`. Every instant that
|
|
27
|
+
-- crosses a `@byok/core` port is a canonical ISO-8601 UTC string
|
|
28
|
+
-- (YYYY-MM-DDTHH:mm:ss.sssZ) and the contract compares them
|
|
29
|
+
-- lexicographically -- `packages/core/src/time.ts` exists to pin exactly
|
|
30
|
+
-- that and to REJECT anything else rather than normalize it. `text` stores
|
|
31
|
+
-- what the port produced, byte for byte, and `<` / `>=` over it is the same
|
|
32
|
+
-- comparison the in-memory reference performs. `timestamptz` would round
|
|
33
|
+
-- trip through Postgres' own serializer, so every read would owe a proof
|
|
34
|
+
-- that the canonical form survived, and a formatting divergence between the
|
|
35
|
+
-- two compositions would be silent rather than loud. The conformance
|
|
36
|
+
-- suite's `canonical instants` dimension asserts a composition's own output
|
|
37
|
+
-- feeds straight back in; `text` makes that unrepresentably true.
|
|
38
|
+
--
|
|
39
|
+
-- 0001 stores its instants as `timestamptz` on purpose and that stays
|
|
40
|
+
-- correct: those are cloud-local expiry fields with no cross-composition
|
|
41
|
+
-- string contract behind them.
|
|
42
|
+
--
|
|
43
|
+
-- 2. Byte counts are `bigint`. `byteSize`, `expectedBytes`, `hardLimitBytes`,
|
|
44
|
+
-- `releasedBytes` and the whole usage surface are declared `bigint` in
|
|
45
|
+
-- `@byok/core`, and the pool installs an int8 parser so they arrive as JS
|
|
46
|
+
-- `bigint` rather than strings (packages/cloud-postgres/src/pool.ts).
|
|
47
|
+
-- `integer` would put a silent 2^31 ceiling under a storage quota; a string
|
|
48
|
+
-- comparison would answer a different question than `>` without throwing.
|
|
49
|
+
--
|
|
50
|
+
-- Row counts the port types declare as `number` (`rev`, `refCount`, `dropped`,
|
|
51
|
+
-- `capacity`) stay `integer` for the same reason in reverse: they are small by
|
|
52
|
+
-- contract, and decoding them as `bigint` would force a cast at every boundary.
|
|
53
|
+
|
|
54
|
+
-- ---------------------------------------------------------------------------
|
|
55
|
+
-- device_stream.acked_at — the one column 0001 could not foresee
|
|
56
|
+
-- ---------------------------------------------------------------------------
|
|
57
|
+
--
|
|
58
|
+
-- 0001 created `device_stream.acked_seq` ahead of its consumer precisely so
|
|
59
|
+
-- this slice would not have to widen a frozen table. It did not create a
|
|
60
|
+
-- companion instant, and `MailboxCursorState` carries a required `updatedAt`
|
|
61
|
+
-- (packages/core/src/mailbox.ts). There are exactly three ways to supply it:
|
|
62
|
+
-- add the column here, invent a second per-device table that would leave
|
|
63
|
+
-- `device_stream.acked_seq` half-owned, or return the reader's own clock and
|
|
64
|
+
-- call a value that was never written "the instant the cursor moved".
|
|
65
|
+
--
|
|
66
|
+
-- The third is a fabricated fact, and the second contradicts the §5 mapping
|
|
67
|
+
-- that puts the mailbox cursor on `device_stream`. So: one additive, nullable
|
|
68
|
+
-- column. Additive in the forward-only sense -- 0001's bytes are untouched, and
|
|
69
|
+
-- a row written before this migration simply has no recorded ack instant until
|
|
70
|
+
-- its next `advanceCursor`.
|
|
71
|
+
ALTER TABLE device_stream ADD COLUMN acked_at text;
|
|
72
|
+
|
|
73
|
+
-- ---------------------------------------------------------------------------
|
|
74
|
+
-- outbox (core.mailbox) — the hosted mailbox rows
|
|
75
|
+
-- ---------------------------------------------------------------------------
|
|
76
|
+
--
|
|
77
|
+
-- `seq` is allocated from `device_stream.next_seq`, the same allocator
|
|
78
|
+
-- `cloud.sequence` uses, because the daemon's redelivery cursor IS that number
|
|
79
|
+
-- (0001's note on that table). Gaps are legal and expected: the allocator bumps
|
|
80
|
+
-- in its own statement, so a rejected append (a replayed `message_id`) burns a
|
|
81
|
+
-- number rather than reusing one. A reused number would make the cursor
|
|
82
|
+
-- ambiguous, which is the failure the allocator exists to prevent; a gap is
|
|
83
|
+
-- merely a gap.
|
|
84
|
+
--
|
|
85
|
+
-- `state` carries MAILBOX_MESSAGE_STATES. Deliberately no CHECK constraint
|
|
86
|
+
-- restating that list -- the port type is the vocabulary's single authority and
|
|
87
|
+
-- a copy here could drift from it silently (the same call 0001 makes for
|
|
88
|
+
-- `task.status`).
|
|
89
|
+
--
|
|
90
|
+
-- UNIQUE (tenant_id, device_id, message_id) is the producer-supplied
|
|
91
|
+
-- idempotency key from `MailboxAppendInput`. Tenant-first, so it is not an
|
|
92
|
+
-- exception to the rule above: a second append of the same envelope returns the
|
|
93
|
+
-- row that already exists instead of enqueuing a duplicate the device would
|
|
94
|
+
-- execute twice.
|
|
95
|
+
CREATE TABLE outbox (
|
|
96
|
+
tenant_id text NOT NULL,
|
|
97
|
+
device_id text NOT NULL,
|
|
98
|
+
seq bigint NOT NULL,
|
|
99
|
+
message_id text NOT NULL,
|
|
100
|
+
body text NOT NULL,
|
|
101
|
+
body_hash text NOT NULL,
|
|
102
|
+
byte_size bigint NOT NULL,
|
|
103
|
+
state text NOT NULL,
|
|
104
|
+
appended_at text NOT NULL,
|
|
105
|
+
PRIMARY KEY (tenant_id, device_id, seq),
|
|
106
|
+
CONSTRAINT outbox_tenant_device_message_key UNIQUE (tenant_id, device_id, message_id)
|
|
107
|
+
);
|
|
108
|
+
|
|
109
|
+
-- The retention sweep's index: `collectRetired` deletes acked rows older than
|
|
110
|
+
-- one cutoff and marks unacked rows older than another, both scoped to a tenant
|
|
111
|
+
-- and optionally a device. Not unique -- an index that made
|
|
112
|
+
-- (tenant, device, state, instant) unique would forbid two messages appended in
|
|
113
|
+
-- the same millisecond.
|
|
114
|
+
CREATE INDEX outbox_retention_idx
|
|
115
|
+
ON outbox (tenant_id, device_id, state, appended_at);
|
|
116
|
+
|
|
117
|
+
-- ---------------------------------------------------------------------------
|
|
118
|
+
-- tenant_stream (core.board) — the per-tenant board sequence
|
|
119
|
+
-- ---------------------------------------------------------------------------
|
|
120
|
+
--
|
|
121
|
+
-- `board_seq` is monotonic PER TENANT and bumps on every board mutation, which
|
|
122
|
+
-- is what makes `list({ afterSeq })` an incremental feed. A global sequence
|
|
123
|
+
-- would work mechanically and leak every other tenant's write rate through the
|
|
124
|
+
-- gaps, which is why §12.3 pins it per tenant.
|
|
125
|
+
--
|
|
126
|
+
-- Allocation is its own statement (`INSERT ... ON CONFLICT DO UPDATE SET
|
|
127
|
+
-- board_seq = board_seq + 1 RETURNING board_seq`), deliberately NOT a
|
|
128
|
+
-- data-modifying CTE inside the board write it feeds. Two concurrent claims
|
|
129
|
+
-- would then lock this row and `board_item` in an order the planner chooses,
|
|
130
|
+
-- and two sessions choosing opposite orders is a deadlock. Allocating first, in
|
|
131
|
+
-- an autocommitted statement that releases its lock immediately, gives every
|
|
132
|
+
-- writer the same lock order and no deadlock edge. The cost is that a rejected
|
|
133
|
+
-- board write burns a number -- the same trade `outbox.seq` makes, and for the
|
|
134
|
+
-- same reason: monotonic is the contract, gapless is not.
|
|
135
|
+
CREATE TABLE tenant_stream (
|
|
136
|
+
tenant_id text NOT NULL,
|
|
137
|
+
board_seq bigint NOT NULL DEFAULT 0,
|
|
138
|
+
PRIMARY KEY (tenant_id)
|
|
139
|
+
);
|
|
140
|
+
|
|
141
|
+
-- ---------------------------------------------------------------------------
|
|
142
|
+
-- board_item (core.board) — the coordination lifecycle
|
|
143
|
+
-- ---------------------------------------------------------------------------
|
|
144
|
+
--
|
|
145
|
+
-- `status` and `holder_id` are two columns, not one enum, because "where it is"
|
|
146
|
+
-- and "who holds it" change independently (§12.3). Every mutation is a
|
|
147
|
+
-- compare-and-set expressed as an `UPDATE ... WHERE` guard whose zero-row
|
|
148
|
+
-- result is the typed rejection; there is no last-write-wins path, and a loser
|
|
149
|
+
-- re-reads the row it lost to so it can re-decide against real state.
|
|
150
|
+
--
|
|
151
|
+
-- `held_since` is NULL exactly when `holder_id` is NULL: the pair is
|
|
152
|
+
-- `BoardAssignee`, and an unheld item has no assignee rather than an assignee
|
|
153
|
+
-- with an empty holder.
|
|
154
|
+
--
|
|
155
|
+
-- No CHECK on `status` -- BOARD_STATUSES and BOARD_TRANSITIONS in
|
|
156
|
+
-- packages/core/src/board.ts are the vocabulary's authority, and the legality
|
|
157
|
+
-- of a move is a two-value question (from, to) that a column constraint cannot
|
|
158
|
+
-- express anyway.
|
|
159
|
+
CREATE TABLE board_item (
|
|
160
|
+
tenant_id text NOT NULL,
|
|
161
|
+
item_id text NOT NULL,
|
|
162
|
+
channel text NOT NULL,
|
|
163
|
+
title text NOT NULL,
|
|
164
|
+
status text NOT NULL,
|
|
165
|
+
holder_id text,
|
|
166
|
+
held_since text,
|
|
167
|
+
board_seq bigint NOT NULL,
|
|
168
|
+
created_at text NOT NULL,
|
|
169
|
+
updated_at text NOT NULL,
|
|
170
|
+
PRIMARY KEY (tenant_id, item_id)
|
|
171
|
+
);
|
|
172
|
+
|
|
173
|
+
-- The incremental feed's ordering index. Not unique: nothing in the contract
|
|
174
|
+
-- promises two items never share a seq, and a unique index here would turn a
|
|
175
|
+
-- harmless allocation race into a write failure.
|
|
176
|
+
CREATE INDEX board_item_feed_idx ON board_item (tenant_id, board_seq);
|
|
177
|
+
|
|
178
|
+
-- ---------------------------------------------------------------------------
|
|
179
|
+
-- attested_record (core.truth) — two write models, one table
|
|
180
|
+
-- ---------------------------------------------------------------------------
|
|
181
|
+
--
|
|
182
|
+
-- `task.terminal` is first-write-wins and immutable; `profile`/`memory` are
|
|
183
|
+
-- per-key snapshots under an `expectedRev` CAS. Both are keyed
|
|
184
|
+
-- (tenant_id, kind, subject_id), so the terminal path's "first fact is never
|
|
185
|
+
-- overwritten" (§12.6.4) is enforced by the primary key itself:
|
|
186
|
+
-- `INSERT ... ON CONFLICT DO NOTHING` plus an equality re-read, never an upsert
|
|
187
|
+
-- that would restamp history while passing a naive "write twice" check.
|
|
188
|
+
--
|
|
189
|
+
-- `subject_id` is the port's `recordKey` -- the task id for a terminal, the
|
|
190
|
+
-- host's key for a snapshot. Named for what the column IS rather than for the
|
|
191
|
+
-- port's accessor, so `kind` reads as the discriminator it is.
|
|
192
|
+
--
|
|
193
|
+
-- The body is stored as a discriminated pair rather than as JSON: `body_kind`
|
|
194
|
+
-- is 'inline' or 'object', and exactly one of `body_inline` /
|
|
195
|
+
-- `body_object_hash` is populated. `TruthBodyRef` is a two-case union in the
|
|
196
|
+
-- contract, and a JSON blob here would let the store hold shapes the contract
|
|
197
|
+
-- cannot name.
|
|
198
|
+
CREATE TABLE attested_record (
|
|
199
|
+
tenant_id text NOT NULL,
|
|
200
|
+
kind text NOT NULL,
|
|
201
|
+
subject_id text NOT NULL,
|
|
202
|
+
rev integer NOT NULL,
|
|
203
|
+
content_hash text NOT NULL,
|
|
204
|
+
byte_size bigint NOT NULL,
|
|
205
|
+
body_kind text NOT NULL,
|
|
206
|
+
body_inline text,
|
|
207
|
+
body_object_hash text,
|
|
208
|
+
label text,
|
|
209
|
+
request_id text,
|
|
210
|
+
written_at text NOT NULL,
|
|
211
|
+
PRIMARY KEY (tenant_id, kind, subject_id)
|
|
212
|
+
);
|
|
213
|
+
|
|
214
|
+
-- ---------------------------------------------------------------------------
|
|
215
|
+
-- device_presence (core.presence) — lossy, TTL-bounded hints
|
|
216
|
+
-- ---------------------------------------------------------------------------
|
|
217
|
+
--
|
|
218
|
+
-- One row per device, overwritten on every publish: presence is a hint, not a
|
|
219
|
+
-- history, and §12.3 forbids deriving coordination, execution, authorization,
|
|
220
|
+
-- billing or recovery state from it.
|
|
221
|
+
--
|
|
222
|
+
-- `expires_at` is authoritative and expiry means ABSENCE. Reads filter on it
|
|
223
|
+
-- rather than deleting the row, so an expired hint is indistinguishable from
|
|
224
|
+
-- one that was never written -- and a read path stays a read path. The
|
|
225
|
+
-- in-memory reference drops the entry lazily on read; both produce the same
|
|
226
|
+
-- observable answer, and only one of them writes to satisfy a `SELECT`.
|
|
227
|
+
CREATE TABLE device_presence (
|
|
228
|
+
tenant_id text NOT NULL,
|
|
229
|
+
device_id text NOT NULL,
|
|
230
|
+
level text NOT NULL,
|
|
231
|
+
detail text,
|
|
232
|
+
observed_at text NOT NULL,
|
|
233
|
+
expires_at text NOT NULL,
|
|
234
|
+
PRIMARY KEY (tenant_id, device_id)
|
|
235
|
+
);
|
|
236
|
+
|
|
237
|
+
-- ---------------------------------------------------------------------------
|
|
238
|
+
-- activity_tail (core.activity) — a bounded, explicitly lossy tail
|
|
239
|
+
-- ---------------------------------------------------------------------------
|
|
240
|
+
--
|
|
241
|
+
-- The whole tail is one row. `entries` is `jsonb` because the tail is read and
|
|
242
|
+
-- written as a unit and is bounded by `capacity` -- a row-per-entry table would
|
|
243
|
+
-- buy per-entry queries the port does not expose and cost a trim on every
|
|
244
|
+
-- append.
|
|
245
|
+
--
|
|
246
|
+
-- `dropped` is the point of the design: lossiness is IN the data, so a reader
|
|
247
|
+
-- can tell "nothing happened" from "we lost the middle" instead of inferring it
|
|
248
|
+
-- from a gap it has to notice. The append is a single upsert that recomputes
|
|
249
|
+
-- both `entries` and `dropped` from the stored row; a concurrent second append
|
|
250
|
+
-- can lose an entry, which is within contract for a store §12.3 declares lossy
|
|
251
|
+
-- and non-authoritative, and is emphatically not within contract anywhere else
|
|
252
|
+
-- in this file.
|
|
253
|
+
CREATE TABLE activity_tail (
|
|
254
|
+
tenant_id text NOT NULL,
|
|
255
|
+
task_id text NOT NULL,
|
|
256
|
+
entries jsonb NOT NULL,
|
|
257
|
+
dropped integer NOT NULL,
|
|
258
|
+
capacity integer NOT NULL,
|
|
259
|
+
expires_at text NOT NULL,
|
|
260
|
+
PRIMARY KEY (tenant_id, task_id)
|
|
261
|
+
);
|
|
262
|
+
|
|
263
|
+
-- ---------------------------------------------------------------------------
|
|
264
|
+
-- object_manifest (core.objects) — metadata only, no bytes
|
|
265
|
+
-- ---------------------------------------------------------------------------
|
|
266
|
+
--
|
|
267
|
+
-- The manifest is the transaction authority; the object store holds the payload
|
|
268
|
+
-- (§12.7.4, §12.7.8). `pending` exists because Postgres and R2 have no shared
|
|
269
|
+
-- transaction: the row is written before the bytes land, and only `committed`
|
|
270
|
+
-- rows may be referenced. `delete_pending` is the tombstone the GC worker
|
|
271
|
+
-- drives, so a failed object-store delete is retryable instead of leaving usage
|
|
272
|
+
-- silently wrong.
|
|
273
|
+
--
|
|
274
|
+
-- PK (tenant_id, hash) -- never (hash). A global content-address key space
|
|
275
|
+
-- would deduplicate better and turn object existence into a cross-tenant
|
|
276
|
+
-- oracle, which is why §12.7.4 forbids it. Two tenants holding the same bytes
|
|
277
|
+
-- get two rows and are billed twice, on purpose.
|
|
278
|
+
--
|
|
279
|
+
-- `ref_count` is DERIVED, never incremented: every reference mutation
|
|
280
|
+
-- recomputes it as `count(*)` over `object_reference`. An increment can drift
|
|
281
|
+
-- -- a retried `addReference` inflates it and strands the object forever, which
|
|
282
|
+
-- is exactly the bug the "idempotent per (hash, refKind, refId)" contract
|
|
283
|
+
-- exists to prevent. A recomputation cannot.
|
|
284
|
+
CREATE TABLE object_manifest (
|
|
285
|
+
tenant_id text NOT NULL,
|
|
286
|
+
hash text NOT NULL,
|
|
287
|
+
byte_size bigint NOT NULL,
|
|
288
|
+
content_type text NOT NULL,
|
|
289
|
+
state text NOT NULL,
|
|
290
|
+
ref_count integer NOT NULL DEFAULT 0,
|
|
291
|
+
created_at text NOT NULL,
|
|
292
|
+
updated_at text NOT NULL,
|
|
293
|
+
delete_pending_at text,
|
|
294
|
+
PRIMARY KEY (tenant_id, hash)
|
|
295
|
+
);
|
|
296
|
+
|
|
297
|
+
-- The GC sweep's index: list by state, and by tombstone age within
|
|
298
|
+
-- `delete_pending`. Not unique.
|
|
299
|
+
CREATE INDEX object_manifest_state_idx
|
|
300
|
+
ON object_manifest (tenant_id, state, delete_pending_at);
|
|
301
|
+
|
|
302
|
+
-- ---------------------------------------------------------------------------
|
|
303
|
+
-- object_reference (core.objects) — what points at an object
|
|
304
|
+
-- ---------------------------------------------------------------------------
|
|
305
|
+
--
|
|
306
|
+
-- `ref_kind` / `ref_id` are opaque to core. The primary key IS the idempotency
|
|
307
|
+
-- contract: re-adding the same reference conflicts instead of double-counting,
|
|
308
|
+
-- which is what lets `ref_count` be a recomputation rather than a guess.
|
|
309
|
+
CREATE TABLE object_reference (
|
|
310
|
+
tenant_id text NOT NULL,
|
|
311
|
+
hash text NOT NULL,
|
|
312
|
+
ref_kind text NOT NULL,
|
|
313
|
+
ref_id text NOT NULL,
|
|
314
|
+
created_at text NOT NULL,
|
|
315
|
+
PRIMARY KEY (tenant_id, hash, ref_kind, ref_id)
|
|
316
|
+
);
|
|
317
|
+
|
|
318
|
+
-- ---------------------------------------------------------------------------
|
|
319
|
+
-- storage_entitlement (core.quota) — the host-issued numeric entitlement
|
|
320
|
+
-- ---------------------------------------------------------------------------
|
|
321
|
+
--
|
|
322
|
+
-- The SDK does not know what a plan is: no tier name, no price, no currency
|
|
323
|
+
-- (§12.7.6, and `packages/core/src/__tests__/constraints.test.ts` asserts the
|
|
324
|
+
-- same about the port source). What crosses this boundary is numbers plus a
|
|
325
|
+
-- monotonic `version` that is CAS-checked on write, so a delayed control-plane
|
|
326
|
+
-- update cannot resurrect an older plan over a newer one.
|
|
327
|
+
--
|
|
328
|
+
-- `downgrade_grace_until` is a canonical instant compared as a string, per the
|
|
329
|
+
-- type note at the top of this file. It is also the column that decides
|
|
330
|
+
-- `blocked` (507, over limit but inside grace) from `suspended` (423, over
|
|
331
|
+
-- limit and grace has ended), so a comparison that read it differently than the
|
|
332
|
+
-- in-memory reference does would change an HTTP status.
|
|
333
|
+
CREATE TABLE storage_entitlement (
|
|
334
|
+
tenant_id text NOT NULL,
|
|
335
|
+
version bigint NOT NULL,
|
|
336
|
+
hard_limit_bytes bigint NOT NULL,
|
|
337
|
+
max_object_bytes bigint NOT NULL,
|
|
338
|
+
max_inline_bytes bigint NOT NULL,
|
|
339
|
+
mailbox_limit_bytes bigint NOT NULL,
|
|
340
|
+
retention_policy_id text NOT NULL,
|
|
341
|
+
downgrade_grace_until text,
|
|
342
|
+
PRIMARY KEY (tenant_id)
|
|
343
|
+
);
|
|
344
|
+
|
|
345
|
+
-- ---------------------------------------------------------------------------
|
|
346
|
+
-- storage_usage (core.quota) — measured usage
|
|
347
|
+
-- ---------------------------------------------------------------------------
|
|
348
|
+
--
|
|
349
|
+
-- Note what is NOT here: `reserved_bytes`. `TenantStorageUsage.reservedBytes`
|
|
350
|
+
-- is DERIVED as `SUM(expected_bytes)` over this tenant's `reserved` rows in
|
|
351
|
+
-- `storage_reservation`. A stored counter has to be incremented on reserve and
|
|
352
|
+
-- decremented on every one of finalize / abort / expire, and any path that
|
|
353
|
+
-- settles a reservation without the matching decrement leaves a tenant
|
|
354
|
+
-- permanently short of quota it has actually released. Deriving it makes that
|
|
355
|
+
-- class of drift unrepresentable, and it makes settlement a pure state
|
|
356
|
+
-- transition on one row.
|
|
357
|
+
--
|
|
358
|
+
-- One row per entitled tenant, seeded by `writeEntitlement` in the same
|
|
359
|
+
-- statement that accepts the entitlement. The invariant that buys:
|
|
360
|
+
-- a `storage_usage` row exists whenever a `storage_entitlement` row does, so
|
|
361
|
+
-- every accounting write is an `UPDATE` with a guard rather than an upsert that
|
|
362
|
+
-- has to restate the guard twice. A tenant with no entitlement reads as all
|
|
363
|
+
-- zeros without a row being written for it.
|
|
364
|
+
CREATE TABLE storage_usage (
|
|
365
|
+
tenant_id text NOT NULL,
|
|
366
|
+
committed_object_bytes bigint NOT NULL DEFAULT 0,
|
|
367
|
+
committed_inline_bytes bigint NOT NULL DEFAULT 0,
|
|
368
|
+
mailbox_bytes bigint NOT NULL DEFAULT 0,
|
|
369
|
+
object_count bigint NOT NULL DEFAULT 0,
|
|
370
|
+
updated_at text NOT NULL,
|
|
371
|
+
PRIMARY KEY (tenant_id)
|
|
372
|
+
);
|
|
373
|
+
|
|
374
|
+
-- ---------------------------------------------------------------------------
|
|
375
|
+
-- storage_reservation (core.quota) — the no-oversell ledger
|
|
376
|
+
-- ---------------------------------------------------------------------------
|
|
377
|
+
--
|
|
378
|
+
-- Reservation exists because Postgres and the object store have no shared
|
|
379
|
+
-- transaction (§12.7.7): every durable write reserves first, uploads second,
|
|
380
|
+
-- finalizes third, and the invariant
|
|
381
|
+
-- `committed + reserved + expected <= hardLimitBytes` is checked at reserve
|
|
382
|
+
-- time. Two concurrent uploads must not both pass that check, which is why the
|
|
383
|
+
-- admission query runs behind a `FOR UPDATE` on the tenant's entitlement row --
|
|
384
|
+
-- see the header of `packages/cloud-postgres/src/stores/core/quota.ts` for why
|
|
385
|
+
-- a lone statement cannot get this right under READ COMMITTED.
|
|
386
|
+
--
|
|
387
|
+
-- `deduplicated` is stored rather than recomputed because
|
|
388
|
+
-- `StorageFinalizeResult.deduplicated` has to answer the same way when a
|
|
389
|
+
-- finalize is replayed against an already-committed reservation. Recomputing it
|
|
390
|
+
-- then would look at a set that now includes this very row and answer `true`
|
|
391
|
+
-- for the write that actually added the bytes.
|
|
392
|
+
--
|
|
393
|
+
-- `settled_at` is NULL exactly while `state = 'reserved'`.
|
|
394
|
+
CREATE TABLE storage_reservation (
|
|
395
|
+
tenant_id text NOT NULL,
|
|
396
|
+
reservation_id text NOT NULL,
|
|
397
|
+
state text NOT NULL,
|
|
398
|
+
kind text NOT NULL,
|
|
399
|
+
expected_bytes bigint NOT NULL,
|
|
400
|
+
content_hash text NOT NULL,
|
|
401
|
+
content_type text NOT NULL,
|
|
402
|
+
created_at text NOT NULL,
|
|
403
|
+
expires_at text NOT NULL,
|
|
404
|
+
settled_at text,
|
|
405
|
+
deduplicated boolean NOT NULL DEFAULT false,
|
|
406
|
+
PRIMARY KEY (tenant_id, reservation_id)
|
|
407
|
+
);
|
|
408
|
+
|
|
409
|
+
-- Two non-unique indexes, one per sweep the port performs: the live-reservation
|
|
410
|
+
-- sum and the TTL expiry both scan (tenant_id, state), and the per-tenant hash
|
|
411
|
+
-- deduplication check scans (tenant_id, content_hash, state).
|
|
412
|
+
CREATE INDEX storage_reservation_state_idx
|
|
413
|
+
ON storage_reservation (tenant_id, state, expires_at);
|
|
414
|
+
|
|
415
|
+
CREATE INDEX storage_reservation_hash_idx
|
|
416
|
+
ON storage_reservation (tenant_id, content_hash, state);
|
|
@@ -0,0 +1,97 @@
|
|
|
1
|
+
-- 0003_cloud_cleanup.sql — S4B-c retention and cross-system GC authority.
|
|
2
|
+
--
|
|
3
|
+
-- Forward-only and additive. 0001/0002 are immutable and checksum-locked by
|
|
4
|
+
-- the migration runner. This file adds no hash-verification state: ADR-024
|
|
5
|
+
-- remains authoritative, so cleanup may observe only key/existence/size/type.
|
|
6
|
+
--
|
|
7
|
+
-- Every table below is tenant-owned and every unique key starts with
|
|
8
|
+
-- tenant_id. tests/sql/control_plane_invariants.sql enforces that catalog rule.
|
|
9
|
+
|
|
10
|
+
-- Replayed mailbox rows retain the exact expired source sequence. The existing
|
|
11
|
+
-- tenant/device/message-id unique key remains the idempotency authority; this
|
|
12
|
+
-- provenance prevents equal bytes from two different dead letters aliasing the
|
|
13
|
+
-- same operator replay id. The source row remains independently discardable.
|
|
14
|
+
ALTER TABLE outbox
|
|
15
|
+
ADD COLUMN replay_source_seq bigint;
|
|
16
|
+
|
|
17
|
+
-- The host names a retention policy in storage_entitlement. Missing policy is
|
|
18
|
+
-- an operator error and cleanup fails closed; there is no hidden default that
|
|
19
|
+
-- can delete data under a window nobody selected.
|
|
20
|
+
CREATE TABLE tenant_retention_policy (
|
|
21
|
+
tenant_id text NOT NULL,
|
|
22
|
+
policy_id text NOT NULL,
|
|
23
|
+
mailbox_acked_retention_ms bigint NOT NULL,
|
|
24
|
+
mailbox_unacked_retention_ms bigint NOT NULL,
|
|
25
|
+
request_receipt_retention_ms bigint NOT NULL,
|
|
26
|
+
object_orphan_grace_ms bigint NOT NULL,
|
|
27
|
+
updated_at text NOT NULL,
|
|
28
|
+
PRIMARY KEY (tenant_id, policy_id),
|
|
29
|
+
CONSTRAINT tenant_retention_policy_nonnegative CHECK (
|
|
30
|
+
mailbox_acked_retention_ms >= 0
|
|
31
|
+
AND mailbox_unacked_retention_ms >= 0
|
|
32
|
+
AND request_receipt_retention_ms >= 0
|
|
33
|
+
AND object_orphan_grace_ms >= 0
|
|
34
|
+
)
|
|
35
|
+
);
|
|
36
|
+
|
|
37
|
+
-- One durable readback row per host-issued job id. The counters are the
|
|
38
|
+
-- provider-neutral metrics surface; a deployment may export them to its own
|
|
39
|
+
-- telemetry system without making that system the cleanup authority.
|
|
40
|
+
CREATE TABLE cleanup_job (
|
|
41
|
+
tenant_id text NOT NULL,
|
|
42
|
+
job_id text NOT NULL,
|
|
43
|
+
kind text NOT NULL,
|
|
44
|
+
state text NOT NULL,
|
|
45
|
+
started_at text NOT NULL,
|
|
46
|
+
finished_at text,
|
|
47
|
+
mailbox_deleted_count bigint NOT NULL DEFAULT 0,
|
|
48
|
+
mailbox_expired_count bigint NOT NULL DEFAULT 0,
|
|
49
|
+
mailbox_released_bytes bigint NOT NULL DEFAULT 0,
|
|
50
|
+
reservations_expired bigint NOT NULL DEFAULT 0,
|
|
51
|
+
ttl_rows_deleted bigint NOT NULL DEFAULT 0,
|
|
52
|
+
objects_tombstoned bigint NOT NULL DEFAULT 0,
|
|
53
|
+
objects_deleted bigint NOT NULL DEFAULT 0,
|
|
54
|
+
object_released_bytes bigint NOT NULL DEFAULT 0,
|
|
55
|
+
orphan_witnesses_created bigint NOT NULL DEFAULT 0,
|
|
56
|
+
missing_objects bigint NOT NULL DEFAULT 0,
|
|
57
|
+
shape_drift bigint NOT NULL DEFAULT 0,
|
|
58
|
+
invalid_object_keys bigint NOT NULL DEFAULT 0,
|
|
59
|
+
operation_errors bigint NOT NULL DEFAULT 0,
|
|
60
|
+
error_message text,
|
|
61
|
+
PRIMARY KEY (tenant_id, job_id)
|
|
62
|
+
);
|
|
63
|
+
|
|
64
|
+
CREATE INDEX cleanup_job_state_idx
|
|
65
|
+
ON cleanup_job (tenant_id, state, started_at);
|
|
66
|
+
|
|
67
|
+
-- Opaque cursor values only. R2 continuation tokens and manifest hashes are
|
|
68
|
+
-- not interchangeable, so cursor_kind is part of the tenant-first key.
|
|
69
|
+
CREATE TABLE gc_cursor (
|
|
70
|
+
tenant_id text NOT NULL,
|
|
71
|
+
cursor_kind text NOT NULL,
|
|
72
|
+
cursor_value text,
|
|
73
|
+
updated_at text NOT NULL,
|
|
74
|
+
PRIMARY KEY (tenant_id, cursor_kind)
|
|
75
|
+
);
|
|
76
|
+
|
|
77
|
+
-- `gc_accounted_bytes` records the accounting fact at the moment a manifest
|
|
78
|
+
-- becomes a tombstone: committed -> byte_size, pending -> 0. It stays NULL for
|
|
79
|
+
-- any legacy/manual delete_pending row whose origin is unknowable, which makes
|
|
80
|
+
-- the worker fail closed instead of guessing whether usage should be reduced.
|
|
81
|
+
-- It is metadata, not a fifth manifest state.
|
|
82
|
+
ALTER TABLE object_manifest
|
|
83
|
+
ADD COLUMN gc_accounted_bytes bigint;
|
|
84
|
+
|
|
85
|
+
ALTER TABLE object_manifest
|
|
86
|
+
ADD COLUMN gc_accounted_object boolean;
|
|
87
|
+
|
|
88
|
+
ALTER TABLE object_manifest
|
|
89
|
+
ADD CONSTRAINT object_manifest_gc_accounted_nonnegative
|
|
90
|
+
CHECK (gc_accounted_bytes IS NULL OR gc_accounted_bytes >= 0);
|
|
91
|
+
|
|
92
|
+
-- Candidate scan: tenant, state and age; the partial predicate keeps live
|
|
93
|
+
-- referenced objects out of the maintenance index. Reference rows are still
|
|
94
|
+
-- scanned again before tombstoning — ref_count alone is not the safety proof.
|
|
95
|
+
CREATE INDEX object_manifest_gc_candidate_idx
|
|
96
|
+
ON object_manifest (tenant_id, state, updated_at, hash)
|
|
97
|
+
WHERE ref_count = 0;
|
|
@@ -0,0 +1,39 @@
|
|
|
1
|
+
-- 0004_device_proof_truth.sql — S6 device proof key and replay authority.
|
|
2
|
+
--
|
|
3
|
+
-- Forward-only. Existing device_public_key rows are the identity proof key;
|
|
4
|
+
-- this migration projects that already-shipped fact into explicit key id and
|
|
5
|
+
-- epoch columns. Runtime verification never supplies a missing default.
|
|
6
|
+
|
|
7
|
+
ALTER TABLE device
|
|
8
|
+
ADD COLUMN proof_key_id text NOT NULL DEFAULT 'identity',
|
|
9
|
+
ADD COLUMN proof_key_epoch integer NOT NULL DEFAULT 0;
|
|
10
|
+
|
|
11
|
+
ALTER TABLE device
|
|
12
|
+
ALTER COLUMN proof_key_id DROP DEFAULT,
|
|
13
|
+
ALTER COLUMN proof_key_epoch DROP DEFAULT,
|
|
14
|
+
ADD CONSTRAINT device_proof_key_epoch_nonnegative CHECK (proof_key_epoch >= 0);
|
|
15
|
+
|
|
16
|
+
-- Dedicated request-bound result authority. The older
|
|
17
|
+
-- device_request_receipts table is the frozen protocol terminal seam and lacks
|
|
18
|
+
-- device/operation/resource/hash fields; widening or reinterpreting it would
|
|
19
|
+
-- create two meanings for one row shape.
|
|
20
|
+
CREATE TABLE proof_request_receipt (
|
|
21
|
+
tenant_id text NOT NULL,
|
|
22
|
+
device_id text NOT NULL,
|
|
23
|
+
request_id text NOT NULL,
|
|
24
|
+
operation text NOT NULL,
|
|
25
|
+
resource text NOT NULL,
|
|
26
|
+
body_sha256 text NOT NULL,
|
|
27
|
+
body_size bigint NOT NULL,
|
|
28
|
+
response_status integer NOT NULL,
|
|
29
|
+
response_body text NOT NULL,
|
|
30
|
+
recorded_at timestamptz NOT NULL,
|
|
31
|
+
PRIMARY KEY (tenant_id, device_id, request_id),
|
|
32
|
+
CONSTRAINT proof_request_receipt_body_size_nonnegative CHECK (body_size >= 0),
|
|
33
|
+
CONSTRAINT proof_request_receipt_status_range CHECK (
|
|
34
|
+
response_status >= 100 AND response_status <= 599
|
|
35
|
+
)
|
|
36
|
+
);
|
|
37
|
+
|
|
38
|
+
CREATE INDEX proof_request_receipt_recorded_idx
|
|
39
|
+
ON proof_request_receipt (tenant_id, recorded_at);
|
|
@@ -0,0 +1,78 @@
|
|
|
1
|
+
-- 0005_skill_packs.sql — the core.skillPacks port tables.
|
|
2
|
+
--
|
|
3
|
+
-- Forward-only, like every migration before it (sprint S4A.6): a correction is
|
|
4
|
+
-- a NEW file, never an edit to this one, and the runner checksums every applied
|
|
5
|
+
-- file against its ledger row (packages/cloud-postgres/src/migrate.ts). 0001
|
|
6
|
+
-- through 0004 are untouched here.
|
|
7
|
+
--
|
|
8
|
+
-- Phase 2 of the skill-pack-delivery-channel plan promotes `skillPacks` from a
|
|
9
|
+
-- bridge port to a mandatory `CoreStores` member, so every Postgres deployment
|
|
10
|
+
-- creates these tables. They are empty on a deployment that never declares the
|
|
11
|
+
-- `skills.pack` capability — storage presence is not capability advertisement.
|
|
12
|
+
--
|
|
13
|
+
-- The type discipline of 0002 applies:
|
|
14
|
+
--
|
|
15
|
+
-- EVERY unique index or constraint on a tenant-owned table starts with
|
|
16
|
+
-- tenant_id.
|
|
17
|
+
--
|
|
18
|
+
-- Both primary keys below start with tenant_id, and this file adds no exception
|
|
19
|
+
-- to the two-entry whitelist 0001 owns.
|
|
20
|
+
--
|
|
21
|
+
-- ---------------------------------------------------------------------------
|
|
22
|
+
-- skill_pack (core.skillPacks) — one row per published pack
|
|
23
|
+
-- ---------------------------------------------------------------------------
|
|
24
|
+
--
|
|
25
|
+
-- The pack-level fields a `SkillPackManifest` needs to be reconstructed on read:
|
|
26
|
+
-- `version` orders two publications of one name, `description` is the catalogue
|
|
27
|
+
-- text, and `content_hash` is the manifest's whole-pack content address (the
|
|
28
|
+
-- installer re-derives and verifies it; this store never hashes). The schema id
|
|
29
|
+
-- is a constant the store supplies from `SKILL_PACK_MANIFEST_SCHEMA_ID` rather
|
|
30
|
+
-- than a stored column, so it cannot drift per row.
|
|
31
|
+
--
|
|
32
|
+
-- PK (tenant_id, name): a pack name is unique within a tenant and a re-publish
|
|
33
|
+
-- under the same name replaces the pack, which is the store's idempotent upsert.
|
|
34
|
+
-- Never (name) alone — a global name space would turn pack existence into a
|
|
35
|
+
-- cross-tenant oracle, the same call object_manifest makes for content hashes.
|
|
36
|
+
--
|
|
37
|
+
-- No stored timestamp: the manifest carries none, so the store writes none and
|
|
38
|
+
-- reads the database clock nowhere.
|
|
39
|
+
CREATE TABLE skill_pack (
|
|
40
|
+
tenant_id text NOT NULL,
|
|
41
|
+
name text NOT NULL,
|
|
42
|
+
version text NOT NULL,
|
|
43
|
+
description text NOT NULL,
|
|
44
|
+
content_hash text NOT NULL,
|
|
45
|
+
PRIMARY KEY (tenant_id, name)
|
|
46
|
+
);
|
|
47
|
+
|
|
48
|
+
-- ---------------------------------------------------------------------------
|
|
49
|
+
-- skill_pack_file (core.skillPacks) — one row per declared file
|
|
50
|
+
-- ---------------------------------------------------------------------------
|
|
51
|
+
--
|
|
52
|
+
-- Each file the pack declares, with the metadata the manifest addresses it by
|
|
53
|
+
-- (`content_hash`, `byte_size`) and the UTF-8 text itself (`content`). A pack
|
|
54
|
+
-- carries Markdown, YAML and static text assets — never binaries, never
|
|
55
|
+
-- archives — so `content` is `text`, not `bytea`: it stores exactly the string
|
|
56
|
+
-- `SkillPackFileContent.content` is typed as and hands the wire handler back
|
|
57
|
+
-- byte for byte.
|
|
58
|
+
--
|
|
59
|
+
-- `byte_size` is `integer`, not `bigint`. `SkillPackFile.byteSize` is a core
|
|
60
|
+
-- `number` bounded by SKILL_PACK_FILE_MAX_BYTES (256 KiB), unlike the quota and
|
|
61
|
+
-- truth byte fields the contract declares as `bigint` (and that the pool's int8
|
|
62
|
+
-- parser therefore decodes as `BigInt`). `integer` decodes to a JS `number`
|
|
63
|
+
-- with no cast at the boundary — the same call 0002 makes for the small
|
|
64
|
+
-- integer row counts (`rev`, `ref_count`).
|
|
65
|
+
--
|
|
66
|
+
-- PK (tenant_id, pack_name, path): a path is unique within a pack, and the
|
|
67
|
+
-- reference store's publish rejects a duplicate path before it reaches here.
|
|
68
|
+
-- The publish replaces the whole file set in one transaction (DELETE then
|
|
69
|
+
-- INSERT), so a re-publish that drops a file leaves no orphan row behind.
|
|
70
|
+
CREATE TABLE skill_pack_file (
|
|
71
|
+
tenant_id text NOT NULL,
|
|
72
|
+
pack_name text NOT NULL,
|
|
73
|
+
path text NOT NULL,
|
|
74
|
+
content_hash text NOT NULL,
|
|
75
|
+
byte_size integer NOT NULL,
|
|
76
|
+
content text NOT NULL,
|
|
77
|
+
PRIMARY KEY (tenant_id, pack_name, path)
|
|
78
|
+
);
|
|
@@ -0,0 +1,18 @@
|
|
|
1
|
+
-- Device-local logical toolset inventory projected through presence.
|
|
2
|
+
--
|
|
3
|
+
-- Nullable is intentional: NULL means a legacy daemon or an inventory that
|
|
4
|
+
-- has not yet been observed, while [] means a current daemon explicitly
|
|
5
|
+
-- reports no configured toolsets. Executable definitions and credentials
|
|
6
|
+
-- never enter this table.
|
|
7
|
+
ALTER TABLE device_presence
|
|
8
|
+
ADD COLUMN configured_toolsets jsonb;
|
|
9
|
+
|
|
10
|
+
ALTER TABLE device_presence
|
|
11
|
+
ADD CONSTRAINT device_presence_configured_toolsets_shape
|
|
12
|
+
CHECK (
|
|
13
|
+
configured_toolsets IS NULL
|
|
14
|
+
OR (
|
|
15
|
+
jsonb_typeof(configured_toolsets) = 'array'
|
|
16
|
+
AND jsonb_array_length(configured_toolsets) <= 64
|
|
17
|
+
)
|
|
18
|
+
);
|