ulda-auth 1.0.1 → 1.1.0

package/docs/API.md

@@ -121,7 +121,7 @@ Creates a content resource and adds one encrypted entry to the master.
 { ok: true, name, subscriptionReady }
 ~~~
 
-The name must not already exist. The master can contain at most 63 content entries. Mode is fixed at creation.
+The name must not already exist. There is no fixed content-entry count limit in a master. Encrypted catalog bytes, operation bytes and the server's admission budgets still apply; see the [manifest](../MANIFEST.md#limits). Mode is fixed at creation.
 
 The operation reserves an encrypted empty resource, then sends one atomic Update for the content and master. The reservation is a separate commit. `subscriptionReady` is the transport's connection state at completion, not a permanent delivery guarantee.
 
@@ -264,3 +264,7 @@ PacketHandle has no public constructor. A plain object supplied to send is not a
 ## Limits
 
 See the [manifest](../MANIFEST.md) for numeric limits and the [security policy](../SECURITY.md) for trust and memory boundaries. Type declarations describe the API but do not replace runtime validation.
+
+## Capacity and compatibility
+
+See [Limits and capacity planning](LIMITS.md) for SDK bounds, interacting server budgets, eager-loading costs, atomicity, timeouts and upgrades from 1.0.x.

package/docs/GUIDE.md

@@ -1,5 +1,24 @@
 # Usage guide
 
+## Large catalogs
+
+A master can reference more than 63 contents without changing application calls. Opening or refreshing reads at most 64 content IDs per request, reducing batches if a response is too large. The SDK verifies every required record before publishing the combined local state. Separate batches are not one database snapshot across concurrent writes.
+
+Large catalogs need a server implementing the [transfer extension](PROTOCOL.md#large-operation-transfer). Large encrypted resources travel in 1 MiB fragments; a large `superDelete()` uploads all proofs before one atomic commit. Uploading a partial operation never deletes a partial catalog.
+
+For workloads whose server commit may exceed the default ten-second ACK deadline, configure a measured timeout, for example:
+
+~~~js
+const ulda = new Ulda(endpoint, { timeout: 60_000 });
+await ulda.connect(masterId, password);
+~~~
+
+This timeout is per transport request, not the time budget for loading the entire catalog. Automatic reconnect has its own overall budget. A timeout after Commit can leave an unknown outcome and never authorizes automatic write replay.
+
+There is no numeric catalog-entry quota in the client, but this is not unlimited storage: the encrypted catalog must fit 64 MiB, a logical transfer must fit 128 MiB including item accounting, and the server must have sufficient subscriptions, Router capacity and memory budgets. Application content retains its 1 MiB JSON limit. The client loads/decrypts all listed content eagerly, so opening and refreshing cost time and memory proportional to the data. Nested masters and lazy loading are not implemented.
+
+Upgrade the server and all clients using a shared catalog before exceeding the old limits. Existing containers keep their format; older clients retain their old catalog limits.
+
 ## 1. Connect to a compatible server
 
 ~~~js
@@ -279,3 +298,7 @@ await ulda.data.notes.done.update(true);
 ~~~
 
 The implementation remains JavaScript. Declarations describe the application's expected shape; they do not add server-side schema validation or validate arbitrary input against your TypeScript type.
+
+## Capacity and compatibility
+
+See [Limits and capacity planning](LIMITS.md) for SDK bounds, interacting server budgets, eager-loading costs, atomicity, timeouts and upgrades from 1.0.x.

package/docs/LIMITS.md

@@ -0,0 +1,102 @@
+# Limits and capacity planning
+
+This page describes ULDA Auth 1.1.0. A catalog has no fixed content-entry count cap. Operations still have byte, memory, queue and time limits. These limits protect both endpoints; they are not throughput guarantees.
+
+## SDK bounds
+
+| Boundary | Limit |
+|---|---|
+| Application JSON per content | 1,048,576 UTF-8 bytes; depth 48; 40,000 visited nodes |
+| Private catalog JSON | 67,104,768 UTF-8 bytes; depth 48 and a byte-derived node bound |
+| Encrypted resource after reassembly | 67,108,864 bytes (64 MiB) |
+| One logical transfer | 134,217,728 bytes (128 MiB), including 256 bytes of accounting per item |
+| Queued incoming binary data | 128 MiB |
+| Ordinary Read / mutation batch / transfer append | At most 64 items |
+| Ordinary message, combined witness and payload bytes | 8,388,608 bytes (8 MiB) |
+| Binary attachments per packet | 128; each at most 1,100,000 bytes |
+| Large-resource fragment | 1,048,576 bytes (1 MiB) |
+| Decoder text header | 65,536 UTF-8 bytes |
+| Queued incoming handlers | 8 |
+| Jobs per relevant outgoing queue | 128 |
+| Content name | 1–256 UTF-16 code units |
+| Master password input | 1–1024 UTF-16 code units |
+| Resource ID accepted by SDK | Positive decimal string of 1–40 digits, without leading zeroes |
+| Chain age | Unsigned 32-bit integer; no automatic chain rotation at exhaustion |
+
+Container overhead, witnesses and catalog metadata consume space. The application JSON maximum does not describe the size of its encrypted wire representation. Bounds are checked together: reaching one maximum does not imply that every other maximum remains available.
+
+The standard PostgreSQL server uses positive BIGINT IDs, whose maximum is `9223372036854775807`. The SDK's wider decimal-string validation does not expand that database range.
+
+These SDK bounds are currently implementation constants, not constructor options. Changing a server's limits alone does not change the SDK's bounds. Hello does not negotiate these budgets.
+
+## Reading a large catalog
+
+1. Read the master by ID. Reassemble large ciphertext when necessary, then authenticate and decrypt locally.
+2. Extract content IDs and private credentials locally. Names and credentials never appear as plaintext request fields.
+3. Read content in groups of at most 64 IDs. Each ordinary request has one ACK containing its result array. Large resources use the transfer extension.
+4. If the server explicitly rejects the size, reduce the read group. Temporary server saturation is a different error and does not trigger recursive splitting or automatic request replay.
+5. Verify every required result before replacing the visible local context. A bad final batch must not publish the earlier batches as a partially refreshed catalog.
+
+Reads subscribe the current connection to each resource. There is no fixed 63-content cap, no single request containing every ID regardless of size, and no durable server-side download cursor. A failed refresh preserves the last verified local context; the application handles the error and can explicitly refresh again. Separate Read batches are not one database snapshot: concurrent writers can change resources between reads.
+
+Loading is eager. All required content is held locally; there is no lazy-loading or nested-master API. Ten thousand contents of 1 MiB each already represent about 10 GiB of plaintext before temporary copies and encryption buffers. A catalog with many small entries and a catalog with the same number of large contents have very different memory costs.
+
+## Mutations and atomicity
+
+Changing one field re-encrypts the entire content. Adding, deleting or replacing a catalog link also re-encrypts the entire master. Catalog growth therefore increases the work and traffic of catalog mutations.
+
+Large atomic mutations are transmitted as `begin`, `append` / `part`, then `commit`. Staging does not write partial resource changes or hold an open SQL transaction between packets. The server checks the complete operation and commits all resource changes in one transaction. A failed operation must not commit a subset. An expired staging area must be sent again only as a new, explicitly initiated operation; an uncertain commit must not be replayed automatically.
+
+`add()` reserves a new ID in one Create, then commits the new content and master together in a second operation. The entire method is **not** one transaction. Interruption can leave an unreferenced resource; there is no automatic orphan garbage collector. `recoverContent()` similarly replaces one master link and intentionally preserves the old resource.
+
+`delete()` and `superDelete()` first refresh and verify the current required resources. Missing or damaged content can prevent deletion. A large `superDelete()` remains atomic, but must fit the server's total operation, lock and staging budgets. It is not a sequence of independently committed deletion chunks.
+
+X mode permits one pending write per resource. S mode prepares a bounded sequence of local steps: with N=5 the next age must remain less than 5 steps ahead of the confirmed age (at most 4 ahead). S does not merge changes from different clients. Conflicting writes require application handling.
+
+## Server budgets also apply
+
+The following are defaults of the companion single-process ULDA server implementing the transfer extension. Operators can configure server admission limits; confirm the actual deployment configuration. Consult that server's `docs/ADMISSION_LIMITS.md` for the complete list.
+
+| Boundary | Default / consequence |
+|---|---|
+| Connections | 1,000; configurable admission ceiling, not measured capacity |
+| Subscription edges | 100,000 globally; 32,768 per connection |
+| Router keys | 32,768 globally; at most 100 waiting jobs per key |
+| Incoming copied bytes | 64 MiB globally |
+| Active read reservations | 256 MiB globally |
+| Outbound bytes | 64 MiB globally; 16 MiB per connection |
+| Staging | 256 MiB globally; accounts for `3 × binary bytes + 256 × item count` |
+| One resource / logical operation | 64 MiB / 128 MiB |
+| Transfer lifetime | 60 seconds absolute by default; not renewed indefinitely by progress |
+| Active transfers | One staged upload and one large read per connection |
+| Waiting commands | 64 per connection |
+| Command rate | 1,000/s with burst 2,000 per decoder; not an authentication or per-IP quota |
+
+Examples of interacting limits:
+
+- A fresh connection needs one subscription for the master and one per content. With the default per-connection ceiling, 32,767 contents is the theoretical maximum for that one connection, before other constraints. This is a **server admission consequence**, not a new SDK catalog-count constant.
+- One hundred connections opening 10,000 contents each need over one million subscription edges, exceeding the default global 100,000 budget.
+- Deleting all resources requires Router capacity for the whole set. Existing unrelated work can reduce available capacity. Transmission in fragments does not remove this requirement.
+- Staging's copy accounting can reject an operation below the logical 128 MiB maximum. Those two maxima are not interchangeable.
+- A valid encrypted resource can exceed the per-connection outbound queue during Changed fan-out. The server can disconnect a slow or over-budget recipient; reconnect and a verified read recover its state. Changed is not a durable message log.
+- The server reserves read memory before database work. Temporary contention waits in a bounded FIFO (default 5 seconds). Exhaustion returns an error; the SDK does not turn saturation into more and smaller requests. A read that cannot fit at all is a size rejection.
+
+Subscriptions are looked up by resource and connection. The current server uses two Maps containing arrays; membership checks and removal scan the relevant list. This avoids scanning every connected client for every update, but fan-out and large subscriber lists still have a cost.
+
+## Timeouts and recovery
+
+The default SDK timeout is 10 seconds per awaited transport operation, configurable from 50 to 60,000 ms. It is not a deadline for the entire multi-request `connect()` or `superDelete()`. Increasing it does not increase server transfer lifetime, SQL deadlines or memory budgets.
+
+Automatic reconnect defaults to at most five attempts within 30 seconds, with delays from 500 to 5,000 ms and jitter. It rereads and verifies state; it does not replay writes. Initial connection failure is not automatically retried in the same way as a lost established connection.
+
+A lost or invalid write ACK can mean `UNKNOWN_OUTCOME`: the server may have committed. Volatile emission is not a delivery guarantee. A lost Create ACK has no built-in ID-recovery service. Save master IDs through the application and handle unknown outcomes explicitly; see [recovery](RECOVERY.md) and [API errors](API.md).
+
+Refresh/reconnect may accept a newer authenticated snapshot without proving every missed chain transition. The server's ID-only deletion notifications are trusted. Secrets are private to SDK internals by API design, but this does not protect them against malicious code controlling the same page or process. See [security](../SECURITY.md).
+
+## Compatibility and upgrades
+
+Version 1.1.0 preserves public method signatures, encrypted container format and the core dependency. It adds bounded multi-request reading and the large-operation transfer extension. A server must implement that extension for large resources or operations; upgrading the SDK alone is insufficient.
+
+Version 1.0.1 and earlier enforce the old small-catalog bound and cannot open catalogs that exceed it. Upgrade the server and **all clients** before allowing catalogs to grow beyond old limits. There is no automatic catalog sharding or conversion into nested masters.
+
+Capacity measurements apply to their stated hardware, payloads, concurrency and transport conditions. A successful small-content load test does not establish performance for large catalogs, registration KDFs, shared hot resources, high fan-out or WAN/TLS traffic.

package/docs/PIPELINES.md

@@ -142,7 +142,8 @@ Refresh does not lock the server. A concurrent writer can still cause conflict a
 ~~~text
 acquire exclusive guard -> refresh all listed resources
 prepare a next-witness deletion proof for each content and the master
-check guards -> one Delete batch
+check guards -> one logical Delete operation
+if larger than one frame: begin -> append proof groups -> commit
 server validates and deletes all or none
 on valid ACK: close SDK and release the context
 ~~~
@@ -152,6 +153,7 @@ A missing or invalid required resource stops preflight. The method cannot discov
 ## Incoming Changed
 
 ~~~text
+if fragmented: bounded begin/append/part/end assembly -> one complete event
 validate event shape and limits -> enqueue in the bounded incoming sequence
 stage known resource updates; authenticate/decrypt and verify lineage
 if master changed: validate its catalog and resolve newly referenced content by Read

package/docs/PROTOCOL.md

@@ -194,12 +194,45 @@ Live Changed requires valid lineage. Explicit refresh/reconnect may accept a new
 
 Oversized Read requests may be split into smaller arrays. Each reply must still match its request. The SDK publishes the combined snapshot only after all required records pass verification. Separate reads do not provide one global database snapshot across concurrent changes.
 
-Atomic Update/Delete batches are never split. The supported limits are in the [manifest](../MANIFEST.md).
+Atomic Update/Delete operations are never split into separate commits. Their transport frames may be split using the extension below. The supported limits are in the [manifest](../MANIFEST.md).
 
-No operation IDs, offline replay, ACL fields, CRDT merge or password-reset protocol are supplied by this contract. A compatible implementation in another language must preserve the byte format, validation rules, atomicity and failure semantics.
+No durable operation IDs, offline replay, ACL fields, CRDT merge or password-reset protocol are supplied by this contract. A compatible implementation in another language must preserve the byte format, validation rules, atomicity and failure semantics.
+
+## Large-operation transfer
+
+The transport advertises `auth: { uldaTransfer: 1 }` in the Socket.IO handshake. This is a framing capability, not authentication or a resource role. Ordinary CRUD shapes remain unchanged. A transfer-capable server is required once an operation exceeds those limits; older servers must not be assumed to support large catalogs.
+
+Content reads use arrays of at most 64 IDs, sequentially. For 300 entries this means five requests, not 300 individual reads. An explicit `too_large` response causes the SDK to bisect a read group. A missing or malformed reply is not treated as a successful partial result. There is no global database snapshot across these requests.
+
+All extension commands use the `ulda:transfer` event and one ACK callback:
+
+| Body | Success ACK / meaning |
+|---|---|
+| `{kind:'begin', action:'create'\|'update'\|'delete', count}` | `{ok:true, token}`; reserve an in-memory operation, no database mutation |
+| `{kind:'append', token, items}` | `{ok:true}`; append up to 64 ordinary action entries within 8 MiB |
+| `{kind:'part', token, resourceId?, witness, total, offset, payload}` | `{ok:true}`; append at most 1 MiB of one encrypted resource; omit resourceId only for Create |
+| `{kind:'commit', token}` | Original action's success ACK, only after the complete atomic commit |
+| `{kind:'cancel', token}` | `{ok:true}`; discard an uncommitted upload |
+| `{kind:'read', resourceId, offset}` | `{ok:true, data:{id,witness,payload,total,offset}}`; at most 1 MiB from a pinned encrypted snapshot |
+
+Tokens are random 32-character lowercase hexadecimal strings scoped to one connection. They are temporary staging references, not persisted resource metadata or idempotency keys. Each connection permits one staged write and one large read. All offsets and totals are byte counts. Parts for a resource are contiguous, starting at zero, with an unchanged ID, witness and total. No plaintext catalog names, credentials or origins are added.
+
+The server checks staging bytes and deadlines before retaining complete data. On Commit it holds every affected resource key, validates all candidate witnesses against current storage, then makes one atomic storage call. A conflict in the last validation group prevents every mutation. No database transaction is held open while the client uploads frames. A disconnect or deadline discards uncommitted staging; a Commit already accepted may finish after disconnect.
+
+The SDK never replays an uncertain mutation. A lost/malformed Commit ACK yields `UNKNOWN_OUTCOME`, even if storage committed. A pre-Commit timeout cannot itself commit staged bytes. The transport disconnects after an interrupted transfer to clear staging. A definite `too_large` rejection of an ordinary write may select staged framing; it is not a replay of an uncertain write.
+
+A singleton Read rejected as too large selects `kind:'read'`. Offset zero pins one bounded ciphertext snapshot; subsequent offsets continue that same version. The server releases it after the final fragment, expiry or disconnect. The SDK requires a complete matching ID/witness/total, then runs ordinary container and cryptographic validation before exposing anything.
+
+Large notifications use ordered `ulda:changed.part` frames without ACKs: `{kind:'begin',count}`, followed by `{kind:'append',items}` and/or `{kind:'part',resourceId,witness,total,offset,payload}`, then `{kind:'end'}`. Append items are updated `{id,witness,payload}` records or deleted `{id}` entries. The client reconstructs and validates the complete logical event before publishing it. Partial, oversized, out-of-order or timed-out assemblies cannot publish partial data.
+
+Servers use ordinary Changed whenever it fits. A legacy recipient that cannot decode a larger change is disconnected rather than left silently stale. Upgrade the server and every client sharing a catalog before growing it beyond the old limits. Existing encrypted records require no format migration; an older SDK still enforces its old catalog limits.
 
 ## Optional diagnostics
 
 A development server may emit `ulda:diagnostic` with `{ action, step, ids, count }`. This is outside CRUD and requires no ACK. The standard transport reads it only when configured with `diagnostics: true`.
 
 Only recognized action/step codes and bounded metadata are accepted. Invalid diagnostic input is ignored. Trace timestamps, request sequence numbers and durations are local diagnostics, not fields added to CRUD or server resource records.
+
+## Capacity and compatibility
+
+See [Limits and capacity planning](LIMITS.md) for SDK bounds, interacting server budgets, eager-loading costs, atomicity, timeouts and upgrades from 1.0.x.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "ulda-auth",
-  "version": "1.0.1",
+  "version": "1.1.0",
   "description": "Client SDK for encrypted JSON resources over ULDA and Socket.IO",
   "license": "SEE LICENSE IN LICENSE.md",
   "type": "module",
@@ -33,7 +33,8 @@
     "docs/RECOVERY.md",
     "docs/SECRET_LIFECYCLE.md",
     "docs/LICENSING.md",
-    "docs/GUIDE.md"
+    "docs/GUIDE.md",
+    "docs/LIMITS.md"
   ],
   "engines": {
     "node": ">=22"

package/src/client.js

@@ -45,6 +45,7 @@ export class Ulda {
   #sentCandidates = new Map();
   #inbound = Promise.resolve();
   #inboundCount = 0;
+  #inboundBytes = 0;
   #masterTail = Promise.resolve();
   #masterTasks = 0;
   #generation = 0;
@@ -88,10 +89,13 @@ export class Ulda {
     });
     this.#transportOff.push(this.#transport.on("changed", (event) => {
       const epoch = this.#connectionEpoch;
-      if (this.#inboundCount >= LIMITS.incoming)
+      const bytes = event.updated.reduce((n, x) => n + x.witness.length + x.payload.length + 256, event.deleted.length * 256);
+      if (this.#inboundCount >= LIMITS.incoming || this.#inboundBytes + bytes > LIMITS.transferBytes)
         return this.#fault(new UldaError("LIMIT"));
+      this.#inboundBytes += bytes;
       this.#serial(() => epoch === this.#connectionEpoch ? this.#changed(event) : undefined)
-        .catch((e) => { if (epoch === this.#connectionEpoch) this.#fault(e); });
+        .catch((e) => { if (epoch === this.#connectionEpoch) this.#fault(e); })
+        .finally(() => { this.#inboundBytes -= bytes; });
     }));
     this.#transportOff.push(this.#transport.on("disconnect", (event) => {
       this.#connectionEpoch++;
@@ -154,7 +158,7 @@ export class Ulda {
     if (!this.#listeners.get("trace")?.size) return;
     try {
       this.#emit("trace", Object.freeze({ time: Date.now(), phase: "client", event: step,
-        ...stepRecord(CLIENT_STEPS, step, ids, count) }));
+        ...stepRecord(CLIENT_STEPS, step, ids.slice(0, LIMITS.batch), count) }));
     } catch { /* Diagnostic callbacks never change operation results. */ }
   }
   async #openRecord(credentials, record, token, kind = "content") {
@@ -163,7 +167,7 @@ export class Ulda {
     // The role comes only from the client's decrypted catalog / chosen login ID.
     this.#trace(`${kind}.decrypt`, [record.id]);
     this.#guard(generation);
-    const result = await open(credentials, record, token);
+    const result = await open(credentials, record, token, kind === "master");
     try { this.#guard(generation); }
     catch (error) { result.origin.fill(0); throw error; }
     this.#trace(`${kind}.opened`, [record.id]);
@@ -250,7 +254,7 @@ export class Ulda {
   #catalog(data) {
     exact(data, ["contents"]);
     check(
-      Array.isArray(data.contents) && data.contents.length < LIMITS.resources,
+      Array.isArray(data.contents),
       "LIMIT",
     );
     const names = new Set(),
@@ -277,11 +281,12 @@ export class Ulda {
     const generation = this.#generation;
     this.#guard(generation);
     this.#trace("write.prepare", [base.id]);
-    const data = json(change(json(base.data)));
+    const catalog = base.id === this.#root?.id;
+    const data = json(change(json(base.data, catalog)), catalog);
     const origin = advance(base.origin);
     try {
       this.#trace("chain.advance", [base.id]);
-      const sealed = await seal(credentials, base.token, origin, data);
+      const sealed = await seal(credentials, base.token, origin, data, catalog);
       this.#guard(generation);
       this.#trace("write.sealed", [base.id]);
       const wire = sealedWire(sealed);
@@ -505,23 +510,29 @@ export class Ulda {
     const map = new Map();
     if (!entries.length) return map;
     this.#trace("content.read", entries.map(x => x.id), entries.length);
-    const records = await this.#readRecords(entries.map((x) => x.id), signal);
-    for (const entry of entries) {
-      checkSignal(signal);
-      const record = records.find((x) => x.id === entry.id);
-      const credentials = await contentCredentials(entry.key, envelope(record.payload).salt);
-      this.#trace("content.key", [entry.id]);
-      const state = {
-        id: entry.id,
-        ...(await this.#openRecord(credentials, record, entry.token)),
-      };
-      await this.#checkProgress(
-        this.#resources.get(entry.id)?.state,
-        state,
-        fresh,
-      );
-      map.set(entry.id, { state, credentials });
-      this.#trace("content.staged", [entry.id]);
+    // Read/decrypt bounded batches; only publish the complete verified map.
+    try {
+      for (let start = 0; start < entries.length; start += LIMITS.batch) {
+        const group = entries.slice(start, start + LIMITS.batch);
+        const records = new Map((await this.#readRecords(group.map(x => x.id), signal)).map(r => [r.id, r]));
+        for (const entry of group) {
+          checkSignal(signal);
+          const record = records.get(entry.id);
+          const credentials = await contentCredentials(entry.key, envelope(record.payload).salt);
+          this.#trace("content.key", [entry.id]);
+          const state = {
+            id: entry.id,
+            ...(await this.#openRecord(credentials, record, entry.token)),
+          };
+          try { await this.#checkProgress(this.#resources.get(entry.id)?.state, state, fresh); }
+          catch (error) { state.origin.fill(0); throw error; }
+          map.set(entry.id, { state, credentials });
+          this.#trace("content.staged", [entry.id]);
+        }
+      }
+    } catch (error) {
+      for (const resource of map.values()) resource.state.origin.fill(0);
+      throw error;
     }
     return map;
   }
@@ -616,7 +627,6 @@ export class Ulda {
         !this.#root.data.contents.some((x) => x.name === content),
         "EXISTS",
       );
-      check(this.#root.data.contents.length + 1 < LIMITS.resources, "LIMIT");
       let pending = this.#adds.get(content);
       if (pending)
         check(

package/src/crypto.js

@@ -103,7 +103,7 @@ function checkToken(token) {
 export const newContentSecret = randomSecret;
 export const newToken = randomSecret;
 export function envelope(payload) {
-  const bytes = binary(payload);
+  const bytes = binary(payload, LIMITS.resourceBytes);
   check(
     bytes.length >= 48 && equal(bytes.subarray(0, 4), MAGIC),
     "INVALID_PACKET",
@@ -116,7 +116,7 @@ export function envelope(payload) {
     header: bytes.subarray(0, 20),
   };
 }
-export async function seal(credentials, token, origin, data) {
+export async function seal(credentials, token, origin, data, catalog = false) {
   checkToken(token);
   const witness = await sign(origin);
   check(credentials.salt instanceof Uint8Array && credentials.salt.length === 16);
@@ -126,11 +126,11 @@ export async function seal(credentials, token, origin, data) {
     JSON.stringify({
       token,
       origin: b64(origin),
-      data: json(data),
+      data: json(data, catalog),
     }),
   );
   try {
-    check(plaintext.length <= LIMITS.payloadBytes - 48, "LIMIT");
+    check(plaintext.length <= (catalog ? LIMITS.resourceBytes : LIMITS.payloadBytes) - 48, "LIMIT");
     const cipher = new Uint8Array(
       await webcrypto().encrypt(
         {
@@ -165,7 +165,7 @@ export function sealedWire(handle) {
   const { witness, payload } = sealed.get(handle);
   return Object.freeze({ witness: witness.slice(), payload: payload.slice() });
 }
-export async function open(credentials, record, expectedToken) {
+export async function open(credentials, record, expectedToken, catalog = false) {
   let plaintext, origin, accepted = false;
   try {
     const witness = binary(record.witness, 167);
@@ -196,7 +196,7 @@ export async function open(credentials, record, expectedToken) {
     const result = {
       origin,
       token: decoded.token,
-      data: json(decoded.data),
+      data: json(decoded.data, catalog),
       witness,
       record: Object.freeze({
         id: record.id,

package/src/diagnostics.js

@@ -23,7 +23,7 @@ export const SERVER_STEPS = new Set([
 export function stepRecord(steps, step, resourceIds = [], count = 0) {
   check(steps.has(step), 'INVALID_PACKET');
   check(Array.isArray(resourceIds) && resourceIds.length <= LIMITS.batch, 'INVALID_PACKET');
-  check(Number.isSafeInteger(count) && count >= 0 && count <= LIMITS.batch, 'INVALID_PACKET');
+  check(Number.isSafeInteger(count) && count >= 0 && (steps === CLIENT_STEPS || count <= LIMITS.batch), 'INVALID_PACKET');
   return Object.freeze({ step, ids: Object.freeze(resourceIds.map(id)), count });
 }
 export function serverDiagnostic(raw) {

package/src/protocol.js

@@ -18,23 +18,23 @@ const put = (action, body) => {
   packets.set(p, { action, body });
   return p;
 };
-function array(items) {
+function array(items, extended = false) {
   check(
-    Array.isArray(items) && items.length > 0 && items.length <= LIMITS.batch,
+    Array.isArray(items) && items.length > 0 && items.length <= (extended ? Math.floor(LIMITS.transferBytes / 256) : LIMITS.batch),
     "INVALID_PACKET",
   );
 }
-function ids(items) {
-  array(items);
+function ids(items, extended = false) {
+  array(items, extended);
   items.forEach(id);
   check(new Set(items).size === items.length, "INVALID_PACKET");
 }
-export function messageSize(items) {
-  let size = 0;
+export function messageSize(items, extended = false) {
+  let size = extended ? items.length * 256 : 0;
   for (const item of items) {
     size += binary(item?.witness, 167).byteLength;
-    if (Object.hasOwn(item, "payload")) size += binary(item.payload).byteLength;
-    check(size <= LIMITS.messageBytes, "LIMIT");
+    if (Object.hasOwn(item, "payload")) size += binary(item.payload, extended ? LIMITS.resourceBytes : LIMITS.payloadBytes).byteLength;
+    check(size <= (extended ? LIMITS.transferBytes : LIMITS.messageBytes), "LIMIT");
   }
 }
 export function emptyPacket(action) {
@@ -47,8 +47,8 @@ export function readPacket(resourceIds) {
 }
 export function writePacket(action, items) {
   check(action === "create" || action === "update");
-  array(items);
-  if (action === "update") ids(items.map((x) => x.id));
+  array(items, true);
+  if (action === "update") ids(items.map((x) => x.id), true);
   const body = items.map((item) =>
     action === "create"
       ? { ...sealedWire(item.sealed) }
@@ -66,16 +66,16 @@ export async function deletionProof(resourceId, origin) {
   return handle;
 }
 export function deletePacket(proofs, updates = []) {
-  array(proofs);
+  array(proofs, true);
   const body = proofs.map((proof) => {
     check(deletionProofs.has(proof), "INVALID_PACKET");
     return { ...deletionProofs.get(proof) };
   });
   check(Array.isArray(updates), "INVALID_PACKET");
   for (const item of updates) body.push({ id: id(item.id), ...sealedWire(item.sealed) });
-  array(body);
-  messageSize(body);
-  ids(body.map((x) => x.id));
+  array(body, true);
+  messageSize(body, true);
+  ids(body.map((x) => x.id), true);
   return put("delete", body);
 }
 export function outgoing(packet) {
@@ -86,7 +86,7 @@ export function outgoing(packet) {
     typeof item === "string" ? item : {
       ...(item.id === undefined ? {} : { id: item.id }),
       witness: binary(item.witness, 167).slice(),
-      ...(item.payload === undefined ? {} : { payload: binary(item.payload).slice() }),
+      ...(item.payload === undefined ? {} : { payload: binary(item.payload, LIMITS.resourceBytes).slice() }),
     }) : {} };
 }
 export function resource(value) {
@@ -97,10 +97,10 @@ export function resource(value) {
   return Object.freeze({
     id: value.id,
     witness: binary(value.witness, 167).slice(),
-    payload: binary(value.payload).slice(),
+    payload: binary(value.payload, LIMITS.resourceBytes).slice(),
   });
 }
-export function requestShape(action, body) {
+export function requestShape(action, body, extended = false) {
   check(actions.has(action), "INVALID_PACKET");
   if (action === "hello" || action === "session.close") {
     exact(body, []);
@@ -110,13 +110,13 @@ export function requestShape(action, body) {
     ids(body);
     return;
   }
-  array(body);
-  if (action !== "delete") messageSize(body);
+  array(body, extended);
+  if (action !== "delete") messageSize(body, extended);
   else {
     check(body.some(item => !Object.hasOwn(item, "payload")), "INVALID_PACKET");
-    messageSize(body);
+    messageSize(body, extended);
   }
-  if (action !== "create") ids(body.map((x) => x.id));
+  if (action !== "create") ids(body.map((x) => x.id), extended);
   body.forEach((x) => {
     exact(
       x,
@@ -130,7 +130,7 @@ export function requestShape(action, body) {
     if (action !== "delete" || Object.hasOwn(x, "payload")) envelope(x.payload);
   });
 }
-export function acknowledgement(action, body, raw) {
+export function acknowledgement(action, body, raw, extended = false) {
   check(
     raw && typeof raw === "object" && !Array.isArray(raw),
     "INVALID_PACKET",
@@ -158,10 +158,10 @@ export function acknowledgement(action, body, raw) {
   check(raw.ok === true, "INVALID_PACKET");
   if (action === "read" || action === "create") {
     exact(raw, ["ok", "data"]);
-    array(raw.data);
+    array(raw.data, extended);
     check(raw.data.length === body.length, "INVALID_PACKET");
     if (action === "create") {
-      ids(raw.data);
+      ids(raw.data, extended);
       return [...raw.data];
     }
     messageSize(raw.data);
@@ -176,7 +176,7 @@ export function acknowledgement(action, body, raw) {
   exact(raw, ["ok"]);
   return undefined;
 }
-export function changed(raw) {
+export function changed(raw, extended = false) {
   check(raw && typeof raw === "object", "INVALID_PACKET");
   exact(
     raw,
@@ -187,10 +187,10 @@ export function changed(raw) {
   check(
     Array.isArray(inputUpdated) &&
       Array.isArray(inputDeleted) &&
-      inputUpdated.length + inputDeleted.length <= LIMITS.batch,
+      inputUpdated.length + inputDeleted.length <= (extended ? Math.floor(LIMITS.transferBytes / 256) : LIMITS.batch),
     "INVALID_PACKET",
   );
-  messageSize(inputUpdated);
+  messageSize(inputUpdated, extended);
   const updated = inputUpdated.map(resource),
     deleted = inputDeleted.map(id);
   const all = [...updated.map((x) => x.id), ...deleted];