@framers/agentos-ext-account-sources 0.5.0 → 0.6.0

package/README.md

@@ -1,6 +1,6 @@
 # @framers/agentos-ext-account-sources
 
-A person's own accounts as sources. This library defines `AccountSourceConnector` and the `SourceRef` every passage carries, with helpers for line ranges, links, bounded body reads, requests under the shared public-sources limiter and the cache lifetime of a read, the contracts, pure rules and PostgreSQL store of a durable, resumable synchronization, and weighted request quotas that every worker shares.
+A person's own accounts as sources. This library defines `AccountSourceConnector` and the `SourceRef` every passage carries, with helpers for line ranges, links, bounded body reads, requests under the shared public-sources limiter and the cache lifetime of a read, the contracts, pure rules, PostgreSQL store and coordinator of a durable, resumable synchronization, and weighted request quotas that every worker shares.
 
 Install `@framers/agentos-ext-account-sources` with its peer `@framers/agentos` 0.13.134 or later. Node.js 20.19 or later is required. The runtime dependencies are `@framers/agentos-ext-public-sources` and `@framers/sql-storage-adapter`.
 
@@ -172,7 +172,7 @@ Pause is the `paused` flag on a connected or syncing connection, not a state: it
 
 ### Cycles and the cursor
 
-`openSourceSyncCycle(input)` opens a checkpoint at revision 0 and throws `invalid_source_sync_checkpoint` for an empty or oversized id or anchor, a generation or cutoff that is not a nonnegative safe integer, or an unknown mode. An `incremental` cycle starts with history from the committed cursor; `bootstrap`, `reconcile` and `recovery` start with a scan, from an anchor read before it. `advanceSourceSyncPage(checkpoint, page, policy)` answers the next checkpoint and the ids to queue, without duplicates:
+`openSourceSyncCycle(input)` opens a checkpoint at revision 0 and throws `invalid_source_sync_checkpoint` for an empty or oversized id or anchor, a generation or cutoff that is not a nonnegative safe integer, or an unknown mode. An `incremental` cycle starts with history from the committed cursor; `bootstrap`, `reconcile` and `recovery` start with a scan, and their history then replays from the checkpoint's `fromCursor`: an anchor read before the scan for `bootstrap` and `recovery`, the committed cursor for `reconcile` (see [The coordinator](#the-coordinator)). `advanceSourceSyncPage(checkpoint, page, policy)` answers the next checkpoint and the ids to queue, without duplicates:
 
 - A scan that reaches `maxScanPages` pages or `maxCandidates` ids with more left keeps the first ids in the order given, marks the checkpoint `bounded` and stops. A finished scan moves the cycle to history from the same anchor.
 - A history page records its cursor as the target. The last page, with no next page token, moves the cycle to `apply`.
@@ -220,7 +220,7 @@ The host supplies seven bindings. The store calls each inside its own transactio
 
 Provider reads, embedding and model calls run between store calls, never inside a binding.
 
-A sync holds a lease. `claim(scope, trigger)` answers a handle only when the owner is active, the `active` binding agrees, no barrier or live lease is set and `sourceSyncDue` holds; `renew(handle)` extends the lease by `leaseMs`; `release(handle, event?)` gives it up, records a deferral or a fixed failure when one is given, and keeps the checkpoint and work for the next claim. `openCycle` opens a checkpoint or answers the one already open, `page` records a page and queues its ids in one transaction (the same page sent again right after answers the same checkpoint; any other stale revision answers null), and `takeWork` marks the next due item attempted before the read. Every call made with a handle checks the owner fence, the lease token and expiry, the generation and the `active` binding in the same transaction; once any of them has moved on, `release(handle)` answers false and every other call answers null.
+A sync holds a lease. `claim(scope, trigger)` answers a handle only when the owner is active, the `active` binding agrees, no barrier or live lease is set and `sourceSyncDue` holds; `renew(handle)` extends the lease by `leaseMs`; `release(handle, event?)` gives it up, records a deferral or a fixed failure when one is given, and keeps the checkpoint and work for the next claim. `openCycle` opens a checkpoint or answers the one already open, `page` records a page and queues its ids in one transaction (the same page sent again right after answers the same checkpoint; any other stale revision answers null), and `takeWork` marks the next due item attempted before the read; an item that reached five attempts without a recorded result is not taken but waits `pollMs` as a `storage_unavailable` retry. Every call made with a handle checks the owner fence, the lease token and expiry, the generation and the `active` binding in the same transaction; once any of them has moved on, `release(handle)` answers false and every other call answers null.
 
 `applyResult(handle, work, result, prepared)` keeps a `retry` result unresolved. Any other result first removes the source's previous copy and every derived result that depends on it, through the fence. A `document` result then replaces it in the same transaction (host write, indexing charge and provenance) when the source is not excluded and fits: the store ranks this connection's retained sources and the new one by activity time, newest first with ties broken by source id, keeps those inside `maxRetainedDocuments`, `maxHeldCharacters` and the allowance, and removes this connection's other sources, never another connection's. A document that does not fit is recorded as skipped with `document_limit`, `held_limit` or `indexing_limit`, and an excluded one with `excluded`. A failing binding rolls the whole application back. The answer carries a new handle, because each removal advances the owner's fence; later calls use it. `finish` commits the cycle's target cursor once every work item is terminal, counting from the stored work, and `progress` records the counts so far.
 
@@ -259,24 +259,86 @@ async function readItem(url: string, attempt: number): Promise<Uint8Array | null
     }
     return { body: await readCapped(response, 2_000_000), retryAfter: null };
   });
-  if (result.kind !== 'completed') return null; // nothing was sent; a deferral says when to try again
+  if (result.kind !== 'completed') return null; // nothing was sent; a deferral or a late start says when to try again
   if (result.value.body === null) {
     await quota.defer(scope, sourceRetryAt(Date.now(), attempt, result.value.retryAfter));
   }
   return result.value.body;
 }
+
+// A scheduled job compacts the tables a page at a time, until nothing more is due.
+let compaction = await quota.compact();
+while (compaction.more) compaction = await quota.compact();
 ```
 
 `SourceQuota` admits provider requests against weighted budgets that every worker shares through PostgreSQL: a project's rolling minute (`projectMinute`), one mailbox's rolling minute (`mailboxMinute`) and a project's UTC day (`projectDay`). Each budget is a core `SqlSpendMeter` under a table prefix of its own (`<prefix>_pm`, `<prefix>_mm` and `<prefix>_pd`), and at most `projectConcurrency` requests of a project and `mailboxConcurrency` of one mailbox are in flight at once. The quota needs a PostgreSQL adapter with transactions and runs no DDL: the host's migration runs, each with `exec`, the seven scripts `sourceQuotaSchema(prefix)` answers, under a prefix of a lower-case letter and at most 27 lower-case letters, digits or underscores (`agentos_source_quota` unless given). Keys are opaque strings of 1 to 512 characters; a mailbox key stays the same for its mailbox and is never an address. The tables hold keys, counts and times, never content.
 
-`run(request, send, signal?)` decides in one transaction under the project's row lock. An attempt id seen before answers `{ kind: 'duplicate' }`, as does an attempt that cannot start its call within a second of its reservation. A recorded provider wait, a full concurrency limit or a budget without room for the method's weight answers `{ kind: 'deferred', reason, retryAt }`, with the wait's own time, the earliest expiry of the slots in the way, 60,001 milliseconds on for a minute budget, or the next UTC midnight for the day; a refusal by one budget rolls back what the others took. Otherwise the transaction consumes the weight in all three budgets and records the attempt's slot, and `run` calls `send` once, outside any transaction, with a signal that aborts after `timeoutMs` or with the caller's signal, then answers `{ kind: 'completed', value }`. Nothing is sent for a duplicate or a deferral. Units once consumed stay consumed: after a failed request, and after a provider wait recorded between the reservation and the call defers the request. A retry is a new `run` with a new attempt id, and pays again. `send` consumes or cancels the response body before it resolves, honors the signal and never retries. The slot frees when `send` settles or, for a process that dies mid-request, once a second to start, `timeoutMs` and `slotGraceMs` have passed; the attempt id stays refused either way.
+`run(request, send, signal?)` decides in one transaction under the project's row lock. An attempt id seen before answers `{ kind: 'duplicate' }`. A recorded provider wait, a full concurrency limit or a budget without room for the method's weight answers `{ kind: 'deferred', reason, retryAt }`, with the wait's own time, the earliest expiry of the slots in the way, 60,001 milliseconds on for a minute budget, or the next UTC midnight for the day; a refusal by one budget rolls back what the others took. Otherwise the transaction consumes the weight in all three budgets and records the attempt's slot, and a second transaction starts the call. An attempt that cannot start within a second of its reservation, after a long wait for the project's lock or a stalled process, answers `{ kind: 'late', retryAt }`: it is retryable with a new attempt id from `retryAt`, a second after the late start was found. Otherwise `run` calls `send` once, outside any transaction, with a signal that aborts after `timeoutMs` or with the caller's signal, then answers `{ kind: 'completed', value }`. Nothing is sent for a duplicate, a deferral or a late start. Units once consumed stay consumed: after a failed request, after a late start, and after a provider wait recorded between the reservation and the call defers the request. A retry is a new `run` with a new attempt id, and pays again. `send` consumes or cancels the response body before it resolves, honors the signal and never retries. The slot frees when `run` finishes, whether it sent or not, or, for a process that dies mid-request, once a second to start, `timeoutMs` and `slotGraceMs` have passed; the attempt id stays refused either way.
 
 `run` throws `source_sync_invalid_response` for a key outside those bounds or a method without a weight, `source_sync_cancelled` when the caller's signal aborted before the call or by the time `send` resolves, `source_sync_network` when `send` resolves after its timeout aborted it, and `source_sync_storage_unavailable` when the store cannot answer. Every worker on the same tables runs the same policy: the first one to use a project records its policy there, and a worker with another policy gets `source_sync_storage_unavailable` too. An error that `send` throws reaches the caller unchanged, unless the store then fails to free the slot.
 
-`defer(scope, retryAt)` records a provider's wait until `retryAt`, in epoch milliseconds, for one mailbox (`{ projectKey, mailboxKey }`) or the whole project (`{ projectKey }`). A shorter wait never replaces a longer one, and every worker honors it, across restarts. `compact()` deletes attempt records a day after their slots expired, mailbox waits that have passed, the meters' reservations settled more than a day ago and the minute budgets' rows idle for a day, at most 1,000 rows of each a call; the daily totals stay. Attempt ids must never repeat: once its record is compacted, an old id is no longer refused.
+`defer(scope, retryAt)` records a provider's wait until `retryAt`, in epoch milliseconds, for one mailbox (`{ projectKey, mailboxKey }`) or the whole project (`{ projectKey }`). A shorter wait never replaces a longer one, and every worker honors it, across restarts. `compact()` deletes attempt records a day after their slots expired, mailbox waits that have passed, the meters' reservations settled more than a day ago and the minute budgets' rows idle for a day, at most 1,000 rows of each table a call; the daily totals stay. It answers what it deleted as `{ attempts, mailboxWaits, reservations, windows, more }`: the rows of each kind, with the three meters' reservations counted together and the two minute budgets' rows together, and `more`, true when a table gave a full 1,000 rows. A host calls it on a schedule, and again while `more` is true. Attempt ids must never repeat: once its record is compacted, an old id is no longer refused.
 
 `sourceRetryAt(now, attempt, retryAfter, random?)` answers when a failed request may be tried again, for attempts 1 to 5: `now` plus one second doubled for each earlier attempt, with up to as much again added as jitter (`random`, from 0 to 1, `Math.random()` unless given), capped at 60 seconds. A `Retry-After` value, in delta seconds or an exact IMF-fixdate, wins when it is later, and is never capped; a value it cannot read is ignored. A `now` that is not a nonnegative safe integer, an attempt outside 1 to 5, a jitter outside 0 to 1 or a time past the latest date throws `source_sync_invalid_response`.
 
+### The coordinator
+
+```ts
+import {
+  AccountSourceSync, sourceSyncCoordinatorSchema, type SourceSyncScope,
+} from '@framers/agentos-ext-account-sources';
+
+// The host's migration, after sourceDerivationSchema() and sourceSyncSchema().
+for (const sql of sourceSyncCoordinatorSchema()) await db.exec(sql);
+
+const sync = new AccountSourceSync<PreparedPassages>({
+  store,     // the SqlAccountSourceSyncStore above
+  lifecycle, // the host's purge recovery and grant-loss handoff
+  // Builds the connector bound to this lease; it sends no request while building.
+  provider: async (currentHandle) => connectorFor(scope, currentHandle).sync,
+  // Embeds outside any transaction; the store commits the result with its document.
+  prepare: async (request, document) => embedPassages(document.read.content.text, request.signal),
+});
+
+const result = await sync.run(scope, 'scheduled');
+// result.kind: completed, idle, yielded, deferred, failed, reconnect, stale or unavailable
+
+// Maintenance, at least once a minute, for every connection.
+let after: SourceSyncScope | null = null;
+do after = (await sync.maintain(after, 100)).next; while (after !== null);
+```
+
+`AccountSourceSync` runs a connection's sync on the SQL store and keeps nothing of its own between calls: the checkpoint, the queued reads, the attempt counts and the retry times are in the store, so any coordinator on the same tables, in this process or another, resumes a run that yielded or stopped. `sourceSyncCoordinatorSchema(prefix)` answers one statement, which adds the nullable JSONB column `operation` to the connections table under the prefix given to `sourceSyncSchema`; run it after that schema, on a new database or an existing one. The store never runs it.
+
+The provider factory receives `currentHandle`, which answers the handle the run holds when it is called; the handle changes after each applied result. The factory builds a provider bound to the lease and sends no request. `prepare` turns a document into the host's prepared value, such as its passages and vectors, outside any transaction, and the store commits that value with the document. `lifecycle.recover(scope)` completes the host's committed purges; `lifecycle.loseGrant(handle, reason)` receives the run's exact handle, compares it atomically before it fences the grant, so a stale run cannot revoke a newer grant, and answers whether it acted. `runBudgetMs` (90,000 unless given, at most 90,000) bounds one run and `operationTimeoutMs` (30,000 unless given, at most 60,000) one provider call or preparation; another value throws `invalid_source_coordinator_policy`. `clock` and `random` (the retry jitter, from 0 to 1) default to `Date.now` and `Math.random`.
+
+`run(scope, trigger, signal?)` calls `lifecycle.recover` and the store's `recoverControl` for the scope, then claims the lease; a run that claims nothing answers `idle`. With no open cycle it opens one: `bootstrap` when no cursor is committed, `reconcile` once `reconcileMs` has passed since the last cycle that was not incremental, otherwise `incremental`. A bootstrap, a reconcile and a recovery call the provider's `begin` for an anchor before their scan. A bootstrap and a recovery replay history from that anchor. A reconcile replays it from the committed cursor: history ids only increase, so that replay also covers the changes made between the cursor and the anchor, and a change to an item the bounded scan does not list still reaches the store. An incremental cycle replays history from the committed cursor without a scan.
+
+The run drains the queued reads before it asks for the next page. Each item is read, unless it is excluded, which records it as skipped; a document is prepared, `applyResult` commits it, and the run goes on with the handle the application answers. When no read is due, a read waiting for its retry time ends the run as `deferred` until then; otherwise the run records the next page, and once the last history page is recorded and every item is terminal, `finish` commits the cursor and the run answers `completed`. A run yields, keeping its checkpoint and reads for the next claim, after `maxChangedPerLease` reads, after `maxHistoryPagesPerLease` history pages, when `runBudgetMs` has passed or when `signal` aborts; a signal aborted before the run starts answers `yielded` at once. It never waits out a retry.
+
+Before each provider call and each preparation the run renews its lease, and the call gets a signal that aborts after `operationTimeoutMs`, at the end of the run budget or a second before the lease expires, whichever comes first. A provider call cut off by that signal counts as a `network` failure and a preparation as `storage_unavailable`; one cut off by the caller's signal yields. The store refuses a result that arrives after the lease, the generation or the owner's fence has moved on, as after a disconnect, and the run then answers `stale`. A failed request is never an empty page, and a bounded scan never proves that an item it did not list is gone.
+
+Attempts are counted before each call, so a process that dies during a call has used that attempt. The anchor read and each page are counted in the connection's `operation` column, under a SHA-256 digest that names the operation, so a new generation, cycle, revision or phase starts a new count; the column holds no page token, cursor or provider id. Each read is counted in its work row. A `network`, `rate_limited`, `provider_unavailable` or `storage_unavailable` failure records a wait of `sourceRetryAt`: one second doubled for each earlier attempt, with up to as much again as jitter and at most 60 seconds, or the provider's own retry time when that is later. The fifth attempt's wait is at least `pollMs`, and the attempt after it starts a new count of five; five attempts that recorded nothing also wait `pollMs`. The connection's next sync is then that wait, which neither a restart nor a manual request shortens.
+
+A `grant_rejected` or `insufficient_scope` failure goes to `lifecycle.loseGrant` with the run's handle, and the run answers `reconnect`, or `stale` when `loseGrant` answers false. A `domain_policy` or `invalid_response` failure releases the lease with that failure, which clears the next scheduled sync until a manual request or another control sets one, and the run answers `failed`. An unknown exception from the provider becomes `invalid_response`, and one from preparation or storage `storage_unavailable`; no exception text reaches a result or an event. Expired history (`history_expired`) opens a `recovery` cycle with a fresh anchor and a new bounded scan; a second expiry in the same run defers the sync as `provider_unavailable` for `pollMs`. Retained items the scan does not list stay until they expire.
+
+| Result | When |
+|---|---|
+| `completed` | The cycle's cursor is committed; carries `examined`, `committed` and `bounded` |
+| `idle` | Nothing was claimed: the connection is not due, paused or held by another lease, or the host refused it |
+| `yielded` | A budget was spent or the signal aborted; the checkpoint and reads wait for the next claim |
+| `deferred` | A retriable failure or a pending retry; carries `reason` and `retryAt`, recorded as the next sync |
+| `failed` | `domain_policy` or `invalid_response`; no scheduled sync until a manual request or another control |
+| `reconnect` | The grant was refused and `loseGrant` fenced it |
+| `stale` | The lease, the generation or the owner's fence moved on; the run committed nothing more |
+| `unavailable` | The store could not be reached; no retry time was recorded, and a lease the run held expires on its own |
+
+`maintain(after?, limit?)` is a maintenance batch: for up to `limit` connections after `after`, in key order and in every state, paused, blocked and disconnected included, it calls `lifecycle.recover`, the store's `recoverControl` and `expire(scope, 100)`. A failure is counted in `failed` and the next connection still runs; no provider is called. It answers `scanned`, `expired`, `failed` and `next`: pass `next` as `after` to the next batch, and start again from null once it is null. Schedule it at least once a minute; reads refuse expired content whether or not it runs.
+
+With an `events` sink the coordinator reports `started` at the claim, `progress` with the count of retained sources after each read, `yielded` and `retry` with the reads made, `succeeded` with the count of retained sources, `reconnect`, and `purged` with the sources a maintenance batch expired. Each event carries `kind`, `state` and `n`, and `yielded`, `retry`, `succeeded` and `reconnect` also the run's duration as `ms`. A sink that throws is ignored.
+
+The coordinator uses three store methods besides those above. `reserveOperation(handle, key)` counts an attempt at the anchor read or a page before it is sent, and answers `{ kind: 'attempt', attempt }`, or `{ kind: 'deferred', reason, retryAt }` while a recorded wait has not passed or after five attempts that recorded nothing. `deferOperation(handle, key, attempt, reason, retryAt)` records the wait after a retriable failure when that attempt is still the one reserved. `pendingRetry(handle)` answers the earliest wait among the cycle's reads.
+
 ## License
 
 Apache-2.0. This pack carries a copy of the repository's license.

package/dist/index.d.ts

@@ -15,5 +15,8 @@ export * from './sync/state.js';
 export { sourceSyncSchema } from './sync/schema.js';
 export { SqlAccountSourceSyncStore, type SourceSyncStoredDocument, type SourceSyncAllowance, type SourceSyncStorageBindings, type SourceSyncStoreOptions, type SourceSyncHandle, type SourceSyncSnapshot, type SourceSyncControl, type SourceSyncApplication, type SourceSyncBarrier, } from './sync/SqlAccountSourceSyncStore.js';
 /** Weighted request budgets, request slots and provider retry times shared by every worker through SQL. */
-export { SourceQuota, sourceQuotaSchema, sourceRetryAt, type SourceQuotaMethod, type SourceQuotaPolicy, type SourceQuotaRequest, type SourceQuotaReason, type SourceQuotaResult, type SourceQuotaOptions, } from './sync/SourceQuota.js';
+export { SourceQuota, sourceQuotaSchema, sourceRetryAt, type SourceQuotaMethod, type SourceQuotaPolicy, type SourceQuotaRequest, type SourceQuotaReason, type SourceQuotaResult, type SourceQuotaOptions, type SourceQuotaCompaction, } from './sync/SourceQuota.js';
+/** The resumable synchronization coordinator, its lifecycle binding and the migration of its control state. */
+export { AccountSourceSync, type AccountSourceSyncOptions, type SourceSyncLifecycle, type SourceSyncRunResult, type SourceSyncMaintenanceResult, } from './sync/AccountSourceSync.js';
+export { sourceSyncCoordinatorSchema, type SourceSyncOperationAttempt, type SourceSyncOperationState, } from './sync/coordinatorState.js';
 //# sourceMappingURL=index.d.ts.map

package/dist/index.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"index.d.ts","sourceRoot":"","sources":["../src/index.ts"],"names":[],"mappings":"AAAA,4EAA4E;AAC5E,cAAc,YAAY,CAAC;AAC3B,yDAAyD;AACzD,cAAc,gBAAgB,CAAC;AAC/B,8DAA8D;AAC9D,cAAc,iBAAiB,CAAC;AAChC,oEAAoE;AACpE,cAAc,cAAc,CAAC;AAC7B,0EAA0E;AAC1E,cAAc,kBAAkB,CAAC;AACjC,yEAAyE;AACzE,cAAc,iBAAiB,CAAC;AAChC,cAAc,iBAAiB,CAAC;AAChC,wGAAwG;AACxG,OAAO,EAAE,gBAAgB,EAAE,MAAM,kBAAkB,CAAC;AACpD,OAAO,EACL,yBAAyB,EACzB,KAAK,wBAAwB,EAAE,KAAK,mBAAmB,EAAE,KAAK,yBAAyB,EACvF,KAAK,sBAAsB,EAAE,KAAK,gBAAgB,EAAE,KAAK,kBAAkB,EAC3E,KAAK,iBAAiB,EAAE,KAAK,qBAAqB,EAAE,KAAK,iBAAiB,GAC3E,MAAM,qCAAqC,CAAC;AAC7C,2GAA2G;AAC3G,OAAO,EACL,WAAW,EAAE,iBAAiB,EAAE,aAAa,EAC7C,KAAK,iBAAiB,EAAE,KAAK,iBAAiB,EAAE,KAAK,kBAAkB,EACvE,KAAK,iBAAiB,EAAE,KAAK,iBAAiB,EAAE,KAAK,kBAAkB,GACxE,MAAM,uBAAuB,CAAC"}
+{"version":3,"file":"index.d.ts","sourceRoot":"","sources":["../src/index.ts"],"names":[],"mappings":"AAAA,4EAA4E;AAC5E,cAAc,YAAY,CAAC;AAC3B,yDAAyD;AACzD,cAAc,gBAAgB,CAAC;AAC/B,8DAA8D;AAC9D,cAAc,iBAAiB,CAAC;AAChC,oEAAoE;AACpE,cAAc,cAAc,CAAC;AAC7B,0EAA0E;AAC1E,cAAc,kBAAkB,CAAC;AACjC,yEAAyE;AACzE,cAAc,iBAAiB,CAAC;AAChC,cAAc,iBAAiB,CAAC;AAChC,wGAAwG;AACxG,OAAO,EAAE,gBAAgB,EAAE,MAAM,kBAAkB,CAAC;AACpD,OAAO,EACL,yBAAyB,EACzB,KAAK,wBAAwB,EAAE,KAAK,mBAAmB,EAAE,KAAK,yBAAyB,EACvF,KAAK,sBAAsB,EAAE,KAAK,gBAAgB,EAAE,KAAK,kBAAkB,EAC3E,KAAK,iBAAiB,EAAE,KAAK,qBAAqB,EAAE,KAAK,iBAAiB,GAC3E,MAAM,qCAAqC,CAAC;AAC7C,2GAA2G;AAC3G,OAAO,EACL,WAAW,EAAE,iBAAiB,EAAE,aAAa,EAC7C,KAAK,iBAAiB,EAAE,KAAK,iBAAiB,EAAE,KAAK,kBAAkB,EACvE,KAAK,iBAAiB,EAAE,KAAK,iBAAiB,EAAE,KAAK,kBAAkB,EAAE,KAAK,qBAAqB,GACpG,MAAM,uBAAuB,CAAC;AAC/B,+GAA+G;AAC/G,OAAO,EACL,iBAAiB,EACjB,KAAK,wBAAwB,EAAE,KAAK,mBAAmB,EACvD,KAAK,mBAAmB,EAAE,KAAK,2BAA2B,GAC3D,MAAM,6BAA6B,CAAC;AACrC,OAAO,EACL,2BAA2B,EAC3B,KAAK,0BAA0B,EAAE,KAAK,wBAAwB,GAC/D,MAAM,4BAA4B,CAAC"}

package/dist/index.js

@@ -16,4 +16,7 @@ export { sourceSyncSchema } from './sync/schema.js';
 export { SqlAccountSourceSyncStore, } from './sync/SqlAccountSourceSyncStore.js';
 /** Weighted request budgets, request slots and provider retry times shared by every worker through SQL. */
 export { SourceQuota, sourceQuotaSchema, sourceRetryAt, } from './sync/SourceQuota.js';
+/** The resumable synchronization coordinator, its lifecycle binding and the migration of its control state. */
+export { AccountSourceSync, } from './sync/AccountSourceSync.js';
+export { sourceSyncCoordinatorSchema, } from './sync/coordinatorState.js';
 //# sourceMappingURL=index.js.map

package/dist/index.js.map

@@ -1 +1 @@
-{"version":3,"file":"index.js","sourceRoot":"","sources":["../src/index.ts"],"names":[],"mappings":"AAAA,4EAA4E;AAC5E,cAAc,YAAY,CAAC;AAC3B,yDAAyD;AACzD,cAAc,gBAAgB,CAAC;AAC/B,8DAA8D;AAC9D,cAAc,iBAAiB,CAAC;AAChC,oEAAoE;AACpE,cAAc,cAAc,CAAC;AAC7B,0EAA0E;AAC1E,cAAc,kBAAkB,CAAC;AACjC,yEAAyE;AACzE,cAAc,iBAAiB,CAAC;AAChC,cAAc,iBAAiB,CAAC;AAChC,wGAAwG;AACxG,OAAO,EAAE,gBAAgB,EAAE,MAAM,kBAAkB,CAAC;AACpD,OAAO,EACL,yBAAyB,GAI1B,MAAM,qCAAqC,CAAC;AAC7C,2GAA2G;AAC3G,OAAO,EACL,WAAW,EAAE,iBAAiB,EAAE,aAAa,GAG9C,MAAM,uBAAuB,CAAC"}
+{"version":3,"file":"index.js","sourceRoot":"","sources":["../src/index.ts"],"names":[],"mappings":"AAAA,4EAA4E;AAC5E,cAAc,YAAY,CAAC;AAC3B,yDAAyD;AACzD,cAAc,gBAAgB,CAAC;AAC/B,8DAA8D;AAC9D,cAAc,iBAAiB,CAAC;AAChC,oEAAoE;AACpE,cAAc,cAAc,CAAC;AAC7B,0EAA0E;AAC1E,cAAc,kBAAkB,CAAC;AACjC,yEAAyE;AACzE,cAAc,iBAAiB,CAAC;AAChC,cAAc,iBAAiB,CAAC;AAChC,wGAAwG;AACxG,OAAO,EAAE,gBAAgB,EAAE,MAAM,kBAAkB,CAAC;AACpD,OAAO,EACL,yBAAyB,GAI1B,MAAM,qCAAqC,CAAC;AAC7C,2GAA2G;AAC3G,OAAO,EACL,WAAW,EAAE,iBAAiB,EAAE,aAAa,GAG9C,MAAM,uBAAuB,CAAC;AAC/B,+GAA+G;AAC/G,OAAO,EACL,iBAAiB,GAGlB,MAAM,6BAA6B,CAAC;AACrC,OAAO,EACL,2BAA2B,GAE5B,MAAM,4BAA4B,CAAC"}

package/dist/sync/AccountSourceSync.d.ts

@@ -0,0 +1,136 @@
+/**
+ * @fileoverview The resumable synchronization coordinator for account sources: it runs one connection's sync under
+ * the SQL store's lease, from its durable checkpoint, through a supplied provider, with durable attempt counts and
+ * retry times, and runs expiry and recovery apart from provider polling.
+ * @module account-sources/sync/AccountSourceSync
+ */
+import type { AccountSourceSyncProvider, SourceSyncScope, SourceSyncTrigger, SourceSyncRequest, SourceSyncDocument, SourceSyncRetryReason, SourceSyncEventSink } from './types.js';
+import type { SqlAccountSourceSyncStore, SourceSyncHandle } from './SqlAccountSourceSyncStore.js';
+/**
+ * The host's lifecycle binding: recovery of its committed purges and the fenced handoff of a lost grant. The
+ * coordinator calls it; it never calls a provider for content.
+ */
+export interface SourceSyncLifecycle {
+    /** Recover committed host purge markers; active() remains false until recovery completes. */
+    recover(scope: SourceSyncScope): Promise<void>;
+    /**
+     * Atomically compare this handle before fencing. A stale request cannot revoke a fresh grant.
+     *
+     * @returns Whether the handle was current and the grant was fenced.
+     */
+    loseGrant(handle: SourceSyncHandle, reason: 'grant_rejected' | 'insufficient_scope'): Promise<boolean>;
+}
+/**
+ * What a coordinator runs with: the store, the provider factory, document preparation, the lifecycle binding and
+ * the optional clock, jitter, event sink and time budgets.
+ *
+ * @typeParam Prepared - The host's in-memory value for one document, passed to the store's `write` binding.
+ */
+export interface AccountSourceSyncOptions<Prepared> {
+    /** The SQL store that holds the lease, checkpoint, work, attempts and retained sources. */
+    readonly store: SqlAccountSourceSyncStore<Prepared>;
+    /**
+     * Memory-only factory bound to this lease/generation. No network call during construction.
+     *
+     * `currentHandle` answers the handle the run holds now: it changes after every applied result.
+     */
+    readonly provider: (currentHandle: () => SourceSyncHandle) => Promise<AccountSourceSyncProvider>;
+    /** Sanitized input only. No storage writes; output remains in memory until applyResult. */
+    readonly prepare: (request: SourceSyncRequest, document: SourceSyncDocument) => Promise<Prepared>;
+    /** The host's purge recovery and grant-loss handoff. */
+    readonly lifecycle: SourceSyncLifecycle;
+    /** Current time as integer UTC epoch milliseconds; `Date.now` unless given. */
+    readonly clock?: () => number;
+    /** Retry jitter from 0 to 1; `Math.random` unless given. */
+    readonly random?: () => number;
+    /** Receives content-free events; a sink that throws is ignored. */
+    readonly events?: SourceSyncEventSink;
+    /** How long one run may work before it yields, from 1 to 90,000 milliseconds; 90,000 unless given. */
+    readonly runBudgetMs?: number;
+    /** The longest one provider call or preparation may take, from 1 to 60,000 milliseconds; 30,000 unless given. */
+    readonly operationTimeoutMs?: number;
+}
+/**
+ * How a run ended: `completed` with its counts once the cursor is committed, `idle` when nothing was claimed,
+ * `yielded` with the checkpoint kept, `deferred` with the retry recorded, `failed` with a fixed reason, `reconnect`
+ * after a lost grant was fenced, `stale` when the lease or fence moved on, and `unavailable` when the store could not
+ * be reached and no retry was recorded.
+ */
+export type SourceSyncRunResult = {
+    readonly kind: 'idle' | 'yielded' | 'stale' | 'reconnect' | 'unavailable';
+} | {
+    readonly kind: 'completed';
+    readonly examined: number;
+    readonly committed: number;
+    readonly bounded: boolean;
+} | {
+    readonly kind: 'deferred';
+    readonly reason: SourceSyncRetryReason;
+    readonly retryAt: number;
+} | {
+    readonly kind: 'failed';
+    readonly reason: 'domain_policy' | 'invalid_response';
+};
+/**
+ * One maintenance batch: the connections visited, the sources expired, the connections whose maintenance failed, and
+ * the scope to pass as `after` to the next batch, null once the last batch is done.
+ */
+export interface SourceSyncMaintenanceResult {
+    readonly scanned: number;
+    readonly expired: number;
+    readonly failed: number;
+    readonly next: SourceSyncScope | null;
+}
+/**
+ * One lease at a time; all durable state and authorization remain in the SQL store.
+ *
+ * `run` performs one connection's sync from its durable checkpoint and yields within its budgets; any coordinator on
+ * the same tables resumes it. `maintain` runs purge recovery and expiry for every connection, apart from provider
+ * polling. The coordinator holds no state between calls.
+ *
+ * @typeParam Prepared - The host's in-memory value for one document, passed to the store's `write` binding.
+ */
+export declare class AccountSourceSync<Prepared = unknown> {
+    private readonly options;
+    private readonly store;
+    private readonly clock;
+    private readonly runBudget;
+    private readonly operationTimeout;
+    /**
+     * @param options - The store, provider factory, preparation, lifecycle binding and optional settings.
+     * @throws {Error} `invalid_source_coordinator_policy` unless `runBudgetMs` is an integer from 1 to 90,000 and
+     *   `operationTimeoutMs` one from 1 to 60,000.
+     */
+    constructor(options: AccountSourceSyncOptions<Prepared>);
+    private now;
+    private emit;
+    private stored;
+    private at;
+    /**
+     * Runs one connection's sync under a claimed lease. It recovers the host's purges and the store's controls, claims
+     * the lease, opens a cycle when none is open (`bootstrap` without a cursor, `reconcile` once `reconcileMs` has
+     * passed since the last non-incremental cycle, otherwise `incremental`), drains the queued reads before asking for
+     * the next page, and commits the cursor once every item is terminal. It yields with the checkpoint kept after the
+     * policy's changed-item or history-page budget, the run budget or an aborted signal, and never waits out a retry.
+     *
+     * @param scope - The connection.
+     * @param trigger - A scheduled poll or a person's request.
+     * @param signal - Aborting it ends the run with `yielded` and aborts the call in flight.
+     * @returns How the run ended; `unavailable` claims no recorded retry.
+     */
+    run(scope: SourceSyncScope, trigger: SourceSyncTrigger, signal?: AbortSignal): Promise<SourceSyncRunResult>;
+    /**
+     * A bounded scheduler batch. The host persists next between ticks to avoid starving high keys.
+     *
+     * Visits up to `limit` connections after `after` in key order, paused, blocked and disconnected ones included, and
+     * for each runs the lifecycle's recovery, the store's control recovery and the expiry of up to 100 sources. A
+     * failure is counted and the next connection still runs. No provider is called.
+     *
+     * @param after - The `next` of the previous batch, or null to start from the first connection.
+     * @param limit - At most this many connections, an integer from 1 to 1,000.
+     * @throws {SourceSyncFault} `storage_unavailable` when the connections cannot be listed, a `limit` outside that
+     *   range included.
+     */
+    maintain(after?: SourceSyncScope | null, limit?: number): Promise<SourceSyncMaintenanceResult>;
+}
+//# sourceMappingURL=AccountSourceSync.d.ts.map

package/dist/sync/AccountSourceSync.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"AccountSourceSync.d.ts","sourceRoot":"","sources":["../../src/sync/AccountSourceSync.ts"],"names":[],"mappings":"AAAA;;;;;GAKG;AAGH,OAAO,KAAK,EACV,yBAAyB,EAAE,eAAe,EAAE,iBAAiB,EAAE,iBAAiB,EAChF,kBAAkB,EACF,qBAAqB,EAAmB,mBAAmB,EAC5E,MAAM,YAAY,CAAC;AACpB,OAAO,KAAK,EACV,yBAAyB,EAAE,gBAAgB,EAC5C,MAAM,gCAAgC,CAAC;AAIxC;;;GAGG;AACH,MAAM,WAAW,mBAAmB;IAClC,6FAA6F;IAC7F,OAAO,CAAC,KAAK,EAAE,eAAe,GAAG,OAAO,CAAC,IAAI,CAAC,CAAC;IAC/C;;;;OAIG;IACH,SAAS,CAAC,MAAM,EAAE,gBAAgB,EAAE,MAAM,EAAE,gBAAgB,GAAG,oBAAoB,GAAG,OAAO,CAAC,OAAO,CAAC,CAAC;CACxG;AACD;;;;;GAKG;AACH,MAAM,WAAW,wBAAwB,CAAC,QAAQ;IAChD,2FAA2F;IAC3F,QAAQ,CAAC,KAAK,EAAE,yBAAyB,CAAC,QAAQ,CAAC,CAAC;IACpD;;;;OAIG;IACH,QAAQ,CAAC,QAAQ,EAAE,CAAC,aAAa,EAAE,MAAM,gBAAgB,KAAK,OAAO,CAAC,yBAAyB,CAAC,CAAC;IACjG,2FAA2F;IAC3F,QAAQ,CAAC,OAAO,EAAE,CAAC,OAAO,EAAE,iBAAiB,EAAE,QAAQ,EAAE,kBAAkB,KAAK,OAAO,CAAC,QAAQ,CAAC,CAAC;IAClG,wDAAwD;IACxD,QAAQ,CAAC,SAAS,EAAE,mBAAmB,CAAC;IACxC,+EAA+E;IAC/E,QAAQ,CAAC,KAAK,CAAC,EAAE,MAAM,MAAM,CAAC;IAC9B,4DAA4D;IAC5D,QAAQ,CAAC,MAAM,CAAC,EAAE,MAAM,MAAM,CAAC;IAC/B,mEAAmE;IACnE,QAAQ,CAAC,MAAM,CAAC,EAAE,mBAAmB,CAAC;IACtC,sGAAsG;IACtG,QAAQ,CAAC,WAAW,CAAC,EAAE,MAAM,CAAC;IAC9B,iHAAiH;IACjH,QAAQ,CAAC,kBAAkB,CAAC,EAAE,MAAM,CAAC;CACtC;AACD;;;;;GAKG;AACH,MAAM,MAAM,mBAAmB,GAC3B;IAAE,QAAQ,CAAC,IAAI,EAAE,MAAM,GAAG,SAAS,GAAG,OAAO,GAAG,WAAW,GAAG,aAAa,CAAA;CAAE,GAC7E;IAAE,QAAQ,CAAC,IAAI,EAAE,WAAW,CAAC;IAAC,QAAQ,CAAC,QAAQ,EAAE,MAAM,CAAC;IAAC,QAAQ,CAAC,SAAS,EAAE,MAAM,CAAC;IAAC,QAAQ,CAAC,OAAO,EAAE,OAAO,CAAA;CAAE,GAChH;IAAE,QAAQ,CAAC,IAAI,EAAE,UAAU,CAAC;IAAC,QAAQ,CAAC,MAAM,EAAE,qBAAqB,CAAC;IAAC,QAAQ,CAAC,OAAO,EAAE,MAAM,CAAA;CAAE,GAC/F;IAAE,QAAQ,CAAC,IAAI,EAAE,QAAQ,CAAC;IAAC,QAAQ,CAAC,MAAM,EAAE,eAAe,GAAG,kBAAkB,CAAA;CAAE,CAAC;AACvF;;;GAGG;AACH,MAAM,WAAW,2BAA2B;IAC1C,QAAQ,CAAC,OAAO,EAAE,MAAM,CAAC;IACzB,QAAQ,CAAC,OAAO,EAAE,MAAM,CAAC;IACzB,QAAQ,CAAC,MAAM,EAAE,MAAM,CAAC;IACxB,QAAQ,CAAC,IAAI,EAAE,eAAe,GAAG,IAAI,CAAC;CACvC;AAYD;;;;;;;;GAQG;AACH,qBAAa,iBAAiB,CAAC,QAAQ,GAAG,OAAO;IAUnC,OAAO,CAAC,QAAQ,CAAC,OAAO;IATpC,OAAO,CAAC,QAAQ,CAAC,KAAK,CAAsC;IAC5D,OAAO,CAAC,QAAQ,CAAC,KAAK,CAAe;IACrC,OAAO,CAAC,QAAQ,CAAC,SAAS,CAAS;IACnC,OAAO,CAAC,QAAQ,CAAC,gBAAgB,CAAS;IAC1C;;;;OAIG;gBAC0B,OAAO,EAAE,wBAAwB,CAAC,QAAQ,CAAC;IAOxE,OAAO,CAAC,GAAG;IAKX,OAAO,CAAC,IAAI;YAKE,MAAM;IAIpB,OAAO,CAAC,EAAE;IAMV;;;;;;;;;;;OAWG;IACG,GAAG,CAAC,KAAK,EAAE,eAAe,EAAE,OAAO,EAAE,iBAAiB,EAAE,MAAM,CAAC,EAAE,WAAW,GAAG,OAAO,CAAC,mBAAmB,CAAC;IAmMjH;;;;;;;;;;;OAWG;IACG,QAAQ,CAAC,KAAK,GAAE,eAAe,GAAG,IAAW,EAAE,KAAK,SAAM,GAAG,OAAO,CAAC,2BAA2B,CAAC;CAcxG"}