@hachej/boring-files 0.2.0 → 0.2.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -10,6 +10,12 @@ Opaque resource view IDs are provider-scoped and bound to the actual backing/inc
10
10
 
11
11
  The provider accepts all-or-nothing create and replace. It refuses working views, per-change requests and deletes (files are removed with the workspace's own tools) before any effect. Logical paths are relative to the workspace root; a path whose real location leaves the root is denied. A single file is replaced atomically by every backend; a multi-file batch only by the SQLite backend, together with its receipt when the journal shares the database. Access control is the host's, at the workspace boundary: there are no per-file permissions inside a workspace, and external revocation fencing is not implemented.
12
12
 
13
+ ### Agent writes and the change feed
14
+
15
+ Agent writes are observed, never receipted ([owner ruling 2](../../docs/architecture/FILES-GIT-EXEC.md#one-place-for-files)). `provider.record({ path, before?, after, source: 'agent', conversationId? }, access)` is what the file guard (`@hachej/boring-agent/file-guard`) calls after a Pi `write` or `edit` succeeded: the file's history gains the written revision with `source: 'agent'` (and the replaced revision, `source: 'observed'`, when `before` carries its bytes), and the change feed an `agent` event. It never writes the publication journal and never mints a receipt. It does not enter the provider's queue (the guard already runs inside it), so calling it from queued work cannot deadlock. History entries (`saves`, `catalog.history`) carry `source`: `publish`, `agent` or `observed` (`keep` and replaced revisions).
16
+
17
+ `provider.changes({ since?, signal }, access)` is one change feed for every file view: an `AsyncIterable` of `{ kind: 'change', path, revision | null, source: 'publish' | 'agent' | 'poll' | 'watch', seq, at }`, ordered by `seq` and replayable from `since` (the last `seq` seen; `changeHead()` is "now"). The provider retains the last `changeBuffer` events (default 1024); a cursor older than that, from another provider lifetime or ahead of this one, and a listener that falls behind get `{ kind: 'resnapshot', seq, at }`: reload everything shown, then continue. Events are invalidation hints; bytes and the journal stay authoritative. Sources: conditional writes (`publish`), `record` (`agent`), `poll` and, while someone listens (and `watchLingerMs` after, default 30 s), the file system's own `watch` (`watch`; a reported directory is `{ revision: null, directory: true }`). The SQLite file system reports writes through its own connection at once and polls every second for other connections. When the file system cannot watch (the virtual and remote environments report `not_supported`), the host calls `provider.poll()` at the end of each agent turn: without paths it hashes every file (not `.git` or `node_modules`); the first call takes the baseline, later ones report changed, new and removed files. A `bash` write gets change events only, never history. After the last listener leaves the watcher lingers and then closes; changes made while no watcher was open are unknown, so the first listener afterwards makes the feed append a `resnapshot` (a client that stays connected is never affected). `changeIncarnation` names the provider lifetime: a cursor sent with another `incarnation` is a `resnapshot` whatever its number. `provider.close()` ends every change stream, closes the watcher and drops the listeners.
18
+
13
19
  Receipts are scoped by workspace identity, principal, scope and initiator. Identical committed retries return the original receipt; changed arguments conflict. Refused operations do not reserve an ID. A receipt from an earlier incarnation of the workspace reads `unknown`. A not-found lookup is not permission to replay an uncertain operation.
14
20
 
15
21
  `@hachej/boring-files/publication` is browser-safe. It captures request bytes, computes a versioned canonical SHA-256 digest with WebCrypto, and parses result shapes. Digests carry `boring-publication-v1:sha256:` and bind that format identifier in the hashed payload. Consumers must also check operation, target, scope and index association against their own request. This module confers no authorization. The SQLite connection (`openNodeConnection` from `@hachej/boring-files/sqlite`) is a separate server entry point.
@@ -138,7 +144,21 @@ const revisionProvider = await connectRevisionProvider({
138
144
 
139
145
  The binding discovers identity and workspace incarnation from the authenticated
140
146
  server. Reconnect when the selected workspace changes. It exposes `list`,
141
- filename `search`, retained `history`, `locate`, `read`, `publish` and `lookup`.
147
+ filename `search`, retained `history`, `locate`, `read`, `publish`, `lookup` and
148
+ `changes({ since?, signal })`, the workspace's change feed.
149
+
150
+ The feed is `GET <endpoint>?op=changes[&since=<seq>&incarnation=<id>]` with the binding header:
151
+ server-sent events (`event: ready` with the current seq and the provider incarnation, then `event: change` or
152
+ `event: resnapshot` with `id:` the seq and the JSON event as `data:`), and a
153
+ comment line every `heartbeatMs` (`createRevisionHandler({ resolve, heartbeatMs })`,
154
+ default 15 s). It is a plain streamed response with no queue (pull-based), so a slow client
155
+ costs the provider's bounded replay buffer and a `resnapshot`, never memory in
156
+ the handler; revocation discards anything unread. It runs on Node and Workers
157
+ alike. The handler holds the workspace lease for the life of the stream and
158
+ releases it when the client disconnects or its access signal aborts; a request
159
+ the host refuses, or whose binding names another workspace or identity, gets no
160
+ event (403 or 409). The client reconnects after the last seq it delivered, with
161
+ backoff, until its `signal` aborts; a refusal ends it with a `RevisionError`.
142
162
  All writes use conditional publication; uploading a new file requires absence,
143
163
  and replacing one requires its exact resource revision. An unknown result must
144
164
  be reconciled by operation ID before another upload to that path.
package/dist/journal.d.ts CHANGED
@@ -38,7 +38,13 @@ export interface StoredVersion {
38
38
  readonly mediaType: string;
39
39
  /** When this revision was saved, in milliseconds since the epoch. Kept the first time the revision is retained. */
40
40
  readonly savedAt?: number;
41
+ /**
42
+ * Who produced the revision: `publish` (a conditional write), `agent` (a Pi `write` or `edit`, recorded by `provider.record`: it
43
+ * has no receipt) or `observed` (retained as found: `keep`, or the replaced revision of a write). Default `publish`.
44
+ */
45
+ readonly source?: HistorySource;
41
46
  }
47
+ export type HistorySource = 'publish' | 'agent' | 'observed';
42
48
  /**
43
49
  * The operation journal plus what a workspace needs beyond receipts: an intent row written before an operation's first effect
44
50
  * (an intent without a completion is an operation of unknown outcome), and the last versions of each file when there is no Git.
@@ -68,6 +74,7 @@ export interface WorkspaceJournal extends OperationJournal {
68
74
  readonly saves: (scope: string, path: string) => {
69
75
  readonly revision: string;
70
76
  readonly savedAt: number;
77
+ readonly source: HistorySource;
71
78
  }[];
72
79
  }
73
80
  export declare function createWorkspaceJournal(db: SqliteConnection, options?: {
package/dist/journal.js CHANGED
@@ -59,6 +59,7 @@ export function committedChanges(changes, ref) {
59
59
  throw new Error('Stored operation has no changes');
60
60
  return [first, ...rest];
61
61
  }
62
+ const historySource = (value) => value === 'agent' || value === 'observed' ? value : 'publish';
62
63
  export function createWorkspaceJournal(db, options = {}) {
63
64
  const limit = options.historyLimit ?? 20;
64
65
  if (!Number.isSafeInteger(limit) || limit < 1)
@@ -74,18 +75,22 @@ export function createWorkspaceJournal(db, options = {}) {
74
75
  ) STRICT;
75
76
  CREATE TABLE IF NOT EXISTS boring_history (
76
77
  scope TEXT NOT NULL, path TEXT NOT NULL, revision TEXT NOT NULL, sequence INTEGER NOT NULL,
77
- bytes BLOB NOT NULL, media_type TEXT NOT NULL, saved_at INTEGER NOT NULL DEFAULT 0,
78
+ bytes BLOB NOT NULL, media_type TEXT NOT NULL, saved_at INTEGER NOT NULL DEFAULT 0, source TEXT NOT NULL DEFAULT 'publish',
78
79
  PRIMARY KEY (scope, path, revision)
79
80
  ) STRICT;
80
81
  `);
81
82
  // A journal created before save times were recorded gets the column; its old rows read as unknown (0).
82
83
  if (!db.all("SELECT name FROM pragma_table_info('boring_history')").some(column => column.name === 'saved_at'))
83
84
  db.exec('ALTER TABLE boring_history ADD COLUMN saved_at INTEGER NOT NULL DEFAULT 0');
85
+ // A journal created before history recorded who wrote each revision: its old rows were all conditional writes.
86
+ if (!db.all("SELECT name FROM pragma_table_info('boring_history')").some(column => column.name === 'source'))
87
+ db.exec("ALTER TABLE boring_history ADD COLUMN source TEXT NOT NULL DEFAULT 'publish'");
84
88
  });
85
89
  const remember = (scope, path, version) => {
86
90
  const last = db.get('SELECT COALESCE(MAX(sequence), 0) AS last FROM boring_history WHERE scope = ? AND path = ?', scope, path);
87
91
  const sequence = Number(last?.last ?? 0) + 1;
88
- db.run('INSERT INTO boring_history (scope, path, revision, sequence, bytes, media_type, saved_at) VALUES (?, ?, ?, ?, ?, ?, ?) ON CONFLICT(scope, path, revision) DO UPDATE SET sequence = excluded.sequence, bytes = excluded.bytes, media_type = excluded.media_type, saved_at = CASE WHEN boring_history.saved_at = 0 THEN excluded.saved_at ELSE boring_history.saved_at END', scope, path, version.revision, sequence, version.bytes, version.mediaType, version.savedAt ?? 0);
92
+ // A revision found again (`observed`) keeps the writer it was first recorded with; a new write of the same bytes names its writer.
93
+ db.run("INSERT INTO boring_history (scope, path, revision, sequence, bytes, media_type, saved_at, source) VALUES (?, ?, ?, ?, ?, ?, ?, ?) ON CONFLICT(scope, path, revision) DO UPDATE SET sequence = excluded.sequence, bytes = excluded.bytes, media_type = excluded.media_type, saved_at = CASE WHEN boring_history.saved_at = 0 THEN excluded.saved_at ELSE boring_history.saved_at END, source = CASE WHEN excluded.source = 'observed' THEN boring_history.source ELSE excluded.source END", scope, path, version.revision, sequence, version.bytes, version.mediaType, version.savedAt ?? 0, historySource(version.source));
89
94
  db.run('DELETE FROM boring_history WHERE scope = ? AND path = ? AND revision NOT IN (SELECT revision FROM boring_history WHERE scope = ? AND path = ? ORDER BY sequence DESC LIMIT ?)', scope, path, scope, path, limit);
90
95
  };
91
96
  const journal = {
@@ -112,7 +117,7 @@ export function createWorkspaceJournal(db, options = {}) {
112
117
  return { revision, bytes: Uint8Array.from(row.bytes), mediaType: text(row.media_type) };
113
118
  },
114
119
  revisions: (scope, path) => db.all('SELECT revision FROM boring_history WHERE scope = ? AND path = ? ORDER BY sequence DESC', scope, path).map(row => text(row.revision)),
115
- saves: (scope, path) => db.all('SELECT revision, saved_at FROM boring_history WHERE scope = ? AND path = ? ORDER BY sequence DESC', scope, path).map(row => ({ revision: text(row.revision), savedAt: Number(row.saved_at) })),
120
+ saves: (scope, path) => db.all('SELECT revision, saved_at, source FROM boring_history WHERE scope = ? AND path = ? ORDER BY sequence DESC', scope, path).map(row => ({ revision: text(row.revision), savedAt: Number(row.saved_at), source: historySource(row.source) })),
116
121
  };
117
122
  registerJournalConnection(journal, db);
118
123
  return journal;
@@ -24,6 +24,41 @@ export interface FileSearchRequest {
24
24
  export interface SavedRevision {
25
25
  readonly revision: string;
26
26
  readonly savedAt: number;
27
+ /** Who wrote it: `publish` (a conditional write), `agent` (a Pi `write` or `edit`: observed, never receipted) or `observed` (retained as found). */
28
+ readonly source?: 'publish' | 'agent' | 'observed';
29
+ }
30
+ /**
31
+ * One entry of a workspace's change feed (ruling 2 of 2026-10-10): an invalidation hint, never content or a receipt. Bytes and
32
+ * the publication journal stay authoritative; a viewer that receives one reads again.
33
+ *
34
+ * - `change`: something at `path` changed. `revision` is the Git blob id now there, or `null` when the path is absent or is a
35
+ * directory (`directory: true`: something below it may have changed). `source`: `publish` (a conditional write), `agent` (a Pi
36
+ * `write` or `edit`), `poll` (found by hashing, for example after a `bash` command) or `watch` (reported by the file system).
37
+ * - `resnapshot`: events were lost (the cursor is older than the retained window, from another provider lifetime, or the watcher
38
+ * overflowed or restarted). Reload everything shown, then continue after this `seq`.
39
+ *
40
+ * `seq` increases strictly within one provider lifetime; `at` is milliseconds since the epoch.
41
+ */
42
+ export type WorkspaceChangeEvent = {
43
+ readonly kind: 'change';
44
+ readonly path: string;
45
+ readonly revision: string | null;
46
+ readonly source: 'publish' | 'agent' | 'poll' | 'watch';
47
+ readonly seq: number;
48
+ readonly at: number;
49
+ readonly directory?: true;
50
+ readonly conversationId?: string;
51
+ } | {
52
+ readonly kind: 'resnapshot';
53
+ readonly seq: number;
54
+ readonly at: number;
55
+ };
56
+ export interface ChangeFeedRequest {
57
+ /** The last `seq` already seen: only later events are delivered (a gap is a `resnapshot`). Absent: live events only. */
58
+ readonly since?: number | undefined;
59
+ /** The provider lifetime `since` came from (`ready` event). A cursor from another lifetime is a `resnapshot`, whatever its number. */
60
+ readonly incarnation?: string | undefined;
61
+ readonly signal?: AbortSignal | undefined;
27
62
  }
28
63
  export interface WorkspaceCatalog {
29
64
  readonly list: (request: FileListRequest, access: ResourceAccess) => Promise<FilePage>;
@@ -44,5 +79,11 @@ export interface RevisionProvider extends ResourceClient {
44
79
  readonly list: (request: FileListRequest, signal?: AbortSignal) => Promise<FilePage>;
45
80
  readonly search: (request: FileSearchRequest, signal?: AbortSignal) => Promise<FilePage>;
46
81
  readonly history: (path: string, signal?: AbortSignal) => Promise<readonly SavedRevision[]>;
82
+ /**
83
+ * The workspace's change feed (`GET ?op=changes`, server-sent events), reconnecting after the last `seq` seen until `signal`
84
+ * aborts; a refused binding (another workspace, revoked access) ends it with a `RevisionError`. Optional: a hand-built provider
85
+ * may have none, and a viewer then refreshes on its own schedule.
86
+ */
87
+ readonly changes?: (request?: ChangeFeedRequest) => AsyncIterable<WorkspaceChangeEvent>;
47
88
  }
48
89
  //# sourceMappingURL=revision-contracts.d.ts.map
@@ -1,11 +1,14 @@
1
1
  import type { ResourceAccess } from './contracts.js';
2
2
  import type { WorkspaceResourceProvider } from './workspace.js';
3
3
  export interface RevisionHandlerOptions {
4
+ /** The host's authentication and workspace selection, once per request. For the change feed the lease is held until the stream ends. */
4
5
  readonly resolve: (request: Request) => Promise<null | {
5
6
  readonly provider: WorkspaceResourceProvider;
6
7
  readonly access: ResourceAccess;
7
8
  readonly release?: () => void | Promise<void>;
8
9
  }>;
10
+ /** How often the change feed sends a comment line so proxies keep it open, in milliseconds (default 15 000). */
11
+ readonly heartbeatMs?: number;
9
12
  }
10
13
  export declare function createRevisionHandler(options: RevisionHandlerOptions): (request: Request) => Promise<Response>;
11
14
  //# sourceMappingURL=revision-handler.d.ts.map
@@ -3,6 +3,90 @@ import { binding, bindingHeader, bindingValue, catalogHeader, filePage, record,
3
3
  import { guardStatus, readJsonBody, RequestGuardError } from './request-guard.js';
4
4
  import { accessSnapshot, identifier } from './publication-input.js';
5
5
  import { RevisionError } from './revision-contracts.js';
6
+ const encoder = new TextEncoder();
7
+ /**
8
+ * `GET ?op=changes[&since=<seq>&incarnation=<id>]`: the workspace's change feed as server-sent events (`event: change | resnapshot`,
9
+ * `id:` the seq, `data:` the JSON event), after an `event: ready` naming the current seq and the provider's incarnation, with a
10
+ * comment every `heartbeatMs` while idle. Plain streamed responses, so it runs on Node and on Workers alike.
11
+ *
12
+ * The stream is pull-based with no queue (`highWaterMark: 0`): the provider is read only while the client is reading, so a slow
13
+ * client costs the provider's bounded replay buffer (and a `resnapshot` when it falls behind), never unbounded memory here. The
14
+ * stream ends when the person disconnects or their access is revoked, discarding anything not yet read, and the workspace lease is
15
+ * released then.
16
+ */
17
+ function changeStream(provider, access, request, heartbeatMs, release) {
18
+ const stop = new AbortController();
19
+ const signal = AbortSignal.any([access.signal, stop.signal, provider.lifetime]);
20
+ let iterator;
21
+ let pending;
22
+ let ready = false, ended = false;
23
+ const finish = () => {
24
+ if (ended)
25
+ return;
26
+ ended = true;
27
+ stop.abort();
28
+ void iterator?.return?.().catch(() => { });
29
+ void release();
30
+ };
31
+ const body = new ReadableStream({
32
+ start: controller => {
33
+ const close = () => { finish(); try {
34
+ controller.close();
35
+ }
36
+ catch { /* already closed or cancelled */ } };
37
+ signal.addEventListener('abort', close, { once: true });
38
+ if (signal.aborted)
39
+ close();
40
+ },
41
+ pull: async (controller) => {
42
+ if (ended)
43
+ return;
44
+ const send = (text) => { try {
45
+ controller.enqueue(encoder.encode(text));
46
+ }
47
+ catch {
48
+ finish();
49
+ } };
50
+ const end = () => { finish(); try {
51
+ controller.close();
52
+ }
53
+ catch { /* already closed */ } };
54
+ try {
55
+ if (!ready) {
56
+ ready = true;
57
+ // Without a cursor the feed starts at the current seq, which the client learns first and resumes from after a disconnect.
58
+ const start = request.since ?? provider.changeHead();
59
+ iterator = provider.changes({ since: start, ...(request.incarnation === undefined ? {} : { incarnation: request.incarnation }), signal }, { ...access, signal })[Symbol.asyncIterator]();
60
+ // An async iterator runs its body only on the first pull, so prime it here: the watcher then starts before `ready`, and a
61
+ // write the client makes after reading `ready` is not missed.
62
+ pending = iterator.next();
63
+ pending.catch(() => { });
64
+ send(`event: ready\ndata: ${JSON.stringify({ seq: start, incarnation: provider.changeIncarnation })}\n\n`);
65
+ return;
66
+ }
67
+ pending ??= iterator.next();
68
+ pending.catch(() => { });
69
+ let timer;
70
+ const beat = new Promise(resolve => { timer = setTimeout(() => resolve('beat'), heartbeatMs); });
71
+ const outcome = await Promise.race([pending, beat]).finally(() => clearTimeout(timer));
72
+ if (signal.aborted)
73
+ return end();
74
+ if (outcome === 'beat')
75
+ return send(': heartbeat\n\n');
76
+ pending = undefined;
77
+ if (outcome.done)
78
+ return end();
79
+ const event = outcome.value;
80
+ send(`id: ${event.seq}\nevent: ${event.kind}\ndata: ${JSON.stringify(event)}\n\n`);
81
+ }
82
+ catch {
83
+ end(); /* the client reconnects from its last seq */
84
+ }
85
+ },
86
+ cancel: () => { finish(); },
87
+ }, { highWaterMark: 0 });
88
+ return new Response(body, { headers: { 'content-type': 'text/event-stream; charset=utf-8', 'cache-control': 'no-store', 'x-accel-buffering': 'no' } });
89
+ }
6
90
  function listRequest(value) {
7
91
  const data = record(value);
8
92
  if ((data.directory !== undefined && typeof data.directory !== 'string') || (data.cursor !== undefined && typeof data.cursor !== 'string') || (data.limit !== undefined && typeof data.limit !== 'number'))
@@ -10,6 +94,9 @@ function listRequest(value) {
10
94
  return { ...(typeof data.directory === 'string' ? { directory: data.directory } : {}), ...(typeof data.cursor === 'string' ? { cursor: data.cursor } : {}), ...(typeof data.limit === 'number' ? { limit: data.limit } : {}) };
11
95
  }
12
96
  export function createRevisionHandler(options) {
97
+ const heartbeatMs = options.heartbeatMs ?? 15_000;
98
+ if (!Number.isSafeInteger(heartbeatMs) || heartbeatMs < 1)
99
+ throw new RangeError('The heartbeat interval must be a positive integer');
13
100
  return async (request) => {
14
101
  let release;
15
102
  const operations = new Set();
@@ -34,7 +121,8 @@ export function createRevisionHandler(options) {
34
121
  return refuse(403);
35
122
  const association = binding({ identity: captured, workspace: selected.provider.identity });
36
123
  const respond = (value) => Response.json(value, { headers: { 'cache-control': 'no-store' } });
37
- if (request.method === 'GET')
124
+ const op = new URL(request.url).searchParams.get('op');
125
+ if (request.method === 'GET' && op === null)
38
126
  return respond(association);
39
127
  const header = request.headers.get(bindingHeader);
40
128
  if (!header || header.length > 16384)
@@ -49,6 +137,26 @@ export function createRevisionHandler(options) {
49
137
  if (requested !== bindingValue(association))
50
138
  return refuse(409);
51
139
  const provider = selected.provider;
140
+ if (request.method === 'GET') {
141
+ if (op !== 'changes')
142
+ return refuse(400);
143
+ const cursor = new URL(request.url).searchParams.get('since');
144
+ if (cursor !== null && !/^\d{1,16}$/.test(cursor))
145
+ return refuse(400);
146
+ const since = cursor === null ? undefined : Number(cursor);
147
+ const incarnation = new URL(request.url).searchParams.get('incarnation') ?? undefined;
148
+ if (incarnation !== undefined && !/^[\w-]{1,64}$/.test(incarnation))
149
+ return refuse(400);
150
+ if (since !== undefined && !Number.isSafeInteger(since))
151
+ return refuse(400);
152
+ // The stream owns the lease from here: it is released when the stream ends, not when this function returns.
153
+ const lease = release;
154
+ release = undefined;
155
+ return changeStream(provider, access, { since, incarnation }, heartbeatMs, async () => { try {
156
+ await lease?.();
157
+ }
158
+ catch { /* the stream already ended */ } });
159
+ }
52
160
  if (!request.headers.has(catalogHeader))
53
161
  return await createResourceHandler({
54
162
  authenticate: async () => access,
@@ -1,6 +1,6 @@
1
1
  import type { ResourceIdentity } from './remote-protocol.js';
2
2
  import type { WorkspaceKey } from './workspace.js';
3
- import type { FilePage, SavedRevision } from './revision-contracts.js';
3
+ import type { FilePage, SavedRevision, WorkspaceChangeEvent } from './revision-contracts.js';
4
4
  export declare const bindingHeader = "x-boring-workspace";
5
5
  export declare const catalogHeader = "x-boring-catalog";
6
6
  export interface RevisionBinding {
@@ -12,4 +12,11 @@ export declare function binding(value: unknown): RevisionBinding;
12
12
  export declare const bindingValue: (value: RevisionBinding) => string;
13
13
  export declare function filePage(value: unknown): FilePage;
14
14
  export declare function savedRevisions(value: unknown): readonly SavedRevision[];
15
+ /** One change-feed event as the server sent it, checked field by field (`data:` of `event: change | resnapshot`). */
16
+ export declare function changeEvent(value: unknown): WorkspaceChangeEvent;
17
+ /** Server-sent events from a response body: `{ event, data }` per message, comments skipped, a line longer than `limit` refused. */
18
+ export declare function serverEvents(body: ReadableStream<Uint8Array>, limit?: number): AsyncGenerator<{
19
+ readonly event: string;
20
+ readonly data: string;
21
+ }>;
15
22
  //# sourceMappingURL=revision-protocol.d.ts.map
@@ -37,6 +37,70 @@ export function savedRevisions(value) {
37
37
  const entry = record(item);
38
38
  if (typeof entry.savedAt !== 'number' || !Number.isSafeInteger(entry.savedAt) || entry.savedAt < 0)
39
39
  throw new RevisionError('invalid', 'Invalid save timestamp');
40
- return { revision: identifier(entry.revision), savedAt: entry.savedAt };
40
+ const source = entry.source === 'publish' || entry.source === 'agent' || entry.source === 'observed' ? entry.source : undefined;
41
+ return { revision: identifier(entry.revision), savedAt: entry.savedAt, ...(source ? { source } : {}) };
41
42
  });
42
43
  }
44
+ const sources = new Set(['publish', 'agent', 'poll', 'watch']);
45
+ const sequence = (value, what) => {
46
+ if (typeof value !== 'number' || !Number.isSafeInteger(value) || value < 0)
47
+ throw new RevisionError('invalid', `Invalid ${what}`);
48
+ return value;
49
+ };
50
+ /** One change-feed event as the server sent it, checked field by field (`data:` of `event: change | resnapshot`). */
51
+ export function changeEvent(value) {
52
+ const data = record(value);
53
+ const seq = sequence(data.seq, 'change sequence'), at = sequence(data.at, 'change time');
54
+ if (data.kind === 'resnapshot')
55
+ return { kind: 'resnapshot', seq, at };
56
+ if (data.kind !== 'change' || typeof data.path !== 'string' || typeof data.source !== 'string' || !sources.has(data.source))
57
+ throw new RevisionError('invalid', 'Invalid change event');
58
+ if (data.directory !== undefined && data.directory !== true)
59
+ throw new RevisionError('invalid', 'Invalid change event');
60
+ const path = data.path === '' && data.directory === true ? '' : catalogPath(data.path);
61
+ const revision = data.revision === null ? null : identifier(data.revision);
62
+ return {
63
+ kind: 'change', path, revision, source: data.source, seq, at,
64
+ ...(data.directory === true ? { directory: true } : {}),
65
+ ...(data.conversationId !== undefined ? { conversationId: identifier(data.conversationId) } : {}),
66
+ };
67
+ }
68
+ /** Server-sent events from a response body: `{ event, data }` per message, comments skipped, a line longer than `limit` refused. */
69
+ export async function* serverEvents(body, limit = 1_048_576) {
70
+ const reader = body.getReader(), decoder = new TextDecoder();
71
+ let pending = '', event = 'message', data = [];
72
+ try {
73
+ for (;;) {
74
+ const { done, value } = await reader.read();
75
+ if (done)
76
+ return;
77
+ pending += decoder.decode(value, { stream: true });
78
+ let newline;
79
+ while ((newline = pending.search(/\r?\n/)) !== -1) {
80
+ const line = pending.slice(0, newline);
81
+ pending = pending.slice(pending[newline] === '\r' ? newline + 2 : newline + 1);
82
+ if (line === '') {
83
+ if (data.length)
84
+ yield { event, data: data.join('\n') };
85
+ event = 'message';
86
+ data = [];
87
+ continue;
88
+ }
89
+ if (line.startsWith(':'))
90
+ continue;
91
+ const colon = line.indexOf(':');
92
+ const field = colon === -1 ? line : line.slice(0, colon);
93
+ const content = colon === -1 ? '' : line.slice(colon + 1).replace(/^ /, '');
94
+ if (field === 'event')
95
+ event = content;
96
+ else if (field === 'data')
97
+ data.push(content);
98
+ }
99
+ if (pending.length > limit)
100
+ throw new RevisionError('invalid', 'Change feed line exceeds the limit');
101
+ }
102
+ }
103
+ finally {
104
+ void reader.cancel().catch(() => { });
105
+ }
106
+ }
package/dist/revision.js CHANGED
@@ -1,5 +1,5 @@
1
1
  import { createResourceClient } from './remote.js';
2
- import { binding, bindingHeader, bindingValue, catalogHeader, filePage, record, savedRevisions } from './revision-protocol.js';
2
+ import { binding, bindingHeader, bindingValue, catalogHeader, changeEvent, filePage, record, savedRevisions, serverEvents } from './revision-protocol.js';
3
3
  import { readJsonBody } from './request-guard.js';
4
4
  import { RevisionError } from './revision-contracts.js';
5
5
  import { observe } from './remote-protocol.js';
@@ -45,6 +45,93 @@ export async function connectRevisionProvider(options) {
45
45
  throw new RevisionError('denied', 'The response belongs to another workspace');
46
46
  return result.value;
47
47
  }
48
+ /** One feed across reconnections: each resumes after the last seq delivered; a refused binding ends it. */
49
+ async function* changes(request = {}) {
50
+ const signal = request.signal;
51
+ let cursor = request.since, incarnation = request.incarnation, staged, delay = 500;
52
+ const pause = () => new Promise(resolve => {
53
+ const timer = setTimeout(done, delay);
54
+ function done() { clearTimeout(timer); signal?.removeEventListener('abort', done); resolve(); }
55
+ signal?.addEventListener('abort', done, { once: true });
56
+ delay = Math.min(delay * 2, 30_000);
57
+ });
58
+ while (!signal?.aborted) {
59
+ const url = new URL(endpoint);
60
+ url.searchParams.set('op', 'changes');
61
+ if (cursor !== undefined) {
62
+ url.searchParams.set('since', String(cursor));
63
+ if (incarnation !== undefined)
64
+ url.searchParams.set('incarnation', incarnation);
65
+ }
66
+ const connection = new AbortController();
67
+ const stop = () => connection.abort();
68
+ signal?.addEventListener('abort', stop, { once: true });
69
+ try {
70
+ let response;
71
+ try {
72
+ response = await fetch(new Request(url, { method: 'GET', redirect: 'error', headers: { accept: 'text/event-stream', [bindingHeader]: bound }, signal: connection.signal }));
73
+ }
74
+ catch {
75
+ if (signal?.aborted)
76
+ return;
77
+ await pause();
78
+ continue;
79
+ }
80
+ if (response.redirected || !response.ok || !response.body) {
81
+ void response.body?.cancel().catch(() => { });
82
+ if (response.status === 403 || response.status === 409)
83
+ throw new RevisionError('denied', 'The selected workspace is unavailable or no longer authorized');
84
+ if (response.status === 400)
85
+ throw new RevisionError('invalid', 'The change feed request was refused');
86
+ await pause();
87
+ continue;
88
+ }
89
+ try {
90
+ for await (const message of serverEvents(response.body)) {
91
+ if (message.event === 'ready') {
92
+ const ready = record(JSON.parse(message.data)), seq = ready.seq;
93
+ // A new provider lifetime is adopted with its resnapshot, not with `ready`: a disconnect in between must reconnect with the
94
+ // old cursor and incarnation, which the server answers with a resnapshot again.
95
+ if (typeof ready.incarnation === 'string') {
96
+ if (cursor === undefined || incarnation === undefined)
97
+ incarnation = ready.incarnation;
98
+ else
99
+ staged = ready.incarnation;
100
+ }
101
+ if (cursor === undefined && typeof seq === 'number' && Number.isSafeInteger(seq))
102
+ cursor = seq;
103
+ delay = 500;
104
+ continue;
105
+ }
106
+ if (message.event !== 'change' && message.event !== 'resnapshot')
107
+ continue;
108
+ const event = changeEvent(JSON.parse(message.data));
109
+ if (event.kind === 'change' && cursor !== undefined && event.seq <= cursor)
110
+ continue;
111
+ if (event.kind === 'resnapshot' && staged !== undefined) {
112
+ incarnation = staged;
113
+ staged = undefined;
114
+ }
115
+ cursor = event.seq;
116
+ yield event;
117
+ }
118
+ }
119
+ catch (error) {
120
+ if (signal?.aborted)
121
+ return;
122
+ if (error instanceof RevisionError && error.kind === 'invalid')
123
+ throw error;
124
+ }
125
+ }
126
+ finally {
127
+ signal?.removeEventListener('abort', stop);
128
+ connection.abort();
129
+ }
130
+ if (signal?.aborted)
131
+ return;
132
+ await pause();
133
+ }
134
+ }
48
135
  if (!client.publish || !client.lookup)
49
136
  throw new Error('Publishing client lacks reconciliation');
50
137
  return {
@@ -53,5 +140,6 @@ export async function connectRevisionProvider(options) {
53
140
  list: async (request, signal) => filePage(await invoke('list', request, signal)),
54
141
  search: async (request, signal) => filePage(await invoke('search', request, signal)),
55
142
  history: async (path, signal) => savedRevisions(await invoke('history', path, signal)),
143
+ changes: request => changes(request),
56
144
  };
57
145
  }
@@ -20,7 +20,10 @@ const within = (base, path) => path === base || path.startsWith(base === '/' ? '
20
20
  // system answers exactly like Pi's environments.
21
21
  const fileError = (code, message, path) => new FileError(code, message, path);
22
22
  const isFileError = (error) => error instanceof FileError;
23
- /** How often a watcher compares snapshots: SQLite reports no changes, so the watcher polls (`mode: 'polling'`). */
23
+ /**
24
+ * How often a watcher compares snapshots. A write through this file system (this connection) is compared at once, right after
25
+ * its transaction; polling remains for writes through another connection to the same database (`mode: 'polling'`).
26
+ */
24
27
  const WATCH_INTERVAL_MS = 1000;
25
28
  /** A FileError whose message starts with the POSIX code, as shells and Git read it. */
26
29
  const failure = (code, errno, path, what) => fileError(code, `${errno}: ${what}, '${path}'`, path);
@@ -65,6 +68,8 @@ export function openSqliteFileSystem(options) {
65
68
  if (found.kind !== 'directory')
66
69
  throw failure('not_directory', 'ENOTDIR', path, 'not a directory');
67
70
  }
71
+ /** Paths written, renamed or removed by this file system since the last comparison: reported as changed even when size and time look alike. */
72
+ const touched = new Set();
68
73
  function bytes(path) {
69
74
  if (existing(path).kind === 'directory')
70
75
  throw failure('is_directory', 'EISDIR', path, 'illegal operation on a directory');
@@ -74,6 +79,7 @@ export function openSqliteFileSystem(options) {
74
79
  return Uint8Array.from(found.bytes);
75
80
  }
76
81
  function put(path, kind, content) {
82
+ touched.add(path);
77
83
  db.run('INSERT INTO boring_workspace_files (workspace, path, parent, kind, bytes, mtime_ms) VALUES (?, ?, ?, ?, ?, ?) ON CONFLICT(workspace, path) DO UPDATE SET kind = excluded.kind, bytes = excluded.bytes, mtime_ms = excluded.mtime_ms', workspace, path, dirname(path), kind, content, now());
78
84
  }
79
85
  function write(path, content) {
@@ -125,13 +131,31 @@ export function openSqliteFileSystem(options) {
125
131
  const incarnation = db.get('SELECT incarnation FROM boring_workspaces WHERE workspace = ?', workspace)?.incarnation;
126
132
  if (typeof incarnation !== 'string')
127
133
  throw new Error('The workspace has no incarnation');
134
+ /** Each open watcher's comparison, run right after a write through this file system commits. */
135
+ const watchers = new Set();
136
+ let notifying = false;
137
+ function notify() {
138
+ if (watchers.size === 0) {
139
+ touched.clear();
140
+ return;
141
+ }
142
+ if (notifying)
143
+ return;
144
+ notifying = true;
145
+ // After the current synchronous work: a batch write inside a caller's transaction is compared once it has committed.
146
+ queueMicrotask(() => { notifying = false; const paths = [...touched]; touched.clear(); for (const check of [...watchers])
147
+ check(paths); });
148
+ }
128
149
  async function run(path, context, mode, work) {
129
150
  if (context.abortSignal?.aborted)
130
151
  return err(fileError('aborted', 'Operation aborted', path));
131
152
  if (typeof path !== 'string' || path.includes('\0'))
132
153
  return err(fileError('invalid', 'Invalid path', String(path)));
133
154
  try {
134
- return ok(db.transaction(mode, () => work(normalize(path.startsWith('/') ? path : `${fs.cwd}/${path}`))));
155
+ const value = db.transaction(mode, () => work(normalize(path.startsWith('/') ? path : `${fs.cwd}/${path}`)));
156
+ if (mode === 'write')
157
+ notify();
158
+ return ok(value);
135
159
  }
136
160
  catch (error) {
137
161
  return err(isFileError(error) ? error : fileError('unknown', error instanceof Error ? error.message : String(error), path));
@@ -264,6 +288,8 @@ export function openSqliteFileSystem(options) {
264
288
  const moved = existing(from);
265
289
  if (from === to)
266
290
  return;
291
+ touched.add(from);
292
+ touched.add(to);
267
293
  if (from === '/' || within(from, to))
268
294
  throw failure('invalid', 'EINVAL', to, 'cannot move a directory into itself');
269
295
  const target = row(to);
@@ -306,8 +332,9 @@ export function openSqliteFileSystem(options) {
306
332
  };
307
333
  return reader;
308
334
  }),
309
- // SQLite reports no changes, so the watcher compares snapshots of the watched rows. The first snapshot is taken before the
310
- // watcher is returned, so a change made after `watch` resolves is always seen.
335
+ // SQLite reports no changes, so the watcher compares snapshots of the watched rows: at once after each write through this file
336
+ // system, and every second for other connections. The first snapshot is taken before the watcher is returned, so a change made
337
+ // after `watch` resolves is always seen.
311
338
  watch: async (targets, onChange, context) => {
312
339
  if (context.abortSignal?.aborted)
313
340
  return err(fileError('aborted', 'Operation aborted'));
@@ -326,7 +353,18 @@ export function openSqliteFileSystem(options) {
326
353
  onChange(change);
327
354
  }
328
355
  catch { /* the host's callback cannot stop the watcher */ } };
329
- const timer = setInterval(() => {
356
+ const hidden = (name, target) => (target.exclude?.hidden && name.startsWith('.')) || target.exclude?.names?.includes(name);
357
+ /** Whether a path this file system just wrote lies under a watched target, as the snapshot would have listed it. */
358
+ const covered = (path) => targets.some(target => {
359
+ const root = normalize(target.path.startsWith('/') ? target.path : `${fs.cwd}/${target.path}`);
360
+ if (path === root)
361
+ return true;
362
+ if (!within(root, path))
363
+ return false;
364
+ const names = path.slice(root.length + (root === '/' ? 0 : 1)).split('/');
365
+ return (target.recursive || names.length === 1) && !names.some(name => hidden(name, target));
366
+ });
367
+ const check = (direct = []) => {
330
368
  if (stopped)
331
369
  return;
332
370
  let next;
@@ -335,16 +373,19 @@ export function openSqliteFileSystem(options) {
335
373
  }
336
374
  catch (error) {
337
375
  report({ error: isFileError(error) ? error : fileError('unknown', String(error)) });
338
- stopped = true;
339
- clearInterval(timer);
376
+ stop();
340
377
  return;
341
378
  }
342
- const changed = [...new Set([...previous.keys(), ...next.keys()])].filter(path => previous.get(path) !== next.get(path));
379
+ const written = new Set(direct.filter(covered));
380
+ const changed = [...new Set([...previous.keys(), ...next.keys(), ...written])].filter(path => written.has(path) || previous.get(path) !== next.get(path));
343
381
  previous = next;
344
382
  if (changed.length)
345
383
  report({ paths: changed });
346
- }, WATCH_INTERVAL_MS);
347
- const watcher = { mode: 'polling', close: async () => { stopped = true; clearInterval(timer); } };
384
+ };
385
+ const timer = setInterval(() => check(), WATCH_INTERVAL_MS);
386
+ const stop = () => { stopped = true; clearInterval(timer); watchers.delete(check); };
387
+ watchers.add(check);
388
+ const watcher = { mode: 'polling', close: async () => { stop(); } };
348
389
  return ok(watcher);
349
390
  },
350
391
  canonicalPath: (path, context) => run(path, context, 'read', absolute => { existing(absolute); return absolute; }),
@@ -366,6 +407,7 @@ export function openSqliteFileSystem(options) {
366
407
  throw failure('permission_denied', 'EPERM', absolute, 'the workspace root cannot be removed');
367
408
  if (found.kind === 'directory' && !options?.recursive && children(absolute).length)
368
409
  throw failure('invalid', 'ENOTEMPTY', absolute, 'directory not empty');
410
+ touched.add(absolute);
369
411
  db.run(`DELETE FROM boring_workspace_files WHERE ${subtree(absolute)}`, ...subtreeArgs(absolute));
370
412
  }),
371
413
  createTempDir: (prefix, context) => temporary(prefix ?? 'tmp-', '', 'directory', context),
@@ -375,7 +417,7 @@ export function openSqliteFileSystem(options) {
375
417
  registerSqliteBatch(fs, {
376
418
  connection: db,
377
419
  bytes: path => { const found = row(normalize(path)); return !found ? null : found.kind === 'directory' ? undefined : bytes(normalize(path)); },
378
- write: (path, content) => { const absolute = normalize(path); makeDirectory(dirname(absolute), true); write(absolute, content); },
420
+ write: (path, content) => { const absolute = normalize(path); makeDirectory(dirname(absolute), true); write(absolute, content); notify(); },
379
421
  });
380
422
  return fs;
381
423
  }
@@ -1,16 +1,32 @@
1
1
  import type { FileSystem } from '@earendil-works/pi-durable/env';
2
2
  import type { PublicationLookupResult, PublicationResult, ReadResult, ResourceAccess, ResourceProvider } from './contracts.js';
3
3
  import type { WorkspaceCatalogOptions } from './workspace-catalog.js';
4
- import type { WorkspaceCatalog } from './revision-contracts.js';
4
+ import type { ChangeFeedRequest, WorkspaceCatalog, WorkspaceChangeEvent } from './revision-contracts.js';
5
+ export type { ChangeFeedRequest, WorkspaceChangeEvent } from './revision-contracts.js';
5
6
  export type { WorkspaceCatalogOptions } from './workspace-catalog.js';
6
- import type { WorkspaceJournal } from './journal.js';
7
+ import type { HistorySource, WorkspaceJournal } from './journal.js';
7
8
  import type { WorkspaceKey } from './workspace-identity.js';
8
9
  export type { WorkspaceKey } from './workspace-identity.js';
9
10
  export interface WorkspaceChange {
10
11
  readonly path: string;
11
- /** `null` when the file is absent. */
12
+ /** `null` when the file is absent (or, from a watcher, when the path is a directory). */
12
13
  readonly revision: string | null;
13
- readonly source: 'provider' | 'poll';
14
+ /** `publish`: a conditional write. `agent`: `record` (a Pi `write` or `edit`). `poll`: found by hashing. `watch`: the file system reported it. */
15
+ readonly source: 'publish' | 'agent' | 'poll' | 'watch';
16
+ }
17
+ /** What `record` is told about an agent write (ruling 2: observed, never receipted). */
18
+ export interface AgentWrite {
19
+ readonly path: string;
20
+ /** The file before the write, when it existed: its revision and the bytes of that revision (kept in history as the replaced version). */
21
+ readonly before?: {
22
+ readonly revision: string;
23
+ readonly bytes: Uint8Array;
24
+ } | null | undefined;
25
+ /** The revision the write produced, as the caller observed it right after. */
26
+ readonly after: string;
27
+ readonly source: 'agent';
28
+ /** The Pi conversation that wrote it, carried on the change event. */
29
+ readonly conversationId?: string | undefined;
14
30
  }
15
31
  /** Runs one piece of work at a time, in order. A failed piece does not stop the ones behind it. */
16
32
  export interface WorkspaceQueue {
@@ -27,25 +43,62 @@ export interface WorkspaceResourceProvider extends ResourceProvider {
27
43
  };
28
44
  /** The workspace's mutation queue. Native tool calls that must not interleave with a conditional write run through it. */
29
45
  readonly queue: WorkspaceQueue;
46
+ /**
47
+ * Record a write the agent already made through the native tools (Pi `write`/`edit`, called by the file guard right after the
48
+ * tool succeeded): the file's history gains the written revision (source `agent`, and the replaced one when `before` carries its
49
+ * bytes) and the change feed an `agent` event. It writes history rows only: no publish, no receipt, no operation or intent rows,
50
+ * so agent writes are observed, not published. It does not enter `queue` (the guard already runs inside it), so it is safe to call from queued work.
51
+ * When the bytes there no longer have the `after` revision (a shell wrote in between), only the change event is recorded.
52
+ */
53
+ readonly record: (write: AgentWrite, access: ResourceAccess) => Promise<void>;
54
+ /**
55
+ * The change feed: every write this provider makes or records, every change `poll` finds and, while someone is listening, every
56
+ * change the file system's `watch` reports (the SQLite file system reports its own connection's writes at once). Ordered by
57
+ * `seq` and replayable from `since` within a bounded window; a gap yields a `resnapshot` event. The iterator ends when
58
+ * `request.signal` or `access.signal` aborts, or the provider closes. Events are invalidation hints: read the bytes again.
59
+ *
60
+ * After the last listener leaves, the watcher stays open for `watchLingerMs` and then closes. Changes made while no watcher was
61
+ * open are unknown, so the first listener afterwards makes the feed append a `resnapshot`: a client that reconnects after a
62
+ * longer absence reloads everything once, and one that stays connected is never affected.
63
+ */
64
+ readonly changes: (request: ChangeFeedRequest, access: ResourceAccess) => AsyncIterable<WorkspaceChangeEvent>;
65
+ /** The seq of the newest change event (the cursor of "now" for `changes({ since })`). */
66
+ readonly changeHead: () => number;
67
+ /** Names this provider's lifetime; a cursor from another one (`ChangeFeedRequest.incarnation`) is answered with a `resnapshot`. */
68
+ readonly changeIncarnation: string;
69
+ /** Ends every change stream, closes the file watcher and drops the change listeners. Idempotent; the provider still reads and publishes. */
70
+ readonly close: () => void;
71
+ /** Aborts when the provider closes, so a change stream nobody is reading still ends and releases its lease. */
72
+ readonly lifetime: AbortSignal;
30
73
  /** Revisions still retained for a file, newest first. */
31
74
  readonly history: (path: string) => string[];
32
75
  /** The same revisions with the time each was saved (milliseconds since the epoch; 0 when unknown), newest first. */
33
76
  readonly saves: (path: string) => {
34
77
  readonly revision: string;
35
78
  readonly savedAt: number;
79
+ readonly source: HistorySource;
36
80
  }[];
37
81
  /** Read the file as it is now and retain that revision in its history (what `present` shows). Serialised with writes. */
38
82
  readonly keep: (path: string, access: ResourceAccess) => Promise<ReadResult>;
39
83
  /** Provider writes and polled detections. The listener must not throw. */
40
84
  readonly onChange: (listener: (change: WorkspaceChange) => void) => () => void;
41
- /** Hash each path now and report those that differ from the last revision this provider wrote or polled. Finds shell writes. */
42
- readonly poll: (paths: readonly string[], signal?: AbortSignal) => Promise<WorkspaceChange[]>;
85
+ /**
86
+ * Hash each path now and report those that differ from the last revision this provider wrote or polled. Finds shell writes.
87
+ * Without `paths`, every file of the workspace (except `.git` and `node_modules`): the first such call takes the baseline and
88
+ * reports nothing; later calls also report new and removed files. A host whose file system cannot `watch` calls it at the end of
89
+ * each agent turn (after `bash`), so the change feed still sees shell writes.
90
+ */
91
+ readonly poll: (paths?: readonly string[], signal?: AbortSignal) => Promise<WorkspaceChange[]>;
43
92
  }
44
93
  export interface WorkspaceProviderOptions {
45
94
  readonly identity: WorkspaceKey;
46
95
  readonly fs: FileSystem;
47
96
  readonly journal: WorkspaceJournal;
48
97
  readonly catalog?: WorkspaceCatalogOptions;
98
+ /** How many change events the feed retains for replay (default 1024). An older cursor gets a `resnapshot`. */
99
+ readonly changeBuffer?: number;
100
+ /** How long the watcher stays open after the last change-feed listener leaves, in milliseconds (default 30 000). */
101
+ readonly watchLingerMs?: number;
49
102
  }
50
103
  /** The Git blob id of `bytes` (`sha1("blob <length>\0" + bytes)`), computed without Git. */
51
104
  export declare function blobRevision(bytes: Uint8Array): Promise<string>;
package/dist/workspace.js CHANGED
@@ -71,14 +71,38 @@ export function createWorkspaceProvider(options) {
71
71
  principal: access.principalId, initiator: access.initiatorId, operation: operationId,
72
72
  });
73
73
  const ownEvidence = (evidence) => evidence.startsWith(`${key.incarnation}:`);
74
- function emit(change) {
75
- known.set(change.path, change.revision);
74
+ // ---- The change feed: a bounded, seq-ordered buffer of invalidation hints ----------------------------------------------------
75
+ const bufferLimit = options.changeBuffer ?? 1024;
76
+ if (!Number.isSafeInteger(bufferLimit) || bufferLimit < 1)
77
+ throw new RangeError('The change buffer must be a positive integer');
78
+ const lingerMs = options.watchLingerMs ?? 30_000;
79
+ // Sequence numbers start from the clock (microseconds), so a cursor from an earlier provider lifetime is older than this one's
80
+ // first event and reads as a gap, never as a position in this lifetime.
81
+ const firstSeq = Date.now() * 1000;
82
+ let head = firstSeq;
83
+ // Names this provider lifetime. A cursor from another lifetime is refused by this, not only by the clock: the clock may go back.
84
+ const incarnationOfFeed = randomUUID();
85
+ const lifetime = new AbortController();
86
+ const buffer = [];
87
+ const waiting = new Set();
88
+ function append(event) {
89
+ head += 1;
90
+ buffer.push(Object.freeze({ ...event, seq: head, at: Date.now() }));
91
+ if (buffer.length > bufferLimit)
92
+ buffer.shift();
93
+ for (const wake of [...waiting])
94
+ wake();
95
+ }
96
+ function emit(change, extra = {}) {
97
+ if (!extra.directory)
98
+ known.set(change.path, change.revision);
76
99
  for (const listener of [...listeners]) {
77
100
  try {
78
101
  listener(change);
79
102
  }
80
103
  catch { /* a listener failure cannot undo a write */ }
81
104
  }
105
+ append({ kind: 'change', ...change, ...extra });
82
106
  }
83
107
  const dirname = (path) => path.slice(0, path.lastIndexOf('/')) || '/';
84
108
  const inside = (base, path) => path === base || path.startsWith(base === '/' ? '/' : `${base}/`);
@@ -182,7 +206,7 @@ export function createWorkspaceProvider(options) {
182
206
  const result = await read({ target: { resource: { providerId, path }, view: { kind: 'published' } }, revision: { kind: 'latest' } }, access);
183
207
  if (result.kind === 'available') {
184
208
  const path = result.snapshot.ref.resource.path;
185
- journal.remember(historyScope, path, { revision: result.snapshot.ref.revision, bytes: result.snapshot.bytes, mediaType: result.snapshot.mediaType, savedAt: await savedAtOf(path) });
209
+ journal.remember(historyScope, path, { revision: result.snapshot.ref.revision, bytes: result.snapshot.bytes, mediaType: result.snapshot.mediaType, savedAt: await savedAtOf(path), source: 'observed' });
186
210
  }
187
211
  return result;
188
212
  });
@@ -276,11 +300,11 @@ export function createWorkspaceProvider(options) {
276
300
  }
277
301
  const evidence = `${key.incarnation}:${randomUUID()}`;
278
302
  const versions = plans.flatMap(({ change, path, before, after, unchanged, beforeSavedAt }) => {
279
- const kept = (revision, bytes, mediaType, savedAt) => ({ scope: historyScope, path, version: { revision, bytes, mediaType, savedAt } });
303
+ const kept = (revision, bytes, mediaType, savedAt, source) => ({ scope: historyScope, path, version: { revision, bytes, mediaType, savedAt, source } });
280
304
  return unchanged ? [] : [
281
305
  ...(change.kind === 'replace' && before?.kind === 'file'
282
- ? [kept(before.revision, before.bytes, journal.version(historyScope, path, before.revision)?.mediaType ?? mediaTypeOf(path), beforeSavedAt)] : []),
283
- kept(after, change.bytes, change.mediaType, Date.now()),
306
+ ? [kept(before.revision, before.bytes, journal.version(historyScope, path, before.revision)?.mediaType ?? mediaTypeOf(path), beforeSavedAt, 'observed')] : []),
307
+ kept(after, change.bytes, change.mediaType, Date.now(), 'publish'),
284
308
  ];
285
309
  });
286
310
  const record = {
@@ -343,7 +367,7 @@ export function createWorkspaceProvider(options) {
343
367
  }
344
368
  for (const { path, after, unchanged } of plans)
345
369
  if (!unchanged)
346
- emit({ path, revision: after, source: 'provider' });
370
+ emit({ path, revision: after, source: 'publish' });
347
371
  return { kind: 'committed', receipt: receipt(request.operationId, access, record) };
348
372
  }
349
373
  catch (error) {
@@ -369,20 +393,63 @@ export function createWorkspaceProvider(options) {
369
393
  ? { kind: 'unknown', operationId: id, reason: 'The operation began and has no recorded outcome' }
370
394
  : { kind: 'not-found' };
371
395
  };
396
+ /** Every file of the workspace, relative to the root (not `.git`, `node_modules` or the provider's temporary files). */
397
+ async function everyFile(signal) {
398
+ const found = [];
399
+ let scanned = 0;
400
+ const walk = async (directory) => {
401
+ const listing = await fs.listDir(directory ? absolute(directory) : root, fsContext(signal));
402
+ if (!listing.ok)
403
+ return true;
404
+ for (const entry of listing.value) {
405
+ if (++scanned > 100_000)
406
+ return false;
407
+ const path = directory ? `${directory}/${entry.name}` : entry.name;
408
+ if (entry.kind === 'directory') {
409
+ if (entry.name !== '.git' && entry.name !== 'node_modules' && !await walk(path))
410
+ return false;
411
+ }
412
+ else if (entry.kind === 'file' && !isTemporary(path))
413
+ found.push(path);
414
+ }
415
+ return true;
416
+ };
417
+ return await walk('') ? found : undefined;
418
+ }
419
+ let baselined = false;
372
420
  const poll = async (paths, signal) => {
373
421
  const changed = [];
374
422
  await sweep();
375
- for (const input of paths) {
423
+ let inputs;
424
+ let report = (path) => known.has(path);
425
+ if (paths === undefined) {
426
+ const files = await everyFile(signal);
427
+ // Too many files to hash on each turn: say so to every listener instead of reporting a partial picture.
428
+ if (files === undefined) {
429
+ append({ kind: 'resnapshot' });
430
+ return changed;
431
+ }
432
+ const present = new Set(files);
433
+ const removed = [...known].filter(([path, revision]) => revision !== null && !present.has(path)).map(([path]) => path);
434
+ inputs = [...files, ...removed];
435
+ if (baselined)
436
+ report = () => true;
437
+ baselined = true;
438
+ }
439
+ else
440
+ inputs = paths;
441
+ for (const input of inputs) {
376
442
  const path = locator({ resource: { providerId, path: input }, view: { kind: 'published' } }).resource.path;
377
443
  const now = await observe(path, signal);
378
444
  if (now.kind === 'error' || now.kind === 'outside' || isTemporary(path))
379
445
  continue;
380
446
  const revision = now.kind === 'file' ? now.revision : null;
381
- const first = !known.has(path);
382
- if (!first && known.get(path) === revision)
447
+ if (known.has(path) && known.get(path) === revision)
448
+ continue;
449
+ if (!known.has(path) && revision === null && paths === undefined)
383
450
  continue;
384
451
  const change = { path, revision, source: 'poll' };
385
- if (first)
452
+ if (!report(path))
386
453
  known.set(path, revision);
387
454
  else {
388
455
  emit(change);
@@ -391,10 +458,181 @@ export function createWorkspaceProvider(options) {
391
458
  }
392
459
  return changed;
393
460
  };
461
+ const record = async (write, context) => {
462
+ const access = accessSnapshot(context);
463
+ if (write.source !== 'agent')
464
+ throw new TypeError('Only agent writes are recorded');
465
+ const path = locator({ resource: { providerId, path: write.path }, view: { kind: 'published' } }).resource.path;
466
+ const after = identifier(write.after);
467
+ const conversationId = write.conversationId === undefined ? undefined : identifier(write.conversationId);
468
+ // Known before the await: a watcher event for this same write, handled meanwhile, then finds nothing new and emits nothing twice.
469
+ if (!isTemporary(path))
470
+ known.set(path, after);
471
+ // Deliberately not `queue.run`: the file guard calls this from inside the queue, and the queue runs one piece at a time.
472
+ const now = await observe(path, access.signal);
473
+ if (now.kind === 'outside' || now.kind === 'error')
474
+ return;
475
+ if (now.kind === 'file' && now.revision === after) {
476
+ const mediaType = journal.version(historyScope, path, after)?.mediaType ?? mediaTypeOf(path);
477
+ const before = write.before ?? undefined;
478
+ // The replaced bytes are kept only when they really are that revision.
479
+ if (before && before.revision !== after && before.bytes instanceof Uint8Array && await blobRevision(before.bytes) === identifier(before.revision)) {
480
+ journal.remember(historyScope, path, { revision: before.revision, bytes: Uint8Array.from(before.bytes), mediaType: journal.version(historyScope, path, before.revision)?.mediaType ?? mediaType, savedAt: 0, source: 'observed' });
481
+ }
482
+ journal.remember(historyScope, path, { revision: after, bytes: now.bytes, mediaType, savedAt: Date.now(), source: 'agent' });
483
+ }
484
+ emit({ path, revision: now.kind === 'file' ? now.revision : null, source: 'agent' }, conversationId === undefined ? {} : { conversationId });
485
+ };
486
+ // ---- The watcher: open while someone listens to the feed (and `lingerMs` after), when the file system can watch ----------------
487
+ let listening = 0;
488
+ let watcher;
489
+ let watchUnsupported = false, watchedBefore = false;
490
+ let linger;
491
+ let watchWork = Promise.resolve();
492
+ const relativeTo = (absolutePath, base) => {
493
+ base = base.replace(/\/+$/, '') || '/';
494
+ if (absolutePath === base)
495
+ return '';
496
+ const prefix = base === '/' ? '/' : `${base}/`;
497
+ return absolutePath.startsWith(prefix) ? absolutePath.slice(prefix.length) : undefined;
498
+ };
499
+ async function watched(input) {
500
+ // The watcher may name the root as given or as resolved (a symbolic link on the way to the workspace).
501
+ const real = await realRoot();
502
+ const path = relativeTo(input, root) ?? (real === undefined ? undefined : relativeTo(input, real));
503
+ if (path === undefined || isTemporary(path) || path.split('/').some(name => name === '.git' || name === 'node_modules'))
504
+ return;
505
+ if (path === '') {
506
+ emit({ path, revision: null, source: 'watch' }, { directory: true });
507
+ return;
508
+ }
509
+ const info = await fs.fileInfo(absolute(path), fsContext());
510
+ if (info.ok && info.value.kind === 'directory') {
511
+ emit({ path, revision: null, source: 'watch' }, { directory: true });
512
+ return;
513
+ }
514
+ const now = await observe(path);
515
+ if (now.kind === 'error' || now.kind === 'outside')
516
+ return;
517
+ const revision = now.kind === 'file' ? now.revision : null;
518
+ if (known.has(path) && known.get(path) === revision)
519
+ return;
520
+ emit({ path, revision, source: 'watch' });
521
+ }
522
+ function onWatch(change) {
523
+ if ('paths' in change) {
524
+ const paths = [...change.paths];
525
+ watchWork = watchWork.then(async () => { for (const path of paths)
526
+ await watched(path).catch(() => undefined); });
527
+ return;
528
+ }
529
+ // Overflow or a failed watcher: what changed is unknown, so every listener reloads. A failed watcher is not reopened (it would
530
+ // fail again, for example on a tree past the environment's limit); the host's `poll()` at turn end remains.
531
+ if ('error' in change) {
532
+ watcher = undefined;
533
+ watchUnsupported = true;
534
+ }
535
+ append({ kind: 'resnapshot' });
536
+ }
537
+ function startWatching() {
538
+ if (lifetime.signal.aborted)
539
+ return;
540
+ if (linger !== undefined) {
541
+ clearTimeout(linger);
542
+ linger = undefined;
543
+ }
544
+ if (watcher || watchUnsupported)
545
+ return;
546
+ // Changes made while nobody watched were not seen: a listener resuming from before that reloads.
547
+ if (watchedBefore)
548
+ append({ kind: 'resnapshot' });
549
+ watchedBefore = true;
550
+ watcher = fs.watch([{ path: root, recursive: true, exclude: { names: ['.git', 'node_modules'] } }], onWatch, fsContext()).then(result => {
551
+ if (result.ok)
552
+ return result.value;
553
+ watchUnsupported = true;
554
+ return undefined;
555
+ }, () => { watchUnsupported = true; return undefined; });
556
+ }
557
+ function stopWatching() {
558
+ const current = watcher;
559
+ watcher = undefined;
560
+ void current?.then(value => value?.close(fsContext()));
561
+ }
562
+ async function* changes(request, context) {
563
+ const access = accessSnapshot(context);
564
+ identifier(access.scopeId);
565
+ const since = request.since;
566
+ if (since !== undefined && (!Number.isSafeInteger(since) || since < 0))
567
+ throw new TypeError('Invalid change cursor');
568
+ if (lifetime.signal.aborted)
569
+ return;
570
+ const signals = [request.signal, access.signal, lifetime.signal].filter((value) => value !== undefined);
571
+ const signal = signals.length ? AbortSignal.any(signals) : new AbortController().signal;
572
+ listening += 1;
573
+ startWatching();
574
+ try {
575
+ let cursor = head;
576
+ if (since !== undefined) {
577
+ const oldest = buffer[0]?.seq ?? head + 1;
578
+ const foreign = request.incarnation !== undefined && request.incarnation !== incarnationOfFeed;
579
+ if (foreign || since > head || since < firstSeq || since < oldest - 1)
580
+ yield { kind: 'resnapshot', seq: head, at: Date.now() };
581
+ else
582
+ cursor = since;
583
+ }
584
+ while (!signal.aborted) {
585
+ const oldest = buffer[0];
586
+ // This listener fell behind the retained window: it reloads, then continues with what is retained.
587
+ if (oldest && oldest.seq > cursor + 1) {
588
+ cursor = oldest.seq - 1;
589
+ yield { kind: 'resnapshot', seq: cursor, at: Date.now() };
590
+ continue;
591
+ }
592
+ const next = buffer.find(event => event.seq > cursor);
593
+ if (next) {
594
+ cursor = next.seq;
595
+ // The access projection: nothing after revocation, never the provider's temporary files.
596
+ if (signal.aborted)
597
+ return;
598
+ if (next.kind === 'change' && isTemporary(next.path))
599
+ continue;
600
+ yield next;
601
+ continue;
602
+ }
603
+ await new Promise(resolve => {
604
+ const wake = () => { waiting.delete(wake); signal.removeEventListener('abort', wake); resolve(); };
605
+ waiting.add(wake);
606
+ signal.addEventListener('abort', wake, { once: true });
607
+ });
608
+ }
609
+ }
610
+ finally {
611
+ listening -= 1;
612
+ if (listening === 0 && watcher && !lifetime.signal.aborted) {
613
+ linger = setTimeout(() => { linger = undefined; if (listening === 0)
614
+ stopWatching(); }, lingerMs);
615
+ linger.unref?.();
616
+ }
617
+ }
618
+ }
394
619
  return {
395
620
  providerId, read, queue, keep, identity: Object.freeze(key),
396
621
  catalog: createWorkspaceCatalog({ fs, ...(options.catalog ? { config: options.catalog } : {}), saves: path => journal.saves(historyScope, path) }),
397
- publication: { publish }, reconciliation: { lookup }, poll,
622
+ publication: { publish }, reconciliation: { lookup }, poll, record, changes, changeHead: () => head, changeIncarnation: incarnationOfFeed,
623
+ lifetime: lifetime.signal,
624
+ close: () => {
625
+ // Every change stream ends, the watcher closes and the listeners are dropped: nothing of this provider keeps running.
626
+ lifetime.abort();
627
+ if (linger !== undefined) {
628
+ clearTimeout(linger);
629
+ linger = undefined;
630
+ }
631
+ stopWatching();
632
+ listeners.clear();
633
+ for (const wake of [...waiting])
634
+ wake();
635
+ },
398
636
  history: path => journal.revisions(historyScope, locator({ resource: { providerId, path }, view: { kind: 'published' } }).resource.path),
399
637
  saves: path => journal.saves(historyScope, locator({ resource: { providerId, path }, view: { kind: 'published' } }).resource.path),
400
638
  onChange: listener => { listeners.add(listener); return () => { listeners.delete(listener); }; },
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "@hachej/boring-files",
3
- "version": "0.2.0",
3
+ "version": "0.2.1",
4
4
  "license": "MIT",
5
5
  "private": false,
6
6
  "type": "module",