@wtfalch/chat 0.3.0 → 0.4.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -256,7 +256,11 @@ reply, and answers `/event` and `/relations`, which is what matrix-js-sdk
256
256
  asks for while it builds a thread. It echoes every accepted send back
257
257
  through sync carrying its transaction id, so a reaction shows up the way it
258
258
  does against a homeserver, and it answers redactions -- which is how a
259
- reaction is taken back.
259
+ reaction is taken back. It also answers `/search`, a plain case-insensitive
260
+ substring match over every message body it has recorded, newest first,
261
+ scoped to `filter.rooms` when the client sends one -- a room seeded with
262
+ `encrypted: true` never matches, the same as a real, unindexed encrypted
263
+ room would not.
260
264
 
261
265
  ## Integration tests
262
266
 
@@ -331,6 +335,36 @@ the next page; it is `null` once there is nothing older. It resumes exactly
331
335
  after the last file returned, so none is skipped or repeated. The last page can
332
336
  come back empty.
333
337
 
338
+ ## Search
339
+
340
+ ([wtfalch/agora#252](https://github.com/wtfalch/agora/issues/252),
341
+ [wtfalch/agora#31](https://github.com/wtfalch/agora/issues/31)).
342
+ `chat.search(query, { roomId, limit })` finds a message by a word or phrase,
343
+ across every room this account is joined to, or just `roomId` when given,
344
+ through Synapse's own `POST /search` (`room_events`, newest first) -- the
345
+ same event-mapping code `timeline()` uses, so a result is a plain
346
+ `ChatTimelineEvent` next to the room it came from. `limit` (default 50)
347
+ stops collection once that many results are in, following Synapse's own
348
+ `next_batch` as far as it takes to get there.
349
+
350
+ Permission filtering is Synapse's own: it only ever searches rooms this
351
+ account has joined, and this package layers nothing of its own on top --
352
+ a person who left a room, or was removed, finds nothing from it on their
353
+ next search, with no filter this package wrote itself.
354
+
355
+ **Encrypted rooms say so, rather than returning nothing.** Crypto stays off
356
+ in this package today (the module comment says why), so a real encrypted
357
+ room's cleartext was never in Synapse's index to begin with -- an empty
358
+ `results` from one would read as "nothing found" when the honest answer is
359
+ "cannot search here". `search` returns
360
+ `{ results, unsearchableRoomIds }`: `unsearchableRoomIds` lists every
361
+ joined room this search could not look inside because it carries
362
+ `m.room.encryption` state. A room-scoped search of an encrypted room is
363
+ `[roomId]` with `results` always empty; a search across every room lists
364
+ every encrypted one it is in, whatever it did or did not find elsewhere.
365
+ This is the one deviation from agora#252's own bare-array shape, a campaign
366
+ decision recorded against it.
367
+
334
368
  ## First-ship local proof
335
369
 
336
370
  See [the browser/Electron fixture](test/acceptance/README.md) for the repeatable
@@ -52,6 +52,12 @@ const TYPING_TIMEOUT_MS = 20_000;
52
52
  * already passed the `contains_url` filter server-side, so a larger page
53
53
  * costs one round trip, not one screenful. */
54
54
  const ROOM_FILES_PAGE_SIZE = 50;
55
+ /** `search`'s own default when a caller passes no `limit`. Unlike
56
+ * `roomFiles`, which walks every page Synapse has when given none, a
57
+ * search with no cap at all would follow `next_batch` across a room's
58
+ * entire history for one keystroke -- a bound by default is the safer
59
+ * read of "search this" than "fetch everything ever said". */
60
+ const SEARCH_LIMIT_DEFAULT = 50;
55
61
  /**
56
62
  * A chat session behind the exchange: a matrix-js-sdk `MatrixClient` the
57
63
  * caller never sees, refreshed on its own, exposing only what a UI needs.
@@ -922,6 +928,72 @@ export function createChatClient(options) {
922
928
  }
923
929
  return { files, next: from };
924
930
  }
931
+ /** Joined rooms this account cannot search inside, because Synapse never
932
+ * indexed their cleartext -- an `m.room.encryption` state event is the
933
+ * one thing that means that (see `ChatSearchResults`). Scoped to
934
+ * `roomId` when given, since that is the only room a room-scoped search
935
+ * could ever have reported unsearchable. The company space is excluded
936
+ * the same way `readRooms` excludes it: it is what `people()` reads,
937
+ * not a room in Chat's sense, and never one `search` was ever asked
938
+ * about. */
939
+ function unsearchableRoomIds(target, roomId) {
940
+ return target
941
+ .getRooms()
942
+ .filter((room) => room.getMyMembership() === KnownMembership.Join)
943
+ .filter((room) => !room.isSpaceRoom())
944
+ .filter((room) => (roomId ? room.roomId === roomId : true))
945
+ .filter((room) => room.hasEncryptionStateEvent())
946
+ .map((room) => room.roomId);
947
+ }
948
+ /** `query`, across `options.roomId` or every room this account is
949
+ * joined to, through Synapse's own `POST /search` -- see
950
+ * `ChatClient.search`'s own doc for what that means for permissions and
951
+ * encryption. `filter.rooms` is the one thing that ever narrows the
952
+ * request itself; everything else Synapse decides on its own, including
953
+ * how many results one page of `next_batch` carries. */
954
+ async function search(query, options) {
955
+ if (!client)
956
+ throw new Error('chat: search called before start()');
957
+ const target = client;
958
+ const roomId = options?.roomId;
959
+ const limit = options?.limit ?? SEARCH_LIMIT_DEFAULT;
960
+ // A room-scoped search of an encrypted room has nothing to ask Synapse
961
+ // at all: its messages were never in the index to begin with, and the
962
+ // one room this search could ever have reported unsearchable already
963
+ // is that room.
964
+ if (roomId && target.getRoom(roomId)?.hasEncryptionStateEvent()) {
965
+ return { results: [], unsearchableRoomIds: [roomId] };
966
+ }
967
+ const mapper = target.getEventMapper();
968
+ const results = [];
969
+ let nextBatch;
970
+ do {
971
+ const body = {
972
+ search_categories: {
973
+ room_events: {
974
+ search_term: query,
975
+ order_by: 'recent',
976
+ ...(roomId ? { filter: { rooms: [roomId] } } : {}),
977
+ },
978
+ },
979
+ };
980
+ const response = await target.search({ body, next_batch: nextBatch });
981
+ const roomEvents = response.search_categories.room_events;
982
+ for (const hit of roomEvents.results ?? []) {
983
+ if (results.length >= limit)
984
+ break;
985
+ const raw = hit.result;
986
+ const room = target.getRoom(raw.room_id);
987
+ if (!room)
988
+ continue;
989
+ const mapped = toTimelineEvent(mapper(raw), room, false);
990
+ if (mapped)
991
+ results.push({ roomId: raw.room_id, event: mapped });
992
+ }
993
+ nextBatch = roomEvents.next_batch;
994
+ } while (nextBatch && results.length < limit);
995
+ return { results, unsearchableRoomIds: unsearchableRoomIds(target, roomId) };
996
+ }
925
997
  /** Builds the Matrix content once, here rather than in `markdown.ts`:
926
998
  * `msgtype`/`format`/`formatted_body` are Matrix's own shape, and this
927
999
  * is the only file in the package that touches matrix-js-sdk (see
@@ -1822,6 +1894,7 @@ export function createChatClient(options) {
1822
1894
  onTimelineChanged,
1823
1895
  paginate,
1824
1896
  roomFiles,
1897
+ search,
1825
1898
  sendMessage,
1826
1899
  editMessage,
1827
1900
  deleteMessage,
@@ -19,6 +19,10 @@ export type FakeRoomSeed = {
19
19
  /** Messages already in the room when the first sync lands, oldest first.
20
20
  * Defaults to none. */
21
21
  messages?: FakeMessageSeed[];
22
+ /** Seeds an `m.room.encryption` state event -- the one thing `search`
23
+ * reads to report a room unsearchable (agora#252). Defaults to
24
+ * `false`. */
25
+ encrypted?: boolean;
22
26
  };
23
27
  export type FakeMessageSeed = {
24
28
  /** Defaults to a minted `$seeded-<n>`. Give one to reply to it. */
@@ -96,6 +96,12 @@ export class FakeChatServer {
96
96
  #tokenSeq = 0;
97
97
  #batchSeq = 0;
98
98
  #eventSeq = 0;
99
+ /** Every event's own place in this server's history, oldest first, keyed
100
+ * by event id -- what `/search`'s "newest first" sorts by. Kept apart
101
+ * from `#events` itself (rather than reading array position) so a
102
+ * redaction rewriting an entry in place never disturbs where it sorts. */
103
+ #insertOrder = new Map();
104
+ #insertSeq = 0;
99
105
  #pending;
100
106
  #parkedWaiters = [];
101
107
  #disposed = false;
@@ -357,6 +363,16 @@ export class FakeChatServer {
357
363
  origin_server_ts: now,
358
364
  content: { name: seed.name },
359
365
  });
366
+ if (seed.encrypted) {
367
+ events.push({
368
+ type: 'm.room.encryption',
369
+ state_key: '',
370
+ event_id: `$encryption-${seed.id}`,
371
+ sender: this.userId,
372
+ origin_server_ts: now,
373
+ content: { algorithm: 'm.megolm.v1.aes-sha2' },
374
+ });
375
+ }
360
376
  return {
361
377
  summary: {},
362
378
  state: { events },
@@ -412,10 +428,14 @@ export class FakeChatServer {
412
428
  #materialise(seed) {
413
429
  if (this.#events.has(seed.id))
414
430
  return;
415
- this.#events.set(seed.id, (seed.messages ?? []).map((message) => this.#messageEvent(message)));
431
+ const events = (seed.messages ?? []).map((message) => this.#messageEvent(message));
432
+ this.#events.set(seed.id, events);
433
+ for (const event of events)
434
+ this.#insertOrder.set(event.event_id, this.#insertSeq++);
416
435
  }
417
436
  #record(roomId, event) {
418
437
  this.#events.set(roomId, [...(this.#events.get(roomId) ?? []), event]);
438
+ this.#insertOrder.set(event.event_id, this.#insertSeq++);
419
439
  }
420
440
  /** One room's incremental entry, carrying just these events -- and,
421
441
  * given some, ephemeral ones (`m.typing`) alongside them, the way a real
@@ -831,6 +851,74 @@ export class FakeChatServer {
831
851
  ...(end ? { end } : {}),
832
852
  }));
833
853
  }
854
+ // `ChatClient.search`'s own wire call (agora#252, agora#31): a plain
855
+ // case-insensitive substring match over every `m.room.message` body
856
+ // this fake has recorded, newest first, scoped to `filter.rooms` when
857
+ // the client sent one. Reads `#events` live rather than a snapshot
858
+ // taken at record time, so a redacted message (its content already
859
+ // emptied by the `redact` route) stops matching immediately, the same
860
+ // as a real search index would once Synapse's own redaction handling
861
+ // catches up to it. `next_batch` pages a deliberately small five at a
862
+ // time, so a unit test can prove `search`'s own paging without seeding
863
+ // hundreds of messages.
864
+ //
865
+ // An encrypted room's messages never match. This fake stores every
866
+ // body as plain text regardless of `encrypted` -- there is no real
867
+ // Megolm session here to encrypt against -- but a real Synapse never
868
+ // has a cleartext body to index for such a room in the first place, so
869
+ // a fake that let one match would prove `search` filters encrypted
870
+ // rooms out itself, which is exactly what it must not do (the fake's
871
+ // own module comment, and `ChatSearchResults`).
872
+ if (url.pathname === '/_matrix/client/v3/search' &&
873
+ init?.method === 'POST' &&
874
+ this.#authorised(init)) {
875
+ let body = {};
876
+ try {
877
+ body = JSON.parse(String(init?.body ?? '{}'));
878
+ }
879
+ catch {
880
+ // Same tolerance as every other body this fake parses.
881
+ }
882
+ const term = (body.search_categories?.room_events?.search_term ?? '').toLowerCase();
883
+ const roomFilter = body.search_categories?.room_events?.filter?.rooms;
884
+ const encryptedRoomIds = new Set(this.#rooms.filter((room) => room.encrypted).map((room) => room.id));
885
+ const flattened = [];
886
+ for (const [roomId, events] of this.#events) {
887
+ if (roomFilter && !roomFilter.includes(roomId))
888
+ continue;
889
+ if (encryptedRoomIds.has(roomId))
890
+ continue;
891
+ for (const event of events)
892
+ flattened.push({ roomId, event });
893
+ }
894
+ const matches = flattened
895
+ .filter((entry) => entry.event.type === 'm.room.message')
896
+ .filter((entry) => {
897
+ const value = entry.event.content.body;
898
+ return typeof value === 'string' && value.toLowerCase().includes(term);
899
+ })
900
+ .sort((a, b) => (this.#insertOrder.get(b.event.event_id) ?? 0) -
901
+ (this.#insertOrder.get(a.event.event_id) ?? 0));
902
+ const pageSize = 5;
903
+ const from = url.searchParams.get('next_batch');
904
+ const start = from?.startsWith('idx-') ? Number(from.slice('idx-'.length)) : 0;
905
+ const chunk = matches.slice(start, start + pageSize);
906
+ const nextIndex = start + chunk.length;
907
+ const nextBatch = nextIndex < matches.length ? `idx-${nextIndex}` : undefined;
908
+ return Promise.resolve(jsonResponse(200, {
909
+ search_categories: {
910
+ room_events: {
911
+ count: matches.length,
912
+ results: chunk.map((entry) => ({
913
+ rank: 1,
914
+ result: { ...entry.event, room_id: entry.roomId },
915
+ })),
916
+ highlights: [],
917
+ ...(nextBatch ? { next_batch: nextBatch } : {}),
918
+ },
919
+ },
920
+ }));
921
+ }
834
922
  const send = url.pathname.match(/^\/_matrix\/client\/v3\/rooms\/([^/]+)\/send\/([^/]+)\/([^/]+)$/);
835
923
  if (send && this.#authorised(init)) {
836
924
  let content = {};
@@ -7,4 +7,4 @@
7
7
  * `README.md` and the chat plan, section 9.
8
8
  */
9
9
  export { createChatClient, MAX_UPLOAD_BYTES } from './client.ts';
10
- export type { ChatAttachment, ChatRoomFilesPage, ChatClient, ChatConnectionStatus, ChatPerson, ChatRoomSummary, ChatUploadProgress, CreateChatClientOptions, ExchangeBody, } from './types.ts';
10
+ export type { ChatAttachment, ChatRoomFilesPage, ChatClient, ChatConnectionStatus, ChatPerson, ChatRoomSummary, ChatSearchResults, ChatTimelineEvent, ChatUploadProgress, CreateChatClientOptions, ExchangeBody, } from './types.ts';
@@ -182,6 +182,24 @@ export type ChatTimelinePage = {
182
182
  * comes back with nothing more behind it. */
183
183
  readonly hasMore: boolean;
184
184
  };
185
+ /** What `search` finds, across the rooms it was allowed to look in.
186
+ * `results` is newest first, each carrying the room it was found in
187
+ * alongside the message itself -- a global search has no single room to
188
+ * hang the result off, so it travels with every one. `unsearchableRoomIds`
189
+ * is every joined room this search could not look inside because it is
190
+ * encrypted: `[roomId]` for a room-scoped search of an encrypted room
191
+ * (`results` is then always empty), or every encrypted room this account
192
+ * is in for a search across all of them. Synapse never indexes an
193
+ * encrypted room's cleartext, so an empty `results` there is silence, not
194
+ * "nothing found" -- a caller reads `unsearchableRoomIds` to tell the two
195
+ * apart (agora#252, agora#31). */
196
+ export type ChatSearchResults = {
197
+ readonly results: readonly {
198
+ roomId: string;
199
+ event: ChatTimelineEvent;
200
+ }[];
201
+ readonly unsearchableRoomIds: readonly string[];
202
+ };
185
203
  /** Whether the sync loop is caught up, working to catch up (after a
186
204
  * network drop, or recovering from a refused sync position with a fresh
187
205
  * full sync), or not connected at all. */
@@ -304,6 +322,23 @@ export type ChatClient = {
304
322
  limit?: number;
305
323
  before?: string;
306
324
  }) => Promise<ChatRoomFilesPage>;
325
+ /** A word or phrase, across every room this account is joined to, or just
326
+ * `options.roomId`, through Synapse's own server-side search
327
+ * (`POST /search`, `room_events`, newest first) -- so permission
328
+ * filtering is Synapse's: a room this account has left never appears,
329
+ * with no filter of this package's own layered on top (agora#31).
330
+ * `limit` stops once that many results are collected, following
331
+ * `next_batch` as far as it takes to get there; defaults to 50.
332
+ *
333
+ * Encrypted rooms are reported, not silently skipped -- see
334
+ * `ChatSearchResults`. That is the one search this package can commit to
335
+ * today: crypto stays off (the module comment says why), so a real
336
+ * encrypted room's cleartext was never in Synapse's index to begin with
337
+ * (agora#252). */
338
+ search: (query: string, options?: {
339
+ roomId?: string;
340
+ limit?: number;
341
+ }) => Promise<ChatSearchResults>;
307
342
  /** Everyone joined to `roomId`, by name, the signed-in person included --
308
343
  * who a message in this room can mention. A room's own membership, not
309
344
  * the company's: mentioning somebody who is not here would notify
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "@wtfalch/chat",
3
- "version": "0.3.0",
3
+ "version": "0.4.0",
4
4
  "description": "A Matrix chat client behind the agora's token exchange.",
5
5
  "license": "MIT",
6
6
  "type": "module",