@tech4good/platform-data 0.1.0-alpha.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -0,0 +1,1011 @@
1
+ # Changes beyond extracting platform-data
2
+
3
+ Written September 10, 2026, for the initial `0.1.0-alpha.0` implementation.
4
+ Moving the shared code into a package also involved fixing bugs and making a
5
+ few design changes. This document explains those changes and why we made them.
6
+ Existing apps will need some updates when they adopt the package.
7
+
8
+ Read [MIGRATION.md](MIGRATION.md) for adoption steps and
9
+ [FUTURE-WORK.md](FUTURE-WORK.md) for open questions and decisions to keep in mind.
10
+ The [README](README.md) explains how to use the package. Record later releases
11
+ separately so this initial history remains clear.
12
+
13
+ ## Which original code this compares against
14
+
15
+ The original code is available at angular-schematics commit
16
+ `fda18e3ab5909a025552bc359e4bafcc12a5c10e` (`feat(shared): extract platform tooling
17
+ package`). Its `src/ng-add/files/src/app/core/` templates still contain the copied
18
+ data infrastructure. For example, from a repository checkout:
19
+
20
+ ```bash
21
+ git show fda18e3:src/ng-add/files/src/app/core/store/app.store.ts.template
22
+ git show fda18e3:src/ng-add/files/src/app/core/firebase/cached-listeners.service.ts.template
23
+ ```
24
+
25
+ We also reviewed MathPets and Mentor, whose code and auth flows differ in places.
26
+ A bug described here existed in the original code we compared; that does not
27
+ mean every version of both apps had it. The tests in temporary copies of those
28
+ apps happened before some of the final changes. They still need a separate
29
+ migration and validation against the finished package.
30
+
31
+ Source and test paths below are relative to the **repository checkout**, not the
32
+ installed npm package. The package ships this document and its companion guides;
33
+ it does not ship all repository tests or planning files. The chronological review
34
+ is `docs-plans/platform-data-extraction.md`; its earlier designs are historical.
35
+
36
+ ## Extraction review: preserve the original flow
37
+
38
+ The September 10 review compared the implementations below against `fda18e3`,
39
+ including the remaining generated app code. The initial extraction rewrote more
40
+ than packaging required. The policy for this first release is to preserve the
41
+ original branches, operation order, and intermediate variables where practical.
42
+ Keep a difference when it implements an agreed change or corrects a specific
43
+ failure; avoid making a function look different just to make it shorter.
44
+
45
+ The subsequent comment review restored explanatory details that the extraction
46
+ had shortened: the null/undefined distinction and optional __hashtagId example,
47
+ query-removal versus deletion, N+1 batching and cache tradeoffs, the load/listener
48
+ lifecycle, mock error/recovery controls, and browser logging commands. Where the
49
+ implementation changed, comments retain the example and explain the current
50
+ behavior rather than repeating an obsolete limitation. This was a comment-only
51
+ source edit; emitted JavaScript was compared with the last validated build.
52
+
53
+ These rewrites were rolled back while retaining the targeted fixes:
54
+
55
+ - **Selectors:** restore the ID `==`/`in` branches and `filteredEntities`, then
56
+ apply remaining filters, ordering and limits. The whole-map rewrite had lost
57
+ the original direct-lookup advantage. Shared comparison helpers can still fix
58
+ timestamp equality and cursor handling without removing those branches.
59
+ - **Batch coordination:** restore separate optimistic and non-optimistic paths.
60
+ Moving preparation inside `try`, reversing rollback order, and remembering
61
+ whether commit succeeded fix failures without combining the two paths.
62
+ - **Firestore query/write construction:** restore explicit orderBy shapes,
63
+ cursor cases, and batch-versus-immediate-write branches. Keep orderBy before
64
+ cursor constraints: the old iteration over object keys could send a cursor
65
+ first when the caller wrote `{ startAt: 0, orderBy: 'score' }`, which the SDK
66
+ rejects. Timestamp conversion and tuple cursor support remain.
67
+ - **Mock change streams:** restore the snapshot → `pairwise` → changes pipeline,
68
+ instead of maintaining a mutable previous-map variable inside an effect.
69
+ Keep raw deletion queries, a pass for records that leave the query, and empty
70
+ initial snapshots needed to establish listener readiness.
71
+ - **Ordinary mock writes:** restore calls to the collection's `add`/`update`
72
+ methods. Routing every single write through batch staging copied and replaced
73
+ a whole collection unnecessarily, and bypassed custom collection methods.
74
+ Only actual batches stage collection replacements for atomic application.
75
+
76
+ The rest of the review found these differences worth retaining:
77
+
78
+ | Area and source mapping | Decision and concrete reason |
79
+ | --- | --- |
80
+ | `app.store.ts.template` → `src/entity-store.ts` | The main load/stream/write branches remain substantially the original code. Keep adapter/token/timestamp substitutions, required creation metadata, preserving `_createdAt` during restoration, and the agreed per-collection reset cleanup. The 50-result cache was already present. |
81
+ | `app.store.ts.template` → `src/query-descriptors.ts` | Keep the original matrix/group/optimize/execute flow. Wider matrix traversal preserves queries after an empty first parent; collection/store/callback checks prevent combining unrelated queries. These are local fixes rather than a new batching algorithm. |
82
+ | `cached-listeners.service.ts.template` → `src/cached-listeners.ts` | Keep pooling, component claims, grace periods and nested traversal. Typed query keys prevent collisions; collection-grouped history supports the agreed reset policy. Explicit nested-observer cleanup and a stream-only generation check prevent teardown from leaving work that can reopen listeners. There is no per-store owner registry or cancellation of ordinary reads. |
83
+ | `stream-multiplexer.ts.template` → `src/stream-multiplexer.ts` | Keep the original packing, sparse-listener and reference-counting algorithm. Waiting for all replacements before retiring old listeners prevents gaps. Tracking retiring listeners and invalidating failed connections makes teardown and later retries work. |
84
+ | `batch-write.service.ts.template` and the Undo helper in `app.store.ts.template` → `src/batch-write.ts`, `src/undo.ts` | Keep preparation-error rollback, reverse-order compensation, and the post-commit guard. The optimistic/non-optimistic branches are restored; sequential inverse writes are needed when writes depend on one another. |
85
+ | Firebase data methods → `firestore/public-api.ts` | Direct SDK snapshots allow shared timestamp conversion and trustworthy document IDs on every read. A shared query builder keeps that boundary consistent. Explicit option/write branches are restored. Alias-based aggregate dispatch corrects a real count/sum/average error. |
86
+ | Mock data methods → `testing/mock-data.ts` | Raw queries must expose deletions to the loader. Subscription-owned effects release on unsubscribe. Shared read preparation consistently applies latency/error/stall controls to reads and aggregates. Keep those helpers, with the original change-stream pipeline and direct ordinary writes restored. |
87
+ | `batch-write.mock.service.ts.template` → `testing/mock-data.ts` | Keep staged atomic batches. The original parallel save functions could partly save a batch before another operation failed. Hard batch deletion, path-based IDs and rejecting a second commit also match the real batch API more closely. This is a real mock behavior change, not necessary merely to publish a package. |
88
+ | Mock persistence in `app.store.ts.template` → `testing/mock-entities.ts` | Configuration replaces the app environment import. Keep validating restored metadata/timestamps, falling back to seeds when storage is invalid, and removing the storage listener on destruction. `replaceAll` is needed so a successful batch or explicit test seed is not overwritten by older saved data. |
89
+ | Firebase timestamps → `src/timestamp.ts`, `firestore/timestamp.ts` | This is the explicitly requested shared timestamp value and adapter. Conversion is substantial new code because core must work without Firebase; it is not a cosmetic rewrite of the loader. |
90
+ | `logging.service.ts.template` → sibling `platform-logging` package | Keep the agreed shared logger API and configuration. Injector-owned state avoids a module-global service redirecting existing loggers; in-memory namespace checks avoid synchronous localStorage reads on each message and allow logging when storage is unavailable. The output sink permits other destinations. This remains a substantial deliberate redesign, not a direct file move. |
91
+ | Generated Firebase/auth facades | Auth flow, profile defaults, guards and navigation stay app-owned and largely unchanged. The real profile method now throws if a newly created profile cannot be read, instead of fabricating a successful profile without required metadata. Keep that explicit failure because authentication must confirm an actual profile. Logout changes implement the agreed listener/auth cleanup policy. |
92
+ | Entity models/mocks and `platform-tooling` registry generation | Generated registries replace manual entity lists as explicitly requested. The parser, typed definitions and production provider replacement are new functionality, not a rewrite of store loading. The entity schematic retains its rules/index insertion steps. |
93
+ | Existing index checker and other app templates | Relative to the tooling-extraction baseline, the checker adds recognition of package Firestore imports and explanatory comments. Other template changes primarily adapt imports, shared timestamp use, provider wiring and test seeds; this review found no additional broad UI or auth-flow rewrite. |
94
+
95
+ Paths beginning `src/`, `firestore/` or `testing/` in the table are relative to
96
+ `packages/data/`. Original filenames are under the baseline's `core/store`,
97
+ `core/firebase` or `core/logging` directories. Splitting `app.store.ts` into
98
+ files still makes a whole-file diff noisy: compare the named functions, not
99
+ only file sizes. The original-source commands above remain the reference.
100
+
101
+ Not every retained change is a packaging requirement. In particular, comparison
102
+ semantics, atomic mock batches and the logging redesign should remain visible
103
+ in release review. The empty-`in` selection policy in section 3.1 is still a
104
+ separate, unresolved behavior decision; the structural restorations did not
105
+ silently change it. Future cleanup/refactoring should be committed after the
106
+ extraction, in separate changes that can be compared against the package's first
107
+ version. This review does not commit or rewrite repository history.
108
+
109
+ ## 1. What moved into the package and how apps use it
110
+
111
+ These changes let the same package work with each app's own models and services:
112
+
113
+ | Before | Result and reason |
114
+ | --- | --- |
115
+ | Store helpers import the app's `AnyEntity` list and combined database/auth service. | `DataEntity` describes the fields all records need; `DataAccess` describes the database methods the stores use. Apps supply their own models and choose their database implementation through `DATA_ACCESS`. |
116
+ | Shared implementations are copied into each application's `core` folders. | Consumers import the package directly. Local `app.store`, cache/listener, multiplexer and batch forwarding files are not generated. Domain stores and named `LoadX`/`StreamX` descriptors remain app-owned. |
117
+ | Firebase, Material feedback, environment configuration and app logging are mixed into the engine. | Core, `/firestore`, `/testing` and `/material` have explicit boundaries. The slash paths are secondary entry points of **one** versioned npm package. Firebase and Material are optional peers used by the corresponding entry points. |
118
+ | Each app's real and mock database services handle both data and auth. | They now extend `FirestoreDataService` or `MockDataService` for data. Auth, creating/checking profiles, and account deletion stay local for now. A later auth package can share more of that code. |
119
+ | Store helpers depend directly on particular snackbar and batch services. | `DATA_FEEDBACK` and `BATCH_WRITE_SERVICE` let the app provide implementations of the `DataFeedback` and `BatchWriteService` interfaces. Core shows no messages by default; `/material` supplies snackbar feedback. |
120
+ | TypeScript says a read or selector always returns a record, even when none exists. | These methods now return `T \| null` in their types. Database updates accept app-defined fields, while the app's store still checks its own entity's field types. |
121
+
122
+ There are still deliberate naming aliases: package `DATABASE_SERVICE` is the
123
+ **same token** as `DATA_ACCESS`, and `DatabaseService` aliases `DataAccess`.
124
+ The app's local database interface adds auth methods to the shared data interface.
125
+ Providing these token names separately does not create separate services: the
126
+ last provider wins. The shared `AnyEntity` alias means `DataEntity`, not a list
127
+ of every model in the app. These aliases do not restore the deleted
128
+ local forwarding files.
129
+
130
+ The build uses Angular's library format and compilation mode. Publish the built
131
+ `dist` folder; a guard prevents accidentally publishing the source folder.
132
+ Apps also install the separate logging package, discussed below. Core and mock
133
+ tests deliberately block Firebase imports to check that neither requires it.
134
+
135
+ The data adapter uses the modular Firebase SDK directly; an app can still use
136
+ AngularFire for its local auth integration. The mock adapter allocates IDs with
137
+ browser crypto without importing Firestore. Real ID allocation creates a document
138
+ reference without writing a document.
139
+
140
+ Evidence: `packages/data/src/{public-api,data-access,model,config,feedback}.ts`,
141
+ `packages/data/test/types.ts`, `packages/data/test/no-firebase-loader.mjs`, and
142
+ the generated `core/firebase` provider/facade templates.
143
+
144
+ ## 2. Required record fields and timestamps without Firebase
145
+
146
+ Every stored record must now include `__id`, `_createdAt`, `_updatedAt` and
147
+ `_deleted`. An early version of the extraction made the last three optional;
148
+ we changed that after review. The loader needs `_deleted` to distinguish active
149
+ and deleted records and `_updatedAt` to fetch changes since the previous load.
150
+ We also require `_createdAt` as the creation date, although the change-loading
151
+ code does not filter on it.
152
+
153
+ `NewDataEntity<T>` is for a record you are about to create. `store.add()` fills
154
+ in missing IDs and metadata; calling the database adapter directly still requires
155
+ an ID. `init()`, direct mock inserts and test seed data must contain complete
156
+ stored records. The store no longer substitutes an empty string when `__id` is
157
+ missing. TypeScript cannot fix old documents or check code outside the app, so
158
+ adoption still needs to check existing data and other code that writes it.
159
+
160
+ Records now use `{ __type: 'DataTimestamp', seconds, nanoseconds }` instead of a
161
+ Firebase `Timestamp`. The factory returns a frozen value that can be saved as
162
+ JSON, so core can work without importing Firebase:
163
+
164
+ - Factories validate the numeric parts; millisecond/date conversion normalizes
165
+ negative/fractional instants. The direct `dataTimestamp(seconds, nanoseconds)`
166
+ constructor requires normalized parts and rejects out-of-range nanoseconds.
167
+ Comparisons and cache keys keep nanosecond precision. Converting to a Date
168
+ or a millisecond number can lose some of that precision.
169
+ - The tag prevents an ordinary domain object containing `seconds` and
170
+ `nanoseconds` from being mistaken for a timestamp.
171
+ - The Firestore adapter converts these values when sending records, filters,
172
+ pagination values and batch writes to Firestore, including values inside
173
+ arrays and nested objects. It converts Firebase timestamps back on reads and
174
+ streams. Other Firebase objects, such as document references, GeoPoints and
175
+ special write commands, are passed through intact.
176
+ - Firestore continues storing native timestamps, including server-generated
177
+ metadata. This changes application values and imports, not the stored timestamp
178
+ schema. While a local write is waiting for the server, reads use Firebase's
179
+ timestamp estimate instead of returning null metadata. A later update supplies
180
+ the server-confirmed value.
181
+
182
+ The adapter also assigns the document's real ID **after** copying its fields.
183
+ For example, a document at `problems/p1` containing an incorrect `__id: 'p2'`
184
+ now reaches the store as `__id: 'p1'`.
185
+
186
+ Evidence: `packages/data/src/timestamp.ts`, `packages/data/firestore/timestamp.ts`,
187
+ `packages/data/firestore/public-api.ts`; timestamp, Firestore unit and emulator
188
+ tests in `packages/data/test/`. See [BACKENDS.md](BACKENDS.md) for what this does
189
+ and does not solve for another database.
190
+
191
+ ## 3. Fixes to which records a query returns
192
+
193
+ A **selector** searches records already loaded into the frontend store. A
194
+ **backend query** asks Firestore or the mock database for records. These should
195
+ agree about filters and ordering, but serve different purposes: the backend
196
+ also needs to report deletions so the frontend can remove old records.
197
+
198
+ The examples below use small, illustrative records with unrelated fields and
199
+ metadata omitted. Source excerpts and diff lines below are copied from the
200
+ original or current implementation, with only leading indentation removed.
201
+ Each has a file and line reference. Diffs compare selected excerpts, sometimes
202
+ from different functions; they are not complete patches. Explanatory comments
203
+ are kept outside the source excerpts. Line numbers refer to this revision;
204
+ search the quoted code or function name if later edits move them.
205
+
206
+ ### 3.1 Asking for a missing ID should return nothing
207
+
208
+ Here, "missing ID" means a concrete value such as `'missing'` that has no record
209
+ in the store. It does **not** mean an `undefined` value from an optional field.
210
+ Suppose the store contains problems `p1` and `p2`:
211
+
212
+ ```ts
213
+ problemStore.selectEntities([['__id', '==', 'missing']], {});
214
+ // Before: [p1, p2]
215
+ // Now: []
216
+ ```
217
+
218
+ The old selector tried to speed up ID lookups by handling them separately. If an
219
+ ID was missing, it fell back to starting with every record. It then skipped the
220
+ ID filter because it assumed that filter had already been handled. A concrete
221
+ ID that is absent should return nothing; an undefined filter has different,
222
+ intentional behavior described below.
223
+
224
+ The original code only entered the single-ID branch if the record already
225
+ existed. Otherwise it could fall through to the all-records branch, then skip
226
+ checking the ID again. The fix keeps the direct lookup and returns `[]` when
227
+ the requested ID has no record.
228
+
229
+ The first extraction rewrote this into a general filter over all records.
230
+ Sharing value comparisons between the mock and selectors was useful, but did
231
+ not require removing the original ID-first branches. That rewrite made the
232
+ code harder to follow and lost the speed benefit of looking up IDs directly.
233
+ The review restored the original structure for **both `__id ==` and `__id in`**.
234
+
235
+ **Current source — exact excerpt:** `packages/data/src/selectors.ts:176`,
236
+ inside `runQuery`:
237
+
238
+ ```ts
239
+ if (uniqSelectEqIds.length === 1) {
240
+ // Exactly one required ID: look it up directly, even if it is not loaded.
241
+ const id = uniqSelectEqIds[0];
242
+ const entity = typeof id === 'string' && Object.hasOwn(entityMap, id) ? entityMap[id] : undefined;
243
+ // Missing IDs have no matches. Do not fall through to all loaded records.
244
+ if (!entity) return [];
245
+ filteredEntities.push(entity);
246
+ } else if (uniqSelectEqIds.length > 1) {
247
+ // One record cannot have two different IDs (e.g. __id == 'a' AND __id == 'b').
248
+ return [];
249
+ } else if (selectIdInQueries.length > 0) {
250
+ // Start with the first ID list, looking up each unique ID directly.
251
+ // Empty lists match nothing; missing IDs and undefined list entries add nothing.
252
+ const firstInIds = selectIdInQueries[0][2];
253
+ if (!Array.isArray(firstInIds)) return [];
254
+ filteredEntities = [...new Set(firstInIds)]
255
+ .filter(id => typeof id === 'string' && Object.hasOwn(entityMap, id))
256
+ .map(id => entityMap[id]);
257
+ } else {
258
+ // Without an ID ==/in filter, consider every record already in the map.
259
+ filteredEntities = Object.values(entityMap);
260
+ }
261
+ ```
262
+
263
+ `runQuery` now follows the same stages as the original `app.store.ts` selector:
264
+
265
+ 1. Collect the ID filters. One `==` ID supplies at most one record; different
266
+ `==` IDs cannot match. An `in` filter supplies records by looking up its IDs.
267
+ Only a query with no defined ID `==`/`in` filter starts with every record.
268
+ 2. Intersect all ID lists and remove deleted records for frontend selection.
269
+ This also applies when the `==` branch supplied the record. For example,
270
+ `__id == 'p1'` **and** `__id in ['p2']` must return nothing.
271
+ 3. Apply the remaining field filters and cursor bounds to `filteredEntities`.
272
+ The remaining-filter pass skips ID `==`/`in` because they are fully handled.
273
+ 4. Sort the matches and apply the limit last.
274
+
275
+ `selectEntities` and the mock-facing `queryEntities` are small wrappers around
276
+ these shared steps. Their only difference is whether to hide deleted records.
277
+ The comparison helpers remain separate to support dotted fields, timestamps,
278
+ arrays, and descending/multiple-field cursors. Those changes are described in
279
+ section 3.3; packaging itself does not require them.
280
+
281
+ Tests check that both ID branches avoid enumerating unrelated records, and
282
+ that missing/deleted IDs, intersecting lists, other filters, ordering, limits
283
+ and optional undefined values work. Duplicate IDs do not duplicate results.
284
+ Without an explicit `orderBy`, the restored `in` branch follows the first ID
285
+ list's order, as the original selector did. The intermediate full-map scan
286
+ instead followed the map's order. Use `orderBy` when result ordering matters.
287
+
288
+ **Separate behavior change: empty ID lists.** The original selector discarded
289
+ `__id in []` filters, including lists that became empty after removing undefined
290
+ entries. It then applied the other filters, or returned all active records if
291
+ none remained. The package currently treats an empty `in` list as matching no
292
+ records. This changes existing behavior and should not be described as merely
293
+ fixing the missing-concrete-ID bug. The intended empty-list behavior is under
294
+ review before release.
295
+
296
+ Undefined filter values remain intentionally ignored in both implementations.
297
+ For example, `__hashtagId == undefined` removes that restriction so an optional
298
+ relationship does not become a filter for an undefined database value.
299
+ Ignoring a filter means applying the remaining filters, not skipping the entire
300
+ query. The original nested-load helper separately skipped entire queries with
301
+ an empty `in` list. The user explicitly chose to keep that load behavior during
302
+ this review. Other nested queries still run, and a skipped load does not clear
303
+ already-loaded records. A regression test preserves this behavior. Potential
304
+ confusion between skipping a load, ignoring a filter and selecting no results
305
+ is recorded in [FUTURE-WORK.md](FUTURE-WORK.md#optional-filters-and-empty-in-lists).
306
+
307
+ ### 3.2 The mock database must report deleted records to the loader
308
+
309
+ The Firestore adapter does not automatically hide deleted records. It applies
310
+ the filters supplied by the store. When the store fetches multiple records,
311
+ `load()` makes **two separate queries**:
312
+
313
+ | Load | Query for active records | Query for deletions |
314
+ | --- | --- | --- |
315
+ | First load | App filters plus `_deleted == false` | App filters plus `_deleted == true` |
316
+ | Later incremental load | App filters plus `_updatedAt > lastLoadTime` and `_deleted == false` | App filters plus `_updatedAt > lastLoadTime` and `_deleted == true` |
317
+
318
+ Active results are added/updated in the frontend store. IDs returned by the
319
+ second query are removed from it. The `_deleted == false` query is only one
320
+ half of this process. This two-query behavior already existed in the original
321
+ template and is also present in MathPets and Mentor; it was not introduced by
322
+ the package extraction.
323
+
324
+ Here is the exact follow-up deletion query and removal code from
325
+ `packages/data/src/entity-store.ts:230`:
326
+
327
+ ```ts
328
+ const entitiesDeletedDelta: S[] = await db.getEntities(
329
+ collectionName,
330
+ [...queryParams,
331
+ ['_updatedAt', '>', dataTimestampFromMillis(lastLoadTime)],
332
+ ['_deleted', '==', true]],
333
+ queryOptions,
334
+ );
335
+
336
+ patchState(store, removeEntities(
337
+ entitiesDeletedDelta.map((e) => e.__id),
338
+ ));
339
+ ```
340
+
341
+ The batched equality/`in` path follows the same active/deleted split. A single
342
+ ID load instead reads the document directly and removes its local record if
343
+ it is absent or marked deleted. Loads with ordering/limits/cursors use full
344
+ queries rather than adding a timestamp cutoff. Data already covered by active
345
+ listeners can avoid these reads.
346
+
347
+ Suppose `p1` is already displayed and the database later marks it deleted.
348
+ The active query will not return it, but the deletion query should return its
349
+ ID so the store removes it. The old mock database called the UI selector,
350
+ which **always hid deleted records, even when explicitly asked for
351
+ `_deleted == true`**. That second query therefore returned nothing in the mock,
352
+ leaving the old record in the frontend store. The mock now uses a separate query
353
+ function that honors the supplied deletion filter:
354
+
355
+ Before: `src/ng-add/files/src/app/core/firebase/firebase.mock.service.ts.template:287` at `fda18e3`.
356
+ Now: `packages/data/testing/mock-data.ts:159` in the current source.
357
+
358
+ ```diff
359
+ -return db.selectEntities(queryParams, queryOptions);
360
+ +return queryEntities((await this.read(name)).entityMap(), params, options) as T[];
361
+ ```
362
+
363
+ The distinction is intentional:
364
+
365
+ ```ts
366
+ // The mock database contains a record marked as deleted.
367
+ const records = { p1: { __id: 'p1', _deleted: true } };
368
+
369
+ queryEntities(records, [['_deleted', '==', true]], {});
370
+ // Returns p1: the loader needs to learn about this deletion.
371
+
372
+ selectEntities(records, [['_deleted', '==', true]], {});
373
+ // Returns []: the frontend selector still hides deleted records.
374
+ ```
375
+
376
+ This changes the mock backend's behavior, not the UI's rule about hiding deleted
377
+ records. A deleted record kept for synchronization is sometimes called a
378
+ *tombstone*.
379
+
380
+ Regression tests now exercise the actual `MockDataService` together with the
381
+ frontend store, for both an ordinary query and the batched equality path. Each
382
+ test loads two records, deletes one through the mock backend, then loads again.
383
+ The deleted record must disappear from the frontend map while the other remains.
384
+ The mock backend keeps the deletion marker so later loads can discover it. The
385
+ tests control the clock to put the deletion after the first load's cutoff.
386
+
387
+ ### 3.3 Ordering and comparisons should mean what the query says
388
+
389
+ **Descending pagination:** suppose the scores are `8, 5, 2`. Starting *after 5*
390
+ in descending order should return `2`. The old cursor check always kept values
391
+ greater than 5, even when sorting in descending order:
392
+
393
+ ```ts
394
+ problemStore.selectEntities([], {
395
+ orderBy: ['score', 'desc'],
396
+ startAfter: 5,
397
+ });
398
+ // Before: the problem with score 8
399
+ // Now: the problem with score 2
400
+ ```
401
+
402
+ The comparison now accounts for the requested direction:
403
+
404
+ Before: `src/ng-add/files/src/app/core/store/app.store.ts.template:1048` at `fda18e3`.
405
+ Now: `packages/data/src/selectors.ts:129` in the current source.
406
+
407
+ ```diff
408
+ -if (queryOptions.startAfter !== undefined && value <= queryOptions.startAfter) {
409
+ - return false;
410
+ -}
411
+ +let result = 0;
412
+ +for (let i = 0; i < cursor.length; i++) {
413
+ + result = compare(fieldValue(entity, orders[i][0]), cursor[i]) * (orders[i][1] === 'desc' ? -1 : 1);
414
+ + if (result) break;
415
+ +}
416
+ +return key === 'startAt' ? result >= 0 : key === 'startAfter' ? result > 0 : key === 'endAt' ? result <= 0 : result < 0;
417
+ ```
418
+
419
+ For multiple sort fields, the cursor supplies values in the same order. For
420
+ example, `orderBy: [['group', 'asc'], ['score', 'desc']]` with
421
+ `startAfter: ['algebra', 5]` means "after algebra/5 in that combined ordering."
422
+ The old cursor code only considered the first sort field. The new code compares
423
+ the supplied fields in order. The Firestore adapter also forwards these cursor
424
+ values separately, converting any shared timestamps to SDK values.
425
+
426
+ Other comparison fixes are easier to see with examples:
427
+
428
+ | Example | Before | Now |
429
+ | --- | --- | --- |
430
+ | Filter on `stats.score` when a record contains `stats: { score: 5 }` | Looked for a property literally named `stats.score` | Reads the nested `score` field |
431
+ | Compare two separately created timestamps representing the same instant | Equality could depend on whether they were the same object | Compares the time values, including nanoseconds |
432
+ | Sort records where one score is `null` | Dropped the record with a null score | Keeps it; distinguishes null from a missing field |
433
+ | Compare numeric `5` and string `'5'` | JavaScript comparisons could blur the distinction | Preserves the types and uses a consistent order for supported values |
434
+
435
+ Local `__name__` filters now refer to the record's `__id`. Arrays and plain
436
+ objects are compared by their contents. Unsupported operators and invalid
437
+ cursor/limit values are rejected when evaluated instead of silently producing
438
+ a plausible result. This is still a subset of Firestore behavior; it does not
439
+ simulate security rules or check whether a deployed index exists.
440
+
441
+ ### 3.4 Different queries should not accidentally share a cache entry
442
+
443
+ The cache gives each query a key, like a label, to remember previous loads and
444
+ reuse active listeners. If two different queries get the same label, one can
445
+ incorrectly reuse the other's history or connection.
446
+
447
+ The old key converted filters to plain strings. For example, these different
448
+ queries could get the same key because both values became the text `5`:
449
+
450
+ ```ts
451
+ [['score', '==', 5]] // A number
452
+ [['score', '==', '5']] // A string
453
+ ```
454
+
455
+ It also omitted `startAt: 0` because `if (queryOptions.startAt)` treats zero as
456
+ false. A query starting at zero could therefore share a key with a query that
457
+ had no starting bound. Finally, `queryParams.sort()` changed the app's original
458
+ array while building the key.
459
+
460
+ The replacement records types as well as values and sorts a new array:
461
+
462
+ Before: `src/ng-add/files/src/app/core/firebase/cached-listeners.service.ts.template:93` at `fda18e3`.
463
+ Now: `packages/data/src/cached-listeners.ts:121` in the current source.
464
+
465
+ ```diff
466
+ -return `${collection}.${queryParams.sort()}.${queryOptionsString}`;
467
+ +return stableQueryKey(collection, queryParams, queryOptions);
468
+ ```
469
+
470
+ Inside `stableQueryKey`, a number is represented as `['number', '5']` and a
471
+ string as `['string', '5']`. Options keep zero values, and timestamps keep both
472
+ seconds and nanoseconds. Reordering the same AND filters still produces the
473
+ same key, without changing the caller's array.
474
+
475
+ ### 3.5 An aggregate's output name should not choose its calculation
476
+
477
+ This request means "count the records and return that count under the name
478
+ `totalProblems`":
479
+
480
+ ```ts
481
+ await db.aggregate('problems', [], {}, {
482
+ totalProblems: ['count'],
483
+ });
484
+ // With three records, the expected result is { totalProblems: 3 }.
485
+ ```
486
+
487
+ The old Firestore code checked whether the **output name** was `'count'`.
488
+ `totalProblems` therefore did not register a count calculation at all. The new
489
+ code reads the operation from the tuple and preserves the chosen output name:
490
+
491
+ Before: `src/ng-add/files/src/app/core/firebase/firebase.service.ts.template:561` at `fda18e3`.
492
+ Now: `packages/data/firestore/public-api.ts:93` in the current source.
493
+
494
+ ```diff
495
+ -for (const key in aggregateParams) {
496
+ - if (key === 'count') {
497
+ - aggregateSpec[key] = count();
498
+ - } else if (key === 'sum') {
499
+ - aggregateSpec[key] = sum(aggregateParams[key][1]);
500
+ - } else if (key === 'average') {
501
+ - aggregateSpec[key] = average(aggregateParams[key][1]);
502
+ - }
503
+ -}
504
+ +for (const [alias, [operator, field]] of Object.entries(params)) {
505
+ + if (operator === 'count') spec[alias] = count();
506
+ + else if (operator === 'sum' && field) spec[alias] = sum(field);
507
+ + else if (operator === 'average' && field) spec[alias] = average(field);
508
+ + else throw new Error(`Invalid aggregate ${alias}: ${operator}`);
509
+ +}
510
+ ```
511
+
512
+ The same applies to names such as `totalScore: ['sum', 'score']`. Invalid
513
+ operations or missing required fields now produce an error. An average with
514
+ no values returns null. The mock already understood output aliases; its
515
+ aggregation was updated to handle the empty-average and validation cases too.
516
+
517
+ ### What stayed the same
518
+
519
+ A filter whose value is `undefined` is still ignored. For example,
520
+ `[['__userId', '==', undefined]]` does **not** mean "return nothing": it removes
521
+ that restriction. Apps still need to check that a user ID exists before relying
522
+ on this filter.
523
+
524
+ Selectors already reused their previous result array when its records had not
525
+ changed, with space for 50 cached queries. That behavior was not added during
526
+ extraction. A comment was corrected: the cache drops the oldest inserted key,
527
+ not necessarily the least recently used one.
528
+
529
+ Source: `packages/data/src/selectors.ts`, `packages/data/firestore/public-api.ts`
530
+ and `packages/data/testing/mock-data.ts`. Tests: `packages/data/test/data.test.mjs`
531
+ (missing IDs, empty lists, deletions, cursor ordering and cache keys),
532
+ `packages/data/test/firestore.test.mjs` (aggregate aliases), and
533
+ `packages/data/test/legacy-query.spec.ts` (existing query cases).
534
+
535
+ ## 4. Fixes to related-data loading and live listeners
536
+
537
+ A **nested query** loads related records after loading a parent record. For
538
+ example: load collections, then load the problems belonging to each collection.
539
+ A **listener** keeps a query up to date as the database changes. The code can
540
+ combine compatible nested queries to avoid opening one listener per parent.
541
+
542
+ These fixes address skipped related data, unnecessary work and listeners that
543
+ could remain active after they should have stopped. Examples are illustrative;
544
+ source excerpts and diff lines are verbatim under the rules in section 3.
545
+
546
+ ### 4.1 One parent with no related queries should not hide the others
547
+
548
+ An app can return a different number of nested queries for each parent:
549
+
550
+ ```ts
551
+ collection => collection.problemCount === 0
552
+ ? []
553
+ : [new LoadProblem(problemStore, [['__collectionId', '==', collection.__id]], {})]
554
+ ```
555
+
556
+ For an empty collection followed by a nonempty collection, that produces:
557
+
558
+ ```ts
559
+ [
560
+ [], // First collection: nothing to load.
561
+ [secondProblemsQuery], // Second collection: load its problems.
562
+ ]
563
+ ```
564
+
565
+ The old grouping code used only the **first parent's array length** to decide
566
+ how many positions to process. Because that array was empty, it processed no
567
+ nested queries at all. If a later parent had two queries and the first had only
568
+ one, the later parent's second query could similarly be skipped.
569
+
570
+ Now it uses the longest array and skips empty positions:
571
+
572
+ Before: `src/ng-add/files/src/app/core/store/app.store.ts.template:1159` at `fda18e3`.
573
+ Now: `packages/data/src/query-descriptors.ts:16` in the current source.
574
+
575
+ ```diff
576
+ -const queriesByQueryType = queriesByEntity[0].map((_, colIndex) => queriesByEntity.map((row) => row[colIndex]));
577
+ +const queriesByQueryType = Array.from({ length: Math.max(0, ...queriesByEntity.map(row => row.length)) }, (_, colIndex) => queriesByEntity.map(row => row[colIndex]).filter(Boolean));
578
+ ```
579
+
580
+ This correction applies to both one-time nested loads and nested streams.
581
+
582
+ ### 4.2 Combine queries only when they ask for compatible data
583
+
584
+ Combining these requests is useful and was already supported:
585
+
586
+ ```ts
587
+ // Both ask the same ProblemStore for problems in a collection.
588
+ ['__collectionId', '==', 'c1']
589
+ ['__collectionId', '==', 'c2']
590
+ // Can become one request: ['__collectionId', 'in', ['c1', 'c2']]
591
+ ```
592
+
593
+ But matching field names are not enough. One request could target `problems`
594
+ and another `problemAttempts`; both can have a `__collectionId` field. The old
595
+ check could combine them, then send the combined request to the first query's
596
+ collection/store. The other data would never be requested from its proper place.
597
+
598
+ After checking that every request has one equality filter and no extra options,
599
+ the code now checks the collection, store and follow-up function too:
600
+
601
+ Before: `src/ng-add/files/src/app/core/store/app.store.ts.template:1217` at `fda18e3`.
602
+ Now: `packages/data/src/query-descriptors.ts:79` in the current source.
603
+
604
+ ```diff
605
+ -return queries.every((q) => q.queryParams[0][0] === firstField && q.queryParams[0][1] === '==');
606
+ +return queries.every((q) => q.collection === queries[0].collection && q.store === queries[0].store && q.loadQueries === queries[0].loadQueries && q.queryParams[0][2] !== undefined && q.queryParams[0][0] === firstField && q.queryParams[0][1] === '==');
607
+ ```
608
+
609
+ Why check `loadQueries`? Two requests might load the same kind of record but
610
+ then load different related records. Keeping only the first follow-up function
611
+ would drop the second request's work. The stream version makes the same check
612
+ for `streamQueries`. Separate function objects are kept separate even if their
613
+ bodies happen to look alike.
614
+
615
+ This does not introduce multiple ProblemStore instances. It prevents combining
616
+ queries that belong to different existing stores or have different follow-up work.
617
+
618
+ ### 4.3 A timestamp alone should not decide whether related data changed
619
+
620
+ Suppose a problem's `__collectionId` changes from `c1` to `c2`. A nested stream
621
+ that follows that field needs to respond to the change. The old check looked
622
+ only at whether `_updatedAt` was a different object:
623
+
624
+ Before: `src/ng-add/files/src/app/core/firebase/cached-listeners.service.ts.template:269` at `fda18e3`.
625
+ Now: `packages/data/src/cached-listeners.ts:310` in the current source.
626
+
627
+ ```diff
628
+ -} else if (currEntities[key]['_updatedAt'] !== prevEntities[key]['_updatedAt']) {
629
+ +} else if (entityChanged(prevEntities[key], currEntities[key])) {
630
+ ```
631
+
632
+ The old check had two weaknesses:
633
+
634
+ - A new record object could have changed fields while reusing the same timestamp
635
+ object. That change was missed.
636
+ - A reread could create a new timestamp object representing exactly the same
637
+ time. That alone was treated as a change and could repeat nested-query work.
638
+
639
+ `entityChanged` compares timestamps by their time values and checks the other
640
+ top-level fields as well. For example:
641
+
642
+ ```ts
643
+ const time = dataTimestampNow();
644
+ const before = { __id: 'p1', __collectionId: 'c1', _updatedAt: time };
645
+ const after = { ...before, __collectionId: 'c2' };
646
+ // Same timestamp object, changed relationship: now detected.
647
+ ```
648
+
649
+ This also avoids treating equal timestamp values as proof that nothing else
650
+ changed during that clock tick. It does not detect every edit made inside the
651
+ same nested object; callers should continue replacing changed objects.
652
+
653
+ ### 4.4 Stopping database listeners must also stop their follow-up work
654
+
655
+ A nested stream watches parent records in the store so it can open new child
656
+ listeners when a relationship changes. That watcher is an Angular effect,
657
+ separate from the database listener itself.
658
+
659
+ Previously, its lifetime was tied to the component's injector. If logout stopped
660
+ database listeners while the component was still alive, the watcher could still
661
+ react to a store change and start related work again. The wrapper now lets
662
+ either component destruction **or global stream teardown** stop the watcher:
663
+
664
+ Before: `src/ng-add/files/src/app/core/firebase/cached-listeners.service.ts.template:241` at `fda18e3`.
665
+ Now: `packages/data/src/cached-listeners.ts:282` in the current source.
666
+
667
+ ```diff
668
+ -toObservable(streamEntities, { injector }).pipe(
669
+ +this.observeNested(streamEntities, injector).pipe(
670
+ ```
671
+
672
+ `observeNested` registers a cleanup callback with the component and with
673
+ `CachedListenersService`. Cleanup completes the subscription and destroys its
674
+ effect. This extends the existing component cleanup to cover global teardown.
675
+
676
+ There is also a stream-generation counter: starting work remembers the current
677
+ counter, and global teardown advances it. A delayed nested callback checks that
678
+ its counter is still current before continuing. This prevents old stream work
679
+ from restarting listeners. Ordinary one-time reads and writes still finish
680
+ normally; see section 5 for that deliberate distinction.
681
+
682
+ Errors from asynchronous follow-up work are now caught by the stream pipeline.
683
+ The two current excerpts below are at separate points: `mergeMap` starts the
684
+ async work inside the pipeline, and the subscription handles its errors:
685
+
686
+ Before: `src/ng-add/files/src/app/core/firebase/cached-listeners.service.ts.template:284` at `fda18e3`.
687
+ Now: `packages/data/src/cached-listeners.ts:325` in the current source; `packages/data/src/cached-listeners.ts:358` in the current source.
688
+
689
+ ```diff
690
+ -).subscribe(async (changes) => {
691
+ +mergeMap(async (changes) => {
692
+ +).subscribe({ error: error => this.reportStreamError(error) });
693
+ ```
694
+
695
+ Previously, a rejected Promise inside the async subscriber could become an
696
+ unhandled rejection. It now reaches `CachedListenersService.errors$`, where an
697
+ app can observe it. If a backend listener fails, its cached connection is also
698
+ removed so a later request can reconnect. This includes a failure reported
699
+ immediately during subscription; it must not leave a dead connection marked
700
+ active. The app still decides whether and when to retry.
701
+
702
+ ### 4.5 Keep old listeners until all their replacements are ready
703
+
704
+ The **multiplexer** manages shared listeners for groups of requested IDs. As
705
+ components come and go, it may replace several small listener groups with fewer,
706
+ larger groups. The replacement listeners need their first database response
707
+ before the old ones can safely close.
708
+
709
+ Suppose the replacement creates two new listeners, A and B:
710
+
711
+ | Event | Before | Now |
712
+ | --- | --- | --- |
713
+ | A receives its first response; B is still waiting | Closed all the old listeners | Keeps the old listeners |
714
+ | B receives its first response | The old listeners were already gone | Now closes the old listeners |
715
+ | Logout happens before either replacement responds | Old listeners waiting for replacement could escape cleanup | Closes both old and new listeners |
716
+
717
+ Closing the old listeners after only A responds can leave some records without
718
+ updates while B is still waiting. The new code counts how many replacements
719
+ have yet to respond. The initialization and first-response check are shown
720
+ together below, though they occur on separate lines in the method:
721
+
722
+ Before: `src/ng-add/files/src/app/core/firebase/stream-multiplexer.ts.template:362` at `fda18e3`.
723
+ Now: `packages/data/src/stream-multiplexer.ts:360` in the current source; `packages/data/src/stream-multiplexer.ts:373` in the current source.
724
+
725
+ ```diff
726
+ -this.teardownOldListeners(oldBatchesToTeardown);
727
+ +let remaining = newChunks.length;
728
+ +if (--remaining === 0) this.teardownOldListeners(oldBatchesToTeardown);
729
+ ```
730
+
731
+ It also keeps old listeners in a `retiring` set until they close. That name means
732
+ "scheduled for replacement but still running." Global teardown checks this set
733
+ as well as the current listeners, so cleanup does not depend on a replacement
734
+ ever responding. A generation check also ignores obsolete listener callbacks.
735
+
736
+ ### What stayed the same
737
+
738
+ Components still share one root store per collection and reuse matching database
739
+ listeners. After the last component using a listener leaves, the normal
740
+ five-second grace period still allows quick navigation back without reopening
741
+ it. Navigation keeps the loaded records and load history. Global teardown stops
742
+ listeners immediately; it does not itself empty the stores.
743
+
744
+ Source: `packages/data/src/query-descriptors.ts` (grouping/combining queries),
745
+ `packages/data/src/cached-listeners.ts` (change detection, watchers and errors),
746
+ and `packages/data/src/stream-multiplexer.ts` (listener replacement). Tests in
747
+ `packages/data/test/data.test.mjs` cover empty first rows, different destinations
748
+ and follow-up functions, component cleanup, failed listeners and replacement
749
+ listeners that have not responded.
750
+
751
+ ## 5. Final reset and logout policy
752
+
753
+ We deliberately chose this behavior during review. Each collection has **one
754
+ shared root store**, used throughout the app.
755
+
756
+ | Operation | Active listeners/nested observers | Entity records | Incremental load history |
757
+ | --- | --- | --- | --- |
758
+ | `teardownAllStreams()` | Stops all, including retiring listeners | Retained | Retained for every collection |
759
+ | `store.reset()` | Unchanged; teardown is separate | Clears this store | Clears both history maps for its collection only |
760
+ | `clearLoadHistory(collection)` | Unchanged | Retained | Clears that collection only |
761
+ | `clearCache()` | Stops all | Retained | Clears all collections |
762
+
763
+ Load history remembers when a query or a batch of related records was last
764
+ loaded, so the next load can ask only for newer changes. Both kinds of history
765
+ are now grouped by collection. Keeping a store keeps that history; resetting
766
+ it makes its next load start fresh. Resetting UserStore leaves other
767
+ collections' records and remembered load times alone.
768
+
769
+ Generated auth uses `clearSessionState()` to stop global listeners, reset
770
+ UserStore and the available UserContextStore, and reset the available telemetry
771
+ session ID. Logout and guest deletion first wait for tracked profile setup to
772
+ finish, then clean up, sign out/delete as appropriate, clear the signed-in user
773
+ and navigate. Apps choose any additional auth-state cleanup.
774
+ Other domain stores retain their records and history.
775
+
776
+ Already-running ordinary reads and writes can still complete and update local
777
+ state/history. This was explicitly accepted; a reset is not a guarantee that a
778
+ pending Promise cannot repopulate a store. It can affect what a selector returns,
779
+ not merely produce a console message. The generated authenticated-user selector
780
+ checks auth identity, so late user data alone does not sign someone back in.
781
+
782
+ Removed intermediate designs:
783
+
784
+ - Automatic store registration and `DataStoreRegistry`/`resetAllStores` behavior.
785
+ - Tracking which independent instance of a store owns each cache entry. We use
786
+ one shared root store per collection instead.
787
+ - A session counter that discarded results from ordinary pending reads/writes
788
+ after logout, and automatic record clearing when a store was destroyed.
789
+
790
+ Entity **model/mock discovery** still exists; it does not register stores for
791
+ automatic auth cleanup. `clearCache()` remains useful for explicit global
792
+ invalidation and tests, but is not the normal logout operation.
793
+
794
+ ### Historical reason for teardown
795
+
796
+ Mentor commit `9516618` (May 26) introduced cache cleanup with the caching
797
+ infrastructure. Commit `5b132ced177bbfae9fee3e5c6c7319b926797814` (July 28,
798
+ `feat(auth): implement passwordless email link authentication state and guards`)
799
+ added logout teardown with a comment explaining that listeners created under
800
+ one user's credentials should not issue denied reads after sign-out/deletion.
801
+
802
+ That commit's reset list covered all **nine** entity stores then present.
803
+ Conversation, ChatMessage and ConvUnit arrived in August 7's `6bfcdde` without
804
+ being added to the list. The omissions therefore do not establish an intentional
805
+ policy of keeping some stores. The history explains the listener problem but
806
+ does not identify a separate incident requiring every cached record to be
807
+ erased. MathPets has its own smaller explicit reset list, and its old global
808
+ teardown also clears history despite a contradictory comment.
809
+
810
+ The chosen package/template policy preserves domain data deliberately. It is
811
+ not identical to Mentor's original apparent reset-all intent. It concerns
812
+ in-memory application stores/listeners, not deleting backend data or clearing
813
+ Firebase SDK persistence. Monitoring requirements are in
814
+ [FUTURE-WORK.md](FUTURE-WORK.md#1-monitor-the-chosen-session-cleanup-policy).
815
+
816
+ Evidence: `packages/data/src/{entity-store,cached-listeners}.ts`; the generated
817
+ `core/store/auth/auth.store.ts.template`, its tests and `app-data.spec.ts.template`;
818
+ the collection-reset, retained-history and late-read tests in
819
+ `packages/data/test/data.test.mjs`.
820
+
821
+ ## 6. Batch writes and Undo corrections
822
+
823
+ Real and mock batches already shared a coordinator before extraction. The package
824
+ keeps that arrangement and its separate optimistic/non-optimistic branches.
825
+ The coordinator waits for asynchronous preparation and catches failures in either
826
+ preparation or saving. The loading flag stays true across both stages and resets
827
+ in `finally`, even on failure.
828
+
829
+ In **optimistic** mode, the frontend changes immediately, before the server
830
+ confirms the write. A preparation failure now reverses those local changes;
831
+ previously they could remain in the store. With optimistic mode off, the frontend
832
+ changes only after the batch saves successfully.
833
+
834
+ Reversing changes now happens in reverse order. For example, if a batch changes
835
+ a title from A to B and then B to C, undoing C to B must happen before undoing B
836
+ to A. Undo prepares these database writes one at a time, saves them, then
837
+ reverses the frontend changes in the same order. The old parallel/forward order
838
+ could restore the wrong state when writes depended on each other. If database
839
+ Undo fails, the frontend is not falsely restored. Undo still makes new writes;
840
+ it cannot cancel a completed write or resolve conflicts with someone else's edit.
841
+
842
+ The batch function remembers whether saving succeeded. If showing a snackbar
843
+ or updating local state fails afterward, it must not reverse optimistic changes
844
+ that the database already saved. The operation can still report an error in
845
+ that case; see the future-work notes.
846
+
847
+ Mock batches prepare and check all changes before applying any of them. If a
848
+ collection is unknown, an updated document is missing, or a simulated failure is
849
+ enabled, the earlier writes are not partially saved. Mock batches also support
850
+ deletion, use the document path's ID, and cannot be committed twice.
851
+
852
+ Source: `packages/data/src/{batch-write,undo}.ts` and
853
+ `packages/data/testing/mock-data.ts`. Tests in `packages/data/test/data.test.mjs`
854
+ cover preparation/save failures, reverse-order Undo and mock batches that must
855
+ save either all their changes or none.
856
+
857
+ ## 7. Mock database and saved development data
858
+
859
+ The app supplies its mock collections and seed data to the shared mock database.
860
+ The package no longer imports every app model/store. An unknown collection
861
+ produces an error. Providers reuse the same mock collection and database
862
+ instances, so the app and its tests see the same data.
863
+
864
+ Mock streams stop watching data when their subscriber unsubscribes. Each
865
+ subscriber tracks its own previous results, so a second subscriber also receives
866
+ the initial records. Reads, counts and aggregates now use the same loading code;
867
+ configured delays, failures and stalled requests behave consistently. Backend
868
+ queries can return deleted records as described in section 3.2.
869
+
870
+ The change stream keeps the original snapshot/pairwise/diff stages. It also
871
+ reports records that leave the query, even if they are not globally deleted.
872
+ A removal carries the current record's deletion marker when available, so the
873
+ frontend can distinguish those cases. Empty initial snapshots are delivered:
874
+ the multiplexer needs them to know that a replacement listener is ready.
875
+
876
+ Ordinary add/update/remove operations use the collection's individual mutation
877
+ methods, as before extraction. Only batch commits stage and replace whole
878
+ collections. This avoids extra full-collection copying on each ordinary write
879
+ and retains custom collection behavior in those methods.
880
+
881
+ Saving mock data in browser storage is opt-in through `MOCK_PERSISTENCE`.
882
+ The package no longer imports an app environment flag to decide. It also works
883
+ without a browser. If saved data is invalid or storage is unavailable, it uses
884
+ seed data and logs the problem. Cleanup removes the storage-event listener.
885
+
886
+ Restoring saved data checks required metadata and supports both shared timestamps
887
+ and the older timestamp JSON format without importing Firebase. Applying a batch
888
+ does not restore older saved data over the new result. This is still a development
889
+ mock; it does not enforce Firestore security rules or serve as a production
890
+ offline database.
891
+
892
+ Source: `packages/data/testing/{mock-data,mock-entities,entity-mocks}.ts`.
893
+ Tests: mock/saved-data cases in `packages/data/test/data.test.mjs` and public
894
+ API type checks in `packages/data/test/types.ts`.
895
+
896
+ ## 8. Generating the app's entity and mock lists
897
+
898
+ Platform-tooling and the schematics now generate the lists of app entities and
899
+ mock collections. Platform-data does not scan application modules at runtime:
900
+
901
+ - Models declare `defineEntity(...)` in their existing `.model.ts` file. Optional
902
+ `.mock.ts` definitions declare mock stores/seeds with `defineEntityMock(...)`.
903
+ - The tooling generator reads source code without executing app modules. It
904
+ writes `src/app/core/store/entities.generated.ts` and
905
+ `entity-mocks.generated.ts` into the app. These are committed so the editor
906
+ can use them, regenerated by development/build/test commands, and checked
907
+ for freshness in CI.
908
+ - The generated entity map replaces the handwritten `AnyEntity` list. The mock
909
+ registry replaces per-entity edits to `FirebaseMockService`. An entity without
910
+ a custom mock gets an empty collection with the correct type. Passing an empty
911
+ seed list deliberately keeps it empty. Invalid/duplicate definitions stop
912
+ generation with an error.
913
+ - Schematics generate the declarations and update the lists. App-specific
914
+ Firestore rules and indexes still require deliberate maintenance.
915
+ - Generated production providers exclude mock lists and seed imports through
916
+ an Angular file replacement. A test build used identifiable mock seed values
917
+ and checked that they were absent from the production bundle.
918
+
919
+ Adding a model does not publish a new library or cause automatic store resets.
920
+ Generation stays in the application during development and production builds.
921
+
922
+ Source: `packages/tooling/src/entity-registry.ts` and its tests,
923
+ `src/shared/entity-registry.ts`, `src/entity/`, and the generated
924
+ `core/firebase/data.providers*.ts.template` files. The package README and
925
+ platform-tooling README describe the commands and file conventions.
926
+
927
+ ## 9. Logging, comments and where tests live
928
+
929
+ General runtime logging lives in the sibling `@tech4good/platform-logging`
930
+ package. Data uses `injectLogger('data')`; apps can use other namespaces with
931
+ the same configured logger and output destination. It belongs outside build-time
932
+ tooling and outside the data-specific API.
933
+
934
+ The intermediate `DataLogger`, `DATA_LOGGER`, no-op data-logger export and
935
+ `providePlatformData` logger option were removed. The app configures logging
936
+ through providers. Each log entry includes its namespace (such as `data`), level
937
+ (such as `error`) and logged values. The old `query` namespace is now `data`.
938
+ Logging is off by default; apps can enable it and optionally save settings and
939
+ use development controls. Generated production configuration leaves those
940
+ controls off. `DATA_FEEDBACK` remains separate because it displays write feedback
941
+ to the user, such as a success snackbar.
942
+
943
+ Exported APIs and cleanup methods now have explanatory comments. Unused imports
944
+ and misleading comments were cleaned up. The original 63 query tests moved into
945
+ the package; generated apps keep tests for their own provider setup and auth
946
+ flows. Tests that expected the missing-concrete-ID bug or used invalid setup
947
+ were corrected rather than skipped. Empty-list expectations were also changed
948
+ to match the new implementation; as noted in section 3.1, that is a separate
949
+ behavior change whose intended policy remains under review.
950
+
951
+ The `simulate-load.spec.ts`, `simulate-stream.spec.ts` and `simulation.mock.ts`
952
+ files also moved from `src/ng-add/files/src/app/core/firebase/` into
953
+ `packages/data/test/`. They test the shared load/stream algorithms, so newly
954
+ generated apps should not each receive their own copy. The simulation steps,
955
+ timing, assertions and detailed scenario comments are preserved. The move
956
+ replaces app-owned `DATABASE_SERVICE`/`provideFirebaseMock` with the package's
957
+ `DATA_ACCESS`/`provideMockData`, uses local data-only fixtures instead of app
958
+ profile models, and adapts Jest spies/matchers to the existing package Jasmine
959
+ runner. No runtime load/stream code changes as part of this move. The fixture
960
+ and specs are private repository tests, not exports of the `/testing` entry point.
961
+
962
+ ## 10. Validation record and its limits
963
+
964
+ The collection-specific load-history cleanup milestone was validated on
965
+ September 10, before the later ID lookup review. That validation recorded:
966
+
967
+ | Scope | Result |
968
+ | --- | --- |
969
+ | Repository `npm test` (includes build) | 224 passed: 65 tooling, 14 logging, 41 data core/timestamp, 3 Firestore unit, 63 legacy query and 38 schematics tests |
970
+ | Disposable Angular 21 app using a fresh installed data tarball | Lint, production build and entity-registry freshness passed; 120 tests in 40 suites |
971
+ | That application's Firestore rules tests | 41 passed in 2 suites; final CLI exit 0 |
972
+ | Earlier shared-timestamp adapter validation | 8 real Firestore emulator tests passed; these were not rerun for the subsequent history-only change |
973
+
974
+ Public API type checks and Firebase-free core tests are part of the package
975
+ suite. The generated-app run used actual installed tarballs, not workspace
976
+ symlinks, and checked retained domain data/history through logout and guest
977
+ deletion and per-collection reset behavior.
978
+
979
+ The later ID lookup update passed the repository suite with **227 tests**,
980
+ including 44 data core/timestamp tests. Its three additional regressions cover
981
+ lookup without enumerating unrelated records, remaining filters and undefined
982
+ optional IDs, and preserving the nested helper's empty-`in` query skip.
983
+
984
+ Restoring the explicit ID `==`/`in` branches and testing mock deletion through
985
+ the real loader then passed **230 repository tests**, including 47 data
986
+ core/timestamp tests. The three added tests cover direct ID-list lookups with
987
+ intersecting/remaining filters, and deletion cleanup through ordinary and
988
+ batched mock loads. These use the actual mock adapter and frontend store rather
989
+ than only testing the raw query helper.
990
+
991
+ The broader extraction review then passed **233 repository tests**: 65 tooling,
992
+ 14 logging, 49 data core/timestamp, 4 Firestore unit/construction, 63 legacy query
993
+ and 38 schematics tests. The three added regressions verify direct ordinary
994
+ mock writes, the restored mock change-stream pipeline (including independent
995
+ subscribers and deletion/query-removal payloads), and Firestore cursor construction
996
+ independent of options-property order. The **8 real Firestore adapter emulator
997
+ tests were rerun and passed** after the adapter branch restoration.
998
+
999
+ The disposable harness disabled Angular's native disk cache after an environment
1000
+ crash, used neutral branding to avoid an existing broad Logout-text test matcher,
1001
+ and used task-local npm/Firebase CLI caches. Repository defaults were not changed;
1002
+ existing build-budget warnings remained. Temporary logs were recorded under
1003
+ `/private/tmp/platform-load-history-validation`, but this summary is retained
1004
+ because those files may disappear.
1005
+
1006
+ These results do not establish final MathPets/Mentor migration compatibility,
1007
+ production composite-index coverage, or observed behavior for every real
1008
+ account-switch/offline scenario. Earlier application adoption trials predated
1009
+ the final strict metadata, timestamp, registry and logout refinements. Rebuild
1010
+ and retest the exact release candidate when adopting it; see
1011
+ [FUTURE-WORK.md](FUTURE-WORK.md) for the next checks.