neotoma 0.22.2 → 0.23.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (67) hide show
  1. package/README.md +2 -2
  2. package/dist/actions.d.ts.map +1 -1
  3. package/dist/actions.js +58 -54
  4. package/dist/actions.js.map +1 -1
  5. package/dist/config.d.ts +19 -0
  6. package/dist/config.d.ts.map +1 -1
  7. package/dist/config.js +79 -3
  8. package/dist/config.js.map +1 -1
  9. package/dist/docs/developer/mcp/compact_instructions.md +2 -0
  10. package/dist/docs/developer/mcp/instructions.md +30 -3
  11. package/dist/docs/developer/mcp/proxy.md +6 -4
  12. package/dist/docs/foundation/core_identity.md +45 -4
  13. package/dist/docs/foundation/developer_release_principles.md +4 -1
  14. package/dist/docs/foundation/layered_architecture.md +10 -0
  15. package/dist/docs/foundation/mmry_comparison.md +4 -4
  16. package/dist/docs/foundation/problem_statement.md +3 -1
  17. package/dist/docs/foundation/product_positioning.md +21 -17
  18. package/dist/docs/foundation/redlines.md +21 -2
  19. package/dist/docs/foundation/scope_decisions.md +32 -10
  20. package/dist/docs/getting_started/what_is_neotoma.md +1 -1
  21. package/dist/docs/icp/icp_from_functionality.md +4 -2
  22. package/dist/docs/icp/icp_reconciliation.md +12 -1
  23. package/dist/docs/operations/configuration.md +54 -42
  24. package/dist/docs/reference/error_codes.md +27 -0
  25. package/dist/docs/specs/MCP_SPEC.md +1 -1
  26. package/dist/docs/subsystems/conflict_resolution.md +11 -6
  27. package/dist/docs/subsystems/observation_architecture.md +2 -2
  28. package/dist/docs/subsystems/reducer.md +29 -3
  29. package/dist/docs/subsystems/schema_registry.md +11 -0
  30. package/dist/docs/testing/automated_test_catalog.md +20 -9
  31. package/dist/mcp_http_session.d.ts +51 -0
  32. package/dist/mcp_http_session.d.ts.map +1 -0
  33. package/dist/mcp_http_session.js +173 -0
  34. package/dist/mcp_http_session.js.map +1 -0
  35. package/dist/reducers/observation_reducer.d.ts.map +1 -1
  36. package/dist/reducers/observation_reducer.js +24 -6
  37. package/dist/reducers/observation_reducer.js.map +1 -1
  38. package/dist/server.d.ts +31 -2
  39. package/dist/server.d.ts.map +1 -1
  40. package/dist/server.js +189 -16
  41. package/dist/server.js.map +1 -1
  42. package/dist/services/field_canonicalization.d.ts +1 -0
  43. package/dist/services/field_canonicalization.d.ts.map +1 -1
  44. package/dist/services/field_canonicalization.js +15 -9
  45. package/dist/services/field_canonicalization.js.map +1 -1
  46. package/dist/services/interpretation.d.ts +2 -0
  47. package/dist/services/interpretation.d.ts.map +1 -1
  48. package/dist/services/interpretation.js +2 -1
  49. package/dist/services/interpretation.js.map +1 -1
  50. package/dist/services/schema_definitions.d.ts.map +1 -1
  51. package/dist/services/schema_definitions.js +108 -0
  52. package/dist/services/schema_definitions.js.map +1 -1
  53. package/dist/services/schema_registry.d.ts +16 -1
  54. package/dist/services/schema_registry.d.ts.map +1 -1
  55. package/dist/services/schema_registry.js.map +1 -1
  56. package/dist/services/skills/instance_skills.d.ts +137 -0
  57. package/dist/services/skills/instance_skills.d.ts.map +1 -0
  58. package/dist/services/skills/instance_skills.js +381 -0
  59. package/dist/services/skills/instance_skills.js.map +1 -0
  60. package/dist/services/store_warning_rule.d.ts +97 -0
  61. package/dist/services/store_warning_rule.d.ts.map +1 -0
  62. package/dist/services/store_warning_rule.js +175 -0
  63. package/dist/services/store_warning_rule.js.map +1 -0
  64. package/dist/tool_definitions.js +1 -1
  65. package/dist/tool_definitions.js.map +1 -1
  66. package/docs/developer/mcp/instructions.md +30 -3
  67. package/package.json +2 -2
@@ -19,8 +19,9 @@ them costly to cross.
19
19
 
20
20
  ## The substantive position
21
21
 
22
- Neotoma exists to provide a deterministic state layer for AI agent memory,
23
- built on write integrity rather than retrieval performance. The architectural
22
+ Neotoma exists to be the system of record for AI agents — the designated
23
+ place where a fact about agent-held state is settled — built on write
24
+ integrity rather than retrieval performance. The architectural
24
25
  commitments named in [`core_identity.md`](core_identity.md) and
25
26
  [`philosophy.md`](philosophy.md) (append-only observations, hash-based entity
26
27
  identity, schema-bound writes, pipeline provenance, replayable timelines,
@@ -180,6 +181,24 @@ copy or product positioning that implies Neotoma is "also" a retrieval system,
180
181
  "also" an embeddings system, or "also" memory-as-cognition. Neotoma is the
181
182
  state-integrity position. Other positions exist; Neotoma is not them.
182
183
 
184
+ <a id="never-say-the-agent-forgot"></a>
185
+ **`never_say_the_agent_forgot` (decided 2026-08-21) is the sharpest instance of
186
+ this redline, and the one crossed most often.** Copy must never say an agent
187
+ *forgot*. Forgetting is the recall framing, and any retrieval vendor answers it
188
+ with better recall — so stating the problem that way argues their case. The
189
+ accurate framing is that nothing was committed to a record: a fact held only in
190
+ context was never written anywhere authoritative, so there is nothing to forget
191
+ and nothing to correct. Retired outright: "Your agents forget. Neotoma makes
192
+ them remember." and "your agents forget what they knew yesterday." The test any
193
+ problem statement must pass is the competitor test — if mem0, Zep, or any
194
+ retrieval vendor could plausibly claim to solve the scenario as written, it
195
+ frames their category and must be rewritten. Verbatim quotation of user research
196
+ is exempt: what a user said is evidence, not Neotoma's own voice. Canonical
197
+ statement in
198
+ [`product_positioning.md`](product_positioning.md) ("Primary message
199
+ (external)"); applied in
200
+ [`problem_statement.md`](problem_statement.md).
201
+
183
202
  ### R14. No pricing model that punishes integrity use
184
203
 
185
204
  If storing more observations (the natural consequence of append-only) costs
@@ -8,8 +8,29 @@ Each decision records: the question, the decision, the date, the rationale, the
8
8
 
9
9
  ## SD-001: Non-Technical GUI and Zero-Config Install
10
10
 
11
- **Date:** 2026-04-09
12
- **Status:** Decided — **Out of scope for MVP and post-MVP through at least v0.5.x**
11
+ **Date:** 2026-04-09 · **Amended:** 2026-09-15
12
+ **Status:** Decided — **Partially superseded.** The *non-technical consumer GUI* remains out of scope. The *install experience* is no longer out of scope: revisit condition 1 has been met and a **guided installation with branded UI** is now the intended onboarding path. See "Amendment" below before applying anything in this decision.
13
+
14
+ ### Amendment (2026-09-15): the install half of this decision is superseded
15
+
16
+ SD-001 bundled two questions that have now been answered differently. Read it as two decisions:
17
+
18
+ | Sub-question | Status |
19
+ | --- | --- |
20
+ | Non-technical graphical **management** UI for non-developer end users | **Still out of scope.** Rationale 2, 3, and 4 hold unchanged. |
21
+ | **Install and onboarding** experience | **Superseded.** A guided installation with branded UI is the intended path. |
22
+
23
+ **Why the install half fell.** This decision's own revisit condition 1 — *"field evidence shows the ICP is bottlenecked by install friction rather than category confusion"* — is met. `docs/icp/general_release_criteria.md` records **0 of 3** round-2 fresh-install attempts clearing the install-path gate, all three by evaluators who wanted the product. That is the condition SD-001 named in advance.
24
+
25
+ **What this does not concede.** Rationale 3 argued that privacy-first and local-first are at odds with zero-config, where zero-config means "sign up and we handle the server." That argument is correct and is **not** overturned: Neotoma still runs on the user's machine, and a guided installer is not a hosted offering. **Guided install is not zero-install.** "Cloud-hosted turnkey offerings" remains out of scope below, and so does account creation as a precondition for using a local instance.
26
+
27
+ **What is superseded, precisely.**
28
+
29
+ - **Rationale 1 is no longer sound as stated.** It reasoned from the ICP — *"this ICP does not need a graphical management UI; they already operate in terminals"* — to the install decision. Under the durable-ICP decision (O1, 2026-09-15, see `../icp/primary_icp.md`), the ICP is the technically fluent operator who does **not** want to own infrastructure. The premise about who the buyer is has changed, so the install conclusion drawn from it does not survive. The *management-UI* conclusion survives on rationale 4's independent grounds (scope discipline protects the State Layer boundary), not on rationale 1.
30
+ - **Rationale 5 is narrowed.** Agent-assisted install remains a good pattern and stays in scope. It is no longer treated as *sufficient*, because it presumes the user already has a capable agent wired up — which is a harness the buyer may not have configured yet, and is precisely the bootstrap the guided path exists to remove.
31
+ - **"One-click installers that hide architecture from the user"** in the out-of-scope list is narrowed to the clause that is actually load-bearing: **hiding the architecture**. A guided installer that is explicit about where data lives, what runs locally, and which agents are granted access does not hide architecture; it explains it at the only moment the user is paying attention.
32
+
33
+ **Design, not implementation.** This amendment records what the product is *for*. It does not specify an installer technology, a distribution format, or a UI framework, and nothing here should be read as a commitment to one.
13
34
 
14
35
  ### Question
15
36
 
@@ -17,11 +38,11 @@ Should Neotoma ship a non-technical graphical management UI and a zero-configura
17
38
 
18
39
  ### Decision
19
40
 
20
- **No.** Non-technical GUI and zero-config install are **out of scope** for the current release trajectory. Neotoma remains a developer-facing / power-user-facing state layer distributed via MCP, installed and configured by the user (or their agent) on their own machine.
41
+ **No.** Non-technical GUI and zero-config install are **out of scope** for the current release trajectory. Neotoma remains a developer-facing system of record for AI agents, distributed via MCP and installed and configured by the user (or their agent) on their own machine.
21
42
 
22
43
  ### Rationale
23
44
 
24
- 1. **ICP is not the non-technical consumer.** Per `docs/icp/primary_icp.md`, Neotoma's target user is the personal agentic OS builder/operator across three operational modes (debugging, building, operating). This ICP does not need a graphical management UI; they already operate in terminals, config files, and MCP clients.
45
+ 1. **ICP is not the non-technical consumer.** Per `docs/icp/primary_icp.md`, Neotoma's target user is the personal agentic OS builder/operator across three operational modes (debugging, building, operating). This ICP does not need a graphical management UI; they already operate in terminals, config files, and MCP clients. **(Superseded in part, 2026-09-15 — see the Amendment above.** The conclusion "not the non-technical consumer" stands. The supporting claim that the ICP is at home in terminals does not: under O1 the durable ICP is technically fluent but not infrastructure-oriented. This rationale no longer supports the *install* half of the decision.)
25
46
 
26
47
  2. **The non-technical consumer market is already contested.** Penfield, provider-native memory (ChatGPT, Claude, Gemini), and eventual OS-level memory (Apple, Google) are all pursuing the non-technical user. Competing there would require surrendering Neotoma's defensible differentiators (privacy-first local-first, determinism, verifiability) because those differentiators are not legible to non-technical users and slow down the zero-config install story.
27
48
 
@@ -29,17 +50,17 @@ Should Neotoma ship a non-technical graphical management UI and a zero-configura
29
50
 
30
51
  4. **Scope discipline protects the State Layer boundary.** A consumer GUI would pressure Neotoma to include strategy and execution concerns (reminders, notifications, task management, personality, chat) that violate the core identity (`core_identity.md` §"What Neotoma Is NOT"). The architectural invariant is clearer when the product surface stays narrow.
31
52
 
32
- 5. **Power users and agents are the right install vector today.** MCP adapters, per-client install flows, and agent-driven setup are the right primitives for Neotoma's ICP. Agent-assisted install ("your agent installs and configures Neotoma for you") is a more interesting zero-config story than a consumer installer, and it aligns with the state-layer-beneath-agents positioning.
53
+ 5. **Power users and agents are the right install vector today.** MCP adapters, per-client install flows, and agent-driven setup are the right primitives for Neotoma's ICP. Agent-assisted install ("your agent installs and configures Neotoma for you") is a more interesting zero-config story than a consumer installer, and it aligns with the state-layer-beneath-agents positioning. **(Narrowed, 2026-09-15.** Agent-assisted install stays in scope and stays valuable. It is no longer treated as sufficient: it assumes the user has already wired up a capable agent, which is the bootstrap problem the guided path addresses.)
33
54
 
34
55
  ### Target Persona (Explicit)
35
56
 
36
57
  Neotoma targets the **personal agentic OS builder/operator**, not the general consumer. This persona:
37
- - Is comfortable with terminals, config files, environment variables, and MCP client setup.
58
+ - Is comfortable with terminals, config files, environment variables, and MCP client setup. **(Superseded 2026-09-15 — see O1 in `../icp/primary_icp.md`.** The durable ICP is technically fluent but not infrastructure-oriented. They can read a config file; they do not want assembling one to be the price of entry.)
38
59
  - Runs multiple AI tools (Claude Desktop, Claude Code, Cursor, ChatGPT) and feels the cross-tool memory tax.
39
60
  - Cares about verifiability, provenance, and data sovereignty enough to trade convenience for them.
40
- - Will accept a more involved install in exchange for local-first guarantees.
61
+ - Will accept a more involved install in exchange for local-first guarantees. **(Superseded 2026-09-15.** They will accept *local-first* in exchange for those guarantees. Field evidence says they will not reliably accept a more involved *install*: 0/3 round-2 fresh installs completed.)
41
62
 
42
- Absence of a non-technical GUI is a **feature for this persona**, not a defect. It signals that Neotoma is a serious substrate, not a consumer app.
63
+ Absence of a non-technical **management** GUI is a **feature for this persona**, not a defect: it signals that Neotoma is serious infrastructure, not a consumer app. **That reasoning does not extend to the install path.** A rough install signals nothing about seriousness — it only selects for the infrastructure builder that O1 de-targeted.
43
64
 
44
65
  ### What Stays In Scope
45
66
 
@@ -53,7 +74,7 @@ Absence of a non-technical GUI is a **feature for this persona**, not a defect.
53
74
 
54
75
  - Consumer-style onboarding wizards, account creation, or sign-up flows.
55
76
  - Graphical memory management targeted at non-technical users.
56
- - One-click installers that hide architecture from the user.
77
+ - Installers that **hide the architecture** from the user — where data lives, what runs locally, and which agents hold access must be explicit. (Narrowed 2026-09-15: a *guided* installer is in scope; an architecture-concealing one is not.)
57
78
  - Cloud-hosted turnkey offerings.
58
79
  - Personality customization, chat UI, or other consumer-assistant features.
59
80
  - User-facing push notifications, reminders, or strategy-level "should I notify?" decisions. **Note:** Substrate-level event emission and webhook delivery to registered agent consumers ARE in scope — see [SD-002](#sd-002-substrate-event-emission-and-webhook-delivery) for the boundary between strategy-level notifications (rejected here) and substrate-level signaling (accepted there).
@@ -71,7 +92,8 @@ Reopen this decision if any of the following become true:
71
92
 
72
93
  - [`core_identity.md`](core_identity.md) — What Neotoma is and is not
73
94
  - [`product_positioning.md`](product_positioning.md) — Positioning, differentiation, and ICP framing
74
- - [`../icp/primary_icp.md`](../icp/primary_icp.md) — Primary ICP definition
95
+ - [`../icp/primary_icp.md`](../icp/primary_icp.md) — Primary ICP definition (durable; carries the O1 decision that amended this one)
96
+ - [`../icp/general_release_criteria.md`](../icp/general_release_criteria.md) — the install-path gate whose 0/3 result met revisit condition 1
75
97
  - [`../private/competitive/penfield_competitive_analysis.md`](../private/competitive/penfield_competitive_analysis.md) — Competitor pursuing the consumer gap
76
98
  - [`../private/insights/penfield_ai_memory_2026_relevance_analysis.md`](../private/insights/penfield_ai_memory_2026_relevance_analysis.md) — Origin of this decision prompt
77
99
 
@@ -48,4 +48,4 @@ You reach the same memory four ways, all backed by one contract:
48
48
 
49
49
  ## What it is not
50
50
 
51
- Neotoma is not a note-taking app or a personal wiki. It is not a hosted, zero-install product; it requires npm and the command line today. It is not a retrieval cache; similarity search is an optional secondary feature, not the point. For who it suits best, see the [ICP](../icp/icp_from_functionality.md).
51
+ Neotoma is not a note-taking app or a personal wiki. It is not a hosted product — it runs on your own machine and your data stays there. Installing it takes npm and the command line today; a guided installation is intended but not yet available. It is not a retrieval cache; similarity search is an optional secondary feature, not the point. For who it suits best, see the [ICP](../icp/icp_from_functionality.md).
@@ -82,17 +82,19 @@ Functionally enabled but not the design center:
82
82
  ## Who Neotoma is not for (functionally)
83
83
 
84
84
  - **Casual note-takers and PKM/Obsidian-style users.** There is no human-first editing experience; the model is agent-written observations with corrections, not freeform documents.
85
- - **Users who need zero-install, hosted onboarding.** Setup requires npm, the CLI, and operator decisions.
85
+ - **Users who need a fully hosted product with no local component.** Neotoma runs on your machine and keeps your data there; that is the architecture, not a stage. Wanting an easier installation is not the same thing and is not a mismatch — see the note below.
86
86
  - **Teams wanting a managed multi-tenant SaaS today.** Storage is local-only in preview; tenancy exists as `user_id` scoping, not a hosted product.
87
87
  - **Platform builders whose core product is the memory or state engine itself.** Neotoma is that engine; it is meant to be built upon, not rebuilt.
88
88
  - **Pure retrieval/RAG use cases.** The system optimizes for deterministic state integrity, versioning, and audit, not similarity recall (semantic search is an optional secondary capability).
89
89
 
90
+ **A note on installation.** This list deliberately no longer excludes people who want a guided installation. Neotoma is for people who want to own their agents' state, not for people who enjoy assembling an install from a terminal — and an earlier version of this list conflated the two. **As of 2026-09-15, setup is npm plus the CLI**; a guided installation is the intended path and is not shipped. Guided is not zero-install: the software runs on your machine either way.
91
+
90
92
  ## Qualification signals
91
93
 
92
94
  Someone is in the primary ICP if most of these are true:
93
95
 
94
96
  - They already use two or more AI agents or assistants and feel the cost of fragmented memory.
95
- - They are comfortable installing an npm CLI and running a local service.
97
+ - They are willing to run a service on their own machine and to own the data it holds. (Today that means installing an npm CLI; a guided installation is intended but not yet available. Comfort with the current install path is not itself a fit signal — it is friction we intend to remove.)
96
98
  - They want their data on their own machine and care that it is never used for training.
97
99
  - They want to correct facts once and trust the correction holds and is traceable.
98
100
  - They run agents they want to constrain and audit, or they intend to build on the API.
@@ -16,8 +16,16 @@ This document reconciles two independently-derived views of who Neotoma is for:
16
16
  - **[ICP from functionality](icp_from_functionality.md)** (functional view): derived only from a first-principles audit of what the code does.
17
17
  - **The existing ICP materials** (market view): `primary_icp.md`, `secondary_icps.md`, `future_icps.md`, `profiles.md`, `developer_release_targeting.md`, `general_release_criteria.md`, `prioritized_pain_points_and_failure_modes.md`, `qualification_survey.md`. These are derived from market research, topology analysis, and go-to-market strategy (TAM, channels, business model). Most are now marked `visibility: internal` (see "Visibility decision" below).
18
18
 
19
+ > **Update, 2026-09-15 — the O1 durable-ICP decision, reconciled against this document rather than re-deciding it.** The canonical roles assigned below still stand and are not reopened: `icp_from_functionality.md` remains the public statement, the market docs remain internal, and this document remains the bridge. What the O1 decision changed is a claim *inside* the agreement table, not the role assignment.
20
+ >
21
+ > The table's "Disposition" row recorded both views as agreeing on *"comfortable with infrastructure-level abstractions."* The operator has ruled that the durable ICP is the person who feels the pain of ad-hoc agent state and **will not build their own state infrastructure**; the infrastructure builder is an early-adopter cohort. That row is corrected below, and it is the only row affected.
22
+ >
23
+ > **This does not weaken the reconciliation's central finding.** The two views were derived from different inputs and still name the same buyer. The disposition row was the one place where both views had inherited the same wrong inference — that because the install path was infrastructural, the user must be too. That is an inference about the delivery mechanism, not about the person, and the functional audit could not have distinguished the two: code can show that install is CLI-driven, but it cannot show whether that is the intended audience or an unfixed defect. Correcting it in both views simultaneously is a reconciliation, not a new divergence.
24
+
19
25
  The two views were produced from different inputs and they agree on the core. That agreement is a useful signal: the audience the market research targets is the same audience the code actually serves.
20
26
 
27
+ > **This document reconciles the functional and market views. It does not reconcile the market documents against each other** — which is where the remaining contradictions live, and which [`README.md`](README.md), the internal ICP index, does. Start there for the per-surface noun table, the retired and mapped informal buyer synonyms, and the open disagreements that need an operator decision. The conclusions below stand unchanged.
28
+
21
29
  ## Agreement on the primary ICP
22
30
 
23
31
  Both views name the same primary archetype, the person who builds and operates agents:
@@ -28,7 +36,7 @@ Both views name the same primary archetype, the person who builds and operates a
28
36
  | Modes | Operator, builder, and debugger are modes of one person | "Three modes, not separate personas": infrastructure engineering, building agent systems, operating across tools |
29
37
  | Tools | Claude Code, Cursor, ChatGPT, Codex, and others, wired together | Same multi-agent stack |
30
38
  | Core pain | Memory does not carry over; facts conflict; corrections do not stick; no audit trail | Chronic "sync tax" plus acute "agent acted on bad state" |
31
- | Disposition | CLI-comfortable, infrastructure-level | "Comfortable with infrastructure-level abstractions" |
39
+ | Disposition | Technically fluent about state; wants to own the data, not the plumbing | Same — "technically fluent without being infrastructure-oriented" (revised 2026-09-15 per O1; previously "comfortable with infrastructure-level abstractions" in both columns) |
32
40
 
33
41
  The functional audit independently confirms the market archetype. No contradiction exists on the primary ICP.
34
42
 
@@ -36,6 +44,7 @@ The functional audit independently confirms the market archetype. No contradicti
36
44
 
37
45
  - **Developers building on a state layer** (functional secondary) maps to **Toolchain Integrators** and the **Internal-Tools Engineer at a model lab** (market secondaries): both are downstream of primary-ICP validation and build on Neotoma's API surface and guarantees.
38
46
  - **Exclusions match.** The functional view's "not for casual note-taking / PKM / hosted-chat users" matches the market view's explicitly "Not Pursued" AI-for-Management-Work cohort and the "Not for" list in the README.
47
+ - **One exclusion was wrong in every view and has been corrected in all of them (2026-09-15).** All three surfaces — `icp_from_functionality.md`, `primary_icp.md` (D4), and the README — excluded people who "need zero-install onboarding." Under O1 that line disqualified the actual buyer. Each now excludes only the durable boundary: a prospect who requires that nothing run on their own machine. The agreement between the views is preserved because the correction was applied to all of them in one pass; a partial fix would have created a real contradiction where none existed.
39
48
 
40
49
  ## Net-new from the functional audit
41
50
 
@@ -66,6 +75,8 @@ The existing ICP strategy docs previously carried no `visibility` frontmatter, s
66
75
 
67
76
  ## Recommended follow-ups for the market materials
68
77
 
78
+ 0. **(New, 2026-09-15)** `profiles.md` (~2,500 lines) has not been audited against the O1 decision. Its persona narratives, pricing tiers, and buyer nominations were written when the archetype was infrastructure-oriented, and some of them almost certainly still describe that person as the buyer. It was left unmodified in this pass deliberately: rewriting persona narratives is a market-research judgement, not a mechanical propagation of a ruling. Treat it as the largest known unreconciled surface.
79
+
69
80
  1. Add the security-operator secondary ICP to `secondary_icps.md`, grounded in the attestation/grants/provenance functionality.
70
81
  2. Note the federated multi-device/small-group tertiary in `future_icps.md`.
71
82
  3. Keep `icp_from_functionality.md` in sync when major capabilities ship that shift who the product serves.
@@ -18,25 +18,37 @@ All state lives under one directory you control, set by `NEOTOMA_DATA_DIR` (defa
18
18
 
19
19
  Resolution order for the data directory and variables: a project-local `.env`, then `~/.config/neotoma/.env`, then built-in defaults.
20
20
 
21
+ ### Under test, the user-level step is refused
22
+
23
+ A test-shaped process — one with `VITEST` set, `NODE_ENV=test`, or `NEOTOMA_REQUIRE_EXPLICIT_DATA_DIR=1` — does **not** fall back to `~/.config/neotoma/.env` for its data directory. If it reaches that step it stops and exits non-zero, naming the variable to set.
24
+
25
+ The reason is that the user-level config names the data directory holding real data. A test process, or a CLI child a test spawns without passing `NEOTOMA_DATA_DIR` through, would otherwise read and write that directory silently and succeed — so the mistake is invisible, and a test can pass on residue left in real data rather than on the code under test.
26
+
27
+ So set `NEOTOMA_DATA_DIR` to a test-scoped directory, and pass it explicitly to every CLI child process a test spawns rather than relying on inheritance. To exercise the user-level fallback deliberately, set `NEOTOMA_ALLOW_USER_ENV_IN_TEST=1`.
28
+
29
+ Nothing changes for normal operation: the interactive CLI, the server, and every non-test entry point resolve the data directory exactly as the order above describes.
30
+
21
31
  ## Environments
22
32
 
23
33
  `NEOTOMA_ENV` selects the profile: `development` (default) or `production`. The profiles use separate database files, source directories, and logs so a dev stack never touches prod data. Production also changes default ports and tightens auth expectations.
24
34
 
25
35
  ## Core variables
26
36
 
27
- | Variable | Purpose | Default |
28
- | --- | --- | --- |
29
- | `NEOTOMA_ENV` | `development` or `production` | `development` |
30
- | `NEOTOMA_DATA_DIR` | Root data directory | local `data/` |
31
- | `NEOTOMA_SQLITE_PATH` | Explicit database file path | `{dataDir}/neotoma.db` (dev) |
32
- | `NEOTOMA_DB_BACKEND` | DB driver: `sqlite` (synchronous, zero-config) or `libsql` (concurrent — statements run off the event loop via worker-hosted driver for local files, or @libsql/client for remote sqld/Turso, so slow queries can't freeze the server; recommended for hosted/agent-heavy/shared instances) | `sqlite` |
33
- | `NEOTOMA_DB_URL` | libsql connection URL (`file:` for embedded local, `http(s)://`/`libsql://` for remote sqld/Turso) | `file:{NEOTOMA_SQLITE_PATH}` |
34
- | `NEOTOMA_DB_AUTH_TOKEN` | Auth token for remote libsql connections | unset |
35
- | `NEOTOMA_DB_READER_WORKERS` | Read-only worker connections for the local `libsql` backend (WAL lets them run concurrently with the writer). Reads go to the least-loaded reader, so raising this adds capacity rather than just slots | `2` |
36
- | `NEOTOMA_DB_STATEMENT_TIMEOUT_MS` | Per-statement budget for reads on that reader pool. On expiry the reader is terminated (the synchronous driver cannot be interrupted mid-statement) and respawns on the next read, so one runaway query cannot hold a pool slot indefinitely. Writes and statements inside a transaction are never timed out. `0` disables | `30000` |
37
- | `NEOTOMA_RAW_STORAGE_DIR` | Content-addressed source files | `{dataDir}/sources` |
38
- | `NEOTOMA_LOGS_DIR` / `NEOTOMA_EVENT_LOG_PATH` | Log directory and event log file | under `{dataDir}/logs` |
39
- | `NEOTOMA_HOST_URL` / `NEOTOMA_PUBLIC_BASE_URL` | Public URL of this instance | auto-discovered or unset |
37
+ | Variable | Purpose | Default |
38
+ | ---------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------- |
39
+ | `NEOTOMA_ENV` | `development` or `production` | `development` |
40
+ | `NEOTOMA_DATA_DIR` | Root data directory | local `data/` |
41
+ | `NEOTOMA_REQUIRE_EXPLICIT_DATA_DIR` | `1` demands an explicit `NEOTOMA_DATA_DIR`: refuse the `~/.config/neotoma/.env` fallback and exit non-zero. Implied by `VITEST` / `NODE_ENV=test` | unset |
42
+ | `NEOTOMA_ALLOW_USER_ENV_IN_TEST` | `1` re-permits that fallback in a test-shaped process, for a test that means to exercise it | unset |
43
+ | `NEOTOMA_SQLITE_PATH` | Explicit database file path | `{dataDir}/neotoma.db` (dev) |
44
+ | `NEOTOMA_DB_BACKEND` | DB driver: `sqlite` (synchronous, zero-config) or `libsql` (concurrent — statements run off the event loop via worker-hosted driver for local files, or @libsql/client for remote sqld/Turso, so slow queries can't freeze the server; recommended for hosted/agent-heavy/shared instances) | `sqlite` |
45
+ | `NEOTOMA_DB_URL` | libsql connection URL (`file:` for embedded local, `http(s)://`/`libsql://` for remote sqld/Turso) | `file:{NEOTOMA_SQLITE_PATH}` |
46
+ | `NEOTOMA_DB_AUTH_TOKEN` | Auth token for remote libsql connections | unset |
47
+ | `NEOTOMA_DB_READER_WORKERS` | Read-only worker connections for the local `libsql` backend (WAL lets them run concurrently with the writer). Reads go to the least-loaded reader, so raising this adds capacity rather than just slots | `2` |
48
+ | `NEOTOMA_DB_STATEMENT_TIMEOUT_MS` | Per-statement budget for reads on that reader pool. On expiry the reader is terminated (the synchronous driver cannot be interrupted mid-statement) and respawns on the next read, so one runaway query cannot hold a pool slot indefinitely. Writes and statements inside a transaction are never timed out. `0` disables | `30000` |
49
+ | `NEOTOMA_RAW_STORAGE_DIR` | Content-addressed source files | `{dataDir}/sources` |
50
+ | `NEOTOMA_LOGS_DIR` / `NEOTOMA_EVENT_LOG_PATH` | Log directory and event log file | under `{dataDir}/logs` |
51
+ | `NEOTOMA_HOST_URL` / `NEOTOMA_PUBLIC_BASE_URL` | Public URL of this instance | auto-discovered or unset |
40
52
 
41
53
  ### When to opt into `NEOTOMA_DB_BACKEND=libsql`
42
54
 
@@ -49,22 +61,22 @@ Before flipping the variable on an existing database, run `npx tsx scripts/valid
49
61
 
50
62
  ## Server and ports
51
63
 
52
- | Variable | Purpose | Default |
53
- | --- | --- | --- |
64
+ | Variable | Purpose | Default |
65
+ | ------------------------------------ | -------------------------- | ----------------------- |
54
66
  | `NEOTOMA_HTTP_PORT` (or `HTTP_PORT`) | HTTP API and HTTP MCP port | `3080` dev, `3180` prod |
55
- | `WS_PORT` | WebSocket MCP bridge port | `8280` |
67
+ | `WS_PORT` | WebSocket MCP bridge port | `8280` |
56
68
 
57
69
  See [Running the Server](running_the_server.md) for transports and processes.
58
70
 
59
71
  ## Auth and access
60
72
 
61
- | Variable | Purpose |
62
- | --- | --- |
63
- | `NEOTOMA_REQUIRE_KEY_FOR_OAUTH` | Require a key for OAuth connections |
64
- | `NEOTOMA_OAUTH_CLIENT_ID` | MCP OAuth client id (hosted mode) |
73
+ | Variable | Purpose |
74
+ | ------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
75
+ | `NEOTOMA_REQUIRE_KEY_FOR_OAUTH` | Require a key for OAuth connections |
76
+ | `NEOTOMA_OAUTH_CLIENT_ID` | MCP OAuth client id (hosted mode) |
65
77
  | `NEOTOMA_OAUTH_TRUSTED_CALLBACK_URLS` | Comma-separated **exact** callback URLs additionally trusted to receive an authorization code when the authorize request arrives via a tunnel. Empty by default. See below. |
66
- | `NEOTOMA_SANDBOX_MODE` | Opt into the public hosted-sandbox profile |
67
- | `NEOTOMA_REFUSE_MODE` | `warn` or `enforce` when a no-auth, non-loopback topology is detected |
78
+ | `NEOTOMA_SANDBOX_MODE` | Opt into the public hosted-sandbox profile |
79
+ | `NEOTOMA_REFUSE_MODE` | `warn` or `enforce` when a no-auth, non-loopback topology is detected |
68
80
 
69
81
  ### Trusted OAuth callback URLs
70
82
 
@@ -86,14 +98,14 @@ Several entries are comma-separated. Rules worth knowing before you set it:
86
98
 
87
99
  A redirect is accepted when the request's callback and a configured entry agree on **scheme, host, port and path** — all four. In detail:
88
100
 
89
- | Part | How it is compared |
90
- | --- | --- |
91
- | Scheme | Must match. `https` does not authorise its `http` twin. |
92
- | Host | Case-**insensitive**. `APP.EXAMPLE.COM` matches `app.example.com`. |
93
- | Port | Must match, but a default port is equivalent to none: `https://app.example.com` = `https://app.example.com:443`. |
94
- | Path | Case-**sensitive**. `/Auth/Callback` does **not** match `/auth/callback`. A trailing slash is insignificant in either direction. Dot segments resolve first, so `/auth/callback/../admin` is compared as `/auth/admin`. An encoded slash (`%2F`) is not a path separator. |
95
- | Query and fragment | **Ignored on both sides**, so the usual `?code=...&state=...` callback matches. |
96
- | Userinfo | Any URL carrying `user:password@` is rejected. |
101
+ | Part | How it is compared |
102
+ | ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
103
+ | Scheme | Must match. `https` does not authorise its `http` twin. |
104
+ | Host | Case-**insensitive**. `APP.EXAMPLE.COM` matches `app.example.com`. |
105
+ | Port | Must match, but a default port is equivalent to none: `https://app.example.com` = `https://app.example.com:443`. |
106
+ | Path | Case-**sensitive**. `/Auth/Callback` does **not** match `/auth/callback`. A trailing slash is insignificant in either direction. Dot segments resolve first, so `/auth/callback/../admin` is compared as `/auth/admin`. An encoded slash (`%2F`) is not a path separator. |
107
+ | Query and fragment | **Ignored on both sides**, so the usual `?code=...&state=...` callback matches. |
108
+ | Userinfo | Any URL carrying `user:password@` is rejected. |
97
109
 
98
110
  If a redirect is refused, the `400` response names the callback URL the server actually compared (scheme, host and path — query and fragment are stripped) and says how many entries are configured. That is usually enough to tell a near miss — a trailing slash, a port, a case difference — from a typo or an unset variable, without needing server logs.
99
111
 
@@ -111,23 +123,23 @@ See [Deployment Modes](deployment.md) and [Agent Access Control](agent_access_co
111
123
 
112
124
  ## Encryption
113
125
 
114
- | Variable | Purpose |
115
- | --- | --- |
116
- | `NEOTOMA_ENCRYPTION_ENABLED` | Turn on AES-256-GCM at-rest column encryption |
117
- | `NEOTOMA_KEY_FILE_PATH` | Path to a 32-byte key file |
118
- | `NEOTOMA_MNEMONIC` / `NEOTOMA_MNEMONIC_PASSPHRASE` | BIP-39 mnemonic key source |
119
- | `NEOTOMA_LOG_ENCRYPTION_ENABLED` | Encrypt the event log |
120
- | `NEOTOMA_MCP_TOKEN_ENCRYPTION_KEY` | Encrypt stored MCP OAuth tokens |
126
+ | Variable | Purpose |
127
+ | -------------------------------------------------- | --------------------------------------------- |
128
+ | `NEOTOMA_ENCRYPTION_ENABLED` | Turn on AES-256-GCM at-rest column encryption |
129
+ | `NEOTOMA_KEY_FILE_PATH` | Path to a 32-byte key file |
130
+ | `NEOTOMA_MNEMONIC` / `NEOTOMA_MNEMONIC_PASSPHRASE` | BIP-39 mnemonic key source |
131
+ | `NEOTOMA_LOG_ENCRYPTION_ENABLED` | Encrypt the event log |
132
+ | `NEOTOMA_MCP_TOKEN_ENCRYPTION_KEY` | Encrypt stored MCP OAuth tokens |
121
133
 
122
134
  See [Encryption and Key Management](encryption.md).
123
135
 
124
136
  ## Search, inspector, and docs
125
137
 
126
- | Variable | Purpose |
127
- | --- | --- |
128
- | `OPENAI_API_KEY` | Enables semantic vector search (embeddings); keyword search works without it |
129
- | `NEOTOMA_INSPECTOR_DISABLE` / `NEOTOMA_PUBLIC_INSPECTOR_URL` / `NEOTOMA_INSPECTOR_BASE_PATH` | Control the bundled Inspector |
130
- | `NEOTOMA_DOCS_SHOW_INTERNAL` | Show `visibility: internal` docs in the in-app `/docs` browser |
138
+ | Variable | Purpose |
139
+ | -------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------- |
140
+ | `OPENAI_API_KEY` | Enables semantic vector search (embeddings); keyword search works without it |
141
+ | `NEOTOMA_INSPECTOR_DISABLE` / `NEOTOMA_PUBLIC_INSPECTOR_URL` / `NEOTOMA_INSPECTOR_BASE_PATH` | Control the bundled Inspector |
142
+ | `NEOTOMA_DOCS_SHOW_INTERNAL` | Show `visibility: internal` docs in the in-app `/docs` browser |
131
143
 
132
144
  ## Limits and mirror
133
145
 
@@ -265,6 +265,7 @@ caller repairs in-turn via `correct` rather than re-submitting the same payload.
265
265
  | `SOURCE_PRIORITY_IGNORED` | 200 | No | The caller set a non-default `source_priority` but no field on the entity type uses a merge strategy that honours it. The write is accepted; the priority value has no effect on the stored snapshot. |
266
266
  | `AUTO_LINK_EDGE_RETRACTED` | 200 | No | A schema-declared reference field's resolved target changed, so a stale auto-linked edge (e.g. a prior `works_at`) was retracted in the same reduction that created the new one (#1963). Positive signal; no action needed. |
267
267
  | `AUTO_LINK_RETRACTION_FAILED` | 200 | No | A stale auto-linked edge should have been retracted but the soft-delete failed. The superseded edge may still be live despite the store succeeding. The message lists the stale `target_entity_id`s; clean up via `delete_relationship` (or re-run the store). |
268
+ | `STORE_WARNING_RULE_NOT_EVALUATED` | 200 | No | A schema-declared `store_warnings` rule could not be evaluated (its `fields` list or `condition` is malformed, uses an unsupported `condition` key, or declares neither spelling). The write is NOT blocked by this; it is a fail-open diagnostic naming the inert rule and the specific path that could not be read (#2165, #2170). |
268
269
 
269
270
  **`MISSING_CONTENT_FIELD`** — fired when an entity's `SchemaDefinition` declares
270
271
  `content_field` (e.g. `"body"` for `plan`, `"content"` for `note`) and the stored
@@ -363,6 +364,32 @@ knowing its own schema, or re-run the store. Example:
363
364
  This code exists so the retraction failure is never silent — a silently-surviving edge is
364
365
  the exact bug #1963 was filed to fix.
365
366
 
367
+ **`STORE_WARNING_RULE_NOT_EVALUATED`** — fired when a schema-declared `store_warnings`
368
+ rule cannot be evaluated. A rule's condition may be spelled as a flat `fields: string[]`
369
+ list (fire when none of the named fields carry a value) or declaratively as
370
+ `condition: { missing_all_of: [...] }`; both spellings evaluate identically. This code
371
+ fires instead of either spelling's own warning when the rule's `fields` or
372
+ `condition.missing_all_of` list is malformed (not a non-empty array of non-empty
373
+ strings), when `condition` declares an unsupported key, or when the rule declares
374
+ neither spelling at all. Before this code existed, an unevaluable rule threw a
375
+ `TypeError` that escaped as `DB_QUERY_FAILED`, making every write of that entity type
376
+ fail — including `commit: false` dry runs, since evaluation happens before the commit
377
+ decision (#2165, #2170). This warning is a fail-open diagnostic: it never blocks the
378
+ write, and it does not claim the entity was persisted — persistence is the response's
379
+ own `success` field and read-back to report. The `message` names the declared rule's own
380
+ `code` and the exact path (`fields` or `condition.missing_all_of`) that could not be
381
+ read, so a schema author can correct or remove the rule. Example:
382
+
383
+ ```json
384
+ {
385
+ "code": "STORE_WARNING_RULE_NOT_EVALUATED",
386
+ "entity_type": "skill",
387
+ "entity_id": "ent_abc…",
388
+ "observation_index": 0,
389
+ "message": "Store-warning rule \"MISSING_CONTENT_FIELD\" declared on this entity type could not be evaluated: `condition` declares no recognised condition key. Supported condition keys: [\"missing_all_of\"]. Correct or remove the rule in the schema; the entity itself does not need to be re-sent for this warning alone. This advisory did NOT block the operation."
390
+ }
391
+ ```
392
+
366
393
  ## ERR_CONSTRAINT_VIOLATION
367
394
 
368
395
  Hard-rejection error returned by `/store` when one or more observations fail a
@@ -1412,7 +1412,7 @@ This enables full explainability: for any fact in the system, you can trace it b
1412
1412
 
1413
1413
  ### 3.12 `correct`
1414
1414
 
1415
- **Purpose:** Create high-priority correction [observation](../vocabulary/canonical_terms.md#observation) to override AI-[extracted](../vocabulary/canonical_terms.md#extraction) fields. Corrections always win in [entity snapshot](../vocabulary/canonical_terms.md#entity-snapshot) computation (priority 1000).
1415
+ **Purpose:** Create a priority-1000 correction [observation](../vocabulary/canonical_terms.md#observation) to override lower-priority AI-[extracted](../vocabulary/canonical_terms.md#extraction) fields. Higher-priority sources remain authoritative; equal-priority ties follow configured source-kind precedence and deterministic recency in [entity snapshot](../vocabulary/canonical_terms.md#entity-snapshot) computation. Read the snapshot and field provenance back before claiming the correction took effect.
1416
1416
 
1417
1417
  **Request Schema:**
1418
1418
 
@@ -53,8 +53,8 @@ obs_b: { status: "complete", observed_at: "2026-01-02T09:00:00Z" }
53
53
 
54
54
  - Primary sort: `source_priority DESC`.
55
55
  - Secondary sort (within the same priority tier): `observation_source` rank ASC (schema-configurable; sensors and workflow outputs rank above LLM summaries by default).
56
- - Tie-breaker: controlled by the field's optional `tie_breaker` key — `"observed_at"` (default; most recent wins) or `"source_priority"` (falls through to `id ASC`).
57
- - Implementation: `highestPriority()` at line 336 of `src/reducers/observation_reducer.ts`.
56
+ - Same-priority ordering: after numeric priority and source-kind rank tie, the reducer chooses `observed_at DESC`, then `created_at DESC`, then `id ASC` as the final stable fallback. This applies even when the field declares `tie_breaker: "source_priority"`; source priority is already the primary trust axis, and same-tier scalar corrections still express replacement intent.
57
+ - Implementation: `highestPriority()` in `src/reducers/observation_reducer.ts`.
58
58
 
59
59
  Use `highest_priority` when some sources are inherently more trustworthy than others —
60
60
  official documents over extracted receipts, sensor data over user text, corrections over
@@ -230,10 +230,15 @@ regardless of which was written more recently.
230
230
 
231
231
  ### `tie_breaker` options
232
232
 
233
- | Value | Behavior within the same `source_priority` tier |
234
- | ------------------------- | -------------------------------------------------- |
235
- | `"observed_at"` (default) | Most recent observation wins |
236
- | `"source_priority"` | Falls through to `id ASC` (stable, not time-based) |
233
+ | Value | `highest_priority` behavior after the priority and source-kind tiers tie | `most_specific` behavior after specificity ties |
234
+ | ------------------------- | ------------------------------------------------------------------------ | -------------------------------------------------------------------- |
235
+ | `"observed_at"` (default) | Newest `observed_at`, then newest `created_at`, then `id ASC` | Source-kind rank, then newest `observed_at` |
236
+ | `"source_priority"` | Same as `"observed_at"`; retained as a compatibility spelling | Numeric source priority, source-kind rank, then newest `observed_at` |
237
+
238
+ For `highest_priority`, numeric source priority is already the primary trust axis, so
239
+ the two configured values intentionally converge once that tier ties. The distinction
240
+ remains meaningful for `most_specific`, where `"source_priority"` adds numeric source
241
+ priority between specificity and source-kind rank.
237
242
 
238
243
  ---
239
244
 
@@ -103,14 +103,14 @@ const correctionObservation = {
103
103
  interpretation_id: null, // No interpretation
104
104
  observed_at: new Date(),
105
105
  specificity_score: 1.0, // Maximum specificity
106
- source_priority: 1000, // Corrections always win
106
+ source_priority: 1000, // Overrides lower-priority observations
107
107
  fields: {
108
108
  name: "Acme Corporation", // Corrected value
109
109
  },
110
110
  user_id: "user_123",
111
111
  };
112
112
  ```
113
- **Key property:** Corrections persist across reinterpretation. Even if a source is reinterpreted with a new model, the correction observation remains and continues to override AI-extracted values.
113
+ **Key property:** Corrections persist across reinterpretation. Even if a source is reinterpreted with a new model, the correction observation remains and continues to override lower-priority AI-extracted values. Higher-priority sources remain authoritative; equal-priority ties follow the schema's source-kind precedence before deterministic recency.
114
114
  ### 2.2 Observation Storage
115
115
  Observations are stored in `observations` table:
116
116
  - Immutable once created
@@ -106,9 +106,12 @@ function highestPriority(
106
106
  field: string,
107
107
  observations: Observation[]
108
108
  ): { value: any; source_observation_id: string } {
109
- const sorted = observations.sort((a, b) =>
110
- b.source_priority - a.source_priority ||
111
- b.observed_at.getTime() - a.observed_at.getTime()
109
+ const sorted = observations.sort((a, b) =>
110
+ b.source_priority - a.source_priority ||
111
+ observationSourceRank(a) - observationSourceRank(b) ||
112
+ b.observed_at.getTime() - a.observed_at.getTime() ||
113
+ b.created_at.getTime() - a.created_at.getTime() ||
114
+ a.id.localeCompare(b.id)
112
115
  );
113
116
  return {
114
117
  value: sorted[0].fields[field],
@@ -116,6 +119,29 @@ function highestPriority(
116
119
  };
117
120
  }
118
121
  ```
122
+
123
+ Same-priority scalar corrections use that same order. A correction does not need
124
+ to invent priority `1001` to replace an earlier normal correction: two correction
125
+ observations at priority `1000` keep higher `observation_source` precedence, then
126
+ the later `observed_at`/`created_at` wins, with `id ASC` used only when both
127
+ timestamps are equal. Correction receipts should still be followed by independent
128
+ snapshot readback; a response status alone is not proof of the effective value.
129
+
130
+ This ordering applies to every field whose active built-in or custom schema
131
+ declares `highest_priority`, and to every same-priority observation on that
132
+ field; it is not limited to observations created by the correction tools. The
133
+ change does not alter `most_specific` fields. Deploying the reducer does not
134
+ rewrite existing snapshots in bulk. A later store or correction recomputes the
135
+ entity it touches, while untouched snapshots keep their prior projection until
136
+ they are explicitly recomputed. For a deliberate rollout, first call
137
+ `POST /recompute_snapshots_by_type` with `dry_run: true`, then recompute each
138
+ affected entity type and verify the returned counts and errors plus representative
139
+ snapshot values and provenance. To roll back, revert the reducer before further
140
+ bulk recomputation. If snapshots were already recomputed, recompute the affected
141
+ types again after the rollback to restore the prior projection. Observations are
142
+ immutable, so this changes which value is materialized without deleting source
143
+ history.
144
+
119
145
  **Use Cases:**
120
146
  - `vendor_name` in invoices (official documents > receipts)
121
147
  - `merchant_name` in receipts (bank statements > receipts)
@@ -62,6 +62,7 @@ interface FieldDefinition {
62
62
  required?: boolean;
63
63
  validator?: string; // Validator function name
64
64
  preserveCase?: boolean; // Preserve case for this field during canonicalization
65
+ preserveWhitespace?: boolean; // Preserve exact whitespace, including line endings
65
66
  description?: string; // Field description
66
67
  converters?: ConverterDefinition[]; // Field type converters
67
68
  }
@@ -632,3 +633,13 @@ Load `docs/subsystems/schema_registry.md` when:
632
633
  - [ ] Reducer uses schema-projection filtering (only schema-defined fields appear in snapshots)
633
634
  - [ ] Tests verify schema versioning, migration, and field removal
634
635
  - [ ] Documentation updated for new schema versions
636
+
637
+ ## Audio transcription contract
638
+
639
+ The built-in `transcription` 2.0.0 declaration retains historical field names and optionality, corrects `file_size_bytes` to a number, and declares capture and transcription provenance. `audio_content_sha256` identifies the input bytes; it is not the transcription entity identity. A recording can have multiple derivations. Source storage remains authoritative for bytes and content hash; observation `source_id` links derived fields to that source. Do not add an ingestion-success field to the transcription.
640
+
641
+ New ingestion clients must verify `audio_content_sha256`, `original_source_file`, `capture_method`, `transcription_engine`, `consent_basis`, `transcription_text`, and `file_size_bytes` after storage, plus the requested source linkage. `consent_basis: "unknown"` records absent attestation and grants no capture authorization. Historical records may omit these fields.
642
+
643
+ String fields can declare `preserveCase: true` and `preserveWhitespace: true` to retain exact transcript text and source names. Without `preserveWhitespace`, existing normalization remains unchanged. This metadata governs new observations only; activating a schema never rewrites historical observations. Lost historical text cannot be recovered from a normalized snapshot alone.
644
+
645
+ The declaration is available from `ENTITY_SCHEMAS.transcription` in `src/services/schema_definitions.ts`. Existing installations must inspect the effective global and user-specific registrations before registering a new version through the existing registry API. Preserve custom fields, reducers and identity declarations when merging an instance's schema. Code deployment alone does not establish that an active custom schema is compatible. Production registration, activation and any historical recovery are separate reviewed operations; this change performs none.