neotoma 0.22.2 → 0.23.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +2 -2
- package/dist/actions.d.ts.map +1 -1
- package/dist/actions.js +58 -54
- package/dist/actions.js.map +1 -1
- package/dist/config.d.ts +19 -0
- package/dist/config.d.ts.map +1 -1
- package/dist/config.js +79 -3
- package/dist/config.js.map +1 -1
- package/dist/docs/developer/mcp/compact_instructions.md +2 -0
- package/dist/docs/developer/mcp/instructions.md +30 -3
- package/dist/docs/developer/mcp/proxy.md +6 -4
- package/dist/docs/foundation/core_identity.md +45 -4
- package/dist/docs/foundation/developer_release_principles.md +4 -1
- package/dist/docs/foundation/layered_architecture.md +10 -0
- package/dist/docs/foundation/mmry_comparison.md +4 -4
- package/dist/docs/foundation/problem_statement.md +3 -1
- package/dist/docs/foundation/product_positioning.md +21 -17
- package/dist/docs/foundation/redlines.md +21 -2
- package/dist/docs/foundation/scope_decisions.md +32 -10
- package/dist/docs/getting_started/what_is_neotoma.md +1 -1
- package/dist/docs/icp/icp_from_functionality.md +4 -2
- package/dist/docs/icp/icp_reconciliation.md +12 -1
- package/dist/docs/operations/configuration.md +54 -42
- package/dist/docs/reference/error_codes.md +27 -0
- package/dist/docs/specs/MCP_SPEC.md +1 -1
- package/dist/docs/subsystems/conflict_resolution.md +11 -6
- package/dist/docs/subsystems/observation_architecture.md +2 -2
- package/dist/docs/subsystems/reducer.md +29 -3
- package/dist/docs/subsystems/schema_registry.md +11 -0
- package/dist/docs/testing/automated_test_catalog.md +20 -9
- package/dist/mcp_http_session.d.ts +51 -0
- package/dist/mcp_http_session.d.ts.map +1 -0
- package/dist/mcp_http_session.js +173 -0
- package/dist/mcp_http_session.js.map +1 -0
- package/dist/reducers/observation_reducer.d.ts.map +1 -1
- package/dist/reducers/observation_reducer.js +24 -6
- package/dist/reducers/observation_reducer.js.map +1 -1
- package/dist/server.d.ts +31 -2
- package/dist/server.d.ts.map +1 -1
- package/dist/server.js +189 -16
- package/dist/server.js.map +1 -1
- package/dist/services/field_canonicalization.d.ts +1 -0
- package/dist/services/field_canonicalization.d.ts.map +1 -1
- package/dist/services/field_canonicalization.js +15 -9
- package/dist/services/field_canonicalization.js.map +1 -1
- package/dist/services/interpretation.d.ts +2 -0
- package/dist/services/interpretation.d.ts.map +1 -1
- package/dist/services/interpretation.js +2 -1
- package/dist/services/interpretation.js.map +1 -1
- package/dist/services/schema_definitions.d.ts.map +1 -1
- package/dist/services/schema_definitions.js +108 -0
- package/dist/services/schema_definitions.js.map +1 -1
- package/dist/services/schema_registry.d.ts +16 -1
- package/dist/services/schema_registry.d.ts.map +1 -1
- package/dist/services/schema_registry.js.map +1 -1
- package/dist/services/skills/instance_skills.d.ts +137 -0
- package/dist/services/skills/instance_skills.d.ts.map +1 -0
- package/dist/services/skills/instance_skills.js +381 -0
- package/dist/services/skills/instance_skills.js.map +1 -0
- package/dist/services/store_warning_rule.d.ts +97 -0
- package/dist/services/store_warning_rule.d.ts.map +1 -0
- package/dist/services/store_warning_rule.js +175 -0
- package/dist/services/store_warning_rule.js.map +1 -0
- package/dist/tool_definitions.js +1 -1
- package/dist/tool_definitions.js.map +1 -1
- package/docs/developer/mcp/instructions.md +30 -3
- package/package.json +2 -2
|
@@ -19,8 +19,9 @@ them costly to cross.
|
|
|
19
19
|
|
|
20
20
|
## The substantive position
|
|
21
21
|
|
|
22
|
-
Neotoma exists to
|
|
23
|
-
|
|
22
|
+
Neotoma exists to be the system of record for AI agents — the designated
|
|
23
|
+
place where a fact about agent-held state is settled — built on write
|
|
24
|
+
integrity rather than retrieval performance. The architectural
|
|
24
25
|
commitments named in [`core_identity.md`](core_identity.md) and
|
|
25
26
|
[`philosophy.md`](philosophy.md) (append-only observations, hash-based entity
|
|
26
27
|
identity, schema-bound writes, pipeline provenance, replayable timelines,
|
|
@@ -180,6 +181,24 @@ copy or product positioning that implies Neotoma is "also" a retrieval system,
|
|
|
180
181
|
"also" an embeddings system, or "also" memory-as-cognition. Neotoma is the
|
|
181
182
|
state-integrity position. Other positions exist; Neotoma is not them.
|
|
182
183
|
|
|
184
|
+
<a id="never-say-the-agent-forgot"></a>
|
|
185
|
+
**`never_say_the_agent_forgot` (decided 2026-08-21) is the sharpest instance of
|
|
186
|
+
this redline, and the one crossed most often.** Copy must never say an agent
|
|
187
|
+
*forgot*. Forgetting is the recall framing, and any retrieval vendor answers it
|
|
188
|
+
with better recall — so stating the problem that way argues their case. The
|
|
189
|
+
accurate framing is that nothing was committed to a record: a fact held only in
|
|
190
|
+
context was never written anywhere authoritative, so there is nothing to forget
|
|
191
|
+
and nothing to correct. Retired outright: "Your agents forget. Neotoma makes
|
|
192
|
+
them remember." and "your agents forget what they knew yesterday." The test any
|
|
193
|
+
problem statement must pass is the competitor test — if mem0, Zep, or any
|
|
194
|
+
retrieval vendor could plausibly claim to solve the scenario as written, it
|
|
195
|
+
frames their category and must be rewritten. Verbatim quotation of user research
|
|
196
|
+
is exempt: what a user said is evidence, not Neotoma's own voice. Canonical
|
|
197
|
+
statement in
|
|
198
|
+
[`product_positioning.md`](product_positioning.md) ("Primary message
|
|
199
|
+
(external)"); applied in
|
|
200
|
+
[`problem_statement.md`](problem_statement.md).
|
|
201
|
+
|
|
183
202
|
### R14. No pricing model that punishes integrity use
|
|
184
203
|
|
|
185
204
|
If storing more observations (the natural consequence of append-only) costs
|
|
@@ -8,8 +8,29 @@ Each decision records: the question, the decision, the date, the rationale, the
|
|
|
8
8
|
|
|
9
9
|
## SD-001: Non-Technical GUI and Zero-Config Install
|
|
10
10
|
|
|
11
|
-
**Date:** 2026-04-09
|
|
12
|
-
**Status:** Decided — **
|
|
11
|
+
**Date:** 2026-04-09 · **Amended:** 2026-09-15
|
|
12
|
+
**Status:** Decided — **Partially superseded.** The *non-technical consumer GUI* remains out of scope. The *install experience* is no longer out of scope: revisit condition 1 has been met and a **guided installation with branded UI** is now the intended onboarding path. See "Amendment" below before applying anything in this decision.
|
|
13
|
+
|
|
14
|
+
### Amendment (2026-09-15): the install half of this decision is superseded
|
|
15
|
+
|
|
16
|
+
SD-001 bundled two questions that have now been answered differently. Read it as two decisions:
|
|
17
|
+
|
|
18
|
+
| Sub-question | Status |
|
|
19
|
+
| --- | --- |
|
|
20
|
+
| Non-technical graphical **management** UI for non-developer end users | **Still out of scope.** Rationale 2, 3, and 4 hold unchanged. |
|
|
21
|
+
| **Install and onboarding** experience | **Superseded.** A guided installation with branded UI is the intended path. |
|
|
22
|
+
|
|
23
|
+
**Why the install half fell.** This decision's own revisit condition 1 — *"field evidence shows the ICP is bottlenecked by install friction rather than category confusion"* — is met. `docs/icp/general_release_criteria.md` records **0 of 3** round-2 fresh-install attempts clearing the install-path gate, all three by evaluators who wanted the product. That is the condition SD-001 named in advance.
|
|
24
|
+
|
|
25
|
+
**What this does not concede.** Rationale 3 argued that privacy-first and local-first are at odds with zero-config, where zero-config means "sign up and we handle the server." That argument is correct and is **not** overturned: Neotoma still runs on the user's machine, and a guided installer is not a hosted offering. **Guided install is not zero-install.** "Cloud-hosted turnkey offerings" remains out of scope below, and so does account creation as a precondition for using a local instance.
|
|
26
|
+
|
|
27
|
+
**What is superseded, precisely.**
|
|
28
|
+
|
|
29
|
+
- **Rationale 1 is no longer sound as stated.** It reasoned from the ICP — *"this ICP does not need a graphical management UI; they already operate in terminals"* — to the install decision. Under the durable-ICP decision (O1, 2026-09-15, see `../icp/primary_icp.md`), the ICP is the technically fluent operator who does **not** want to own infrastructure. The premise about who the buyer is has changed, so the install conclusion drawn from it does not survive. The *management-UI* conclusion survives on rationale 4's independent grounds (scope discipline protects the State Layer boundary), not on rationale 1.
|
|
30
|
+
- **Rationale 5 is narrowed.** Agent-assisted install remains a good pattern and stays in scope. It is no longer treated as *sufficient*, because it presumes the user already has a capable agent wired up — which is a harness the buyer may not have configured yet, and is precisely the bootstrap the guided path exists to remove.
|
|
31
|
+
- **"One-click installers that hide architecture from the user"** in the out-of-scope list is narrowed to the clause that is actually load-bearing: **hiding the architecture**. A guided installer that is explicit about where data lives, what runs locally, and which agents are granted access does not hide architecture; it explains it at the only moment the user is paying attention.
|
|
32
|
+
|
|
33
|
+
**Design, not implementation.** This amendment records what the product is *for*. It does not specify an installer technology, a distribution format, or a UI framework, and nothing here should be read as a commitment to one.
|
|
13
34
|
|
|
14
35
|
### Question
|
|
15
36
|
|
|
@@ -17,11 +38,11 @@ Should Neotoma ship a non-technical graphical management UI and a zero-configura
|
|
|
17
38
|
|
|
18
39
|
### Decision
|
|
19
40
|
|
|
20
|
-
**No.** Non-technical GUI and zero-config install are **out of scope** for the current release trajectory. Neotoma remains a developer-facing
|
|
41
|
+
**No.** Non-technical GUI and zero-config install are **out of scope** for the current release trajectory. Neotoma remains a developer-facing system of record for AI agents, distributed via MCP and installed and configured by the user (or their agent) on their own machine.
|
|
21
42
|
|
|
22
43
|
### Rationale
|
|
23
44
|
|
|
24
|
-
1. **ICP is not the non-technical consumer.** Per `docs/icp/primary_icp.md`, Neotoma's target user is the personal agentic OS builder/operator across three operational modes (debugging, building, operating). This ICP does not need a graphical management UI; they already operate in terminals, config files, and MCP clients.
|
|
45
|
+
1. **ICP is not the non-technical consumer.** Per `docs/icp/primary_icp.md`, Neotoma's target user is the personal agentic OS builder/operator across three operational modes (debugging, building, operating). This ICP does not need a graphical management UI; they already operate in terminals, config files, and MCP clients. **(Superseded in part, 2026-09-15 — see the Amendment above.** The conclusion "not the non-technical consumer" stands. The supporting claim that the ICP is at home in terminals does not: under O1 the durable ICP is technically fluent but not infrastructure-oriented. This rationale no longer supports the *install* half of the decision.)
|
|
25
46
|
|
|
26
47
|
2. **The non-technical consumer market is already contested.** Penfield, provider-native memory (ChatGPT, Claude, Gemini), and eventual OS-level memory (Apple, Google) are all pursuing the non-technical user. Competing there would require surrendering Neotoma's defensible differentiators (privacy-first local-first, determinism, verifiability) because those differentiators are not legible to non-technical users and slow down the zero-config install story.
|
|
27
48
|
|
|
@@ -29,17 +50,17 @@ Should Neotoma ship a non-technical graphical management UI and a zero-configura
|
|
|
29
50
|
|
|
30
51
|
4. **Scope discipline protects the State Layer boundary.** A consumer GUI would pressure Neotoma to include strategy and execution concerns (reminders, notifications, task management, personality, chat) that violate the core identity (`core_identity.md` §"What Neotoma Is NOT"). The architectural invariant is clearer when the product surface stays narrow.
|
|
31
52
|
|
|
32
|
-
5. **Power users and agents are the right install vector today.** MCP adapters, per-client install flows, and agent-driven setup are the right primitives for Neotoma's ICP. Agent-assisted install ("your agent installs and configures Neotoma for you") is a more interesting zero-config story than a consumer installer, and it aligns with the state-layer-beneath-agents positioning.
|
|
53
|
+
5. **Power users and agents are the right install vector today.** MCP adapters, per-client install flows, and agent-driven setup are the right primitives for Neotoma's ICP. Agent-assisted install ("your agent installs and configures Neotoma for you") is a more interesting zero-config story than a consumer installer, and it aligns with the state-layer-beneath-agents positioning. **(Narrowed, 2026-09-15.** Agent-assisted install stays in scope and stays valuable. It is no longer treated as sufficient: it assumes the user has already wired up a capable agent, which is the bootstrap problem the guided path addresses.)
|
|
33
54
|
|
|
34
55
|
### Target Persona (Explicit)
|
|
35
56
|
|
|
36
57
|
Neotoma targets the **personal agentic OS builder/operator**, not the general consumer. This persona:
|
|
37
|
-
- Is comfortable with terminals, config files, environment variables, and MCP client setup.
|
|
58
|
+
- Is comfortable with terminals, config files, environment variables, and MCP client setup. **(Superseded 2026-09-15 — see O1 in `../icp/primary_icp.md`.** The durable ICP is technically fluent but not infrastructure-oriented. They can read a config file; they do not want assembling one to be the price of entry.)
|
|
38
59
|
- Runs multiple AI tools (Claude Desktop, Claude Code, Cursor, ChatGPT) and feels the cross-tool memory tax.
|
|
39
60
|
- Cares about verifiability, provenance, and data sovereignty enough to trade convenience for them.
|
|
40
|
-
- Will accept a more involved install in exchange for local-first guarantees.
|
|
61
|
+
- Will accept a more involved install in exchange for local-first guarantees. **(Superseded 2026-09-15.** They will accept *local-first* in exchange for those guarantees. Field evidence says they will not reliably accept a more involved *install*: 0/3 round-2 fresh installs completed.)
|
|
41
62
|
|
|
42
|
-
Absence of a non-technical GUI is a **feature for this persona**, not a defect
|
|
63
|
+
Absence of a non-technical **management** GUI is a **feature for this persona**, not a defect: it signals that Neotoma is serious infrastructure, not a consumer app. **That reasoning does not extend to the install path.** A rough install signals nothing about seriousness — it only selects for the infrastructure builder that O1 de-targeted.
|
|
43
64
|
|
|
44
65
|
### What Stays In Scope
|
|
45
66
|
|
|
@@ -53,7 +74,7 @@ Absence of a non-technical GUI is a **feature for this persona**, not a defect.
|
|
|
53
74
|
|
|
54
75
|
- Consumer-style onboarding wizards, account creation, or sign-up flows.
|
|
55
76
|
- Graphical memory management targeted at non-technical users.
|
|
56
|
-
-
|
|
77
|
+
- Installers that **hide the architecture** from the user — where data lives, what runs locally, and which agents hold access must be explicit. (Narrowed 2026-09-15: a *guided* installer is in scope; an architecture-concealing one is not.)
|
|
57
78
|
- Cloud-hosted turnkey offerings.
|
|
58
79
|
- Personality customization, chat UI, or other consumer-assistant features.
|
|
59
80
|
- User-facing push notifications, reminders, or strategy-level "should I notify?" decisions. **Note:** Substrate-level event emission and webhook delivery to registered agent consumers ARE in scope — see [SD-002](#sd-002-substrate-event-emission-and-webhook-delivery) for the boundary between strategy-level notifications (rejected here) and substrate-level signaling (accepted there).
|
|
@@ -71,7 +92,8 @@ Reopen this decision if any of the following become true:
|
|
|
71
92
|
|
|
72
93
|
- [`core_identity.md`](core_identity.md) — What Neotoma is and is not
|
|
73
94
|
- [`product_positioning.md`](product_positioning.md) — Positioning, differentiation, and ICP framing
|
|
74
|
-
- [`../icp/primary_icp.md`](../icp/primary_icp.md) — Primary ICP definition
|
|
95
|
+
- [`../icp/primary_icp.md`](../icp/primary_icp.md) — Primary ICP definition (durable; carries the O1 decision that amended this one)
|
|
96
|
+
- [`../icp/general_release_criteria.md`](../icp/general_release_criteria.md) — the install-path gate whose 0/3 result met revisit condition 1
|
|
75
97
|
- [`../private/competitive/penfield_competitive_analysis.md`](../private/competitive/penfield_competitive_analysis.md) — Competitor pursuing the consumer gap
|
|
76
98
|
- [`../private/insights/penfield_ai_memory_2026_relevance_analysis.md`](../private/insights/penfield_ai_memory_2026_relevance_analysis.md) — Origin of this decision prompt
|
|
77
99
|
|
|
@@ -48,4 +48,4 @@ You reach the same memory four ways, all backed by one contract:
|
|
|
48
48
|
|
|
49
49
|
## What it is not
|
|
50
50
|
|
|
51
|
-
Neotoma is not a note-taking app or a personal wiki. It is not a hosted
|
|
51
|
+
Neotoma is not a note-taking app or a personal wiki. It is not a hosted product — it runs on your own machine and your data stays there. Installing it takes npm and the command line today; a guided installation is intended but not yet available. It is not a retrieval cache; similarity search is an optional secondary feature, not the point. For who it suits best, see the [ICP](../icp/icp_from_functionality.md).
|
|
@@ -82,17 +82,19 @@ Functionally enabled but not the design center:
|
|
|
82
82
|
## Who Neotoma is not for (functionally)
|
|
83
83
|
|
|
84
84
|
- **Casual note-takers and PKM/Obsidian-style users.** There is no human-first editing experience; the model is agent-written observations with corrections, not freeform documents.
|
|
85
|
-
- **Users who need
|
|
85
|
+
- **Users who need a fully hosted product with no local component.** Neotoma runs on your machine and keeps your data there; that is the architecture, not a stage. Wanting an easier installation is not the same thing and is not a mismatch — see the note below.
|
|
86
86
|
- **Teams wanting a managed multi-tenant SaaS today.** Storage is local-only in preview; tenancy exists as `user_id` scoping, not a hosted product.
|
|
87
87
|
- **Platform builders whose core product is the memory or state engine itself.** Neotoma is that engine; it is meant to be built upon, not rebuilt.
|
|
88
88
|
- **Pure retrieval/RAG use cases.** The system optimizes for deterministic state integrity, versioning, and audit, not similarity recall (semantic search is an optional secondary capability).
|
|
89
89
|
|
|
90
|
+
**A note on installation.** This list deliberately no longer excludes people who want a guided installation. Neotoma is for people who want to own their agents' state, not for people who enjoy assembling an install from a terminal — and an earlier version of this list conflated the two. **As of 2026-09-15, setup is npm plus the CLI**; a guided installation is the intended path and is not shipped. Guided is not zero-install: the software runs on your machine either way.
|
|
91
|
+
|
|
90
92
|
## Qualification signals
|
|
91
93
|
|
|
92
94
|
Someone is in the primary ICP if most of these are true:
|
|
93
95
|
|
|
94
96
|
- They already use two or more AI agents or assistants and feel the cost of fragmented memory.
|
|
95
|
-
- They are
|
|
97
|
+
- They are willing to run a service on their own machine and to own the data it holds. (Today that means installing an npm CLI; a guided installation is intended but not yet available. Comfort with the current install path is not itself a fit signal — it is friction we intend to remove.)
|
|
96
98
|
- They want their data on their own machine and care that it is never used for training.
|
|
97
99
|
- They want to correct facts once and trust the correction holds and is traceable.
|
|
98
100
|
- They run agents they want to constrain and audit, or they intend to build on the API.
|
|
@@ -16,8 +16,16 @@ This document reconciles two independently-derived views of who Neotoma is for:
|
|
|
16
16
|
- **[ICP from functionality](icp_from_functionality.md)** (functional view): derived only from a first-principles audit of what the code does.
|
|
17
17
|
- **The existing ICP materials** (market view): `primary_icp.md`, `secondary_icps.md`, `future_icps.md`, `profiles.md`, `developer_release_targeting.md`, `general_release_criteria.md`, `prioritized_pain_points_and_failure_modes.md`, `qualification_survey.md`. These are derived from market research, topology analysis, and go-to-market strategy (TAM, channels, business model). Most are now marked `visibility: internal` (see "Visibility decision" below).
|
|
18
18
|
|
|
19
|
+
> **Update, 2026-09-15 — the O1 durable-ICP decision, reconciled against this document rather than re-deciding it.** The canonical roles assigned below still stand and are not reopened: `icp_from_functionality.md` remains the public statement, the market docs remain internal, and this document remains the bridge. What the O1 decision changed is a claim *inside* the agreement table, not the role assignment.
|
|
20
|
+
>
|
|
21
|
+
> The table's "Disposition" row recorded both views as agreeing on *"comfortable with infrastructure-level abstractions."* The operator has ruled that the durable ICP is the person who feels the pain of ad-hoc agent state and **will not build their own state infrastructure**; the infrastructure builder is an early-adopter cohort. That row is corrected below, and it is the only row affected.
|
|
22
|
+
>
|
|
23
|
+
> **This does not weaken the reconciliation's central finding.** The two views were derived from different inputs and still name the same buyer. The disposition row was the one place where both views had inherited the same wrong inference — that because the install path was infrastructural, the user must be too. That is an inference about the delivery mechanism, not about the person, and the functional audit could not have distinguished the two: code can show that install is CLI-driven, but it cannot show whether that is the intended audience or an unfixed defect. Correcting it in both views simultaneously is a reconciliation, not a new divergence.
|
|
24
|
+
|
|
19
25
|
The two views were produced from different inputs and they agree on the core. That agreement is a useful signal: the audience the market research targets is the same audience the code actually serves.
|
|
20
26
|
|
|
27
|
+
> **This document reconciles the functional and market views. It does not reconcile the market documents against each other** — which is where the remaining contradictions live, and which [`README.md`](README.md), the internal ICP index, does. Start there for the per-surface noun table, the retired and mapped informal buyer synonyms, and the open disagreements that need an operator decision. The conclusions below stand unchanged.
|
|
28
|
+
|
|
21
29
|
## Agreement on the primary ICP
|
|
22
30
|
|
|
23
31
|
Both views name the same primary archetype, the person who builds and operates agents:
|
|
@@ -28,7 +36,7 @@ Both views name the same primary archetype, the person who builds and operates a
|
|
|
28
36
|
| Modes | Operator, builder, and debugger are modes of one person | "Three modes, not separate personas": infrastructure engineering, building agent systems, operating across tools |
|
|
29
37
|
| Tools | Claude Code, Cursor, ChatGPT, Codex, and others, wired together | Same multi-agent stack |
|
|
30
38
|
| Core pain | Memory does not carry over; facts conflict; corrections do not stick; no audit trail | Chronic "sync tax" plus acute "agent acted on bad state" |
|
|
31
|
-
| Disposition |
|
|
39
|
+
| Disposition | Technically fluent about state; wants to own the data, not the plumbing | Same — "technically fluent without being infrastructure-oriented" (revised 2026-09-15 per O1; previously "comfortable with infrastructure-level abstractions" in both columns) |
|
|
32
40
|
|
|
33
41
|
The functional audit independently confirms the market archetype. No contradiction exists on the primary ICP.
|
|
34
42
|
|
|
@@ -36,6 +44,7 @@ The functional audit independently confirms the market archetype. No contradicti
|
|
|
36
44
|
|
|
37
45
|
- **Developers building on a state layer** (functional secondary) maps to **Toolchain Integrators** and the **Internal-Tools Engineer at a model lab** (market secondaries): both are downstream of primary-ICP validation and build on Neotoma's API surface and guarantees.
|
|
38
46
|
- **Exclusions match.** The functional view's "not for casual note-taking / PKM / hosted-chat users" matches the market view's explicitly "Not Pursued" AI-for-Management-Work cohort and the "Not for" list in the README.
|
|
47
|
+
- **One exclusion was wrong in every view and has been corrected in all of them (2026-09-15).** All three surfaces — `icp_from_functionality.md`, `primary_icp.md` (D4), and the README — excluded people who "need zero-install onboarding." Under O1 that line disqualified the actual buyer. Each now excludes only the durable boundary: a prospect who requires that nothing run on their own machine. The agreement between the views is preserved because the correction was applied to all of them in one pass; a partial fix would have created a real contradiction where none existed.
|
|
39
48
|
|
|
40
49
|
## Net-new from the functional audit
|
|
41
50
|
|
|
@@ -66,6 +75,8 @@ The existing ICP strategy docs previously carried no `visibility` frontmatter, s
|
|
|
66
75
|
|
|
67
76
|
## Recommended follow-ups for the market materials
|
|
68
77
|
|
|
78
|
+
0. **(New, 2026-09-15)** `profiles.md` (~2,500 lines) has not been audited against the O1 decision. Its persona narratives, pricing tiers, and buyer nominations were written when the archetype was infrastructure-oriented, and some of them almost certainly still describe that person as the buyer. It was left unmodified in this pass deliberately: rewriting persona narratives is a market-research judgement, not a mechanical propagation of a ruling. Treat it as the largest known unreconciled surface.
|
|
79
|
+
|
|
69
80
|
1. Add the security-operator secondary ICP to `secondary_icps.md`, grounded in the attestation/grants/provenance functionality.
|
|
70
81
|
2. Note the federated multi-device/small-group tertiary in `future_icps.md`.
|
|
71
82
|
3. Keep `icp_from_functionality.md` in sync when major capabilities ship that shift who the product serves.
|
|
@@ -18,25 +18,37 @@ All state lives under one directory you control, set by `NEOTOMA_DATA_DIR` (defa
|
|
|
18
18
|
|
|
19
19
|
Resolution order for the data directory and variables: a project-local `.env`, then `~/.config/neotoma/.env`, then built-in defaults.
|
|
20
20
|
|
|
21
|
+
### Under test, the user-level step is refused
|
|
22
|
+
|
|
23
|
+
A test-shaped process — one with `VITEST` set, `NODE_ENV=test`, or `NEOTOMA_REQUIRE_EXPLICIT_DATA_DIR=1` — does **not** fall back to `~/.config/neotoma/.env` for its data directory. If it reaches that step it stops and exits non-zero, naming the variable to set.
|
|
24
|
+
|
|
25
|
+
The reason is that the user-level config names the data directory holding real data. A test process, or a CLI child a test spawns without passing `NEOTOMA_DATA_DIR` through, would otherwise read and write that directory silently and succeed — so the mistake is invisible, and a test can pass on residue left in real data rather than on the code under test.
|
|
26
|
+
|
|
27
|
+
So set `NEOTOMA_DATA_DIR` to a test-scoped directory, and pass it explicitly to every CLI child process a test spawns rather than relying on inheritance. To exercise the user-level fallback deliberately, set `NEOTOMA_ALLOW_USER_ENV_IN_TEST=1`.
|
|
28
|
+
|
|
29
|
+
Nothing changes for normal operation: the interactive CLI, the server, and every non-test entry point resolve the data directory exactly as the order above describes.
|
|
30
|
+
|
|
21
31
|
## Environments
|
|
22
32
|
|
|
23
33
|
`NEOTOMA_ENV` selects the profile: `development` (default) or `production`. The profiles use separate database files, source directories, and logs so a dev stack never touches prod data. Production also changes default ports and tightens auth expectations.
|
|
24
34
|
|
|
25
35
|
## Core variables
|
|
26
36
|
|
|
27
|
-
| Variable
|
|
28
|
-
|
|
|
29
|
-
| `NEOTOMA_ENV`
|
|
30
|
-
| `NEOTOMA_DATA_DIR`
|
|
31
|
-
| `
|
|
32
|
-
| `
|
|
33
|
-
| `
|
|
34
|
-
| `
|
|
35
|
-
| `
|
|
36
|
-
| `
|
|
37
|
-
| `
|
|
38
|
-
| `
|
|
39
|
-
| `
|
|
37
|
+
| Variable | Purpose | Default |
|
|
38
|
+
| ---------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------- |
|
|
39
|
+
| `NEOTOMA_ENV` | `development` or `production` | `development` |
|
|
40
|
+
| `NEOTOMA_DATA_DIR` | Root data directory | local `data/` |
|
|
41
|
+
| `NEOTOMA_REQUIRE_EXPLICIT_DATA_DIR` | `1` demands an explicit `NEOTOMA_DATA_DIR`: refuse the `~/.config/neotoma/.env` fallback and exit non-zero. Implied by `VITEST` / `NODE_ENV=test` | unset |
|
|
42
|
+
| `NEOTOMA_ALLOW_USER_ENV_IN_TEST` | `1` re-permits that fallback in a test-shaped process, for a test that means to exercise it | unset |
|
|
43
|
+
| `NEOTOMA_SQLITE_PATH` | Explicit database file path | `{dataDir}/neotoma.db` (dev) |
|
|
44
|
+
| `NEOTOMA_DB_BACKEND` | DB driver: `sqlite` (synchronous, zero-config) or `libsql` (concurrent — statements run off the event loop via worker-hosted driver for local files, or @libsql/client for remote sqld/Turso, so slow queries can't freeze the server; recommended for hosted/agent-heavy/shared instances) | `sqlite` |
|
|
45
|
+
| `NEOTOMA_DB_URL` | libsql connection URL (`file:` for embedded local, `http(s)://`/`libsql://` for remote sqld/Turso) | `file:{NEOTOMA_SQLITE_PATH}` |
|
|
46
|
+
| `NEOTOMA_DB_AUTH_TOKEN` | Auth token for remote libsql connections | unset |
|
|
47
|
+
| `NEOTOMA_DB_READER_WORKERS` | Read-only worker connections for the local `libsql` backend (WAL lets them run concurrently with the writer). Reads go to the least-loaded reader, so raising this adds capacity rather than just slots | `2` |
|
|
48
|
+
| `NEOTOMA_DB_STATEMENT_TIMEOUT_MS` | Per-statement budget for reads on that reader pool. On expiry the reader is terminated (the synchronous driver cannot be interrupted mid-statement) and respawns on the next read, so one runaway query cannot hold a pool slot indefinitely. Writes and statements inside a transaction are never timed out. `0` disables | `30000` |
|
|
49
|
+
| `NEOTOMA_RAW_STORAGE_DIR` | Content-addressed source files | `{dataDir}/sources` |
|
|
50
|
+
| `NEOTOMA_LOGS_DIR` / `NEOTOMA_EVENT_LOG_PATH` | Log directory and event log file | under `{dataDir}/logs` |
|
|
51
|
+
| `NEOTOMA_HOST_URL` / `NEOTOMA_PUBLIC_BASE_URL` | Public URL of this instance | auto-discovered or unset |
|
|
40
52
|
|
|
41
53
|
### When to opt into `NEOTOMA_DB_BACKEND=libsql`
|
|
42
54
|
|
|
@@ -49,22 +61,22 @@ Before flipping the variable on an existing database, run `npx tsx scripts/valid
|
|
|
49
61
|
|
|
50
62
|
## Server and ports
|
|
51
63
|
|
|
52
|
-
| Variable
|
|
53
|
-
|
|
|
64
|
+
| Variable | Purpose | Default |
|
|
65
|
+
| ------------------------------------ | -------------------------- | ----------------------- |
|
|
54
66
|
| `NEOTOMA_HTTP_PORT` (or `HTTP_PORT`) | HTTP API and HTTP MCP port | `3080` dev, `3180` prod |
|
|
55
|
-
| `WS_PORT`
|
|
67
|
+
| `WS_PORT` | WebSocket MCP bridge port | `8280` |
|
|
56
68
|
|
|
57
69
|
See [Running the Server](running_the_server.md) for transports and processes.
|
|
58
70
|
|
|
59
71
|
## Auth and access
|
|
60
72
|
|
|
61
|
-
| Variable
|
|
62
|
-
|
|
|
63
|
-
| `NEOTOMA_REQUIRE_KEY_FOR_OAUTH`
|
|
64
|
-
| `NEOTOMA_OAUTH_CLIENT_ID`
|
|
73
|
+
| Variable | Purpose |
|
|
74
|
+
| ------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
75
|
+
| `NEOTOMA_REQUIRE_KEY_FOR_OAUTH` | Require a key for OAuth connections |
|
|
76
|
+
| `NEOTOMA_OAUTH_CLIENT_ID` | MCP OAuth client id (hosted mode) |
|
|
65
77
|
| `NEOTOMA_OAUTH_TRUSTED_CALLBACK_URLS` | Comma-separated **exact** callback URLs additionally trusted to receive an authorization code when the authorize request arrives via a tunnel. Empty by default. See below. |
|
|
66
|
-
| `NEOTOMA_SANDBOX_MODE`
|
|
67
|
-
| `NEOTOMA_REFUSE_MODE`
|
|
78
|
+
| `NEOTOMA_SANDBOX_MODE` | Opt into the public hosted-sandbox profile |
|
|
79
|
+
| `NEOTOMA_REFUSE_MODE` | `warn` or `enforce` when a no-auth, non-loopback topology is detected |
|
|
68
80
|
|
|
69
81
|
### Trusted OAuth callback URLs
|
|
70
82
|
|
|
@@ -86,14 +98,14 @@ Several entries are comma-separated. Rules worth knowing before you set it:
|
|
|
86
98
|
|
|
87
99
|
A redirect is accepted when the request's callback and a configured entry agree on **scheme, host, port and path** — all four. In detail:
|
|
88
100
|
|
|
89
|
-
| Part
|
|
90
|
-
|
|
|
91
|
-
| Scheme
|
|
92
|
-
| Host
|
|
93
|
-
| Port
|
|
94
|
-
| Path
|
|
95
|
-
| Query and fragment | **Ignored on both sides**, so the usual `?code=...&state=...` callback matches.
|
|
96
|
-
| Userinfo
|
|
101
|
+
| Part | How it is compared |
|
|
102
|
+
| ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
103
|
+
| Scheme | Must match. `https` does not authorise its `http` twin. |
|
|
104
|
+
| Host | Case-**insensitive**. `APP.EXAMPLE.COM` matches `app.example.com`. |
|
|
105
|
+
| Port | Must match, but a default port is equivalent to none: `https://app.example.com` = `https://app.example.com:443`. |
|
|
106
|
+
| Path | Case-**sensitive**. `/Auth/Callback` does **not** match `/auth/callback`. A trailing slash is insignificant in either direction. Dot segments resolve first, so `/auth/callback/../admin` is compared as `/auth/admin`. An encoded slash (`%2F`) is not a path separator. |
|
|
107
|
+
| Query and fragment | **Ignored on both sides**, so the usual `?code=...&state=...` callback matches. |
|
|
108
|
+
| Userinfo | Any URL carrying `user:password@` is rejected. |
|
|
97
109
|
|
|
98
110
|
If a redirect is refused, the `400` response names the callback URL the server actually compared (scheme, host and path — query and fragment are stripped) and says how many entries are configured. That is usually enough to tell a near miss — a trailing slash, a port, a case difference — from a typo or an unset variable, without needing server logs.
|
|
99
111
|
|
|
@@ -111,23 +123,23 @@ See [Deployment Modes](deployment.md) and [Agent Access Control](agent_access_co
|
|
|
111
123
|
|
|
112
124
|
## Encryption
|
|
113
125
|
|
|
114
|
-
| Variable
|
|
115
|
-
|
|
|
116
|
-
| `NEOTOMA_ENCRYPTION_ENABLED`
|
|
117
|
-
| `NEOTOMA_KEY_FILE_PATH`
|
|
118
|
-
| `NEOTOMA_MNEMONIC` / `NEOTOMA_MNEMONIC_PASSPHRASE` | BIP-39 mnemonic key source
|
|
119
|
-
| `NEOTOMA_LOG_ENCRYPTION_ENABLED`
|
|
120
|
-
| `NEOTOMA_MCP_TOKEN_ENCRYPTION_KEY`
|
|
126
|
+
| Variable | Purpose |
|
|
127
|
+
| -------------------------------------------------- | --------------------------------------------- |
|
|
128
|
+
| `NEOTOMA_ENCRYPTION_ENABLED` | Turn on AES-256-GCM at-rest column encryption |
|
|
129
|
+
| `NEOTOMA_KEY_FILE_PATH` | Path to a 32-byte key file |
|
|
130
|
+
| `NEOTOMA_MNEMONIC` / `NEOTOMA_MNEMONIC_PASSPHRASE` | BIP-39 mnemonic key source |
|
|
131
|
+
| `NEOTOMA_LOG_ENCRYPTION_ENABLED` | Encrypt the event log |
|
|
132
|
+
| `NEOTOMA_MCP_TOKEN_ENCRYPTION_KEY` | Encrypt stored MCP OAuth tokens |
|
|
121
133
|
|
|
122
134
|
See [Encryption and Key Management](encryption.md).
|
|
123
135
|
|
|
124
136
|
## Search, inspector, and docs
|
|
125
137
|
|
|
126
|
-
| Variable
|
|
127
|
-
|
|
|
128
|
-
| `OPENAI_API_KEY`
|
|
129
|
-
| `NEOTOMA_INSPECTOR_DISABLE` / `NEOTOMA_PUBLIC_INSPECTOR_URL` / `NEOTOMA_INSPECTOR_BASE_PATH` | Control the bundled Inspector
|
|
130
|
-
| `NEOTOMA_DOCS_SHOW_INTERNAL`
|
|
138
|
+
| Variable | Purpose |
|
|
139
|
+
| -------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------- |
|
|
140
|
+
| `OPENAI_API_KEY` | Enables semantic vector search (embeddings); keyword search works without it |
|
|
141
|
+
| `NEOTOMA_INSPECTOR_DISABLE` / `NEOTOMA_PUBLIC_INSPECTOR_URL` / `NEOTOMA_INSPECTOR_BASE_PATH` | Control the bundled Inspector |
|
|
142
|
+
| `NEOTOMA_DOCS_SHOW_INTERNAL` | Show `visibility: internal` docs in the in-app `/docs` browser |
|
|
131
143
|
|
|
132
144
|
## Limits and mirror
|
|
133
145
|
|
|
@@ -265,6 +265,7 @@ caller repairs in-turn via `correct` rather than re-submitting the same payload.
|
|
|
265
265
|
| `SOURCE_PRIORITY_IGNORED` | 200 | No | The caller set a non-default `source_priority` but no field on the entity type uses a merge strategy that honours it. The write is accepted; the priority value has no effect on the stored snapshot. |
|
|
266
266
|
| `AUTO_LINK_EDGE_RETRACTED` | 200 | No | A schema-declared reference field's resolved target changed, so a stale auto-linked edge (e.g. a prior `works_at`) was retracted in the same reduction that created the new one (#1963). Positive signal; no action needed. |
|
|
267
267
|
| `AUTO_LINK_RETRACTION_FAILED` | 200 | No | A stale auto-linked edge should have been retracted but the soft-delete failed. The superseded edge may still be live despite the store succeeding. The message lists the stale `target_entity_id`s; clean up via `delete_relationship` (or re-run the store). |
|
|
268
|
+
| `STORE_WARNING_RULE_NOT_EVALUATED` | 200 | No | A schema-declared `store_warnings` rule could not be evaluated (its `fields` list or `condition` is malformed, uses an unsupported `condition` key, or declares neither spelling). The write is NOT blocked by this; it is a fail-open diagnostic naming the inert rule and the specific path that could not be read (#2165, #2170). |
|
|
268
269
|
|
|
269
270
|
**`MISSING_CONTENT_FIELD`** — fired when an entity's `SchemaDefinition` declares
|
|
270
271
|
`content_field` (e.g. `"body"` for `plan`, `"content"` for `note`) and the stored
|
|
@@ -363,6 +364,32 @@ knowing its own schema, or re-run the store. Example:
|
|
|
363
364
|
This code exists so the retraction failure is never silent — a silently-surviving edge is
|
|
364
365
|
the exact bug #1963 was filed to fix.
|
|
365
366
|
|
|
367
|
+
**`STORE_WARNING_RULE_NOT_EVALUATED`** — fired when a schema-declared `store_warnings`
|
|
368
|
+
rule cannot be evaluated. A rule's condition may be spelled as a flat `fields: string[]`
|
|
369
|
+
list (fire when none of the named fields carry a value) or declaratively as
|
|
370
|
+
`condition: { missing_all_of: [...] }`; both spellings evaluate identically. This code
|
|
371
|
+
fires instead of either spelling's own warning when the rule's `fields` or
|
|
372
|
+
`condition.missing_all_of` list is malformed (not a non-empty array of non-empty
|
|
373
|
+
strings), when `condition` declares an unsupported key, or when the rule declares
|
|
374
|
+
neither spelling at all. Before this code existed, an unevaluable rule threw a
|
|
375
|
+
`TypeError` that escaped as `DB_QUERY_FAILED`, making every write of that entity type
|
|
376
|
+
fail — including `commit: false` dry runs, since evaluation happens before the commit
|
|
377
|
+
decision (#2165, #2170). This warning is a fail-open diagnostic: it never blocks the
|
|
378
|
+
write, and it does not claim the entity was persisted — persistence is the response's
|
|
379
|
+
own `success` field and read-back to report. The `message` names the declared rule's own
|
|
380
|
+
`code` and the exact path (`fields` or `condition.missing_all_of`) that could not be
|
|
381
|
+
read, so a schema author can correct or remove the rule. Example:
|
|
382
|
+
|
|
383
|
+
```json
|
|
384
|
+
{
|
|
385
|
+
"code": "STORE_WARNING_RULE_NOT_EVALUATED",
|
|
386
|
+
"entity_type": "skill",
|
|
387
|
+
"entity_id": "ent_abc…",
|
|
388
|
+
"observation_index": 0,
|
|
389
|
+
"message": "Store-warning rule \"MISSING_CONTENT_FIELD\" declared on this entity type could not be evaluated: `condition` declares no recognised condition key. Supported condition keys: [\"missing_all_of\"]. Correct or remove the rule in the schema; the entity itself does not need to be re-sent for this warning alone. This advisory did NOT block the operation."
|
|
390
|
+
}
|
|
391
|
+
```
|
|
392
|
+
|
|
366
393
|
## ERR_CONSTRAINT_VIOLATION
|
|
367
394
|
|
|
368
395
|
Hard-rejection error returned by `/store` when one or more observations fail a
|
|
@@ -1412,7 +1412,7 @@ This enables full explainability: for any fact in the system, you can trace it b
|
|
|
1412
1412
|
|
|
1413
1413
|
### 3.12 `correct`
|
|
1414
1414
|
|
|
1415
|
-
**Purpose:** Create
|
|
1415
|
+
**Purpose:** Create a priority-1000 correction [observation](../vocabulary/canonical_terms.md#observation) to override lower-priority AI-[extracted](../vocabulary/canonical_terms.md#extraction) fields. Higher-priority sources remain authoritative; equal-priority ties follow configured source-kind precedence and deterministic recency in [entity snapshot](../vocabulary/canonical_terms.md#entity-snapshot) computation. Read the snapshot and field provenance back before claiming the correction took effect.
|
|
1416
1416
|
|
|
1417
1417
|
**Request Schema:**
|
|
1418
1418
|
|
|
@@ -53,8 +53,8 @@ obs_b: { status: "complete", observed_at: "2026-01-02T09:00:00Z" }
|
|
|
53
53
|
|
|
54
54
|
- Primary sort: `source_priority DESC`.
|
|
55
55
|
- Secondary sort (within the same priority tier): `observation_source` rank ASC (schema-configurable; sensors and workflow outputs rank above LLM summaries by default).
|
|
56
|
-
-
|
|
57
|
-
- Implementation: `highestPriority()`
|
|
56
|
+
- Same-priority ordering: after numeric priority and source-kind rank tie, the reducer chooses `observed_at DESC`, then `created_at DESC`, then `id ASC` as the final stable fallback. This applies even when the field declares `tie_breaker: "source_priority"`; source priority is already the primary trust axis, and same-tier scalar corrections still express replacement intent.
|
|
57
|
+
- Implementation: `highestPriority()` in `src/reducers/observation_reducer.ts`.
|
|
58
58
|
|
|
59
59
|
Use `highest_priority` when some sources are inherently more trustworthy than others —
|
|
60
60
|
official documents over extracted receipts, sensor data over user text, corrections over
|
|
@@ -230,10 +230,15 @@ regardless of which was written more recently.
|
|
|
230
230
|
|
|
231
231
|
### `tie_breaker` options
|
|
232
232
|
|
|
233
|
-
| Value |
|
|
234
|
-
| ------------------------- |
|
|
235
|
-
| `"observed_at"` (default) |
|
|
236
|
-
| `"source_priority"` |
|
|
233
|
+
| Value | `highest_priority` behavior after the priority and source-kind tiers tie | `most_specific` behavior after specificity ties |
|
|
234
|
+
| ------------------------- | ------------------------------------------------------------------------ | -------------------------------------------------------------------- |
|
|
235
|
+
| `"observed_at"` (default) | Newest `observed_at`, then newest `created_at`, then `id ASC` | Source-kind rank, then newest `observed_at` |
|
|
236
|
+
| `"source_priority"` | Same as `"observed_at"`; retained as a compatibility spelling | Numeric source priority, source-kind rank, then newest `observed_at` |
|
|
237
|
+
|
|
238
|
+
For `highest_priority`, numeric source priority is already the primary trust axis, so
|
|
239
|
+
the two configured values intentionally converge once that tier ties. The distinction
|
|
240
|
+
remains meaningful for `most_specific`, where `"source_priority"` adds numeric source
|
|
241
|
+
priority between specificity and source-kind rank.
|
|
237
242
|
|
|
238
243
|
---
|
|
239
244
|
|
|
@@ -103,14 +103,14 @@ const correctionObservation = {
|
|
|
103
103
|
interpretation_id: null, // No interpretation
|
|
104
104
|
observed_at: new Date(),
|
|
105
105
|
specificity_score: 1.0, // Maximum specificity
|
|
106
|
-
source_priority: 1000, //
|
|
106
|
+
source_priority: 1000, // Overrides lower-priority observations
|
|
107
107
|
fields: {
|
|
108
108
|
name: "Acme Corporation", // Corrected value
|
|
109
109
|
},
|
|
110
110
|
user_id: "user_123",
|
|
111
111
|
};
|
|
112
112
|
```
|
|
113
|
-
**Key property:** Corrections persist across reinterpretation. Even if a source is reinterpreted with a new model, the correction observation remains and continues to override AI-extracted values.
|
|
113
|
+
**Key property:** Corrections persist across reinterpretation. Even if a source is reinterpreted with a new model, the correction observation remains and continues to override lower-priority AI-extracted values. Higher-priority sources remain authoritative; equal-priority ties follow the schema's source-kind precedence before deterministic recency.
|
|
114
114
|
### 2.2 Observation Storage
|
|
115
115
|
Observations are stored in `observations` table:
|
|
116
116
|
- Immutable once created
|
|
@@ -106,9 +106,12 @@ function highestPriority(
|
|
|
106
106
|
field: string,
|
|
107
107
|
observations: Observation[]
|
|
108
108
|
): { value: any; source_observation_id: string } {
|
|
109
|
-
const sorted = observations.sort((a, b) =>
|
|
110
|
-
b.source_priority - a.source_priority ||
|
|
111
|
-
|
|
109
|
+
const sorted = observations.sort((a, b) =>
|
|
110
|
+
b.source_priority - a.source_priority ||
|
|
111
|
+
observationSourceRank(a) - observationSourceRank(b) ||
|
|
112
|
+
b.observed_at.getTime() - a.observed_at.getTime() ||
|
|
113
|
+
b.created_at.getTime() - a.created_at.getTime() ||
|
|
114
|
+
a.id.localeCompare(b.id)
|
|
112
115
|
);
|
|
113
116
|
return {
|
|
114
117
|
value: sorted[0].fields[field],
|
|
@@ -116,6 +119,29 @@ function highestPriority(
|
|
|
116
119
|
};
|
|
117
120
|
}
|
|
118
121
|
```
|
|
122
|
+
|
|
123
|
+
Same-priority scalar corrections use that same order. A correction does not need
|
|
124
|
+
to invent priority `1001` to replace an earlier normal correction: two correction
|
|
125
|
+
observations at priority `1000` keep higher `observation_source` precedence, then
|
|
126
|
+
the later `observed_at`/`created_at` wins, with `id ASC` used only when both
|
|
127
|
+
timestamps are equal. Correction receipts should still be followed by independent
|
|
128
|
+
snapshot readback; a response status alone is not proof of the effective value.
|
|
129
|
+
|
|
130
|
+
This ordering applies to every field whose active built-in or custom schema
|
|
131
|
+
declares `highest_priority`, and to every same-priority observation on that
|
|
132
|
+
field; it is not limited to observations created by the correction tools. The
|
|
133
|
+
change does not alter `most_specific` fields. Deploying the reducer does not
|
|
134
|
+
rewrite existing snapshots in bulk. A later store or correction recomputes the
|
|
135
|
+
entity it touches, while untouched snapshots keep their prior projection until
|
|
136
|
+
they are explicitly recomputed. For a deliberate rollout, first call
|
|
137
|
+
`POST /recompute_snapshots_by_type` with `dry_run: true`, then recompute each
|
|
138
|
+
affected entity type and verify the returned counts and errors plus representative
|
|
139
|
+
snapshot values and provenance. To roll back, revert the reducer before further
|
|
140
|
+
bulk recomputation. If snapshots were already recomputed, recompute the affected
|
|
141
|
+
types again after the rollback to restore the prior projection. Observations are
|
|
142
|
+
immutable, so this changes which value is materialized without deleting source
|
|
143
|
+
history.
|
|
144
|
+
|
|
119
145
|
**Use Cases:**
|
|
120
146
|
- `vendor_name` in invoices (official documents > receipts)
|
|
121
147
|
- `merchant_name` in receipts (bank statements > receipts)
|
|
@@ -62,6 +62,7 @@ interface FieldDefinition {
|
|
|
62
62
|
required?: boolean;
|
|
63
63
|
validator?: string; // Validator function name
|
|
64
64
|
preserveCase?: boolean; // Preserve case for this field during canonicalization
|
|
65
|
+
preserveWhitespace?: boolean; // Preserve exact whitespace, including line endings
|
|
65
66
|
description?: string; // Field description
|
|
66
67
|
converters?: ConverterDefinition[]; // Field type converters
|
|
67
68
|
}
|
|
@@ -632,3 +633,13 @@ Load `docs/subsystems/schema_registry.md` when:
|
|
|
632
633
|
- [ ] Reducer uses schema-projection filtering (only schema-defined fields appear in snapshots)
|
|
633
634
|
- [ ] Tests verify schema versioning, migration, and field removal
|
|
634
635
|
- [ ] Documentation updated for new schema versions
|
|
636
|
+
|
|
637
|
+
## Audio transcription contract
|
|
638
|
+
|
|
639
|
+
The built-in `transcription` 2.0.0 declaration retains historical field names and optionality, corrects `file_size_bytes` to a number, and declares capture and transcription provenance. `audio_content_sha256` identifies the input bytes; it is not the transcription entity identity. A recording can have multiple derivations. Source storage remains authoritative for bytes and content hash; observation `source_id` links derived fields to that source. Do not add an ingestion-success field to the transcription.
|
|
640
|
+
|
|
641
|
+
New ingestion clients must verify `audio_content_sha256`, `original_source_file`, `capture_method`, `transcription_engine`, `consent_basis`, `transcription_text`, and `file_size_bytes` after storage, plus the requested source linkage. `consent_basis: "unknown"` records absent attestation and grants no capture authorization. Historical records may omit these fields.
|
|
642
|
+
|
|
643
|
+
String fields can declare `preserveCase: true` and `preserveWhitespace: true` to retain exact transcript text and source names. Without `preserveWhitespace`, existing normalization remains unchanged. This metadata governs new observations only; activating a schema never rewrites historical observations. Lost historical text cannot be recovered from a normalized snapshot alone.
|
|
644
|
+
|
|
645
|
+
The declaration is available from `ENTITY_SCHEMAS.transcription` in `src/services/schema_definitions.ts`. Existing installations must inspect the effective global and user-specific registrations before registering a new version through the existing registry API. Preserve custom fields, reducers and identity declarations when merging an instance's schema. Code deployment alone does not establish that an active custom schema is compatible. Production registration, activation and any historical recovery are separate reviewed operations; this change performs none.
|