neotoma 0.3.9 → 0.3.10
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +119 -325
- package/dist/cli/bootstrap.js +0 -0
- package/dist/cli/index.d.ts.map +1 -1
- package/dist/cli/index.js +330 -16
- package/dist/cli/index.js.map +1 -1
- package/dist/repositories/sqlite/local_db_adapter.d.ts.map +1 -1
- package/dist/repositories/sqlite/local_db_adapter.js +180 -165
- package/dist/repositories/sqlite/local_db_adapter.js.map +1 -1
- package/package.json +7 -3
package/README.md
CHANGED
|
@@ -1,327 +1,160 @@
|
|
|
1
|
-
# Neotoma
|
|
1
|
+
# Neotoma
|
|
2
2
|
|
|
3
|
-
|
|
3
|
+
Deterministic state layer for AI agents. Open-source. Local-first. MIT licensed. For a guided overview, see [neotoma.io](https://neotoma.io).
|
|
4
4
|
|
|
5
|
-
|
|
5
|
+
## Why this exists
|
|
6
6
|
|
|
7
|
-
|
|
7
|
+
Production agents fail because their state has no invariant. Without a state invariant:
|
|
8
8
|
|
|
9
|
-
|
|
9
|
+
- Context drifts across sessions
|
|
10
|
+
- Facts conflict across tools
|
|
11
|
+
- Decisions execute without a reproducible trail
|
|
10
12
|
|
|
11
|
-
|
|
13
|
+
These are not hypothetical. They happen every day in production agent systems:
|
|
12
14
|
|
|
13
|
-
|
|
15
|
+
- An agent references an outdated contract clause retrieved from a stale embedding.
|
|
16
|
+
- Two tools record different versions of the same entity.
|
|
17
|
+
- An automated decision cannot be reproduced during debugging.
|
|
18
|
+
- Context drifts across sessions and the agent silently changes behavior.
|
|
14
19
|
|
|
15
|
-
|
|
20
|
+
Agent state must obey invariants. Without them, every downstream action inherits the uncertainty of the state it reads.
|
|
16
21
|
|
|
17
|
-
|
|
22
|
+
## The state invariant
|
|
18
23
|
|
|
19
|
-
|
|
24
|
+
Neotoma enforces a deterministic state invariant. State is **versioned**, **schema-bound**, **replayable**, and **auditable**. Every mutation is recorded. Every state change can be inspected or replayed. No silent mutation. No implicit overwrite.
|
|
20
25
|
|
|
21
|
-
|
|
26
|
+
Agents need a deterministic state layer. RAG retrieves documents. Neotoma enforces state evolution. Neotoma treats memory as state evolution, not retrieval. State evolves through a four-stage pipeline. Every stage is versioned with full provenance.
|
|
22
27
|
|
|
23
|
-
|
|
24
|
-
|
|
25
|
-
It's not a note-taking app or "second brain." Not provider-controlled ChatGPT Memory or Claude Projects (those are conversation-only and platform-locked; Neotoma is structured personal data memory with entity resolution and timelines, cross-platform via MCP). Not a vector store or RAG layer. Not an autonomous agent. It is the memory layer agents read and write; you control what goes in and what stays.
|
|
26
|
-
|
|
27
|
-
---
|
|
28
|
-
|
|
29
|
-
## Three Foundations
|
|
30
|
-
|
|
31
|
-
Neotoma is built on three architectural choices that provider memory cannot offer:
|
|
32
|
-
|
|
33
|
-
| Foundation | What it means |
|
|
34
|
-
| ------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
35
|
-
| **Privacy-first** | User-controlled memory, end-to-end encryption and row-level security, never used for training. Your data remains yours. |
|
|
36
|
-
| **Deterministic** | Same input always produces same output. Schema-first extraction, hash-based entity IDs, full provenance. No hallucinations or probabilistic behavior. |
|
|
37
|
-
| **Cross-platform** | Works with ChatGPT, Claude, Cursor, and Claude Code via MCP. One memory system across tools; no platform lock-in. Localhost-agent friendly. |
|
|
38
|
-
|
|
39
|
-
**These enable:** Immutable audit trail and time-travel queries, cryptographic integrity, event-sourced history, entity resolution across documents and agent data, timeline generation, dual-path storing (files + agent interactions), and persistent memory without context-window limits.
|
|
40
|
-
|
|
41
|
-
**System architecture:**
|
|
28
|
+
## Architecture
|
|
42
29
|
|
|
43
30
|
```mermaid
|
|
44
31
|
graph LR
|
|
45
|
-
Sources[
|
|
46
|
-
Ingest --> Obs[Observations]
|
|
32
|
+
Sources["Sources (files, messages, APIs)"] --> Obs[Observations]
|
|
47
33
|
Obs --> Entities[Entity Resolution]
|
|
48
|
-
Entities --> Snapshots[Entity Snapshots]
|
|
34
|
+
Entities --> Snapshots["Entity Snapshots (versioned)"]
|
|
49
35
|
Snapshots --> Graph[Memory Graph]
|
|
50
36
|
Graph <--> MCP[MCP Protocol]
|
|
51
|
-
MCP --> ChatGPT
|
|
52
37
|
MCP --> Claude
|
|
53
38
|
MCP --> Cursor
|
|
39
|
+
MCP --> Codex
|
|
54
40
|
```
|
|
55
41
|
|
|
56
|
-
|
|
57
|
-
|
|
58
|
-
|
|
59
|
-
|
|
60
|
-
| Problem | How Neotoma addresses it |
|
|
61
|
-
| ------------------------------------ | ------------------------------------------------------------------------------------------------------------- |
|
|
62
|
-
| Personal data is fragmented | Dual-path storing from file uploads and agent interactions into one source of truth. |
|
|
63
|
-
| Provider memory is conversation-only | Structured personal data memory with entity resolution and timelines across documents and agent-created data. |
|
|
64
|
-
| No trust when agents act | Explicit, named operations; visible inputs; reconstructable history. Replay and audit trail. |
|
|
65
|
-
| No cross-data reasoning | One graph: sources, entities, events, typed relationships. |
|
|
66
|
-
| Entity fragmentation | Hash-based canonical IDs unify "Acme Corp" across all personal data. |
|
|
67
|
-
| No temporal reasoning | Automatic timeline generation from date fields. |
|
|
68
|
-
| Platform lock-in | MCP-based access; works with any compatible AI tool. |
|
|
69
|
-
|
|
70
|
-
---
|
|
71
|
-
|
|
72
|
-
## Core Terminology
|
|
73
|
-
|
|
74
|
-
| Term | Definition |
|
|
75
|
-
| --------------- | --------------------------------------------------------------------------------------------- |
|
|
76
|
-
| Truth Layer | Deterministic, immutable structured memory substrate that tools and agents read and write. |
|
|
77
|
-
| Source | Raw data (file, text, URL, or structured JSON) stored with content-addressed deduplication. |
|
|
78
|
-
| Observation | Granular fact extracted from a source; reducers merge observations into entity snapshots. |
|
|
79
|
-
| Entity | Canonical representation of a person, company, task, or other object with a deterministic ID. |
|
|
80
|
-
| Entity snapshot | Current truth for an entity computed from all related observations. |
|
|
81
|
-
| Provenance | Origin tracking (source, timestamp, operation) so each value is traceable. |
|
|
82
|
-
| Memory graph | Graph of sources, observations, entities, events, and typed relationships. |
|
|
83
|
-
|
|
84
|
-
For the full glossary, see [Core terminology](https://neotoma.io/#terminology).
|
|
85
|
-
|
|
86
|
-
---
|
|
87
|
-
|
|
88
|
-
## Who It's For
|
|
89
|
-
|
|
90
|
-
| Who | What they need | Example data to remember |
|
|
91
|
-
| ------------------------------------- | ------------------------------------------------------------------------ | ----------------------------------------------------------------------- |
|
|
92
|
-
| AI-native individual operators | Memory that follows across daily tools and sessions | Tasks, preferences, notes, recurring reminders, contacts, deadlines |
|
|
93
|
-
| Knowledge workers with scattered data | Durable context across documents and sessions, with evidence and lineage | Source documents, extracted entities, citations, key quotes, timelines |
|
|
94
|
-
| Builders of agentic systems | Structured memory agents can read and write with provenance | Session histories, accumulated facts, decisions, runbooks, tool configs |
|
|
95
|
-
|
|
96
|
-
**Why Neotoma:** One memory graph across documents and agent-created data; agents remember context without re-explanation; full provenance and audit trail; works with any MCP-compatible tool; privacy-first and user-controlled. The same substrate serves both human-in-the-loop use and agent frameworks or toolchains that need deterministic memory and provenance.
|
|
97
|
-
|
|
98
|
-
**Suggested GitHub topics:** `ai-agents` `ai-memory` `memory-system` `entity-resolution` `event-sourcing` `truth-layer` `mcp` `provenance` `privacy-first`
|
|
99
|
-
|
|
100
|
-
---
|
|
101
|
-
|
|
102
|
-
## Security Defaults
|
|
103
|
-
|
|
104
|
-
Neotoma stores personal data and requires secure configuration.
|
|
105
|
-
|
|
106
|
-
**Authentication:** Local auth is the active mode (dev stub or key-based when encryption is enabled). Session tokens are deprecated. See [OAuth implementation](docs/developer/mcp_oauth_implementation.md) for current local flow details.
|
|
107
|
-
|
|
108
|
-
**Authorization:** Local data isolation and explicit operation-level access controls are enforced in local-only mode.
|
|
109
|
-
|
|
110
|
-
**Data protection:** User-controlled data with full export and deletion control; never used for training or provider access. Local backend supports optional encryption at rest (key file or mnemonic). End-to-end encryption planned (v2.0.0).
|
|
111
|
-
|
|
112
|
-
**Verify your setup:** Run `npm run doctor` for environment, database, RLS, and security checks. See [Health check](docs/operations/health_check.md), [Auth](docs/subsystems/auth.md), [Privacy](docs/subsystems/privacy.md), and [Compliance](docs/legal/compliance.md).
|
|
113
|
-
|
|
114
|
-
---
|
|
115
|
-
|
|
116
|
-
## Current Status
|
|
117
|
-
|
|
118
|
-
**Version:** v0.3.0
|
|
119
|
-
**Status:** Completed (reconciliation release establishing current baseline)
|
|
120
|
-
|
|
121
|
-
**Developer preview:** Planned during dogfooding once core invariants are stable.
|
|
122
|
-
|
|
123
|
-
**What's implemented:** Sources-first architecture with content-addressed storage, dual-path storing (file uploads + agent interactions), observations architecture, entity resolution with hash-based IDs, schema registry system, auto-enhancement, timeline generation, optional entity semantic search for `retrieve_entities` and `retrieve_entity_by_identifier` (local embeddings), MCP integration (ChatGPT, Claude, Cursor), full provenance and audit trail, React frontend, CLI. See [Release roadmap](#release-roadmap) and [docs/releases/](docs/releases/) for details.
|
|
124
|
-
|
|
125
|
-
**Next steps:** Review current uncommitted changes, apply pending migrations, audit the test suite, and plan v0.4.0 realistically based on the current baseline.
|
|
126
|
-
|
|
127
|
-
---
|
|
128
|
-
|
|
129
|
-
## Developer preview
|
|
130
|
-
|
|
131
|
-
The developer preview exposes the **core contract only**: CLI for humans, MCP for agents, OpenAPI as the single source of truth. There is no web app in scope for the preview; the frontend in this repo exists for development and future use.
|
|
132
|
-
|
|
133
|
-
### What is guaranteed (even in preview)
|
|
134
|
-
|
|
135
|
-
- **No silent data loss** – Operations either succeed and are recorded or fail with explicit errors.
|
|
136
|
-
- **Explicit, inspectable state mutations** – Every change is a named operation with visible inputs; state is reconstructable from the audit trail.
|
|
137
|
-
- **Auditable operations** – Full provenance; CLI and MCP map to the same underlying contract. If it cannot be expressed via the contract, it is not real.
|
|
138
|
-
- **Same contract for CLI and MCP** – Both use the same OpenAPI-backed operations; MCP tool calls log the equivalent CLI invocation (redaction-aware).
|
|
139
|
-
|
|
140
|
-
### What is not guaranteed yet
|
|
141
|
-
|
|
142
|
-
- Stable schemas
|
|
143
|
-
- Deterministic extraction across versions
|
|
144
|
-
- Long-term replay compatibility
|
|
145
|
-
- Backward compatibility
|
|
146
|
-
|
|
147
|
-
Breaking changes should be expected.
|
|
148
|
-
|
|
149
|
-
### Who this is for
|
|
150
|
-
|
|
151
|
-
- Developers comfortable with CLI-first workflows
|
|
152
|
-
- People building or operating agentic systems
|
|
153
|
-
- Anyone who treats personal data like production infrastructure
|
|
154
|
-
|
|
155
|
-
### Who this is not for (yet)
|
|
156
|
-
|
|
157
|
-
- UI-first users
|
|
158
|
-
- Casual note-taking
|
|
159
|
-
- Anyone expecting reliability guarantees today
|
|
160
|
-
|
|
161
|
-
**Storage:** The developer preview supports **local storage only** (SQLite + local file storage). See [Developer preview storage](docs/developer/developer_preview_storage.md) and [Getting started](docs/developer/getting_started.md) for setup.
|
|
162
|
-
|
|
163
|
-
---
|
|
164
|
-
|
|
165
|
-
## Release Roadmap
|
|
166
|
-
|
|
167
|
-
### Completed Releases
|
|
168
|
-
|
|
169
|
-
- **v0.2.0** – Minimal ingestion + correction loop (`in_testing`). [docs/releases/v0.2.0/](docs/releases/v0.2.0/)
|
|
170
|
-
- **v0.2.1** – Documentation generation system (`in_progress`). [docs/releases/v0.2.1/](docs/releases/v0.2.1/)
|
|
171
|
-
- **v0.2.2** – `list_capabilities` MCP action (`planning`). [docs/releases/v0.2.2/](docs/releases/v0.2.2/)
|
|
172
|
-
- **v0.2.15** – Complete architecture migration (`implemented`, pending migrations). [docs/releases/v0.2.15/](docs/releases/v0.2.15/)
|
|
173
|
-
- **v0.3.0** – Reconciliation release (`completed`). [docs/releases/v0.3.0/](docs/releases/v0.3.0/)
|
|
174
|
-
|
|
175
|
-
### Future Planning
|
|
176
|
-
|
|
177
|
-
Future releases will be planned realistically based on the v0.3.0 baseline. Previous aspirational releases (v0.4.0 through v2.2.0) have been archived to [docs/releases/archived/aspirational/](docs/releases/archived/aspirational/) and can be revisited for future planning.
|
|
178
|
-
|
|
179
|
-
Full release index: [docs/releases/](docs/releases/).
|
|
180
|
-
|
|
181
|
-
---
|
|
182
|
-
|
|
183
|
-
## Documentation Index
|
|
184
|
-
|
|
185
|
-
The **primary entrypoint** for all documentation is the index and navigation guide. All contributors and AI assistants working on the repo should load it first.
|
|
42
|
+
- **Deterministic.** Same observations always produce the same versioned entity snapshots. No ordering sensitivity.
|
|
43
|
+
- **Immutable.** Append-only observations. Corrections add new data, never erase.
|
|
44
|
+
- **Replayable.** Inspect any entity at any point in time. Diff versions. Reconstruct history from the observation log.
|
|
45
|
+
- **Structure-first.** Schema-first extraction with deterministic retrieval. Optional similarity search when embeddings are configured.
|
|
186
46
|
|
|
187
|
-
|
|
47
|
+
### Three foundations
|
|
188
48
|
|
|
189
|
-
|
|
49
|
+
| Foundation | What it means |
|
|
50
|
+
| ------------------ | --------------------------------------------------------------------------------------------------------------------------------------- |
|
|
51
|
+
| **Privacy-first** | Your data stays local. Never used for training. User-controlled storage, optional encryption at rest. Full export and deletion control. |
|
|
52
|
+
| **Deterministic** | Same input always produces same output. Schema-first extraction, hash-based entity IDs, full provenance. No silent mutation. |
|
|
53
|
+
| **Cross-platform** | One memory graph across Claude, Cursor, Codex, and CLI. MCP-based access. No platform lock-in. Works alongside native memory. |
|
|
190
54
|
|
|
191
|
-
|
|
192
|
-
- [Philosophy](docs/foundation/philosophy.md) – Principles and invariants
|
|
193
|
-
- [Problem statement](docs/foundation/problem_statement.md) – Why Neotoma exists
|
|
194
|
-
- [Layered architecture](docs/foundation/layered_architecture.md) – Truth / Strategy / Execution layers
|
|
55
|
+
## State guarantees
|
|
195
56
|
|
|
196
|
-
|
|
57
|
+
Most AI memory systems optimize storage or retrieval. Neotoma enforces state integrity.
|
|
197
58
|
|
|
198
|
-
|
|
199
|
-
|
|
200
|
-
|
|
201
|
-
|
|
59
|
+
| Property | Platform | Retrieval / RAG | Files | Deterministic |
|
|
60
|
+
| ------------------------------------ | --------- | --------------- | ---------- | ------------- |
|
|
61
|
+
| Deterministic state evolution | ✗ | ✗ | ✗ | ✓ |
|
|
62
|
+
| Versioned history | ✗ | ✗ | ⚠ manual | ✓ |
|
|
63
|
+
| Replayable timeline | ✗ | ✗ | ✗ | ✓ |
|
|
64
|
+
| Auditable change log | ✗ | ✗ | ⚠ partial | ✓ |
|
|
65
|
+
| Schema constraints | ✗ | ✗ | ✗ | ✓ |
|
|
66
|
+
| Silent mutation risk | ⚠ common | ⚠ common | ⚠ common | prevented |
|
|
67
|
+
| Conflicting facts risk | ⚠ common | ⚠ common | ⚠ possible | prevented |
|
|
68
|
+
| Reproducible state reconstruction | ✗ | ✗ | ✗ | ✓ |
|
|
69
|
+
| Human inspectability (diffs/lineage) | ⚠ partial | ⚠ partial | ⚠ partial | ✓ |
|
|
70
|
+
| Zero-setup onboarding | ✓ | ✗ | ✗ | ✗ |
|
|
71
|
+
| Semantic similarity search | ✗ | ✓ | ✗ | ✓ |
|
|
72
|
+
| Direct human editability | ✗ | ✗ | ✓ | ✗ |
|
|
202
73
|
|
|
203
|
-
**
|
|
74
|
+
**Platform:** Claude, ChatGPT, Gemini, Copilot. **Retrieval:** Mem0, Zep, LangChain Memory. **Files:** Markdown files, JSON stores, CRDT docs. **Deterministic:** Neotoma (reference implementation).
|
|
204
75
|
|
|
205
|
-
|
|
206
|
-
- [Developer preview storage](docs/developer/developer_preview_storage.md) – Local-only stance for the preview
|
|
207
|
-
- [CLI overview](docs/developer/cli_overview.md) – CLI workflows
|
|
208
|
-
- [CLI reference](docs/developer/cli_reference.md) – Commands and scripts
|
|
209
|
-
- [MCP overview](docs/developer/mcp_overview.md) – MCP entry points and setup links
|
|
210
|
-
- [Development workflow](docs/developer/development_workflow.md) – Git, branches, PRs
|
|
211
|
-
- [MCP developer docs](docs/developer/mcp/) – Instructions, tool descriptions, unauthenticated use
|
|
76
|
+
## Quick start
|
|
212
77
|
|
|
213
|
-
|
|
78
|
+
### Agent-driven install (recommended)
|
|
214
79
|
|
|
215
|
-
|
|
216
|
-
- [Subsystems](docs/subsystems/) – Ingestion, observation architecture, reducer, relationships, auth, events, errors, privacy
|
|
217
|
-
- [Feature units](docs/feature_units/) – Standards, manifests, execution instructions
|
|
218
|
-
- [Testing](docs/testing/) – [Automated test catalog](docs/testing/automated_test_catalog.md), [testing standard](docs/testing/testing_standard.md)
|
|
219
|
-
- [Operations](docs/operations/) – [Runbook](docs/operations/runbook.md) (startup, health, shutdown), [Health check](docs/operations/health_check.md) (`npm run doctor`), [Troubleshooting](docs/operations/troubleshooting.md)
|
|
220
|
-
- [API](docs/api/rest_api.md), [Error codes](docs/reference/error_codes.md), [Vocabulary](docs/vocabulary/canonical_terms.md)
|
|
80
|
+
Agents install Neotoma themselves. Paste this prompt into Claude, Codex, or Cursor:
|
|
221
81
|
|
|
222
|
-
|
|
82
|
+
```
|
|
83
|
+
Install Neotoma by following these instructions: https://github.com/markmhendrickson/neotoma/blob/main/install.md
|
|
223
84
|
|
|
224
|
-
|
|
85
|
+
Discover candidate data from available context — including any facts this tool already remembers about me — preview it for approval, and save only what I confirm.
|
|
86
|
+
```
|
|
225
87
|
|
|
226
|
-
|
|
88
|
+
The agent handles npm install, initialization, and MCP configuration. **Manual install:**
|
|
227
89
|
|
|
228
90
|
```bash
|
|
229
|
-
# Global install (recommended for MCP server use)
|
|
230
91
|
npm install -g neotoma
|
|
231
|
-
|
|
232
|
-
# Initialize (creates data directories, database; init can prompt for encryption)
|
|
233
92
|
neotoma init
|
|
234
|
-
|
|
235
|
-
# Start the API server
|
|
236
|
-
neotoma api start
|
|
93
|
+
neotoma mcp config
|
|
237
94
|
```
|
|
238
95
|
|
|
239
|
-
|
|
96
|
+
More options: [Docker](docs/developer/docker.md) | [CLI reference](docs/developer/cli_reference.md) | [Getting started](docs/developer/getting_started.md)
|
|
240
97
|
|
|
241
|
-
|
|
98
|
+
## Example
|
|
242
99
|
|
|
243
100
|
```bash
|
|
244
|
-
|
|
101
|
+
neotoma store --json='[{"entity_type":"task","title":"Submit expense report","status":"open"}]'
|
|
245
102
|
neotoma entities list --type task
|
|
246
|
-
|
|
247
|
-
# Explicit remote API mode (fail if API is unreachable)
|
|
248
|
-
neotoma --api-only entities list --type task
|
|
103
|
+
neotoma upload ./invoice.pdf
|
|
249
104
|
```
|
|
250
105
|
|
|
251
|
-
|
|
106
|
+
Results reflect versioned entity state with full provenance. Agents perform the same operations through MCP tool calls (`store`, `retrieve_entities`, `retrieve_entity_by_identifier`).
|
|
252
107
|
|
|
253
|
-
|
|
254
|
-
neotoma mcp config
|
|
255
|
-
```
|
|
256
|
-
|
|
257
|
-
For first-run agent-guided setup (install + init + preview-before-save confirmation), see [Agent onboarding](docs/developer/agent_onboarding.md).
|
|
258
|
-
|
|
259
|
-
Marketing site: **https://neotoma.io** (GitHub Pages, deployed from **main**; custom domain set in repo Settings → Pages). See [Deployment](docs/infrastructure/deployment.md#marketing-site-neotomaio).
|
|
260
|
-
|
|
261
|
-
### Option 2: Clone source checkout (for development)
|
|
262
|
-
|
|
263
|
-
```bash
|
|
264
|
-
git clone https://github.com/markmhendrickson/neotoma.git
|
|
265
|
-
cd neotoma
|
|
266
|
-
npm install
|
|
267
|
-
npm test
|
|
268
|
-
```
|
|
108
|
+
## Interfaces
|
|
269
109
|
|
|
270
|
-
|
|
110
|
+
Three interfaces. One state invariant. Every interface provides the same deterministic behavior regardless of how you access the state layer.
|
|
271
111
|
|
|
272
|
-
|
|
112
|
+
| Interface | Description |
|
|
113
|
+
| -------------- | ------------------------------------------------------------------------------------------------------------------------------ |
|
|
114
|
+
| **REST API** | Full HTTP interface for application integration. Entities, relationships, observations, schema, timeline, and version history. |
|
|
115
|
+
| **MCP Server** | Model Context Protocol for Claude, Cursor, and Codex. Agents store and retrieve state through structured tool calls. |
|
|
116
|
+
| **CLI** | Command-line for scripting and direct access. Inspect entities, replay timelines, and manage state from the terminal. |
|
|
273
117
|
|
|
274
|
-
|
|
275
|
-
git clone https://github.com/markmhendrickson/neotoma.git
|
|
276
|
-
cd neotoma
|
|
277
|
-
docker build -t neotoma .
|
|
118
|
+
All three map to the same OpenAPI-backed operations. MCP tool calls log the equivalent CLI invocation.
|
|
278
119
|
|
|
279
|
-
|
|
280
|
-
--name neotoma \
|
|
281
|
-
-p 3080:3080 \
|
|
282
|
-
-v neotoma-data:/app/data \
|
|
283
|
-
neotoma
|
|
120
|
+
## Who this is for
|
|
284
121
|
|
|
285
|
-
|
|
286
|
-
|
|
122
|
+
| Who | What they need |
|
|
123
|
+
| ------------------------------- | ------------------------------------------------------------------ |
|
|
124
|
+
| **AI infrastructure engineers** | State integrity guarantees for agent runtimes and orchestration |
|
|
125
|
+
| **Agent system builders** | Deterministic state and provenance layer for agents and toolchains |
|
|
126
|
+
| **AI-native operators** | State that follows across every tool and session |
|
|
287
127
|
|
|
288
|
-
|
|
289
|
-
|
|
290
|
-
```json
|
|
291
|
-
{
|
|
292
|
-
"mcpServers": {
|
|
293
|
-
"neotoma": {
|
|
294
|
-
"command": "docker",
|
|
295
|
-
"args": ["exec", "-i", "neotoma", "node", "dist/index.js"]
|
|
296
|
-
}
|
|
297
|
-
}
|
|
298
|
-
}
|
|
299
|
-
```
|
|
128
|
+
Not for casual note-taking. Not for UI-first users expecting reliability guarantees today.
|
|
300
129
|
|
|
301
|
-
|
|
130
|
+
## Current status
|
|
302
131
|
|
|
303
|
-
|
|
304
|
-
docker exec neotoma neotoma store \
|
|
305
|
-
--json='[{"entity_type":"task","title":"Submit expense report","status":"open"}]'
|
|
132
|
+
**Version:** v0.3.9 · **Releases:** 10 · **License:** MIT
|
|
306
133
|
|
|
307
|
-
|
|
308
|
-
```
|
|
134
|
+
### What is guaranteed (even in preview)
|
|
309
135
|
|
|
310
|
-
|
|
136
|
+
- **No silent data loss.** Operations either succeed and are recorded or fail with explicit errors.
|
|
137
|
+
- **Explicit, inspectable state mutations.** Every change is a named operation with visible inputs. State is reconstructable from the audit trail.
|
|
138
|
+
- **Auditable operations.** Full provenance. CLI and MCP map to the same underlying contract.
|
|
139
|
+
- **Same contract for CLI and MCP.** Both use the same OpenAPI-backed operations.
|
|
311
140
|
|
|
312
|
-
|
|
141
|
+
### What is not guaranteed yet
|
|
313
142
|
|
|
314
|
-
|
|
143
|
+
- Stable schemas
|
|
144
|
+
- Deterministic extraction across versions
|
|
145
|
+
- Long-term replay compatibility
|
|
146
|
+
- Backward compatibility
|
|
315
147
|
|
|
316
|
-
|
|
317
|
-
2. In a conversation, tell your assistant: "Remind me to review my subscription Friday."
|
|
318
|
-
3. In the same conversation, ask it to list your open tasks.
|
|
148
|
+
Breaking changes should be expected. **Storage:** Local-only (SQLite + local file storage). See [Developer preview storage](docs/developer/developer_preview_storage.md).
|
|
319
149
|
|
|
320
|
-
|
|
150
|
+
## Security defaults
|
|
321
151
|
|
|
322
|
-
|
|
152
|
+
Neotoma stores user data and requires secure configuration.
|
|
323
153
|
|
|
324
|
-
|
|
154
|
+
- **Authentication:** Local auth (dev stub or key-based when encryption is enabled).
|
|
155
|
+
- **Authorization:** Local data isolation and explicit operation-level access controls.
|
|
156
|
+
- **Data protection:** User-controlled data with full export and deletion control. Never used for training. Optional encryption at rest.
|
|
157
|
+
- **Verify your setup:** Run `npm run doctor` for environment, database, and security checks. See [Auth](docs/subsystems/auth.md), [Privacy](docs/subsystems/privacy.md), [Compliance](docs/legal/compliance.md).
|
|
325
158
|
|
|
326
159
|
## Development
|
|
327
160
|
|
|
@@ -332,100 +165,61 @@ npm run dev # MCP server (stdio)
|
|
|
332
165
|
npm run dev:ui # Frontend
|
|
333
166
|
npm run dev:server # API only (MCP at /mcp)
|
|
334
167
|
npm run dev:full # API + UI + build watch
|
|
335
|
-
npm run dev:ws # WebSocket MCP bridge
|
|
336
|
-
npm run watch:prod:tunnel # Production mode + HTTPS tunnel (local storage, remote access)
|
|
337
168
|
```
|
|
338
169
|
|
|
339
|
-
For `watch:prod:tunnel`, no `.env` is required - server auto-discovers tunnel URL. Tunnel prefers Cloudflare when `cloudflared` is installed; else ngrok. Optional: `NEOTOMA_HOST_URL` (for fixed domain), `NEOTOMA_BEARER_TOKEN`. Full tunnel and env: [Tunnels](docs/developer/tunnels.md), [Getting started – Production with tunnel](docs/developer/getting_started.md#start-development-server).
|
|
340
|
-
|
|
341
170
|
**CLI:**
|
|
342
171
|
|
|
343
172
|
```bash
|
|
344
173
|
npm run cli # Run via npm (no global install)
|
|
345
174
|
npm run cli:dev # Dev mode (tsx; picks up source changes)
|
|
346
175
|
npm run setup:cli # Build and link so `neotoma` is available globally
|
|
347
|
-
# Or manually: npm run build:server && npm install -g . (or npm link)
|
|
348
176
|
```
|
|
349
177
|
|
|
350
|
-
|
|
351
|
-
|
|
352
|
-
Interactive `neotoma` startup is connect-only. It discovers healthy local API instances, applies `--env` as a preference, and prompts when multiple instances are available.
|
|
178
|
+
**Testing:** `npm test` | `npm run test:integration` | `npm run test:e2e` | `npm run test:agent-mcp` | `npm run type-check` | `npm run lint` · **Source checkout:**
|
|
353
179
|
|
|
354
|
-
|
|
355
|
-
|
|
356
|
-
|
|
357
|
-
|
|
358
|
-
|
|
180
|
+
```bash
|
|
181
|
+
git clone https://github.com/markmhendrickson/neotoma.git
|
|
182
|
+
cd neotoma
|
|
183
|
+
npm install
|
|
184
|
+
npm test
|
|
185
|
+
```
|
|
359
186
|
|
|
360
|
-
|
|
187
|
+
**Prerequisites:** Node.js v18.x or v20.x (LTS), npm v9+. No `.env` required for local storage. See [Getting started](docs/developer/getting_started.md).
|
|
361
188
|
|
|
362
|
-
## Using with AI
|
|
189
|
+
## Using with AI tools (MCP)
|
|
363
190
|
|
|
364
|
-
Neotoma exposes
|
|
191
|
+
Neotoma exposes state via MCP. Local storage only in preview. Local built-in auth.
|
|
365
192
|
|
|
366
193
|
**Setup:**
|
|
367
194
|
|
|
368
195
|
- [Cursor MCP setup](docs/developer/mcp_cursor_setup.md)
|
|
369
196
|
- [Claude Code MCP setup](docs/developer/mcp_claude_code_setup.md)
|
|
370
197
|
- [ChatGPT Custom GPT setup](docs/developer/mcp_chatgpt_setup.md)
|
|
371
|
-
- [OAuth implementation](docs/developer/mcp_oauth_implementation.md)
|
|
372
|
-
|
|
373
|
-
**Representative actions:** `store`, `reinterpret`, `correct`, `retrieve_entities`, `retrieve_entity_snapshot`, `merge_entities`, `list_observations`, `create_relationship`, `list_relationships`, `list_timeline_events`, `retrieve_graph_neighborhood`, `retrieve_file_url`, schema tools (`analyze_schema_candidates`, `get_schema_recommendations`, `update_schema_incremental`, `register_schema`). Full list: [MCP spec](docs/specs/MCP_SPEC.md).
|
|
374
|
-
|
|
375
|
-
To use the Neotoma MCP server from another workspace, see [Cursor MCP setup](docs/developer/mcp_cursor_setup.md).
|
|
376
|
-
|
|
377
|
-
---
|
|
378
198
|
|
|
379
|
-
|
|
199
|
+
**Agent behavior contract:** Store first, retrieve before storing, extract entities from user input, create tasks for commitments. Full instructions: [MCP instructions](docs/developer/mcp/instructions.md) and [CLI agent instructions](docs/developer/cli_agent_instructions.md).
|
|
380
200
|
|
|
381
|
-
|
|
382
|
-
|
|
383
|
-
- **Store first:** Persist the conversation turn before responding.
|
|
384
|
-
- **Bounded retrieval:** Retrieve likely related entities before storing new ones.
|
|
385
|
-
- **Entity extraction:** Extract and store relevant people, tasks, events, places, and relationships from user input.
|
|
386
|
-
- **Task creation:** Create tasks when users express intent, obligations, deadlines, or reminders.
|
|
387
|
-
- **External data safety:** Store relevant entities from external tool results before responding.
|
|
388
|
-
|
|
389
|
-
Full instructions: [MCP instructions](docs/developer/mcp/instructions.md) and [CLI agent instructions](docs/developer/cli_agent_instructions.md).
|
|
390
|
-
|
|
391
|
-
---
|
|
392
|
-
|
|
393
|
-
## Core Principles
|
|
394
|
-
|
|
395
|
-
1. **Deterministic** – Same input → same output. Hash-based IDs, no randomness in core components.
|
|
396
|
-
2. **Schema-first** – Type-driven extraction and storage, not freeform blobs.
|
|
397
|
-
3. **Explainable** – Every field traces to source (document or agent interaction).
|
|
398
|
-
4. **Entity-unified** – Canonical IDs across all personal data.
|
|
399
|
-
5. **Timeline-aware** – Chronological ordering from date fields.
|
|
400
|
-
6. **Cross-platform** – MCP for any compatible AI tool.
|
|
401
|
-
7. **Privacy-first** – User-controlled memory, encryption, RLS.
|
|
402
|
-
8. **Immutable** – Truth does not change after storage; corrections create new observations.
|
|
403
|
-
9. **Provenance** – Full audit trail; event-sourced, replayable history.
|
|
404
|
-
10. **Explicit control** – Ingestion only of what you explicitly provide; no background scanning.
|
|
405
|
-
11. **Four-layer model** – Source → Observation → Entity → Entity Snapshot.
|
|
406
|
-
|
|
407
|
-
---
|
|
408
|
-
|
|
409
|
-
## Local-only storage
|
|
410
|
-
|
|
411
|
-
Neotoma is intentionally local-only in this active codebase (SQLite + local file storage).
|
|
412
|
-
If remote backend support is needed later, recover it from git history.
|
|
413
|
-
|
|
414
|
-
---
|
|
201
|
+
**Representative actions:** `store`, `retrieve_entities`, `retrieve_entity_snapshot`, `merge_entities`, `list_observations`, `create_relationship`, `list_relationships`, `list_timeline_events`, `retrieve_graph_neighborhood`. Full list: [MCP spec](docs/specs/MCP_SPEC.md).
|
|
415
202
|
|
|
416
203
|
## Related posts
|
|
417
204
|
|
|
418
205
|
- [Neotoma developer release](https://markmhendrickson.com/posts/neotoma-developer-release)
|
|
206
|
+
- [Your AI remembers your vibe but not your work](https://markmhendrickson.com/posts/your-ai-remembers-your-vibe-but-not-your-work)
|
|
419
207
|
- [Building a truth layer for persistent agent memory](https://markmhendrickson.com/posts/truth-layer-agent-memory)
|
|
420
208
|
- [Agent memory has a truth problem](https://markmhendrickson.com/posts/agent-memory-truth-problem)
|
|
421
209
|
- [Why agent memory needs more than RAG](https://markmhendrickson.com/posts/why-agent-memory-needs-more-than-rag)
|
|
422
210
|
|
|
423
|
-
|
|
211
|
+
## Documentation
|
|
424
212
|
|
|
425
|
-
|
|
213
|
+
Full documentation is organized at [neotoma.io/docs](https://neotoma.io/docs) and in the `docs/` directory.
|
|
214
|
+
|
|
215
|
+
**Foundational:** [Core identity](docs/foundation/core_identity.md), [Philosophy](docs/foundation/philosophy.md), [Problem statement](docs/foundation/problem_statement.md), [Architecture](docs/architecture/architecture.md)
|
|
426
216
|
|
|
427
|
-
|
|
217
|
+
**Developer:** [Getting started](docs/developer/getting_started.md), [CLI reference](docs/developer/cli_reference.md), [MCP overview](docs/developer/mcp_overview.md), [Development workflow](docs/developer/development_workflow.md)
|
|
428
218
|
|
|
429
|
-
|
|
219
|
+
**Specs:** [MCP spec](docs/specs/MCP_SPEC.md), [Schema](docs/subsystems/schema.md), [REST API](docs/api/rest_api.md), [Terminology](https://neotoma.io/terminology)
|
|
220
|
+
|
|
221
|
+
**Operations:** [Runbook](docs/operations/runbook.md), [Health check](docs/operations/health_check.md) (`npm run doctor`), [Troubleshooting](docs/operations/troubleshooting.md)
|
|
222
|
+
|
|
223
|
+
## Contributing
|
|
430
224
|
|
|
431
|
-
MIT
|
|
225
|
+
Neotoma is in active development. For questions or collaboration, open an issue or discussion. See [CONTRIBUTING.md](CONTRIBUTING.md) and [SECURITY.md](SECURITY.md). **License:** MIT
|
package/dist/cli/bootstrap.js
CHANGED
|
File without changes
|
package/dist/cli/index.d.ts.map
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"index.d.ts","sourceRoot":"","sources":["../../src/cli/index.ts"],"names":[],"mappings":";AAqBA,OAAO,EAeL,KAAK,WAAW,EAEjB,MAAM,aAAa,CAAC;AAyDrB,8HAA8H;AAC9H,MAAM,MAAM,iBAAiB,GAAG;IAAE,OAAO,EAAE,MAAM,EAAE,CAAC;IAAC,QAAQ,EAAE;QAAE,OAAO,EAAE,MAAM,CAAA;KAAE,CAAA;CAAE,CAAC;AAIrF,wBAAgB,oBAAoB,CAAC,MAAM,EAAE,MAAM,EAAE,eAAe,SAAmB,GAAG,MAAM,CAE/F;AAED;;;;GAIG;AACH,wBAAgB,+BAA+B,CAAC,uBAAuB,EAAE,MAAM,GAAG,MAAM,CAEvF;AAED;;;GAGG;AACH,wBAAgB,sBAAsB,CAAC,KAAK,EAAE,MAAM,GAAG,MAAM,CAQ5D;AAED,wBAAgB,kBAAkB,CAAC,CAAC,EAAE,MAAM,EAAE,CAAC,EAAE,GAAG,MAAM,GAAG,SAAS,CAErE;AAiaD,wBAAsB,YAAY,CAAC,QAAQ,EAAE,MAAM,GAAG,OAAO,CAAC,MAAM,GAAG,IAAI,CAAC,CAkB3E;
|
|
1
|
+
{"version":3,"file":"index.d.ts","sourceRoot":"","sources":["../../src/cli/index.ts"],"names":[],"mappings":";AAqBA,OAAO,EAeL,KAAK,WAAW,EAEjB,MAAM,aAAa,CAAC;AAyDrB,8HAA8H;AAC9H,MAAM,MAAM,iBAAiB,GAAG;IAAE,OAAO,EAAE,MAAM,EAAE,CAAC;IAAC,QAAQ,EAAE;QAAE,OAAO,EAAE,MAAM,CAAA;KAAE,CAAA;CAAE,CAAC;AAIrF,wBAAgB,oBAAoB,CAAC,MAAM,EAAE,MAAM,EAAE,eAAe,SAAmB,GAAG,MAAM,CAE/F;AAED;;;;GAIG;AACH,wBAAgB,+BAA+B,CAAC,uBAAuB,EAAE,MAAM,GAAG,MAAM,CAEvF;AAED;;;GAGG;AACH,wBAAgB,sBAAsB,CAAC,KAAK,EAAE,MAAM,GAAG,MAAM,CAQ5D;AAED,wBAAgB,kBAAkB,CAAC,CAAC,EAAE,MAAM,EAAE,CAAC,EAAE,GAAG,MAAM,GAAG,SAAS,CAErE;AAiaD,wBAAsB,YAAY,CAAC,QAAQ,EAAE,MAAM,GAAG,OAAO,CAAC,MAAM,GAAG,IAAI,CAAC,CAkB3E;AA8kBD,wBAAgB,aAAa,CAAC,GAAG,EAAE,OAAO,GAAG,IAAI,CAIhD;AA8CD,wBAAgB,sBAAsB,CAAC,MAAM,EAAE;IAC7C,OAAO,EAAE,MAAM,CAAC;IAChB,WAAW,EAAE,MAAM,CAAC;IACpB,KAAK,EAAE,MAAM,CAAC;IACd,aAAa,EAAE,MAAM,CAAC;IACtB,QAAQ,EAAE,MAAM,CAAC;IACjB,OAAO,CAAC,EAAE,OAAO,CAAC;CACnB,GAAG,MAAM,CAWT;AA0oBD,OAAO,EAAE,cAAc,EAAE,MAAM,iBAAiB,CAAC;AAEjD,OAAO,EAAE,cAAc,EAAE,CAAC;AAwB1B,wBAAgB,2BAA2B,CAAC,IAAI,EAAE,MAAM,EAAE,GAAG,SAAgB,GAAG,MAAM,EAAE,CA0CvF;AAitSD,wBAAsB,0BAA0B,CAAC,MAAM,EAAE;IACvD,QAAQ,EAAE,MAAM,GAAG,IAAI,CAAC;IACxB,OAAO,EAAE,MAAM,GAAG,IAAI,CAAC;IACvB,OAAO,CAAC,EAAE,MAAM,CAAC;IACjB,GAAG,CAAC,EAAE,MAAM,CAAC;IACb,iBAAiB,CAAC,EAAE,MAAM,CAAC;CAC5B,GAAG,OAAO,CAAC;IACV,cAAc,EAAE,MAAM,CAAC;IACvB,uBAAuB,EAAE,MAAM,CAAC;IAChC,0BAA0B,EAAE,MAAM,GAAG,IAAI,CAAC;IAC1C,oBAAoB,EAAE,MAAM,GAAG,IAAI,CAAC;CACrC,CAAC,CAmCD;AA4cD,KAAK,UAAU,GAAG;IAChB,cAAc,EAAE,MAAM,CAAC;IACvB,mBAAmB,EAAE,MAAM,CAAC;IAC5B,aAAa,EAAE,MAAM,CAAC;IACtB,YAAY,EAAE,MAAM,CAAC;IACrB,kBAAkB,EAAE,MAAM,CAAC;IAC3B,qBAAqB,EAAE,MAAM,CAAC;CAC/B,CAAC;AAEF,0DAA0D;AAC1D,MAAM,MAAM,iBAAiB,GAAG;IAAE,IAAI,EAAE,UAAU,GAAG,IAAI,CAAC;IAAC,GAAG,EAAE,UAAU,GAAG,IAAI,CAAC;IAAC,OAAO,CAAC,EAAE,MAAM,CAAA;CAAE,CAAC;AA+DtG,wBAAgB,uBAAuB,IAAI,MAAM,EAAE,CAOlD;AAED,wBAAgB,gBAAgB,CAAC,SAAS,EAAE,WAAW,EAAE,EAAE,aAAa,CAAC,EAAE,MAAM,GAAG,MAAM,EAAE,CA6B3F;AAED,KAAK,yBAAyB,GAAG;IAC/B,cAAc,EAAE;QAAE,MAAM,EAAE,OAAO,CAAC;QAAC,MAAM,EAAE,OAAO,CAAC;QAAC,KAAK,EAAE,OAAO,CAAA;KAAE,CAAC;IACrE,WAAW,EAAE;QAAE,MAAM,EAAE,OAAO,CAAC;QAAC,MAAM,EAAE,OAAO,CAAC;QAAC,KAAK,EAAE,OAAO,CAAA;KAAE,CAAC;CACnE,CAAC;AA8DF,MAAM,MAAM,iBAAiB,GAAG;IAC9B,aAAa,EAAE,OAAO,CAAC;IACvB,aAAa,EAAE,OAAO,CAAC;IACvB,OAAO,EAAE,MAAM,CAAC;IAChB,WAAW,EAAE,MAAM,CAAC;IACpB,SAAS,EAAE,SAAS,GAAG,MAAM,CAAC;CAC/B,CAAC;AAEF,kFAAkF;AAClF,wBAAsB,oBAAoB,CACxC,QAAQ,EAAE,MAAM,GAAG,IAAI,GACtB,OAAO,CAAC,iBAAiB,GAAG,IAAI,CAAC,CAuCnC;AAED;;;GAGG;AACH,wBAAsB,wBAAwB,CAAC,QAAQ,EAAE,MAAM,GAAG,IAAI,GAAG,OAAO,CAAC,OAAO,CAAC,CAOxF;AAED,wBAAgB,yBAAyB,CACvC,UAAU,EAAE,eAAe,EAAE,EAC7B,OAAO,EAAE,yBAAyB,GAAG,IAAI,EACzC,WAAW,CAAC,EAAE,iBAAiB,GAAG,IAAI,EACtC,QAAQ,CAAC,EAAE,MAAM,GAAG,IAAI,GACvB,MAAM,EAAE,CAqGV;AAsGD,KAAK,eAAe,GAAG;IAAE,KAAK,EAAE,MAAM,EAAE,CAAC;IAAC,KAAK,EAAE,MAAM,CAAA;CAAE,CAAC;AAU1D,wBAAgB,oBAAoB,CAClC,OAAO,EAAE,MAAM,EACf,mBAAmB,EAAE,MAAM,EAAE,GAAG,iBAAiB,EACjD,IAAI,CAAC,EAAE,KAAK,GAAG,MAAM,GACpB,eAAe,CA0DjB;AAED,KAAK,eAAe,GAAG;IAAE,IAAI,EAAE,MAAM,CAAC;IAAC,MAAM,EAAE,OAAO,CAAC;IAAC,OAAO,EAAE,OAAO,CAAA;CAAE,CAAC;AAS3E;;;GAGG;AACH,wBAAgB,sBAAsB,CACpC,IAAI,EAAE;IACJ,YAAY,EAAE,eAAe,CAAC;IAC9B,QAAQ,CAAC,EAAE,MAAM,CAAC;IAClB,QAAQ,CAAC,EAAE,MAAM,EAAE,CAAC;IACpB,UAAU,CAAC,EAAE,MAAM,EAAE,CAAC;IACtB,eAAe,CAAC,EAAE,MAAM,CAAC;IACzB,iBAAiB,CAAC,EAAE,MAAM,EAAE,CAAC;IAC7B,eAAe,CAAC,EAAE,MAAM,EAAE,CAAC;IAC3B,iBAAiB,CAAC,EAAE,OAAO,CAAC;CAC7B,EACD,eAAe,EAAE,MAAM,GACtB;IAAE,MAAM,EAAE,MAAM,CAAC;IAAC,SAAS,EAAE,MAAM,CAAA;CAAE,CAyEvC;AAgED,kFAAkF;AAClF,wBAAgB,sBAAsB,IAAI,MAAM,EAAE,CAEjD;AA8FD,wBAAsB,MAAM,CAAC,IAAI,GAAE,MAAM,EAAiB,GAAG,OAAO,CAAC,IAAI,CAAC,CA6kCzE"}
|