datanotes 2.0.0 → 3.0.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (181) hide show
  1. package/NOTICE +9 -0
  2. package/README.md +207 -28
  3. package/SECURITY.md +92 -45
  4. package/dist/THIRD-PARTY-NOTICES.txt +682 -0
  5. package/dist/api/catalog.d.cts +69 -0
  6. package/dist/api/catalog.d.ts +50 -2
  7. package/dist/api/cli-options.d.cts +39 -0
  8. package/dist/api/cli-options.d.ts +39 -0
  9. package/dist/api/client.d.cts +89 -0
  10. package/dist/api/client.d.ts +19 -30
  11. package/dist/api/docs.d.cts +211 -0
  12. package/dist/api/docs.d.ts +211 -0
  13. package/dist/api/guidance.d.cts +81 -0
  14. package/dist/api/guidance.d.ts +81 -0
  15. package/dist/api/index.cjs +6610 -2247
  16. package/dist/api/index.d.cts +13 -0
  17. package/dist/api/index.d.ts +10 -7
  18. package/dist/api/index.js +6610 -2247
  19. package/dist/api/local.cjs +118 -6
  20. package/dist/api/local.d.cts +95 -0
  21. package/dist/api/local.d.ts +70 -6
  22. package/dist/api/local.js +104 -6
  23. package/dist/api/mcp.cjs +17384 -0
  24. package/dist/api/mcp.d.cts +108 -0
  25. package/dist/api/mcp.d.ts +108 -0
  26. package/dist/api/mcp.js +17334 -0
  27. package/dist/api/otlp.d.cts +65 -0
  28. package/dist/api/redact.d.cts +17 -0
  29. package/dist/api/redact.d.ts +17 -0
  30. package/dist/api/requests.d.cts +26 -0
  31. package/dist/api/requests.d.ts +14 -16
  32. package/dist/api/results.d.cts +16 -0
  33. package/dist/api/results.d.ts +2 -1
  34. package/dist/api/tokens.d.cts +7 -0
  35. package/dist/datanotes.cjs +62911 -28409
  36. package/dist/docs/administration/configuration.md +59 -0
  37. package/dist/docs/administration/docker.md +80 -0
  38. package/dist/docs/administration/git-sync.md +71 -0
  39. package/dist/docs/administration/limits.md +69 -0
  40. package/dist/docs/administration/run-the-service.md +219 -0
  41. package/dist/docs/administration/status.md +61 -0
  42. package/dist/docs/administration/use-with-obsidian.md +75 -0
  43. package/dist/docs/api/operations/add_column.md +49 -0
  44. package/dist/docs/api/operations/add_constraint.md +55 -0
  45. package/dist/docs/api/operations/adopt_rows.md +50 -0
  46. package/dist/docs/api/operations/align_relations.md +41 -0
  47. package/dist/docs/api/operations/alter_column.md +46 -0
  48. package/dist/docs/api/operations/alter_table.md +67 -0
  49. package/dist/docs/api/operations/batch.md +48 -0
  50. package/dist/docs/api/operations/changes_stream.md +28 -0
  51. package/dist/docs/api/operations/check_table.md +29 -0
  52. package/dist/docs/api/operations/clone_table.md +48 -0
  53. package/dist/docs/api/operations/comment_on.md +42 -0
  54. package/dist/docs/api/operations/configure_database.md +52 -0
  55. package/dist/docs/api/operations/create_table.md +78 -0
  56. package/dist/docs/api/operations/create_view.md +54 -0
  57. package/dist/docs/api/operations/create_webhook.md +37 -0
  58. package/dist/docs/api/operations/define_base.md +52 -0
  59. package/dist/docs/api/operations/delete_rows.md +43 -0
  60. package/dist/docs/api/operations/delete_webhook.md +31 -0
  61. package/dist/docs/api/operations/describe_database.md +30 -0
  62. package/dist/docs/api/operations/describe_history.md +40 -0
  63. package/dist/docs/api/operations/describe_semantic_index.md +24 -0
  64. package/dist/docs/api/operations/describe_table.md +38 -0
  65. package/dist/docs/api/operations/docs.md +13 -0
  66. package/dist/docs/api/operations/drop_column.md +41 -0
  67. package/dist/docs/api/operations/drop_constraint.md +39 -0
  68. package/dist/docs/api/operations/drop_table.md +43 -0
  69. package/dist/docs/api/operations/drop_view.md +38 -0
  70. package/dist/docs/api/operations/engine_status.md +23 -0
  71. package/dist/docs/api/operations/graph_neighbors.md +32 -0
  72. package/dist/docs/api/operations/graph_path.md +32 -0
  73. package/dist/docs/api/operations/graph_traverse.md +43 -0
  74. package/dist/docs/api/operations/identify_session.md +24 -0
  75. package/dist/docs/api/operations/infer_table.md +43 -0
  76. package/dist/docs/api/operations/insert_row.md +48 -0
  77. package/dist/docs/api/operations/list_changes.md +35 -0
  78. package/dist/docs/api/operations/list_folder.md +28 -0
  79. package/dist/docs/api/operations/list_link_targets.md +44 -0
  80. package/dist/docs/api/operations/list_migrations.md +35 -0
  81. package/dist/docs/api/operations/list_tables.md +32 -0
  82. package/dist/docs/api/operations/list_webhooks.md +23 -0
  83. package/dist/docs/api/operations/manual_index.md +14 -0
  84. package/dist/docs/api/operations/manual_page.md +22 -0
  85. package/dist/docs/api/operations/merge_rows.md +44 -0
  86. package/dist/docs/api/operations/move_rows.md +52 -0
  87. package/dist/docs/api/operations/openapi.md +13 -0
  88. package/dist/docs/api/operations/openapi_post.md +20 -0
  89. package/dist/docs/api/operations/read_note.md +29 -0
  90. package/dist/docs/api/operations/recompute_columns.md +44 -0
  91. package/dist/docs/api/operations/record_tool_call.md +36 -0
  92. package/dist/docs/api/operations/rename_column.md +44 -0
  93. package/dist/docs/api/operations/rename_value.md +45 -0
  94. package/dist/docs/api/operations/restore_database.md +51 -0
  95. package/dist/docs/api/operations/restore_table.md +46 -0
  96. package/dist/docs/api/operations/search_notes.md +28 -0
  97. package/dist/docs/api/operations/select_rows.md +61 -0
  98. package/dist/docs/api/operations/semantic_search.md +34 -0
  99. package/dist/docs/api/operations/semantic_search_docs.md +26 -0
  100. package/dist/docs/api/operations/set_table_bases.md +44 -0
  101. package/dist/docs/api/operations/status.md +13 -0
  102. package/dist/docs/api/operations/stop_engine.md +23 -0
  103. package/dist/docs/api/operations/sync_semantic_index.md +33 -0
  104. package/dist/docs/api/operations/table_changes.md +31 -0
  105. package/dist/docs/api/operations/tags/auxiliary.md +14 -0
  106. package/dist/docs/api/operations/tags/change-feed.md +8 -0
  107. package/dist/docs/api/operations/tags/history.md +17 -0
  108. package/dist/docs/api/operations/tags/queries.md +15 -0
  109. package/dist/docs/api/operations/tags/rows-dml.md +16 -0
  110. package/dist/docs/api/operations/tags/schema-ddl.md +22 -0
  111. package/dist/docs/api/operations/tags/service.md +15 -0
  112. package/dist/docs/api/operations/tags/tokens.md +13 -0
  113. package/dist/docs/api/operations/tokens_create.md +34 -0
  114. package/dist/docs/api/operations/tokens_delete.md +29 -0
  115. package/dist/docs/api/operations/tokens_list.md +21 -0
  116. package/dist/docs/api/operations/tokens_me.md +20 -0
  117. package/dist/docs/api/operations/tokens_rotate.md +29 -0
  118. package/dist/docs/api/operations/tokens_update.md +34 -0
  119. package/dist/docs/api/operations/undo_migration.md +46 -0
  120. package/dist/docs/api/operations/update_row.md +60 -0
  121. package/dist/docs/api/operations/update_rows.md +46 -0
  122. package/dist/docs/api/operations/vacuum_table.md +37 -0
  123. package/dist/docs/api.md +135 -0
  124. package/dist/docs/develop/build-an-app.md +320 -0
  125. package/dist/docs/develop/build-on-datanotes.md +138 -0
  126. package/dist/docs/develop/change-feed-and-webhooks.md +70 -0
  127. package/dist/docs/develop/client.md +116 -0
  128. package/dist/docs/develop/concurrency.md +153 -0
  129. package/dist/docs/develop/http-api.md +79 -0
  130. package/dist/docs/develop/telemetry.md +86 -0
  131. package/dist/docs/guides/agents/docs-offline.md +99 -0
  132. package/dist/docs/guides/agents/how-agents-use-datanotes.md +124 -0
  133. package/dist/docs/guides/agents/prompts.md +495 -0
  134. package/dist/docs/guides/agents/reading-errors.md +112 -0
  135. package/dist/docs/guides/data-quality/check-constraints.md +379 -0
  136. package/dist/docs/guides/data-quality/check-table.md +63 -0
  137. package/dist/docs/guides/data-quality/hand-edits.md +507 -0
  138. package/dist/docs/guides/data-quality/sync-between-devices.md +51 -0
  139. package/dist/docs/guides/data-quality/validation.md +90 -0
  140. package/dist/docs/guides/database-objects/bases.md +480 -0
  141. package/dist/docs/guides/database-objects/catalogs-and-schemas.md +897 -0
  142. package/dist/docs/guides/database-objects/columns.md +1332 -0
  143. package/dist/docs/guides/database-objects/relations.md +1454 -0
  144. package/dist/docs/guides/database-objects/tables.md +1404 -0
  145. package/dist/docs/guides/database-objects/views.md +1003 -0
  146. package/dist/docs/guides/history/back-up.md +195 -0
  147. package/dist/docs/guides/history/change-data-feed.md +247 -0
  148. package/dist/docs/guides/history/journal-and-undo.md +262 -0
  149. package/dist/docs/guides/history/restore.md +341 -0
  150. package/dist/docs/guides/history/table-history.md +657 -0
  151. package/dist/docs/guides/history/vacuum.md +35 -0
  152. package/dist/docs/guides/read-and-write/automatic-links.md +768 -0
  153. package/dist/docs/guides/read-and-write/batch.md +139 -0
  154. package/dist/docs/guides/read-and-write/import-notes.md +896 -0
  155. package/dist/docs/guides/read-and-write/joins-and-aggregates.md +251 -0
  156. package/dist/docs/guides/read-and-write/read-and-filter.md +536 -0
  157. package/dist/docs/guides/read-and-write/write-rows.md +573 -0
  158. package/dist/docs/guides/search/link-graph.md +146 -0
  159. package/dist/docs/guides/search/semantic.md +133 -0
  160. package/dist/docs/guides/search/text.md +142 -0
  161. package/dist/docs/guides/security/security-model.md +110 -0
  162. package/dist/docs/guides/security/tokens.md +60 -0
  163. package/dist/docs/guides/security/who-did-what.md +42 -0
  164. package/dist/docs/index.md +113 -0
  165. package/dist/docs/llms.txt +33 -0
  166. package/dist/docs/reference/cli.md +136 -0
  167. package/dist/docs/reference/compatibility.md +63 -0
  168. package/dist/docs/reference/errors.md +72 -0
  169. package/dist/docs/reference/glossary.md +161 -0
  170. package/dist/docs/reference/operations/auxiliary.md +116 -0
  171. package/dist/docs/reference/operations/history.md +168 -0
  172. package/dist/docs/reference/operations/queries.md +169 -0
  173. package/dist/docs/reference/operations/rows.md +185 -0
  174. package/dist/docs/reference/operations/schema.md +316 -0
  175. package/dist/docs/reference/operations.md +96 -0
  176. package/dist/docs/reference/settings.md +31 -0
  177. package/dist/docs/release-notes.md +67 -0
  178. package/dist/docs/start/connect-agents.md +204 -0
  179. package/dist/docs/start/how-it-works.md +317 -0
  180. package/dist/docs/start/quick-start.md +126 -0
  181. package/package.json +59 -14
@@ -0,0 +1,146 @@
1
+ # Walk the link graph
2
+
3
+ > Read the links between notes as a graph: neighbors, walks filtered by label and table, and the shortest path between two notes.
4
+
5
+ The engine reads your notes and their links as a labelled graph. Edges from link columns have the column’s name as their label, such as `cast` or `genre`; other resolved links have the label `link`, and each row has its table as its label.
6
+
7
+ The vault of the examples has three actors, two classic films, a viewing and two plain notes.
8
+
9
+ ## See what a note links to and what links to it
10
+
11
+ `graph_neighbors` returns a note’s links in both directions. Each link has a label: the link column it comes from (`cast`), or `link` for a link in the text.
12
+
13
+ **`graph_neighbors`**
14
+
15
+ ```json
16
+ { "from_nodes": { "table": "actor", "where": { "name": "Cary Grant" } } }
17
+ ```
18
+
19
+ Answer:
20
+
21
+ ```json
22
+ {
23
+ "nodes": [
24
+ {
25
+ "path": "datanotes/main/public/actor/Cary Grant.md",
26
+ "table": "main.public.actor",
27
+ "counts": { "in:cast": 2, "in:link": 1 },
28
+ "edges": [
29
+ {
30
+ "path": "datanotes/main/public/film/Charade.md",
31
+ "table": "main.public.film",
32
+ "label": "cast",
33
+ "direction": "in"
34
+ },
35
+ {
36
+ "path": "datanotes/main/public/film/His Girl Friday.md",
37
+ "table": "main.public.film",
38
+ "label": "cast",
39
+ "direction": "in"
40
+ },
41
+ { "path": "Notes/Film night.md", "label": "link", "direction": "in" }
42
+ ]
43
+ }
44
+ ]
45
+ }
46
+ ```
47
+
48
+ ## Walk the links from a note
49
+
50
+ `graph_traverse` follows links breadth-first, in the `direction` you choose: `out`, `in` or `both`. Here it starts from Audrey Hepburn, goes up to two steps in both directions and keeps only films. It finds Charade, the film she’s in, and His Girl Friday, which the Charade note links to. `via` shows the route to each one.
51
+
52
+ **`graph_traverse`**
53
+
54
+ ```json
55
+ {
56
+ "from_nodes": { "paths": ["datanotes/main/public/actor/Audrey Hepburn.md"] },
57
+ "direction": "both",
58
+ "tables": ["film"],
59
+ "max_depth": 2
60
+ }
61
+ ```
62
+
63
+ Answer:
64
+
65
+ ```json
66
+ {
67
+ "start": ["datanotes/main/public/actor/Audrey Hepburn.md"],
68
+ "nodes": [
69
+ {
70
+ "path": "datanotes/main/public/film/Charade.md",
71
+ "table": "main.public.film",
72
+ "depth": 1,
73
+ "via": [
74
+ {
75
+ "from": "datanotes/main/public/actor/Audrey Hepburn.md",
76
+ "label": "cast",
77
+ "direction": "in",
78
+ "to": "datanotes/main/public/film/Charade.md"
79
+ }
80
+ ]
81
+ },
82
+ {
83
+ "path": "datanotes/main/public/film/His Girl Friday.md",
84
+ "table": "main.public.film",
85
+ "depth": 2,
86
+ "via": [
87
+ {
88
+ "from": "datanotes/main/public/actor/Audrey Hepburn.md",
89
+ "label": "cast",
90
+ "direction": "in",
91
+ "to": "datanotes/main/public/film/Charade.md"
92
+ },
93
+ {
94
+ "from": "datanotes/main/public/film/Charade.md",
95
+ "label": "link",
96
+ "direction": "out",
97
+ "to": "datanotes/main/public/film/His Girl Friday.md"
98
+ }
99
+ ]
100
+ }
101
+ ],
102
+ "truncated": false
103
+ }
104
+ ```
105
+
106
+ To read or total those rows, pass the paths to `select_rows` with `where: {"path": {"in": [...]}}`.
107
+
108
+ ## Find how two notes are connected
109
+
110
+ `graph_path` finds the shortest chain of links between two notes, or between two sets of notes.
111
+
112
+ **`graph_path`**
113
+
114
+ ```json
115
+ {
116
+ "from_nodes": {
117
+ "paths": ["Notes/Rewatch ideas.md", "datanotes/main/public/actor/Audrey Hepburn.md"]
118
+ },
119
+ "to_nodes": { "paths": ["Notes/Film night.md"] }
120
+ }
121
+ ```
122
+
123
+ Answer:
124
+
125
+ ```json
126
+ {
127
+ "found": true,
128
+ "length": 2,
129
+ "via": [
130
+ {
131
+ "from": "datanotes/main/public/actor/Audrey Hepburn.md",
132
+ "label": "cast",
133
+ "direction": "in",
134
+ "to": "datanotes/main/public/film/Charade.md"
135
+ },
136
+ {
137
+ "from": "datanotes/main/public/film/Charade.md",
138
+ "label": "link",
139
+ "direction": "in",
140
+ "to": "Notes/Film night.md"
141
+ }
142
+ ],
143
+ "from": "datanotes/main/public/actor/Audrey Hepburn.md",
144
+ "to": "Notes/Film night.md"
145
+ }
146
+ ```
@@ -0,0 +1,133 @@
1
+ # Find notes by meaning
2
+
3
+ > Turn on semantic search with a local or hosted embedding model, and search your notes by what they say.
4
+
5
+ Semantic search finds notes by meaning instead of exact words. The engine splits each note into passages, sends them to an embedding model you choose and stores the resulting vectors in the vault. Agents and programs then search with the [`semantic_search`](https://datanotes.dev/reference/operations/queries/#semantic_search) operation. It’s off by default, because note text goes to the model’s provider.
6
+
7
+ ## Turn on semantic search
8
+
9
+ Turn it on in the engine’s settings file, `settings.json` in its store (`datanotes store path` shows where):
10
+
11
+ 1. Stop the engine.
12
+ 2. Pick a provider and a model from the [embedding providers](#choose-an-embedding-provider) table.
13
+ 3. Set `semanticEnabled`, `semanticProvider` and `semanticModel` in the settings file.
14
+ 4. If the provider needs an API key, [pass the key to the engine](#pass-the-api-key-to-the-engine).
15
+ 5. Start the engine.
16
+
17
+ The settings below use Ollama, which keeps your notes on your computer:
18
+
19
+ ```json
20
+ {
21
+ "semanticEnabled": true,
22
+ "semanticProvider": "ollama",
23
+ "semanticModel": "nomic-embed-text"
24
+ }
25
+ ```
26
+
27
+ Leave `semanticBaseUrl` empty to use the provider’s default address. The [settings reference](https://datanotes.dev/reference/settings/) lists every key and its default.
28
+
29
+ ## Choose an embedding provider
30
+
31
+ The provider decides where your note text goes. Local providers keep it on your computer; hosted ones send it over the network:
32
+
33
+ | `semanticProvider` | Where notes go | Default address | Example model |
34
+ | ------------------ | ------------------------------------------------------ | ------------------------------ | ------------------------------------------------------------ |
35
+ | `ollama` | Stays on this computer | `http://127.0.0.1:11434` | `nomic-embed-text`, `qwen3-embedding:0.6b` |
36
+ | `lmstudio` | Stays on this computer | `http://127.0.0.1:1234/v1` | `text-embedding-nomic-embed-text-v1.5` |
37
+ | `openrouter` | OpenRouter and the model’s provider (needs an API key) | `https://openrouter.ai/api/v1` | `openai/text-embedding-3-small`, `qwen/qwen3-embedding-0.6b` |
38
+ | `openai` | OpenAI (needs an API key) | `https://api.openai.com/v1` | `text-embedding-3-small` |
39
+ | `custom` | Any OpenAI-compatible `/embeddings` API | None: set `semanticBaseUrl` | The model your API serves |
40
+
41
+ ## Pass the API key to the engine
42
+
43
+ The engine never stores API keys in the settings file. Set the `DATANOTES_EMBEDDING_KEY` environment variable before you start the engine, or pass a file that holds the key:
44
+
45
+ ```bash
46
+ npx -y datanotes serve --database your_database_folder \
47
+ --embedding-key-file your_key_file
48
+ ```
49
+
50
+ ## Choose which notes to index
51
+
52
+ By default the engine indexes every note in the vault. Two settings narrow the scope:
53
+
54
+ * **`semanticScope`**: `"vault"` (the default) indexes every note; `"tables"` indexes only table rows
55
+ * **`semanticExclude`**: folders to leave out
56
+
57
+ The engine always skips `_archive` folders.
58
+
59
+ ## How the engine builds the index
60
+
61
+ The engine splits each note into passages of at most about 1,500 characters. The splitting follows these rules:
62
+
63
+ * **First passage**: holds the title and the front matter, up to 800 characters, without the keys `id` and `table`
64
+ * **Heading sections**: each section starts a new passage
65
+ * **Long sections**: the engine splits them at paragraph boundaries
66
+ * **Limit**: 40 passages per note
67
+
68
+ The engine sends the passages to the model and stores the vectors as int8 values in `semantic/`, in its store. It updates the index 10s after edits stop, and 5s after it starts. It embeds again only the notes whose content changed. If you change the provider, the address or the model, the engine starts a new index.
69
+
70
+ ## Search from agents and programs
71
+
72
+ Agents and programs use three operations. All three work through the [HTTP API](https://datanotes.dev/develop/http-api/), the [JavaScript client](https://datanotes.dev/develop/client/) and the [Model Context Protocol (MCP) server](https://datanotes.dev/start/connect-agents/):
73
+
74
+ * **[`semantic_search`](https://datanotes.dev/reference/operations/queries/#semantic_search)**: takes a `query`, with optional `tables`, `folder`, `min_score` and `limit`; returns one result per note, its best passage, best first
75
+ * **[`describe_semantic_index`](https://datanotes.dev/reference/operations/auxiliary/#describe_semantic_index)**: shows the index status
76
+ * **[`sync_semantic_index`](https://datanotes.dev/reference/operations/auxiliary/#sync_semantic_index)**: brings the index up to date, or rebuilds it with `rebuild: true`
77
+
78
+ This example asks for the five notes closest in meaning to a question. `db` is a [client](https://datanotes.dev/develop/client/) connected to the engine:
79
+
80
+ ```ts
81
+ const r = await db.call("semantic_search", {
82
+ query: "films that scare without a single spoken word",
83
+ limit: 5,
84
+ });
85
+ ```
86
+
87
+ ## Search these docs by meaning
88
+
89
+ With semantic search on, the engine also embeds the sections of these docs, in an index of their own. `search_docs` then merges its ranking by words with the engine’s ranking by meaning. [Search by words and by meaning](https://datanotes.dev/guides/agents/docs-offline/#search-by-words-and-by-meaning) explains how.
90
+
91
+ ## Search notes by meaning
92
+
93
+ `semantic_search` finds notes by meaning rather than by words. It’s off by default, because it sends note text to an embedding provider. Until you turn it on in the engine’s settings, the call answers that it’s off; [Turn on semantic search](https://datanotes.dev/guides/search/semantic/#turn-on-semantic-search) explains how. `describe_semantic_index` reports the state of the index, and `sync_semantic_index` brings it up to date once it’s on.
94
+
95
+ **`semantic_search`**
96
+
97
+ ```json
98
+ { "query": "the Friday we watched the Cary Grant film" }
99
+ ```
100
+
101
+ Refused:
102
+
103
+ ```text
104
+ semantic search is off: set semanticEnabled, semanticProvider and semanticModel in the engine settings
105
+ Hint: Check the arguments of semantic_search: query (string, required), limit (integer), tables (list of strings), folder (string), min_score (number).
106
+ Docs: https://datanotes.dev/reference/operations/queries.md#semantic_search (offline: read_docs("reference/operations/queries#semantic_search"))
107
+ ```
108
+
109
+ **`describe_semantic_index`** (no arguments)
110
+
111
+ Answer:
112
+
113
+ ```json
114
+ {
115
+ "enabled": false,
116
+ "provider": "ollama",
117
+ "model": "",
118
+ "files": 0,
119
+ "passages": 0,
120
+ "pending": 6,
121
+ "running": false
122
+ }
123
+ ```
124
+
125
+ **`sync_semantic_index`** (no arguments)
126
+
127
+ Refused:
128
+
129
+ ```text
130
+ semantic search is off: set semanticEnabled, semanticProvider and semanticModel in the engine settings
131
+ Hint: Check the arguments of sync_semantic_index: rebuild (boolean).
132
+ Docs: https://datanotes.dev/reference/operations/auxiliary.md#sync_semantic_index (offline: read_docs("reference/operations/auxiliary#sync_semantic_index"))
133
+ ```
@@ -0,0 +1,142 @@
1
+ # Read and search notes
2
+
3
+ > Read any note in the vault, list folders and search their text.
4
+
5
+ Not every note is a row, so the operations on this page work on the whole vault: plain notes and rows alike. The vault has three actors, two classic films, a viewing and two plain notes.
6
+
7
+ ## Read a note
8
+
9
+ `read_note` returns a note’s Markdown in `content`, with its front matter, tags, file dates and size, parsed.
10
+
11
+ **`read_note`**
12
+
13
+ ```json
14
+ { "path": "Notes/Film night.md" }
15
+ ```
16
+
17
+ Answer:
18
+
19
+ ```json
20
+ {
21
+ "path": "Notes/Film night.md",
22
+ "content": "---\ntags: [films]\n---\n# Film night\n\nEvery other Friday. Next up: [[Charade]]. October’s theme is [[Cary Grant]].\n",
23
+ "front_matter": { "tags": ["films"] },
24
+ "tags": ["films"],
25
+ "modified_at": "<date-time>",
26
+ "created_at": "<date-time>",
27
+ "size": 115
28
+ }
29
+ ```
30
+
31
+ ## List a folder
32
+
33
+ `list_folder` lists a folder’s sub-folders and files, as vault paths, and `""` is the vault root. The listing hides the files and folders whose names start with a dot, such as `.obsidian`.
34
+
35
+ **`list_folder`**
36
+
37
+ ```json
38
+ { "folder": "" }
39
+ ```
40
+
41
+ Answer:
42
+
43
+ ```json
44
+ { "folder": "", "folders": ["Notes", "datanotes"], "files": [] }
45
+ ```
46
+
47
+ **`list_folder`**
48
+
49
+ ```json
50
+ { "folder": "datanotes/main/public" }
51
+ ```
52
+
53
+ Answer:
54
+
55
+ ```json
56
+ {
57
+ "folder": "datanotes/main/public",
58
+ "folders": [
59
+ "datanotes/main/public/actor",
60
+ "datanotes/main/public/film",
61
+ "datanotes/main/public/watch"
62
+ ],
63
+ "files": []
64
+ }
65
+ ```
66
+
67
+ ## Search text
68
+
69
+ `search_notes` finds notes whose path or text contains every word of the query, ignoring case and accents. It returns the matching passages.
70
+
71
+ **`search_notes`**
72
+
73
+ ```json
74
+ { "query": "grant", "context_length": 30 }
75
+ ```
76
+
77
+ Answer (62 lines)
78
+
79
+ ```json
80
+ {
81
+ "query": "grant",
82
+ "results": [
83
+ {
84
+ "path": "datanotes/main/public/actor/Cary Grant.md",
85
+ "score": 6,
86
+ "matches": [
87
+ {
88
+ "start": 81,
89
+ "end": 86,
90
+ "context": "-bd1b-d5bf80031dd6\nname: Cary Grant\n---\n\n"
91
+ }
92
+ ]
93
+ },
94
+ {
95
+ "path": "datanotes/main/public/film/Charade.md",
96
+ "score": 2,
97
+ "matches": [
98
+ {
99
+ "start": 102,
100
+ "end": 107,
101
+ "context": "le: Charade\ncast:\n - \"[[Cary Grant]]\"\n - \"[[Audrey Hepburn]]\"\n-"
102
+ },
103
+ {
104
+ "start": 147,
105
+ "end": 152,
106
+ "context": "[Audrey Hepburn]]\"\n---\n[[Cary Grant]] again, 23 years after [[His"
107
+ }
108
+ ]
109
+ },
110
+ {
111
+ "path": "datanotes/main/public/film/His Girl Friday.md",
112
+ "score": 1,
113
+ "matches": [
114
+ {
115
+ "start": 110,
116
+ "end": 115,
117
+ "context": "Girl Friday\ncast:\n - \"[[Cary Grant]]\"\n - \"[[Rosalind Russell]]\""
118
+ }
119
+ ]
120
+ },
121
+ {
122
+ "path": "Notes/Film night.md",
123
+ "score": 1,
124
+ "matches": [
125
+ {
126
+ "start": 104,
127
+ "end": 109,
128
+ "context": "]]. October’s theme is [[Cary Grant]].\n"
129
+ }
130
+ ]
131
+ },
132
+ {
133
+ "path": "Notes/Rewatch ideas.md",
134
+ "score": 1,
135
+ "matches": [
136
+ { "start": 7, "end": 12, "context": "A Cary Grant weekend: every film with him," }
137
+ ]
138
+ }
139
+ ],
140
+ "total": 5
141
+ }
142
+ ```
@@ -0,0 +1,110 @@
1
+ # Security model
2
+
3
+ > Security model, credentials, and how to report a vulnerability.
4
+
5
+ Datanotes is an engine that reads and writes a folder of Markdown notes, the vault. It runs on your own machine, for you and the agents you choose. This page explains:
6
+
7
+ * How to report a vulnerability
8
+ * Which versions get security fixes
9
+ * What the engine exposes on the network
10
+ * What each token can do
11
+ * How the engine guards your files and its store
12
+ * How to limit what an agent can do
13
+ * Which features send data off your machine
14
+
15
+ ## Report a vulnerability
16
+
17
+ Report vulnerabilities only through GitHub Private Vulnerability Reporting. Open a [new private advisory](https://github.com/davidefornelli/datanotes/security/advisories/new) on the repository. It’s the one channel for security reports. Don’t report them in public issues or discussions. You’ll get an answer within a week.
18
+
19
+ Include these details in your report:
20
+
21
+ * The Datanotes version
22
+ * How to reproduce the problem
23
+ * What an attacker gains
24
+
25
+ ## Supported versions
26
+
27
+ Security fixes go to the latest release of Datanotes 3.
28
+
29
+ | Version | Supported |
30
+ | ------- | -------------------- |
31
+ | 3.x | Yes |
32
+ | 2.x | No: it was a preview |
33
+ | 1.x | No: it was a preview |
34
+
35
+ Update to the latest 3.x release to get a fix.
36
+
37
+ ## What the engine exposes on the network
38
+
39
+ The engine’s HTTP server listens on `127.0.0.1`, so only your machine can reach it, unless you start it with `--host`. It answers no Cross-Origin Resource Sharing (CORS) preflight, so web pages can’t call it.
40
+
41
+ Three routes answer without a token:
42
+
43
+ * **`GET /`**: the status and the version
44
+ * **`GET /datanotes/openapi.json`**: the API’s description
45
+ * **`/datanotes/docs`**: an interactive console over that description, with its script and stylesheet
46
+
47
+ These routes contain no data from your vault: the website publishes the same description. The console’s page loads nothing from other hosts, because its Content Security Policy (CSP) allows the same origin only. Its calls need a token, which the browser keeps in local storage until you log out.
48
+
49
+ To reach the engine from another device, such as a phone or a hosted assistant, publish it through a reverse proxy or a tunnel, over HTTPS. Give each device or agent its own named token. Publish the Model Context Protocol (MCP) server, `datanotes mcp serve`, the same way: it holds no credentials and passes each agent’s token on to the engine.
50
+
51
+ ## Credentials and what they can do
52
+
53
+ The engine accepts two kinds of token, and the owner token can do everything a named token can. Prefer a named token per device, program or agent: it has only the permissions it needs, and the journal names it. MCP clients use the same tokens, because the MCP server passes them to the engine.
54
+
55
+ ### Owner token
56
+
57
+ The owner token is `httpServerToken` in the settings file. It can do everything: read and write data, change the schema, manage tokens and webhooks, undo, and read the engine’s status. `datanotes status` reads it from the settings file. Keep it on the machine.
58
+
59
+ ### Named token
60
+
61
+ The owner creates a named token with `POST /datanotes/tokens/create/`, and the engine stores only its hash, in `tokens.json` in its store, outside the vault. A named token has these permissions:
62
+
63
+ * **Read**: everything
64
+ * **Write rows**: only the tables in its `write` patterns
65
+ * **Change the schema**: only with `ddl`
66
+ * **Maintain**: with `maintain`, it may also undo, restore, vacuum, configure the database and rebuild the semantic index; use it for an agent’s own token
67
+
68
+ A named token can’t create webhooks, manage tokens or read the engine’s status, and without `maintain` it can’t undo.
69
+
70
+ ### Who made each write
71
+
72
+ The engine journals every write with its actor. The client sends the agent’s name, model and session, so they’re self-reported. The server sets the credential, `auth` (`owner`, or the named token’s name), and no client can claim another one.
73
+
74
+ ## File access inside the vault
75
+
76
+ The engine resolves every path inside the vault. Both its operations and its file host refuse a path that contains `..`, a backslash, a drive letter or a NUL character.
77
+
78
+ The engine follows symbolic links inside the vault. Don’t link to folders you wouldn’t give the engine.
79
+
80
+ ## A private store
81
+
82
+ The engine’s store holds its secrets: the owner token in `settings.json` and the webhooks’ secrets in `webhooks.json`. It also holds the journal and the hashes of the named tokens. On Linux and macOS, the engine keeps the store private to the user that runs it:
83
+
84
+ * **Folders**: the data home and each store are created with mode `0700`
85
+ * **Files**: `settings.json`, `webhooks.json` and the other files of the store are created with mode `0600`
86
+ * **At start**: the engine tightens a data home, a store, a settings file or a `webhooks.json` that an older version left readable by others
87
+
88
+ On Windows, the store lives in your user profile, which other users can’t read. The Docker image runs the engine with `umask 077`, so every file it creates is private too.
89
+
90
+ ## Agents and prompt injection
91
+
92
+ An agent reads your notes, and a note can hold text written to manipulate it: a prompt injection. A web page you clipped or an email you saved can tell the agent to delete rows or to send your data out. The engine can’t tell such text from your own notes.
93
+
94
+ Limit what each agent can do:
95
+
96
+ * **Give it a named token**: grant only the tables it writes, and leave out `ddl` and `maintain` unless it needs them
97
+ * **Give it only the tools it needs**: start the MCP server with `datanotes mcp --profile`, `--tools` or `--except`
98
+ * **Keep the webhook tools off**: they’re off by default, because a webhook sends change events to any URL
99
+
100
+ Every write is journaled with the agent’s name, so you can review it and undo it.
101
+
102
+ ## Data that leaves your machine
103
+
104
+ Nothing leaves your machine unless you turn it on. These features send data out:
105
+
106
+ * **Semantic search**: sends note text to the embedding provider you configure. It’s off by default (`semanticEnabled`), and you choose the provider with `semanticProvider` and `semanticBaseUrl`. A local provider, such as Ollama or LM Studio, keeps the text on your machine.
107
+ * **Telemetry**: sends logs to the OpenTelemetry Protocol (OTLP) endpoint you configure. It’s off by default (`telemetryEnabled`). The logs contain operations, routes, note paths, actors, tool names and error messages, which may quote a value, but not note bodies. The settings file stores `telemetryHeaders`, which may hold an API key; set `OTEL_EXPORTER_OTLP_HEADERS` instead to keep the key out of it.
108
+ * **Webhooks**: the owner creates them, and the engine posts change events to the URLs they name.
109
+ * **Git sync**: with `--git-sync`, the engine pushes the vault to its git remote.
110
+ * **Store backup**: it’s off by default (`storeBackupEnabled`). When on, the engine pushes its store to the git remote you configure. The backup holds the journal, the transaction logs, the blobs, the change feed and the named tokens’ hashes. The journal, the logs and the feed quote your notes’ values. The backup leaves out `settings.json` and `webhooks.json`, which hold the secrets. Use a private repository.
@@ -0,0 +1,60 @@
1
+ # Give each program its own token
2
+
3
+ > Create named tokens with their own permissions, so the engine limits what each program or agent can do and the journal says who wrote what.
4
+
5
+ A named token lets a program, a device or an agent call the engine with only the permissions it needs. The owner token can do everything; a named token reads everything and writes only what you allow. The journal records the token’s name with every write it makes, so you can tell who changed what.
6
+
7
+ To see a token at work in a complete program, follow [Build an app on Datanotes](https://datanotes.dev/develop/build-an-app/). For the ways to build on the engine, read [Build on Datanotes](https://datanotes.dev/develop/build-on-datanotes/). [Who did what](https://datanotes.dev/guides/security/who-did-what/) shows how the journal names each token.
8
+
9
+ ## Create a token
10
+
11
+ You create a token with the owner token, and the engine answers with the new token. The engine shows it once and keeps only its Secure Hash Algorithm (SHA-256) hash, in `tokens.json` in its store:
12
+
13
+ ```bash
14
+ curl -X POST http://127.0.0.1:27150/datanotes/tokens/create/ \
15
+ -H "Authorization: Bearer your_owner_token" \
16
+ -H "Content-Type: application/json" \
17
+ -d '{"name": "film-diary", "write": ["main.public.*"]}'
18
+ ```
19
+
20
+ This request creates a token named `film-diary` that may write rows of every table in the `main.public` schema, such as `film` and `actor`. The answer holds the token in `token` and the registry’s entry in `entry`. The name is kebab-case, and `owner` is reserved.
21
+
22
+ ## Manage your tokens
23
+
24
+ The owner manages tokens with these routes under `/datanotes/`:
25
+
26
+ * **`tokens/`**: lists the tokens, with their permissions and when each one was last used
27
+ * **`tokens/update/`**: changes a token’s description, permissions or `autolink` default
28
+ * **`tokens/rotate/`**: gives a token a new value; the old one stops working at once
29
+ * **`tokens/delete/`**: revokes a token
30
+
31
+ `datanotes status` lists every token with its permissions, never its value.
32
+
33
+ ## Choose what a token can do
34
+
35
+ A named token can always read. Every other permission comes from its entry:
36
+
37
+ * **Row writes**: only on tables that match a pattern in `write`: `catalog.schema.table`, `catalog.schema.*`, `catalog.*` or `*`
38
+ * **Schema changes**: only with `"ddl": true`
39
+ * **Maintenance**: only with `"maintain": true`, meant for an agent’s own token; it covers configuration, undo, restore, vacuum and rebuilding the semantic index
40
+
41
+ Whatever its permissions, a named token can’t create or delete tokens and webhooks, list tokens, rotate them or read the engine’s status: those stay the owner’s. A token may list webhooks and read its own entry with `tokens/me/`. With `tokens/update/`, it may change its own `description` and `autolink`.
42
+
43
+ ## Turn automatic links off for a sync program
44
+
45
+ By default, the text a write puts in a note links the notes it mentions. The engine calls this [automatic links](https://datanotes.dev/guides/read-and-write/automatic-links/). A sync program copies text from elsewhere, such as tasks, calendar events or receipts. That text should stay as it came, so give its token `"autolink": false`:
46
+
47
+ ```bash
48
+ curl -X POST http://127.0.0.1:27150/datanotes/tokens/create/ \
49
+ -H "Authorization: Bearer your_owner_token" \
50
+ -H "Content-Type: application/json" \
51
+ -d '{"name": "tasks-sync", "write": ["main.public.task"], "autolink": false}'
52
+ ```
53
+
54
+ A token’s `autolink` is the default of its writes:
55
+
56
+ * **`insert_row`, `update_row` and `update_rows`** take it, inside a `batch` too.
57
+ * **A write’s own `autolink`** still wins over it.
58
+ * **Without it**, the token’s writes follow the database’s `autolink` setting.
59
+
60
+ Turn it off for the tokens of sync programs and of modules that write text for you. Keep it unset for an agent’s token. To change it later, call `tokens/update/` with `autolink`. Pass `null` to follow the database setting again. `tokens/` and `tokens/me/` list it with the other fields of the entry.
@@ -0,0 +1,42 @@
1
+ # See who did what
2
+
3
+ > Every write records who made it: the token, the program or agent, its model and session. Read it in the journal, the history and the change feed.
4
+
5
+ Every write records its actor: the credential that made it, and, when the caller says so, the program or agent behind it, its model and its session. The journal, the history and the change feed show it, so you can tell who changed what and undo one agent’s work without touching the rest.
6
+
7
+ ## What the actor records
8
+
9
+ The actor has these fields:
10
+
11
+ * **`auth`**: the credential, `owner` or the name of a [named token](https://datanotes.dev/guides/security/tokens/); the engine sets it, and no header can claim it
12
+ * **`harness`** and **`harness_version`**: the program or MCP client, such as `claude-code`
13
+ * **`model`**: the model behind an agent
14
+ * **`session`**: the agent’s session
15
+ * **`tool`**: the tool the agent called
16
+
17
+ A program sends the other fields with the `X-Datanotes-Actor` header, or the client’s `actor` option, as [Record who makes each write](https://datanotes.dev/develop/http-api/#record-who-makes-each-write) shows. `datanotes mcp` fills them from the MCP connection, as [Every write names the agent that made it](https://datanotes.dev/guides/agents/how-agents-use-datanotes/#every-write-names-the-agent-that-made-it) explains.
18
+
19
+ ## See which token made a change
20
+
21
+ Every write records the credential as `auth` in its actor: `owner`, or the token’s name. When a program forwards an agent’s `X-Datanotes-Actor` header, the engine records both, as “harness / model via token”. A program that writes for itself, without the header, appears under its token’s name.
22
+
23
+ `list_migrations` shows this name in the `by` field of each change, and `list_changes` shows the whole actor.
24
+
25
+ ## Report what an agent does
26
+
27
+ A program that serves its own Model Context Protocol (MCP) server can report its agents to the engine, with any token:
28
+
29
+ * **`POST /datanotes/sessions/identify/`**: names the model and harness behind an agent’s session; later writes that carry the session in `X-Datanotes-Actor` record them
30
+ * **`POST /datanotes/tool-calls/record/`**: records a tool call on the change feed, with `served_by` naming who served it
31
+
32
+ Tool calls appear on the change feed as events with `kind: "tool_call"`, sent only to consumers that ask for that kind. The `call` field holds `tool`, `ok`, `served_by`, `duration_ms`, `summary` and `session`, the arguments cut at 300 characters, the value cut at 1,000, and the result’s size in `result_chars` and `result_tokens`.
33
+
34
+ ## Read who did what
35
+
36
+ Three operations read the record:
37
+
38
+ * **`list_migrations`**: the journal, newest first, each write with `by`, the actor’s name
39
+ * **`describe_history`**: the commits of one table or of every table, with the full actor; it needs history on
40
+ * **`list_changes`**: every change with its actor, and with `kinds: ["tool_call"]` every tool call an MCP server reported
41
+
42
+ `restore_database` with `before_migration` brings every table back to the moment before an agent’s first write: [Restore a table or the database](https://datanotes.dev/guides/history/restore/) shows it.