sensemaking 0.9.4 → 0.9.5

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -1,18 +1,12 @@
1
1
  # sensemaking
2
2
 
3
- Query and search a tree of markdown notes: SQL over frontmatter and links, ranked search over
4
- the prose — words, links, and meaning fused. A CLI that starts, answers, and exits — no
5
- server, no build step.
3
+ Query and search your markdown notes with context-aware progressive disclosure: SQL over frontmatter, links, and text, plus semantic search and link-graph ranking. No server, no build step.
6
4
 
7
5
  ## Problem
8
6
 
9
- Working with AI agents produces piles of small notes. Past a few dozen, finding the right ones
10
- means grepping or reading whole folders into context. The structure that makes notes navigable —
11
- frontmatter, wikilinks, headings — is exactly what an agent needs, but nothing exposes it as a
12
- query surface.
7
+ Markdown notes accumulate: research, decisions, meeting notes, agent output. Past a few dozen, finding the ones relevant to what you're doing means grepping or reading whole folders into context. The structure that makes notes navigable (frontmatter, wikilinks, headings) is exactly what a query needs, but nothing exposes it as a query surface.
13
8
 
14
- `sense` indexes all of it into SQLite and reconciles against file timestamps on every query, so
15
- results are never stale and nothing has to be running.
9
+ `sense` indexes all of it into SQLite and reconciles against file timestamps on every query, so results are never stale and nothing has to be running.
16
10
 
17
11
  ## Quick start
18
12
 
@@ -21,8 +15,7 @@ npm install -g sensemaking
21
15
  cd your-notes && sense init
22
16
  ```
23
17
 
24
- Needs Node 22.16 or newer: that is the first release whose built-in SQLite carries FTS5, which
25
- `sense search` indexes prose with.
18
+ Needs Node 22.16 or newer: that is the first release whose built-in SQLite carries FTS5, which `sense search` indexes prose with.
26
19
 
27
20
  ```bash
28
21
  sense map # orient: fields, hub notes, recent changes
@@ -42,14 +35,9 @@ Every file becomes rows in these tables, plus whatever an enabled feature adds o
42
35
  | `links` | `src`, `target` as written, `dst` resolved (`NULL` = dead link) | graph |
43
36
  | `sections` | heading, `level`, `start_line`, `end_line`, `tokens` estimate | structure |
44
37
 
45
- Results are references path, title, summary, excerpt never file contents. Reading happens
46
- afterward through the filesystem, scoped to the line ranges `peek` returns. This is the
47
- [just-in-time context pattern](https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents):
48
- the agent holds lightweight identifiers and loads payloads only when needed.
38
+ Results are references (path, title, summary, excerpt), never file contents. Reading happens afterward through the filesystem, scoped to the line ranges `peek` returns. This is the [just-in-time context pattern](https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents): the agent holds lightweight identifiers and loads payloads only when needed.
49
39
 
50
- Output size is a contract, measured per release ([BENCHMARKING.md](BENCHMARKING.md)): `map` is
51
- fixed-size, a `find` row is tens of tokens, and a `peek` stays flat however large the note is —
52
- so what it saves over reading grows with the file, and a small note is cheaper read whole.
40
+ Output size is a contract, measured per release ([BENCHMARKING.md](BENCHMARKING.md)): `map` is fixed-size, a `find` row is tens of tokens, and a `peek` stays flat however large the note is. What it saves over reading grows with the file; a small note is cheaper to read whole.
53
41
 
54
42
  ```sql
55
43
  -- filter and search compose in one query
@@ -65,7 +53,7 @@ ORDER BY bm25(content, 10.0, 5.0, 1.0) LIMIT 10
65
53
  |---|---|
66
54
  | `map` | doc count, frontmatter field coverage, top hubs by link rank, recent changes |
67
55
  | `search "<text>" [--preset name] [--include glob] [--where "<sql>"] [--k n] [--lexical]` | words + links + vectors, one fused ranked list; `via` labels each row's evidence |
68
- | `peek <path>` | frontmatter + heading outline (`[L143-162, ~380t]`) + links both ways first 20 per list, each with its total; the `links` table has the rest |
56
+ | `peek <path>` | frontmatter + heading outline (`[L143-162, ~380t]`) + links both ways: first 20 per list, each with its total; the `links` table has the rest |
69
57
  | `query "<sql>" [params...]` | ad-hoc SQL over all the tables; `?` binds positional args |
70
58
  | `<name> [params...]` | run a query saved in the config; `--list` names them |
71
59
  | `init` | write a starter `sense.config.json` |
@@ -74,21 +62,11 @@ ORDER BY bm25(content, 10.0, 5.0, 1.0) LIMIT 10
74
62
  | `rebuild` | delete the cache and re-crawl |
75
63
  | `watch` | keep the index warm in the background (optional; see [WATCH.md](WATCH.md)) |
76
64
 
77
- `search` runs one text through every engine its scope has FTS5 word match (BM25-ranked,
78
- bare words AND-join, operators are yours), a personalized-PageRank walk over the link graph,
79
- and vector similarity — fused into one list. `via` labels each row's evidence (`match`,
80
- `link`, `vector`, combinations); `similarity` is the cosine against the best-matching chunk;
81
- `lines` points at the section that earned the row — a direct read range. Rows that only
82
- vectors produced are the "these words aren't in the tree, this is what's near in meaning"
83
- signal. `--preset` picks a named settings bundle from the config, `--lexical` skips vectors
84
- for one command, `--where` filters on frontmatter. `--format json` on any reporting command returns
85
- structured output; `--version` and `--help` do what they say.
65
+ `search` runs one text through every engine its scope has: FTS5 word match (BM25-ranked, bare words AND-join, operators are yours), a personalized-PageRank walk over the link graph, and vector similarity, fused into one list. `via` labels each row's evidence (`match`, `link`, `vector`, combinations); `similarity` is the cosine against the best-matching chunk; `lines` points at the section that earned the row (a direct read range). A `vector`-only row means the search words don't appear in that note; it showed up because the model judged it semantically related. `--preset` picks a named settings bundle from the config, `--lexical` skips vectors for one command, `--where` filters on frontmatter. `--format json` on any reporting command returns structured output; `--version` and `--help` do what they say.
86
66
 
87
67
  ## Config
88
68
 
89
- `sense init` writes `sense.config.json`; discovery walks up from cwd like git
90
- (`--config <path>` overrides). Three keys: **presets** (named, self-contained setting
91
- bundles), **queries** (saved commands), and the version.
69
+ `sense init` writes `sense.config.json`; discovery walks up from cwd like git (`--config <path>` overrides).
92
70
 
93
71
  ```json
94
72
  {
@@ -106,46 +84,27 @@ bundles), **queries** (saved commands), and the version.
106
84
  }
107
85
  ```
108
86
 
109
- A preset bundles `include`/`exclude` globs (which files), `k` (result count), `semantic`
110
- (vectors, on unless `false`), and `where` (a standing SQL filter). Bare commands use
111
- `default`; `--preset` names another; flags override single fields. **Indexing derives from
112
- the presets**: a file is indexed if any preset includes it, and embedded if any covering
113
- preset has semantic on so a `semantic: false` preset's files cost no vectors, and
114
- `status` shows each preset's coverage. Editing a preset rebuilds the cache and says which
115
- preset caused it.
87
+ | key | holds |
88
+ |---|---|
89
+ | `presets` | named bundles of `include`/`exclude` globs, `k` (result count), `semantic` (vectors, on unless `false`), `where` (a standing SQL filter). A file is indexed if any preset includes it, embedded if any covering preset has `semantic` on. A `semantic: false` preset's files cost no vectors, and `status` shows each preset's coverage. |
90
+ | `queries` | a SQL string (`?` binds positional args) or a saved search with its settings baked in: `sense hot` needs no flags. `sense check` runs them all, so a typo'd column fails at check time instead of mid-task. |
91
+ | `version` | schema version; older configs auto-migrate on load, noted on stderr. |
116
92
 
117
- A query entry is a SQL string (`?` binds positional args) or a saved search with its
118
- settings baked in — `sense hot` needs no flags. `sense check` runs them all, so a typo'd
119
- column fails at check time instead of mid-task. Older config versions auto-migrate on
120
- load, noted on stderr.
93
+ Bare commands use the `default` preset; `--preset` names another; flags override single fields. Editing a preset rebuilds the cache and says which preset caused it.
121
94
 
122
- Vectors use the built-in static model (downloaded to `~/.cache/sensemaking` on first use,
123
- never in the package); an optional top-level `"embed": { "model", "type", "url", "key" }`
124
- block points at any Model2Vec model, local path, or OpenAI-compatible endpoint (Ollama,
125
- LM Studio, hosted). Embedding happens on the first semantic search, with progress.
95
+ Vectors use the built-in static model (downloaded to `~/.cache/sensemaking` on first use, never in the package); an optional top-level `"embed": { "model", "type", "url", "key" }` block points at any Model2Vec model, local path, or OpenAI-compatible endpoint (Ollama, LM Studio, hosted). Embedding happens on the first semantic search, with progress.
126
96
 
127
- `has(field, value)` is the one custom SQL function: array membership on JSON-array fields,
128
- substring on strings, false on missing keys. Frontmatter parsing is lenient — syntax errors are
129
- per-file warnings and the values are still indexed, so one bad note never costs you the crawl.
97
+ `has(field, value)` is the one custom SQL function: array membership on JSON-array fields, substring on strings, false on missing keys. Frontmatter parsing is lenient: syntax errors are per-file warnings, and the values are still indexed, so one bad note never costs you the crawl.
130
98
 
131
99
  ## Scale
132
100
 
133
- Every query starts with a freshness check against the cache in `.sense/`; only changed files are
134
- re-parsed. What to expect as a tree grows:
101
+ Every query starts with a freshness check against the cache in `.sense/`; only changed files are re-parsed. What to expect as a tree grows:
135
102
 
136
- - **Work is linear in note count** — crawl, reconcile, and the freshness check every invocation
137
- pays. That check is the floor cost of a query and the first thing to watch on a large tree.
138
- - **Output is flat.** `map`, `peek`, and a `find` row cost the same on a small tree as a large
139
- one: context cost is bounded by what you ask for, not by how much there is.
140
- - **Bulk changes are paid by whoever queries next.** `sense watch` moves that re-parse into the
141
- background ([WATCH.md](WATCH.md)) — it changes latency, never answers, since every query
142
- reconciles for itself. `sense rebuild` starts the cache over.
103
+ - **Work is linear in note count.** Crawl, reconcile, and the freshness check are what every invocation pays; that check is the floor cost of a query and the first thing to watch on a large tree.
104
+ - **Output is flat.** `map`, `peek`, and a search row cost the same on a small tree as a large one: context cost is bounded by what you ask for, not by how much there is.
105
+ - **Bulk changes are paid by whoever queries next.** `sense watch` moves that re-parse into the background ([WATCH.md](WATCH.md)): it changes latency, never answers, since every query reconciles for itself. `sense rebuild` starts the cache over.
143
106
 
144
- These are release gates rather than hopes: every release regenerates the numbers on pinned
145
- corpora spanning a 4x range in note count plus a stress tree that packs the worst measured
146
- shapes into one place — a megabyte-scale note, heading-dense outlines, dense link graphs,
147
- hundreds of frontmatter fields. A row that grows faster than linearly, or a token count
148
- that grows at all, blocks the release. Current figures: [BENCHMARKING.md](BENCHMARKING.md).
107
+ These are release gates rather than hopes: every release regenerates the numbers on pinned corpora spanning a 4x range in note count plus a stress tree that packs the worst measured shapes into one place: a megabyte-scale note, heading-dense outlines, dense link graphs, hundreds of frontmatter fields. A row that grows faster than linearly, or a token count that grows at all, blocks the release. Current figures: [BENCHMARKING.md](BENCHMARKING.md).
149
108
 
150
109
  ## For AI agents
151
110
 
@@ -153,40 +112,25 @@ that grows at all, blocks the release. Current figures: [BENCHMARKING.md](BENCHM
153
112
  npx skills add kmalakoff/sensemaking # -g for global, -a claude-code to target
154
113
  ```
155
114
 
156
- Two skills: `sense` for querying a tree what each command is for, FTS5 syntax, reading the
157
- `via`/`score`/`similarity` columns, worked examples — and `sense-setup` for making one, where
158
- features, frontmatter conventions, and note size are decisions with consequences either way.
115
+ Two skills: `sense` for querying a tree (what each command is for, FTS5 syntax, reading the `via`/`score`/`similarity` columns, worked examples) and `sense-setup` for making one, where features, frontmatter conventions, and note size are decisions with consequences either way.
159
116
 
160
117
  ## Prior art
161
118
 
162
- - [Effective context engineering for AI agents](https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents)
163
- (Anthropic): agents should hold lightweight identifiers file paths, links and load payloads
164
- just in time, because context is a finite resource. The commands implement that pattern as a CLI.
165
- - [llm-wiki](https://gist.github.com/karpathy/442a6bf555914893e9891c11519de94f) (Karpathy): an
166
- agent-maintained wiki navigated by an `index.md` and links, which he notes needs real search
167
- infrastructure past a few hundred pages. `sense map` derives that index from the notes instead of
168
- maintaining it; `find` is the hybrid local search it calls for.
169
- - Agent memory patterns — llm-wiki's raw/wiki split, Claude Code's dreaming-style nightly
170
- consolidation — are trees of small notes with metadata, links, and layers of differing
171
- authority. sense is the query layer such patterns need (filter by metadata and age, scope
172
- by layer, surface near-duplicates semantically), not an implementation of any one of them.
119
+ - [Effective context engineering for AI agents](https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents) (Anthropic): agents should hold lightweight identifiers (file paths, links) and load payloads just in time, because context is a finite resource. The commands implement that pattern as a CLI.
120
+ - [llm-wiki](https://gist.github.com/karpathy/442a6bf555914893e9891c11519de94f) (Karpathy): an agent-maintained wiki navigated by an `index.md` and links, which he notes needs real search infrastructure past a few hundred pages. `sense map` derives that index from the notes instead of maintaining it; `find` is the hybrid local search it calls for.
121
+ - Agent memory patterns (llm-wiki's raw/wiki split, Claude Code's dreaming-style nightly consolidation) are trees of small notes with metadata, links, and layers of differing authority. sense is the query layer such patterns need: filter by metadata and age, scope by layer, surface near-duplicates semantically. It isn't an implementation of any one of them.
173
122
 
174
123
  ## Alternatives
175
124
 
176
- - **Obsidian Bases/Dataview** same filters, but only inside the running app; agents can't
177
- query it headless.
178
- - **Index-on-build tools (MarkdownDB)** query a snapshot; `sense` reconciles on every query.
179
- - **Note CLIs (zk)** fixed schema; `sense` filters on arbitrary frontmatter.
180
- - **RAG / vector stores** similarity can't express `WHERE status = 'active'`. Here vectors
181
- are one signal inside `search`: same SQLite file, filters compose, every row labels its
182
- evidence (`via`), and a preset turns vectors off per layer of the tree no second store,
183
- no daemon, no native builds.
184
-
185
- Dependencies: [yaml](https://github.com/eemeli/yaml),
186
- [remove-markdown](https://github.com/zuchka/remove-markdown),
187
- [fast-glob](https://github.com/mrmlnc/fast-glob),
188
- [@huggingface/tokenizers](https://github.com/huggingface/tokenizers.js) (pure JS), and
189
- Node's built-in SQLite. No native builds.
125
+ - **Obsidian Bases/Dataview:** same filters, but only inside the running app; agents can't query it headless.
126
+ - **Index-on-build tools (MarkdownDB):** query a snapshot; `sense` reconciles on every query.
127
+ - **Note CLIs (zk):** fixed schema; `sense` filters on arbitrary frontmatter.
128
+ - **Graph/LSP tools (IWE):** structural queries over a markdown graph via LSP/CLI/MCP, retrieval by structure rather than similarity; no SQL, no vector search.
129
+ - **Markdown vector stores (markdown-vdb):** hybrid BM25 + vector search over markdown files, no frontmatter filtering; `sense` treats vectors as one signal alongside SQL, not the whole store.
130
+ - **RAG / vector stores:** similarity can't express `WHERE status = 'active'`. Here vectors are one signal inside `search`: same SQLite file, filters compose, every row labels its evidence (`via`), and a preset turns vectors off per layer of the tree. No second store, no daemon, no native builds.
131
+ - **Document-OS apps (Anytype, Logseq, SilverBullet, Capacities):** full applications with their own UI and storage. `sense` is headless: your files stay files, there's no app to run.
132
+
133
+ Dependencies: [yaml](https://github.com/eemeli/yaml), [remove-markdown](https://github.com/zuchka/remove-markdown), [fast-glob](https://github.com/mrmlnc/fast-glob), [@huggingface/tokenizers](https://github.com/huggingface/tokenizers.js) (pure JS), and Node's built-in SQLite. No native builds.
190
134
 
191
135
  ## License
192
136
 
package/package.json CHANGED
@@ -1,7 +1,7 @@
1
1
  {
2
2
  "name": "sensemaking",
3
- "version": "0.9.4",
4
- "description": "Query and search a tree of markdown notes: SQL over frontmatter and links, ranked search over the prose words, links, and meaning fused",
3
+ "version": "0.9.5",
4
+ "description": "Query and search your markdown notes with context-aware progressive disclosure: SQL over frontmatter, links, and text, plus semantic search and link-graph ranking. No server, no build step",
5
5
  "keywords": [
6
6
  "markdown",
7
7
  "frontmatter",
@@ -11,10 +11,19 @@
11
11
  "query",
12
12
  "search",
13
13
  "semantic-search",
14
+ "hybrid-search",
14
15
  "knowledge-base",
15
16
  "rag",
16
- "obsidian",
17
+ "markdown-database",
18
+ "document-os",
17
19
  "notes",
20
+ "obsidian",
21
+ "markdowndb",
22
+ "iwe",
23
+ "silverbullet",
24
+ "logseq",
25
+ "anytype",
26
+ "zk",
18
27
  "agent",
19
28
  "agent-memory",
20
29
  "memory-consolidation",
@@ -1,7 +1,6 @@
1
1
  # sense: worked examples
2
2
 
3
- Outputs below are illustrative. Every result is a reference; reading happens afterward, through
4
- the filesystem, on the paths that earned it.
3
+ Outputs below are illustrative. Every result is a reference; reading happens afterward, through the filesystem, on the paths that earned it.
5
4
 
6
5
  ## A. "Do the notes say anything about X?"
7
6
 
@@ -18,9 +17,7 @@ sense search "pricing OR billing OR invoicing" --k 10 --format json
18
17
  ]
19
18
  ```
20
19
 
21
- Tens of tokens per row; often the `summary` answers the question with no read at all. A row with
22
- `via: "link"` never contained the terms — it is linked from notes that did. `lines`, when
23
- set, is the section that earned the row, a direct `Read` range.
20
+ Tens of tokens per row; often the `summary` answers the question with no read at all. A row with `via: "link"` never contained the terms. It is linked from notes that did. `lines`, when set, is the section that earned the row, a direct `Read` range.
24
21
 
25
22
  ## B. Known-field filtering (no search)
26
23
 
@@ -28,11 +25,9 @@ set, is the section that earned the row, a direct `Read` range.
28
25
  sense query "SELECT path, title, status FROM frontmatter WHERE status = 'active' AND has(tags, ?)" pricing --format json
29
26
  ```
30
27
 
31
- Plain SQL on discovered columns. Combine with search by joining `content` and adding
32
- `AND content MATCH ?` — filter and rank in one query.
28
+ Plain SQL on discovered columns. Combine with search by joining `content` and adding `AND content MATCH ?`: filter and rank in one query.
33
29
 
34
- Per-member counts on an array field (GROUP BY on the raw column would split `["a","b"]` from
35
- `["b","a"]`):
30
+ Per-member counts on an array field (GROUP BY on the raw column would split `["a","b"]` from `["b","a"]`):
36
31
 
37
32
  ```
38
33
  sense query "SELECT j.value AS tag, COUNT(*) n FROM frontmatter, json_each(frontmatter.tags) j GROUP BY j.value ORDER BY n DESC"
@@ -55,8 +50,7 @@ links out (7): notes/pricing-model.md, ...
55
50
  backlinks (2): notes/_index.md, notes/roadmap.md
56
51
  ```
57
52
 
58
- The note is ~4,400 tokens; the peek is ~500. If only one section matters, `Read` its line range
59
- (~400 tokens) — a tenth of the file.
53
+ The note is ~4,400 tokens; the peek is ~500. If only one section matters, `Read` its line range (~400 tokens), a tenth of the file.
60
54
 
61
55
  ## D. The graph
62
56
 
@@ -89,10 +83,7 @@ sense search "children dying from poor nutrition" --k 3 --format json
89
83
  ]
90
84
  ```
91
85
 
92
- A `via: "vector"` row never contained the terms it is semantically near them; `similarity` is
93
- the cosine against the chunk `lines` names, a direct `Read` range. Vector rows appear whenever
94
- the scope's preset has semantic on (the default); a result of only vector rows means the words
95
- themselves are nowhere in the scope.
86
+ A `via: "vector"` row never contained the terms. It is semantically near them; `similarity` is the cosine against the chunk `lines` names, a direct `Read` range. Vector rows appear whenever the scope's preset has semantic on (the default); a result of only vector rows means the words themselves are nowhere in the scope.
96
87
 
97
88
  ## Consequences
98
89
 
@@ -1,48 +1,23 @@
1
1
  ---
2
2
  name: sense
3
- description: Query a markdown tree with the sense CLI filter notes by frontmatter, full-text search the prose, follow wikilinks/backlinks, and read note outlines. Use when the user wants to query, filter, count, search, or report on a folder of markdown notes, when you need to find which notes discuss a topic before reading them, when you want a note's backlinks or structure, when a directory has a sense.config.json, or when asked to add a named query to one.
3
+ description: Query a markdown tree with the sense CLI: filter notes by frontmatter, full-text search the prose, follow wikilinks/backlinks, and read note outlines. Use when the user wants to query, filter, count, search, or report on a folder of markdown notes, when you need to find which notes discuss a topic before reading them, when you want a note's backlinks or structure, when a directory has a sense.config.json, or when asked to add a named query to one.
4
4
  ---
5
5
 
6
6
  # sense
7
7
 
8
- SQL over a markdown tree, kept fresh by a filesystem check on every query. Every file becomes
9
- rows in `frontmatter` (one column per key, plus `path`/`_mtime`/`_size`/`_rank`), `content`
10
- (FTS5: `title`, `summary`, `text`), `links` (`src`, `target`, `dst` — `NULL` dst = dead link),
11
- and `sections` (heading outline with line ranges and token estimates). Features add their own
12
- storage; `map` and `status` report which are on.
8
+ SQL over a markdown tree, kept fresh by a filesystem check on every query. Every file becomes rows in `frontmatter` (one column per key, plus `path`/`_mtime`/`_size`/`_rank`), `content` (FTS5: `title`, `summary`, `text`), `links` (`src`, `target`, `dst`; `NULL` dst = dead link), and `sections` (heading outline with line ranges and token estimates). Features add their own storage; `map` and `status` report which are on.
13
9
 
14
10
  ## What each tool is for
15
11
 
16
- Every result is a reference (path, metadata, excerpt), never file contents; prose enters
17
- context only when you Read it. Costs: `map` is fixed-size, a `search` row is tens of tokens,
18
- and a `peek` stays flat however large the note is. Which tool fits is a property of the
19
- question:
20
-
21
- - A deterministic, factual answer over known fields counts, filters, "which notes have
22
- X" is SQL: `sense query`, a named query, or `search --where`. Enumerates every match;
23
- same result regardless of phrasing.
24
- - Locating notes about something is `search` one text through every engine the scope
25
- has: word match (bare words AND-join — one absent word = zero lexical rows; write
26
- `a OR b OR c` for any-word), link-graph expansion, and vector similarity, fused into one
27
- ranked list. Read `via` per row: `match` rows contained your words; `vector`-only rows
28
- did not — they are the "these words aren't in the tree; this is what's near in meaning"
29
- signal. Vector rows are conceptual similarity, not typo-tolerance; false positives are
30
- expected, labeled, and bounded by `--k`. `--lexical` skips vectors for one command when
31
- word-presence is the question.
32
- - `map` answers "what is this tree" — fields, hub notes, recent changes — when the tree is
33
- unfamiliar.
34
- - `peek <path>` prices a file before you pay for it: outline with `[L143-162, ~380t]`
35
- ranges, links both ways. Every list shows its first 20 with the true total; the
36
- `sections` and `links` tables hold the rest, so a peek costs a few hundred tokens on any
37
- note — heading-dense monsters included.
38
- - When you know the file and need its contents, `Read` it — sense adds nothing there. On
39
- large files peek's ranges let you read just one section; small files are often cheaper
40
- whole.
41
-
42
- Output defaults to a table, built for humans; `--format json` returns the same rows
43
- machine-parseable. That also makes a saved query usable as a CI/hook gate with zero added
44
- mechanism: `[ "$(sense <query> --format json)" = "[]" ]` is true exactly when the query
45
- returned no rows.
12
+ Every result is a reference (path, metadata, excerpt), never file contents; prose enters context only when you Read it. Costs: `map` is fixed-size, a `search` row is tens of tokens, and a `peek` stays flat however large the note is. Which tool fits is a property of the question:
13
+
14
+ - A deterministic, factual answer over known fields (counts, filters, "which notes have X") is SQL: `sense query`, a named query, or `search --where`. Enumerates every match; same result regardless of phrasing.
15
+ - Locating notes about something is `search`, one text through every engine the scope has: word match (bare words AND-join, one absent word = zero lexical rows; write `a OR b OR c` for any-word), link-graph expansion, and vector similarity, fused into one ranked list. Read `via` per row: `match` rows contained your words; `vector`-only rows did not. A `vector`-only row means the search words don't appear in that note; it showed up because the model judged it semantically related. Vector rows are conceptual similarity, not typo-tolerance; false positives are expected, labeled, and bounded by `--k`. `--lexical` skips vectors for one command when word-presence is the question.
16
+ - `map` answers "what is this tree" (fields, hub notes, recent changes) when the tree is unfamiliar.
17
+ - `peek <path>` prices a file before you pay for it: outline with `[L143-162, ~380t]` ranges, links both ways. Every list shows its first 20 with the true total; the `sections` and `links` tables hold the rest, so a peek costs a few hundred tokens on any note, heading-dense monsters included.
18
+ - When you know the file and need its contents, `Read` it. sense adds nothing there. On large files peek's ranges let you read just one section; small files are often cheaper whole.
19
+
20
+ Output defaults to a table, built for humans; `--format json` returns the same rows machine-parseable. That also makes a saved query usable as a CI/hook gate with zero added mechanism: `[ "$(sense <query> --format json)" = "[]" ]` is true exactly when the query returned no rows.
46
21
 
47
22
  ## Commands
48
23
 
@@ -56,54 +31,15 @@ sense <name> [params...] # named query or saved search fr
56
31
  sense --list | status | rebuild | check
57
32
  ```
58
33
 
59
- - Terms pass verbatim to FTS5 MATCH. Bare words AND-join one absent word means zero rows
60
- so write `OR` yourself when you want any-word matching; double-quote punctuated terms
61
- (`"customer-facing"`, `"founder's"`); invalid syntax is an error, not a rewrite. The same
62
- rules apply to search commands you write into subagent briefs.
63
- - When a search misses, the recall levers are: OR-in synonyms and concrete instances (the
64
- index only knows the words in the files a note about a specific tool rarely names its
65
- category), raise `--k` (a row costs tens of tokens), and widen the scope (`--preset`, or
66
- `--include` for an ad-hoc glob). Vector rows already cover the meaning-over-words gap by
67
- default. Each widening adds candidates and dilutes ranking, so the noise trade-off runs
68
- both ways.
69
- - A frontmatter query enumerates its matches deterministically; search ranks by term overlap,
70
- so results shift as phrasing shifts. Trade-off: a query needs a known field, search doesn't.
71
- - The `via` column says what produced each row — `match` (words hit), `link` (connected to
72
- notes that hit), `vector` (near in meaning), and combinations. The `lines` column, when
73
- set, points at the section that earned the row — the best-matching chunk on vector rows,
74
- the term cluster's section on large lexical notes — and is a direct `Read` range; null
75
- means the whole note is the reference.
76
- - Scope comes from presets: bare `search` uses the config's `default` preset; `--preset
77
- <name>` picks another (unknown names error, listing what's declared); `--include <glob>`
78
- is an ad-hoc scope that replaces the preset's globs for one command. `--where` takes any
79
- SQL condition against frontmatter alias `f` — not only field equality:
80
- `"f.status = 'active' AND has(f.tags, 'x')"`, `"datetime(f.created) >= datetime(?)"` —
81
- and filters within the scope. `sense status` shows every preset with its coverage.
82
- - `score` is a rank-fusion value: it ranks rows within one result set and is not comparable
83
- across queries, not a relevance magnitude — it encodes how many signals fired and at what
84
- rank, so a perfect lexical hit and a weak vector-only hit can read the same number. With
85
- vectors active, rows carry `similarity`: the cosine (-1 to 1) of the query against that
86
- file's best-matching chunk — the same chunk the `lines` range points at. It orders vector
87
- evidence within a result set; the range it spans depends on the corpus and the embedding
88
- model, and compresses on small trees, where even a nonsense query has a moderately near
89
- neighbour somewhere. Compare similarities within a result set rather than against a fixed
90
- cutoff carried between trees.
91
- - Absence evidence lives in the labels: `search --lexical` (or a semantic-off scope)
92
- returns 0 rows when the words are nowhere in the tree. Default `search` always returns
93
- up to `k` rows — nearest-neighbour search has a nearest neighbour for any input — so a
94
- result of only `via: vector` rows IS the absence signal for the words themselves;
95
- `similarity` and the snippet are the evidence for judging whether a vector row is a real
96
- conceptual hit.
97
- - Besides SQL strings, a config entry can save a whole search:
98
- `"hot": { "search": "pricing OR billing", "preset": "raw", "k": 20 }` runs as
99
- `sense hot` — the scenario's settings ride along with the name, so repeat runs need no
100
- flags. An invocation-level `--preset`, `--k`, `--where`, or `--lexical` overrides the
101
- saved value; `--list` marks these entries `(search)`.
102
- - `sense check` prepares every saved query and probes every saved search lexically with
103
- k=1, so a typo'd column, stale SQL, bad FTS5 syntax, or unknown preset fails at check
104
- time instead of silently mid-task. It reports row counts; whether an empty result is
105
- good or bad is the reader's judgment — a dead-link query returning rows means broken
106
- citations to fix, and the agent reads that directly.
34
+ - Terms pass verbatim to FTS5 MATCH. Bare words AND-join (one absent word means zero rows), so write `OR` yourself when you want any-word matching; double-quote punctuated terms (`"customer-facing"`, `"founder's"`); invalid syntax is an error, not a rewrite. The same rules apply to search commands you write into subagent briefs.
35
+ - When a search misses, the recall levers are: OR-in synonyms and concrete instances (the index only knows the words in the files; a note about a specific tool rarely names its category), raise `--k` (a row costs tens of tokens), and widen the scope (`--preset`, or `--include` for an ad-hoc glob). Vector rows already cover the meaning-over-words gap by default. Each widening adds candidates and dilutes ranking, so the noise trade-off runs both ways.
36
+ - A frontmatter query enumerates its matches deterministically; search ranks by term overlap, so results shift as phrasing shifts. Trade-off: a query needs a known field, search doesn't.
37
+ - The `via` column says what produced each row: `match` (words hit), `link` (connected to notes that hit), `vector` (near in meaning), and combinations. The `lines` column, when set, points at the section that earned the row (the best-matching chunk on vector rows, the term cluster's section on large lexical notes) and is a direct `Read` range; null means the whole note is the reference.
38
+ - Scope comes from presets: bare `search` uses the config's `default` preset; `--preset <name>` picks another (unknown names error, listing what's declared); `--include <glob>` is an ad-hoc scope that replaces the preset's globs for one command. `--where` takes any SQL condition against frontmatter alias `f`, not only field equality: `"f.status = 'active' AND has(f.tags, 'x')"`, `"datetime(f.created) >= datetime(?)"`, and filters within the scope. `sense status` shows every preset with its coverage.
39
+ - `score` is a rank-fusion value: it ranks rows within one result set and is not comparable across queries, not a relevance magnitude. It encodes how many signals fired and at what rank, so a perfect lexical hit and a weak vector-only hit can read the same number. With vectors active, rows carry `similarity`: the cosine (-1 to 1) of the query against that file's best-matching chunk (the same chunk the `lines` range points at). It orders vector evidence within a result set; the range it spans depends on the corpus and the embedding model, and compresses on small trees, where even a nonsense query has a moderately near neighbour somewhere. Compare similarities within a result set rather than against a fixed cutoff carried between trees.
40
+ - Absence evidence lives in the labels: `search --lexical` (or a semantic-off scope) returns 0 rows when the words are nowhere in the tree. Default `search` always returns up to `k` rows (nearest-neighbour search has a nearest neighbour for any input), so a result of only `via: vector` rows IS the absence signal for the words themselves; `similarity` and the snippet are the evidence for judging whether a vector row is a real conceptual hit.
41
+ - Besides SQL strings, a config entry can save a whole search: `"hot": { "search": "pricing OR billing", "preset": "raw", "k": 20 }` runs as `sense hot`. The scenario's settings ride along with the name, so repeat runs need no flags. An invocation-level `--preset`, `--k`, `--where`, or `--lexical` overrides the saved value; `--list` marks these entries `(search)`.
42
+ - `sense check` prepares every saved query and probes every saved search lexically with k=1, so a typo'd column, stale SQL, bad FTS5 syntax, or unknown preset fails at check time instead of silently mid-task. It reports row counts; whether an empty result is good or bad is the reader's judgment: a dead-link query returning rows means broken citations to fix, and the agent reads that directly.
107
43
 
108
44
  ## SQL
109
45
 
@@ -119,55 +55,22 @@ sense query "SELECT heading, start_line, tokens FROM sections WHERE path = ?" a.
119
55
  sense query "SELECT j.value, COUNT(*) n FROM frontmatter, json_each(frontmatter.tags) j GROUP BY j.value ORDER BY n DESC" # count per array member
120
56
  ```
121
57
 
122
- - `content MATCH` takes FTS5 syntax: `a OR b`, `"phrase"`, `pref*`, `NEAR(a b, 5)`,
123
- `summary: term`. Stemmed; markdown stripped at index time. Double-quote any term with
124
- punctuation bare `customer-facing` errors (`-` reads as a column filter), bare
125
- apostrophes are syntax errors: write `"customer-facing"`, `"founder's"`.
126
- - Rank with `ORDER BY bm25(content, 10.0, 5.0, 1.0)` (title > summary > body); excerpt with
127
- `snippet(content, -1, '«', '»', '…', 10)`. snippet() re-tokenizes each matched doc and its
128
- cost grows superlinearly with doc size measured ~10 s per query on a tree holding one
129
- 1 MB note. `search` bounds this itself (docs past 16 KB get an equivalent excerpt another
130
- way); in hand-written SQL, guard it: `CASE WHEN length(text) <= 16384 THEN snippet(...)
131
- END`, or select `title`/`summary` instead of an excerpt.
132
- - Select `content.title`/`content.summary` (always exist, empty when absent) rather than
133
- `f.title`/`f.summary` (discovered columns — error on trees that never declare them).
134
- - Frontmatter values keep their YAML type: strings are TEXT, whole numbers and booleans are
135
- INTEGER (`true` stores as 1, so `WHERE flag = 1` matches and `WHERE flag = 'true'` matches
136
- nothing), fractions are REAL, lists and maps are JSON text. `map` prints the observed type
137
- per field, and a field showing two types (`integer,text`) has drifted across notes.
138
- - `has(field, value)`: array membership on JSON-array fields, substring on strings, false on NULL
139
- — the `includes()` convention. Substring means `has(f.status, 'active')` also matches
140
- `inactive`; exact scalar match is `f.status = ?`, deliberate substring is `LIKE`, exact array
141
- membership is `EXISTS (SELECT 1 FROM json_each(f.tags) WHERE value = ?)`.
142
- To aggregate per member instead, use `json_each(frontmatter.<field>)` (above) -- GROUP BY on the
143
- raw column splits `["a","b"]` and `["b","a"]` into separate buckets.
144
- - Date fields are stored as written. Compare through `datetime()`, which normalizes ISO 8601
145
- timezone offsets to UTC: `WHERE datetime(created) >= datetime(?)`. Bare string comparison
146
- is only safe when every note uses the same offset.
147
- - To bound what a query puts into context: `snippet()` excerpts just the matching text,
148
- `LIMIT` caps row counts, and selecting `path`/`title`/`summary` keeps rows small.
149
- `SELECT text FROM content` returns the tree's entire prose (sense warns past 50 KB).
150
- Aggregates (`COUNT`, `GROUP BY`) are already bounded. `SELECT * FROM frontmatter` is always
151
- safe — prose is not a frontmatter column.
58
+ - `content MATCH` takes FTS5 syntax: `a OR b`, `"phrase"`, `pref*`, `NEAR(a b, 5)`, `summary: term`. Stemmed; markdown stripped at index time. Double-quote any term with punctuation. Bare `customer-facing` errors (`-` reads as a column filter), bare apostrophes are syntax errors: write `"customer-facing"`, `"founder's"`.
59
+ - Rank with `ORDER BY bm25(content, 10.0, 5.0, 1.0)` (title > summary > body); excerpt with `snippet(content, -1, '«', '»', '…', 10)`. snippet() re-tokenizes each matched doc and its cost grows superlinearly with doc size, measured ~10 s per query on a tree holding one 1 MB note. `search` bounds this itself (docs past 16 KB get an equivalent excerpt another way); in hand-written SQL, guard it: `CASE WHEN length(text) <= 16384 THEN snippet(...) END`, or select `title`/`summary` instead of an excerpt.
60
+ - Select `content.title`/`content.summary` (always exist, empty when absent) rather than `f.title`/`f.summary` (discovered columns; error on trees that never declare them).
61
+ - Frontmatter values keep their YAML type: strings are TEXT, whole numbers and booleans are INTEGER (`true` stores as 1, so `WHERE flag = 1` matches and `WHERE flag = 'true'` matches nothing), fractions are REAL, lists and maps are JSON text. `map` prints the observed type per field, and a field showing two types (`integer,text`) has drifted across notes.
62
+ - `has(field, value)`: array membership on JSON-array fields, substring on strings, false on NULL. This is the `includes()` convention. Substring means `has(f.status, 'active')` also matches `inactive`; exact scalar match is `f.status = ?`, deliberate substring is `LIKE`, exact array membership is `EXISTS (SELECT 1 FROM json_each(f.tags) WHERE value = ?)`. To aggregate per member instead, use `json_each(frontmatter.<field>)` (above) -- GROUP BY on the raw column splits `["a","b"]` and `["b","a"]` into separate buckets.
63
+ - Date fields are stored as written. Compare through `datetime()`, which normalizes ISO 8601 timezone offsets to UTC: `WHERE datetime(created) >= datetime(?)`. Bare string comparison is only safe when every note uses the same offset.
64
+ - To bound what a query puts into context: `snippet()` excerpts just the matching text, `LIMIT` caps row counts, and selecting `path`/`title`/`summary` keeps rows small. `SELECT text FROM content` returns the tree's entire prose (sense warns past 50 KB). Aggregates (`COUNT`, `GROUP BY`) are already bounded. `SELECT * FROM frontmatter` is always safe: prose is not a frontmatter column.
152
65
 
153
66
  Worked traces: [EXAMPLES.md](EXAMPLES.md).
154
67
 
155
68
  ## Setup and upkeep
156
69
 
157
- - Missing CLI: `npm install -g sensemaking`. Missing config: `sense init` at the tree root.
158
- Discovery walks up from cwd; `--config <path>` overrides. Setting up or restructuring a
159
- tree (presets, frontmatter conventions, note design) is the `sense-setup` skill.
160
- - `map` and `status` report each preset's coverage (files matched, embedded count) —
161
- indexing derives from presets, so the coverage numbers are how you see what a config
162
- actually indexes and embeds. A scope with fewer signals just uses fewer (a semantic-off
163
- preset searches lexically); a saved search naming an unknown preset errors at `check`.
70
+ - Missing CLI: `npm install -g sensemaking`. Missing config: `sense init` at the tree root. Discovery walks up from cwd; `--config <path>` overrides. Setting up or restructuring a tree (presets, frontmatter conventions, note design) is the `sense-setup` skill.
71
+ - `map` and `status` report each preset's coverage (files matched, embedded count). Indexing derives from presets, so the coverage numbers are how you see what a config actually indexes and embeds. A scope with fewer signals just uses fewer (a semantic-off preset searches lexically); a saved search naming an unknown preset errors at `check`.
164
72
  - Save a query into `sense.config.json` only when it will be reused; run ad-hoc otherwise.
165
- - A one-line `summary:` per note is optional and pays twice: it appears in result rows and is a
166
- weighted search field. Date comparisons work for dates written as ISO 8601 (`2026-08-12`, or
167
- with time and offset) the only format `datetime()` parses. Field names in examples
168
- (`status`, `tags`, `created`) are illustrative; your tree defines its own.
169
- - Reserved frontmatter keys (dropped with a warning): `path`, `_mtime`, `_size`, `_rank`,
170
- `content`, `links`, `sections`.
171
- - Exit codes: `0` ok, `1` error (SQLite message verbatim), `2` usage (unknown query, wrong
172
- param count).
173
- - Doubted cache: `sense rebuild`. Rarely needed — every query reconciles first.
73
+ - A one-line `summary:` per note is optional and pays twice: it appears in result rows and is a weighted search field. Date comparisons work for dates written as ISO 8601 (`2026-08-12`, or with time and offset), the only format `datetime()` parses. Field names in examples (`status`, `tags`, `created`) are illustrative; your tree defines its own.
74
+ - Reserved frontmatter keys (dropped with a warning): `path`, `_mtime`, `_size`, `_rank`, `content`, `links`, `sections`.
75
+ - Exit codes: `0` ok, `1` error (SQLite message verbatim), `2` usage (unknown query, wrong param count).
76
+ - Doubted cache: `sense rebuild`. Rarely needed; every query reconciles first.
@@ -1,13 +1,10 @@
1
1
  # sense-setup: worked configurations
2
2
 
3
- Four tree shapes, each with its config and the commands an agent actually runs. Field
4
- names and folder names are illustrative — your tree defines its own.
3
+ Four tree shapes, each with its config and the commands an agent actually runs. Field names and folder names are illustrative; your tree defines its own.
5
4
 
6
5
  ## A. Compiled wiki over immutable sources (the llm-wiki pattern)
7
6
 
8
- `raw/` holds ingested sources big, never hand-edited. `wiki/` holds agent-compiled
9
- pages — linked, curated. The human drops sources and asks questions; the agent compiles
10
- and cites.
7
+ `raw/` holds ingested sources: big, never hand-edited. `wiki/` holds agent-compiled pages: linked, curated. The human drops sources and asks questions; the agent compiles and cites.
11
8
 
12
9
  ```json
13
10
  {
@@ -31,17 +28,11 @@ sense search "rotary embeddings" --preset raw # cite from sources; lexical, k=
31
28
  sense dead-links # rows are broken citations to fix
32
29
  ```
33
30
 
34
- What the shape buys: bare search never ranks raw noise above compiled pages; raw pays no
35
- vector/link cost; the compile queue, stub list, and citation integrity are one saved
36
- query each. The maintenance loop is `uncompiled` → write the wiki page citing its sources
37
- → `dead-links` stays empty.
31
+ What the shape buys: bare search never ranks raw noise above compiled pages; raw pays no vector/link cost; the compile queue, stub list, and citation integrity are one saved query each. The maintenance loop is `uncompiled` → write the wiki page citing its sources → `dead-links` stays empty.
38
32
 
39
33
  ## B. Nightly agent memory, consolidated (the dreaming pattern)
40
34
 
41
- `memory/` accumulates small notes written at session end, each with `project`,
42
- `created`, and `kind` (observation / steer / decision) frontmatter. A consolidation agent
43
- runs periodically: prune, merge, surface contradictions for the human. Retired notes move
44
- to `archive/` — still queryable, no longer embedded or ranked.
35
+ `memory/` accumulates small notes written at session end, each with `project`, `created`, and `kind` (observation / steer / decision) frontmatter. A consolidation agent runs periodically: prune, merge, surface contradictions for the human. Retired notes move to `archive/`: still queryable, no longer embedded or ranked.
45
36
 
46
37
  ```json
47
38
  {
@@ -69,17 +60,11 @@ sense retirement # old + uncited → move to arch
69
60
  sense unfiled # rows are notes missing a project — file them
70
61
  ```
71
62
 
72
- Presets are structural (live vs archived); per-project filtering is metadata (`project = ?`)
73
- — one tree serves every project. Semantic search over the memory preset is the
74
- near-duplicate detector: search a new note's own summary and read `similarity` within the
75
- results. Whether an old steer was overridden is a question for the human — found by
76
- search, never decided by it.
63
+ Presets are structural (live vs archived); per-project filtering is metadata (`project = ?`). One tree serves every project. Semantic search over the memory preset is the near-duplicate detector: search a new note's own summary and read `similarity` within the results. Whether an old steer was overridden is a question for the human, found by search, never decided by it.
77
64
 
78
65
  ## C. Evidence corpus: claims trace to sources
79
66
 
80
- `sources/` (immutable imports), `notes/` (one reading note per source), `reviews/`
81
- (synthesis whose claims must cite notes). The same shape fits incident reports and
82
- postmortems, user research and findings, due diligence and memos.
67
+ `sources/` (immutable imports), `notes/` (one reading note per source), `reviews/` (synthesis whose claims must cite notes). The same shape fits incident reports and postmortems, user research and findings, due diligence and memos.
83
68
 
84
69
  ```json
85
70
  {
@@ -105,8 +90,7 @@ sense unread # the reading queue: sources no
105
90
 
106
91
  ## D. A plain vault: zero configuration
107
92
 
108
- Someone else's Obsidian vault, heterogeneous, no structure worth declaring the
109
- `sense init` starter untouched. The workflow is discovery:
93
+ Someone else's Obsidian vault, heterogeneous with no structure worth declaring. The `sense init` starter is left untouched. The workflow is discovery:
110
94
 
111
95
  ```
112
96
  sense map # fields in use, hub notes, recent changes
@@ -115,7 +99,4 @@ sense query "SELECT j.value AS tag, COUNT(*) n FROM frontmatter, json_each(front
115
99
  sense peek "Plugins/dataview.md" # outline + links before reading
116
100
  ```
117
101
 
118
- Presets earn their place only when a tree has parts deserving different treatment; a tree
119
- that is one kind of thing needs none of the vocabulary above. On a big vault, raise
120
- `default`'s `k` and read `lines` ranges instead of whole files — or start from the
121
- starter's `large` preset.
102
+ Presets earn their place only when a tree has parts deserving different treatment; a tree that is one kind of thing needs none of the vocabulary above. On a big vault, raise `default`'s `k` and read `lines` ranges instead of whole files, or start from the starter's `large` preset.
@@ -1,32 +1,21 @@
1
1
  ---
2
2
  name: sense-setup
3
- description: Set up the sense CLI on a markdown tree and make the tree-design decisions that shape it sense init, presets (which files, which settings, vectors on or off), and the trade-offs of frontmatter conventions, summaries, folder layout, and note size. Use when creating or restructuring a markdown knowledge base, running sense init, editing sense.config.json, configuring search scope or vectors, or deciding how notes should be written for an agent to query later.
3
+ description: Set up the sense CLI on a markdown tree and make the tree-design decisions that shape it: sense init, presets (which files, which settings, vectors on or off), and the trade-offs of frontmatter conventions, summaries, folder layout, and note size. Use when creating or restructuring a markdown knowledge base, running sense init, editing sense.config.json, configuring search scope or vectors, or deciding how notes should be written for an agent to query later.
4
4
  ---
5
5
 
6
6
  # sense: setup and tree design
7
7
 
8
- Querying an existing tree is the `sense` skill. This one covers making a tree:
9
- installing, writing presets, and the design decisions a tree owner faces.
10
- Worked configurations for common tree shapes: [EXAMPLES.md](EXAMPLES.md).
8
+ Querying an existing tree is the `sense` skill. This one covers making a tree: installing, writing presets, and the design decisions a tree owner faces. Worked configurations for common tree shapes: [EXAMPLES.md](EXAMPLES.md).
11
9
 
12
10
  ## Setup
13
11
 
14
- - `npm install -g sensemaking`, then `sense init` at the tree root writes
15
- `sense.config.json` — two presets (`default`, and `large` showing what a big
16
- vault tunes), everything on. Config discovery walks up from cwd;
17
- `--config <path>` overrides.
12
+ - `npm install -g sensemaking`, then `sense init` at the tree root writes `sense.config.json`: two presets (`default`, and `large` showing what a big vault tunes), everything on. Config discovery walks up from cwd; `--config <path>` overrides.
18
13
  - Globs resolve relative to the config file, never the cwd.
19
- - `sense status` and `sense map` show each preset's coverage (files matched,
20
- embedded count), so what a config actually indexes is always visible in
21
- output. A config edit that changes coverage rebuilds the cache and names the
22
- preset that caused it on stderr.
14
+ - `sense status` and `sense map` show each preset's coverage (files matched, embedded count), so what a config actually indexes is always visible in output. A config edit that changes coverage rebuilds the cache and names the preset that caused it on stderr.
23
15
 
24
16
  ## Presets
25
17
 
26
- A preset is a named, self-contained bundle of settings. `default` (required) is
27
- what bare commands use; every other preset is addressed by name
28
- (`sense search "..." --preset raw`, or `"preset": "raw"` in a saved search).
29
- No inheritance: what a preset states is all it does.
18
+ A preset is a named, self-contained bundle of settings. `default` (required) is what bare commands use; every other preset is addressed by name (`sense search "..." --preset raw`, or `"preset": "raw"` in a saved search). No inheritance: what a preset states is all it does.
30
19
 
31
20
  | field | means | default |
32
21
  |---|---|---|
@@ -35,76 +24,26 @@ No inheritance: what a preset states is all it does.
35
24
  | `semantic` | vectors for this preset's files and searches | on; only ever written as `false` |
36
25
  | `where` | a standing SQL filter on frontmatter | none |
37
26
 
38
- **Indexing derives from presets.** A file is indexed if any preset includes it;
39
- it is embedded if any covering preset has semantic on. Consequences worth
40
- designing around:
27
+ **Indexing derives from presets.** A file is indexed if any preset includes it; it is embedded if any covering preset has semantic on. Consequences worth designing around:
41
28
 
42
- - A layer of the tree covered only by a `semantic: false` preset (raw sources,
43
- archives, generated output) is fully searchable lexically and by SQL but
44
- costs no vector work — the main scale lever.
29
+ - A layer of the tree covered only by a `semantic: false` preset (raw sources, archives, generated output) is fully searchable lexically and by SQL but costs no vector work. This is the main scale lever.
45
30
  - Files no preset includes are not indexed at all.
46
31
  - Presets may overlap; they are views, not partitions.
47
- - The first semantic search embeds everything covered (progress on stderr;
48
- minutes on tens of thousands of notes, seconds on small trees). Vectors use
49
- the built-in static model unless a top-level
50
- `"embed": { "model", "type": "static"|"api", "url", "key" }` block points at
51
- a Model2Vec model, local path, or OpenAI-compatible endpoint. `static`
52
- handles paraphrase and reworded concepts; tight domain jargon ("heart
53
- attack" for "myocardial infarction") is where an `api` transformer model
54
- tends to do better — measured in BENCHMARKING.md, "Retrieval quality".
55
- - Global `features` (`links`, `sections`, `rank`) still toggle tree-wide;
56
- most trees never touch them.
32
+ - The first semantic search embeds everything covered (progress on stderr; minutes on tens of thousands of notes, seconds on small trees). Vectors use the built-in static model unless a top-level `"embed": { "model", "type": "static"|"api", "url", "key" }` block points at a Model2Vec model, local path, or OpenAI-compatible endpoint. `static` handles paraphrase and reworded concepts; tight domain jargon ("heart attack" for "myocardial infarction") is where an `api` transformer model tends to do better, measured in BENCHMARKING.md, "Retrieval quality".
33
+ - Global `features` (`links`, `sections`, `rank`) still toggle tree-wide; most trees never touch them.
57
34
 
58
- **Large vaults**: everything except the vector build is measured linear to
59
- 100k notes with no tuning (BENCHMARKING.md). The knobs that matter are `k`
60
- (more, smaller results — rows carry `lines` section ranges, so agents read
61
- sections, not files) and `semantic: false` on the layers that don't earn
62
- vectors.
35
+ **Large vaults**: everything except the vector build is measured linear to 100k notes with no tuning (BENCHMARKING.md). The knobs that matter are `k` (more, smaller results; rows carry `lines` section ranges, so agents read sections, not files) and `semantic: false` on the layers that don't earn vectors.
63
36
 
64
37
  ## Tree design decisions
65
38
 
66
- These belong to the tree's owner. sense works with any of them and reads no
67
- instruction files of its own; each choice only changes what queries can do.
68
-
69
- - **Frontmatter fields.** Columns are discovered per tree whatever keys notes
70
- declare become queryable. Consistent fields across notes make SQL filters
71
- and named queries possible (`WHERE status = 'active'`). SQLite's compiled
72
- column limit (2,000; sqlite.org/limits.html) bounds distinct keys per tree
73
- the crawl stops with an error naming the count and the levers. Reserved keys
74
- (dropped with a warning): `path`, `_mtime`, `_size`, `_rank`, `content`,
75
- `links`, `sections`. Values keep their YAML type: strings TEXT, whole numbers
76
- and booleans INTEGER (`true` is 1), fractions REAL, lists and maps JSON text;
77
- `map` prints the observed type per field.
78
- - **Presets are path-shaped; frontmatter is state-shaped.** A preset's coverage
79
- must be computable from the path alone (it decides indexing, baked into the
80
- cache). Volatile state (`status`, `project`, dates) lives in frontmatter and
81
- filters at query time (`where`, `has()`, `datetime()`). A state worth
82
- different *indexing* (retired memory, superseded sources) is a state worth
83
- moving the file — the archive-folder pattern in EXAMPLES.md.
84
- - **What a note omits is also a filter.** A layer that deliberately carries none
85
- of the fields the saved views filter on is excluded from all of them without
86
- any view naming the layer. Sparse fields cut both ways: less of the tree
87
- filters when you want breadth, and exactly this separation when layers
88
- differ in authority.
89
- - **Dates.** `datetime()` comparisons work for dates written as ISO 8601 —
90
- the only format it parses. A tree that mixes date formats can store them,
91
- but can't compare them in SQL.
92
- - **Summaries.** A one-line `summary:` is optional and pays twice: it shows in
93
- every result row (often answering a question with no file read) and is a
94
- weighted search field ranked above body text. The cost is writing and
95
- maintaining the line as notes change.
96
- - **Folder shape.** Globs find the files, paths are queryable text, links
97
- resolve by basename at any depth — but presets make folders meaningful:
98
- a folder is the natural unit that gets its own coverage and settings.
99
- - **Note size.** Many small notes: precise search hits, whole-file reads stay
100
- cheap, more links to maintain. Fewer large notes: `sections`, `peek`, and
101
- the `lines` column carry the cost down to line-range reads. Both work.
102
- - **Recurring questions.** Save a scenario an agent will repeat: a SQL string
103
- for filters and reports, or a saved search
104
- (`"hot": { "search": "...", "preset": "raw", "k": 5 }`). Either runs as
105
- `sense <name>`, and `sense check` validates both kinds against the real
106
- tree, so a broken saved scenario fails at check time, not mid-task.
107
- - **Where decisions live.** Choices that should outlive one conversation can
108
- be recorded in the agent's own instruction or skill files, or in a note in
109
- the tree itself; a one-off search over an existing corpus needs none of
110
- that.
39
+ These belong to the tree's owner. sense works with any of them and reads no instruction files of its own; each choice only changes what queries can do.
40
+
41
+ - **Frontmatter fields.** Columns are discovered per tree: whatever keys notes declare become queryable. Consistent fields across notes make SQL filters and named queries possible (`WHERE status = 'active'`). SQLite's compiled column limit (2,000; sqlite.org/limits.html) bounds distinct keys per tree. The crawl stops with an error naming the count and the levers. Reserved keys (dropped with a warning): `path`, `_mtime`, `_size`, `_rank`, `content`, `links`, `sections`. Values keep their YAML type: strings TEXT, whole numbers and booleans INTEGER (`true` is 1), fractions REAL, lists and maps JSON text; `map` prints the observed type per field.
42
+ - **Presets are path-shaped; frontmatter is state-shaped.** A preset's coverage must be computable from the path alone (it decides indexing, baked into the cache). Volatile state (`status`, `project`, dates) lives in frontmatter and filters at query time (`where`, `has()`, `datetime()`). A state worth different *indexing* (retired memory, superseded sources) is a state worth moving the file: the archive-folder pattern in EXAMPLES.md.
43
+ - **What a note omits is also a filter.** A layer that deliberately carries none of the fields the saved views filter on is excluded from all of them without any view naming the layer. Sparse fields cut both ways: less of the tree filters when you want breadth, and exactly this separation when layers differ in authority.
44
+ - **Dates.** `datetime()` comparisons work for dates written as ISO 8601, the only format it parses. A tree that mixes date formats can store them, but can't compare them in SQL.
45
+ - **Summaries.** A one-line `summary:` is optional and pays twice: it shows in every result row (often answering a question with no file read) and is a weighted search field ranked above body text. The cost is writing and maintaining the line as notes change.
46
+ - **Folder shape.** Globs find the files, paths are queryable text, links resolve by basename at any depth, but presets make folders meaningful: a folder is the natural unit that gets its own coverage and settings.
47
+ - **Note size.** Many small notes: precise search hits, whole-file reads stay cheap, more links to maintain. Fewer large notes: `sections`, `peek`, and the `lines` column carry the cost down to line-range reads. Both work.
48
+ - **Recurring questions.** Save a scenario an agent will repeat: a SQL string for filters and reports, or a saved search (`"hot": { "search": "...", "preset": "raw", "k": 5 }`). Either runs as `sense <name>`, and `sense check` validates both kinds against the real tree, so a broken saved scenario fails at check time, not mid-task.
49
+ - **Where decisions live.** Choices that should outlive one conversation can be recorded in the agent's own instruction or skill files, or in a note in the tree itself; a one-off search over an existing corpus needs none of that.