sensemaking 0.9.4 → 0.9.5
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +35 -91
- package/package.json +12 -3
- package/skills/sense/EXAMPLES.md +6 -15
- package/skills/sense/SKILL.md +33 -130
- package/skills/sense-setup/EXAMPLES.md +8 -27
- package/skills/sense-setup/SKILL.md +21 -82
package/README.md
CHANGED
|
@@ -1,18 +1,12 @@
|
|
|
1
1
|
# sensemaking
|
|
2
2
|
|
|
3
|
-
Query and search
|
|
4
|
-
the prose — words, links, and meaning fused. A CLI that starts, answers, and exits — no
|
|
5
|
-
server, no build step.
|
|
3
|
+
Query and search your markdown notes with context-aware progressive disclosure: SQL over frontmatter, links, and text, plus semantic search and link-graph ranking. No server, no build step.
|
|
6
4
|
|
|
7
5
|
## Problem
|
|
8
6
|
|
|
9
|
-
|
|
10
|
-
means grepping or reading whole folders into context. The structure that makes notes navigable —
|
|
11
|
-
frontmatter, wikilinks, headings — is exactly what an agent needs, but nothing exposes it as a
|
|
12
|
-
query surface.
|
|
7
|
+
Markdown notes accumulate: research, decisions, meeting notes, agent output. Past a few dozen, finding the ones relevant to what you're doing means grepping or reading whole folders into context. The structure that makes notes navigable (frontmatter, wikilinks, headings) is exactly what a query needs, but nothing exposes it as a query surface.
|
|
13
8
|
|
|
14
|
-
`sense` indexes all of it into SQLite and reconciles against file timestamps on every query, so
|
|
15
|
-
results are never stale and nothing has to be running.
|
|
9
|
+
`sense` indexes all of it into SQLite and reconciles against file timestamps on every query, so results are never stale and nothing has to be running.
|
|
16
10
|
|
|
17
11
|
## Quick start
|
|
18
12
|
|
|
@@ -21,8 +15,7 @@ npm install -g sensemaking
|
|
|
21
15
|
cd your-notes && sense init
|
|
22
16
|
```
|
|
23
17
|
|
|
24
|
-
Needs Node 22.16 or newer: that is the first release whose built-in SQLite carries FTS5, which
|
|
25
|
-
`sense search` indexes prose with.
|
|
18
|
+
Needs Node 22.16 or newer: that is the first release whose built-in SQLite carries FTS5, which `sense search` indexes prose with.
|
|
26
19
|
|
|
27
20
|
```bash
|
|
28
21
|
sense map # orient: fields, hub notes, recent changes
|
|
@@ -42,14 +35,9 @@ Every file becomes rows in these tables, plus whatever an enabled feature adds o
|
|
|
42
35
|
| `links` | `src`, `target` as written, `dst` resolved (`NULL` = dead link) | graph |
|
|
43
36
|
| `sections` | heading, `level`, `start_line`, `end_line`, `tokens` estimate | structure |
|
|
44
37
|
|
|
45
|
-
Results are references
|
|
46
|
-
afterward through the filesystem, scoped to the line ranges `peek` returns. This is the
|
|
47
|
-
[just-in-time context pattern](https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents):
|
|
48
|
-
the agent holds lightweight identifiers and loads payloads only when needed.
|
|
38
|
+
Results are references (path, title, summary, excerpt), never file contents. Reading happens afterward through the filesystem, scoped to the line ranges `peek` returns. This is the [just-in-time context pattern](https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents): the agent holds lightweight identifiers and loads payloads only when needed.
|
|
49
39
|
|
|
50
|
-
Output size is a contract, measured per release ([BENCHMARKING.md](BENCHMARKING.md)): `map` is
|
|
51
|
-
fixed-size, a `find` row is tens of tokens, and a `peek` stays flat however large the note is —
|
|
52
|
-
so what it saves over reading grows with the file, and a small note is cheaper read whole.
|
|
40
|
+
Output size is a contract, measured per release ([BENCHMARKING.md](BENCHMARKING.md)): `map` is fixed-size, a `find` row is tens of tokens, and a `peek` stays flat however large the note is. What it saves over reading grows with the file; a small note is cheaper to read whole.
|
|
53
41
|
|
|
54
42
|
```sql
|
|
55
43
|
-- filter and search compose in one query
|
|
@@ -65,7 +53,7 @@ ORDER BY bm25(content, 10.0, 5.0, 1.0) LIMIT 10
|
|
|
65
53
|
|---|---|
|
|
66
54
|
| `map` | doc count, frontmatter field coverage, top hubs by link rank, recent changes |
|
|
67
55
|
| `search "<text>" [--preset name] [--include glob] [--where "<sql>"] [--k n] [--lexical]` | words + links + vectors, one fused ranked list; `via` labels each row's evidence |
|
|
68
|
-
| `peek <path>` | frontmatter + heading outline (`[L143-162, ~380t]`) + links both ways
|
|
56
|
+
| `peek <path>` | frontmatter + heading outline (`[L143-162, ~380t]`) + links both ways: first 20 per list, each with its total; the `links` table has the rest |
|
|
69
57
|
| `query "<sql>" [params...]` | ad-hoc SQL over all the tables; `?` binds positional args |
|
|
70
58
|
| `<name> [params...]` | run a query saved in the config; `--list` names them |
|
|
71
59
|
| `init` | write a starter `sense.config.json` |
|
|
@@ -74,21 +62,11 @@ ORDER BY bm25(content, 10.0, 5.0, 1.0) LIMIT 10
|
|
|
74
62
|
| `rebuild` | delete the cache and re-crawl |
|
|
75
63
|
| `watch` | keep the index warm in the background (optional; see [WATCH.md](WATCH.md)) |
|
|
76
64
|
|
|
77
|
-
`search` runs one text through every engine its scope has
|
|
78
|
-
bare words AND-join, operators are yours), a personalized-PageRank walk over the link graph,
|
|
79
|
-
and vector similarity — fused into one list. `via` labels each row's evidence (`match`,
|
|
80
|
-
`link`, `vector`, combinations); `similarity` is the cosine against the best-matching chunk;
|
|
81
|
-
`lines` points at the section that earned the row — a direct read range. Rows that only
|
|
82
|
-
vectors produced are the "these words aren't in the tree, this is what's near in meaning"
|
|
83
|
-
signal. `--preset` picks a named settings bundle from the config, `--lexical` skips vectors
|
|
84
|
-
for one command, `--where` filters on frontmatter. `--format json` on any reporting command returns
|
|
85
|
-
structured output; `--version` and `--help` do what they say.
|
|
65
|
+
`search` runs one text through every engine its scope has: FTS5 word match (BM25-ranked, bare words AND-join, operators are yours), a personalized-PageRank walk over the link graph, and vector similarity, fused into one list. `via` labels each row's evidence (`match`, `link`, `vector`, combinations); `similarity` is the cosine against the best-matching chunk; `lines` points at the section that earned the row (a direct read range). A `vector`-only row means the search words don't appear in that note; it showed up because the model judged it semantically related. `--preset` picks a named settings bundle from the config, `--lexical` skips vectors for one command, `--where` filters on frontmatter. `--format json` on any reporting command returns structured output; `--version` and `--help` do what they say.
|
|
86
66
|
|
|
87
67
|
## Config
|
|
88
68
|
|
|
89
|
-
`sense init` writes `sense.config.json`; discovery walks up from cwd like git
|
|
90
|
-
(`--config <path>` overrides). Three keys: **presets** (named, self-contained setting
|
|
91
|
-
bundles), **queries** (saved commands), and the version.
|
|
69
|
+
`sense init` writes `sense.config.json`; discovery walks up from cwd like git (`--config <path>` overrides).
|
|
92
70
|
|
|
93
71
|
```json
|
|
94
72
|
{
|
|
@@ -106,46 +84,27 @@ bundles), **queries** (saved commands), and the version.
|
|
|
106
84
|
}
|
|
107
85
|
```
|
|
108
86
|
|
|
109
|
-
|
|
110
|
-
|
|
111
|
-
`
|
|
112
|
-
|
|
113
|
-
|
|
114
|
-
`status` shows each preset's coverage. Editing a preset rebuilds the cache and says which
|
|
115
|
-
preset caused it.
|
|
87
|
+
| key | holds |
|
|
88
|
+
|---|---|
|
|
89
|
+
| `presets` | named bundles of `include`/`exclude` globs, `k` (result count), `semantic` (vectors, on unless `false`), `where` (a standing SQL filter). A file is indexed if any preset includes it, embedded if any covering preset has `semantic` on. A `semantic: false` preset's files cost no vectors, and `status` shows each preset's coverage. |
|
|
90
|
+
| `queries` | a SQL string (`?` binds positional args) or a saved search with its settings baked in: `sense hot` needs no flags. `sense check` runs them all, so a typo'd column fails at check time instead of mid-task. |
|
|
91
|
+
| `version` | schema version; older configs auto-migrate on load, noted on stderr. |
|
|
116
92
|
|
|
117
|
-
|
|
118
|
-
settings baked in — `sense hot` needs no flags. `sense check` runs them all, so a typo'd
|
|
119
|
-
column fails at check time instead of mid-task. Older config versions auto-migrate on
|
|
120
|
-
load, noted on stderr.
|
|
93
|
+
Bare commands use the `default` preset; `--preset` names another; flags override single fields. Editing a preset rebuilds the cache and says which preset caused it.
|
|
121
94
|
|
|
122
|
-
Vectors use the built-in static model (downloaded to `~/.cache/sensemaking` on first use,
|
|
123
|
-
never in the package); an optional top-level `"embed": { "model", "type", "url", "key" }`
|
|
124
|
-
block points at any Model2Vec model, local path, or OpenAI-compatible endpoint (Ollama,
|
|
125
|
-
LM Studio, hosted). Embedding happens on the first semantic search, with progress.
|
|
95
|
+
Vectors use the built-in static model (downloaded to `~/.cache/sensemaking` on first use, never in the package); an optional top-level `"embed": { "model", "type", "url", "key" }` block points at any Model2Vec model, local path, or OpenAI-compatible endpoint (Ollama, LM Studio, hosted). Embedding happens on the first semantic search, with progress.
|
|
126
96
|
|
|
127
|
-
`has(field, value)` is the one custom SQL function: array membership on JSON-array fields,
|
|
128
|
-
substring on strings, false on missing keys. Frontmatter parsing is lenient — syntax errors are
|
|
129
|
-
per-file warnings and the values are still indexed, so one bad note never costs you the crawl.
|
|
97
|
+
`has(field, value)` is the one custom SQL function: array membership on JSON-array fields, substring on strings, false on missing keys. Frontmatter parsing is lenient: syntax errors are per-file warnings, and the values are still indexed, so one bad note never costs you the crawl.
|
|
130
98
|
|
|
131
99
|
## Scale
|
|
132
100
|
|
|
133
|
-
Every query starts with a freshness check against the cache in `.sense/`; only changed files are
|
|
134
|
-
re-parsed. What to expect as a tree grows:
|
|
101
|
+
Every query starts with a freshness check against the cache in `.sense/`; only changed files are re-parsed. What to expect as a tree grows:
|
|
135
102
|
|
|
136
|
-
- **Work is linear in note count
|
|
137
|
-
|
|
138
|
-
- **
|
|
139
|
-
one: context cost is bounded by what you ask for, not by how much there is.
|
|
140
|
-
- **Bulk changes are paid by whoever queries next.** `sense watch` moves that re-parse into the
|
|
141
|
-
background ([WATCH.md](WATCH.md)) — it changes latency, never answers, since every query
|
|
142
|
-
reconciles for itself. `sense rebuild` starts the cache over.
|
|
103
|
+
- **Work is linear in note count.** Crawl, reconcile, and the freshness check are what every invocation pays; that check is the floor cost of a query and the first thing to watch on a large tree.
|
|
104
|
+
- **Output is flat.** `map`, `peek`, and a search row cost the same on a small tree as a large one: context cost is bounded by what you ask for, not by how much there is.
|
|
105
|
+
- **Bulk changes are paid by whoever queries next.** `sense watch` moves that re-parse into the background ([WATCH.md](WATCH.md)): it changes latency, never answers, since every query reconciles for itself. `sense rebuild` starts the cache over.
|
|
143
106
|
|
|
144
|
-
These are release gates rather than hopes: every release regenerates the numbers on pinned
|
|
145
|
-
corpora spanning a 4x range in note count plus a stress tree that packs the worst measured
|
|
146
|
-
shapes into one place — a megabyte-scale note, heading-dense outlines, dense link graphs,
|
|
147
|
-
hundreds of frontmatter fields. A row that grows faster than linearly, or a token count
|
|
148
|
-
that grows at all, blocks the release. Current figures: [BENCHMARKING.md](BENCHMARKING.md).
|
|
107
|
+
These are release gates rather than hopes: every release regenerates the numbers on pinned corpora spanning a 4x range in note count plus a stress tree that packs the worst measured shapes into one place: a megabyte-scale note, heading-dense outlines, dense link graphs, hundreds of frontmatter fields. A row that grows faster than linearly, or a token count that grows at all, blocks the release. Current figures: [BENCHMARKING.md](BENCHMARKING.md).
|
|
149
108
|
|
|
150
109
|
## For AI agents
|
|
151
110
|
|
|
@@ -153,40 +112,25 @@ that grows at all, blocks the release. Current figures: [BENCHMARKING.md](BENCHM
|
|
|
153
112
|
npx skills add kmalakoff/sensemaking # -g for global, -a claude-code to target
|
|
154
113
|
```
|
|
155
114
|
|
|
156
|
-
Two skills: `sense` for querying a tree
|
|
157
|
-
`via`/`score`/`similarity` columns, worked examples — and `sense-setup` for making one, where
|
|
158
|
-
features, frontmatter conventions, and note size are decisions with consequences either way.
|
|
115
|
+
Two skills: `sense` for querying a tree (what each command is for, FTS5 syntax, reading the `via`/`score`/`similarity` columns, worked examples) and `sense-setup` for making one, where features, frontmatter conventions, and note size are decisions with consequences either way.
|
|
159
116
|
|
|
160
117
|
## Prior art
|
|
161
118
|
|
|
162
|
-
- [Effective context engineering for AI agents](https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents)
|
|
163
|
-
|
|
164
|
-
|
|
165
|
-
- [llm-wiki](https://gist.github.com/karpathy/442a6bf555914893e9891c11519de94f) (Karpathy): an
|
|
166
|
-
agent-maintained wiki navigated by an `index.md` and links, which he notes needs real search
|
|
167
|
-
infrastructure past a few hundred pages. `sense map` derives that index from the notes instead of
|
|
168
|
-
maintaining it; `find` is the hybrid local search it calls for.
|
|
169
|
-
- Agent memory patterns — llm-wiki's raw/wiki split, Claude Code's dreaming-style nightly
|
|
170
|
-
consolidation — are trees of small notes with metadata, links, and layers of differing
|
|
171
|
-
authority. sense is the query layer such patterns need (filter by metadata and age, scope
|
|
172
|
-
by layer, surface near-duplicates semantically), not an implementation of any one of them.
|
|
119
|
+
- [Effective context engineering for AI agents](https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents) (Anthropic): agents should hold lightweight identifiers (file paths, links) and load payloads just in time, because context is a finite resource. The commands implement that pattern as a CLI.
|
|
120
|
+
- [llm-wiki](https://gist.github.com/karpathy/442a6bf555914893e9891c11519de94f) (Karpathy): an agent-maintained wiki navigated by an `index.md` and links, which he notes needs real search infrastructure past a few hundred pages. `sense map` derives that index from the notes instead of maintaining it; `find` is the hybrid local search it calls for.
|
|
121
|
+
- Agent memory patterns (llm-wiki's raw/wiki split, Claude Code's dreaming-style nightly consolidation) are trees of small notes with metadata, links, and layers of differing authority. sense is the query layer such patterns need: filter by metadata and age, scope by layer, surface near-duplicates semantically. It isn't an implementation of any one of them.
|
|
173
122
|
|
|
174
123
|
## Alternatives
|
|
175
124
|
|
|
176
|
-
- **Obsidian Bases/Dataview
|
|
177
|
-
|
|
178
|
-
- **
|
|
179
|
-
- **
|
|
180
|
-
- **
|
|
181
|
-
|
|
182
|
-
|
|
183
|
-
|
|
184
|
-
|
|
185
|
-
Dependencies: [yaml](https://github.com/eemeli/yaml),
|
|
186
|
-
[remove-markdown](https://github.com/zuchka/remove-markdown),
|
|
187
|
-
[fast-glob](https://github.com/mrmlnc/fast-glob),
|
|
188
|
-
[@huggingface/tokenizers](https://github.com/huggingface/tokenizers.js) (pure JS), and
|
|
189
|
-
Node's built-in SQLite. No native builds.
|
|
125
|
+
- **Obsidian Bases/Dataview:** same filters, but only inside the running app; agents can't query it headless.
|
|
126
|
+
- **Index-on-build tools (MarkdownDB):** query a snapshot; `sense` reconciles on every query.
|
|
127
|
+
- **Note CLIs (zk):** fixed schema; `sense` filters on arbitrary frontmatter.
|
|
128
|
+
- **Graph/LSP tools (IWE):** structural queries over a markdown graph via LSP/CLI/MCP, retrieval by structure rather than similarity; no SQL, no vector search.
|
|
129
|
+
- **Markdown vector stores (markdown-vdb):** hybrid BM25 + vector search over markdown files, no frontmatter filtering; `sense` treats vectors as one signal alongside SQL, not the whole store.
|
|
130
|
+
- **RAG / vector stores:** similarity can't express `WHERE status = 'active'`. Here vectors are one signal inside `search`: same SQLite file, filters compose, every row labels its evidence (`via`), and a preset turns vectors off per layer of the tree. No second store, no daemon, no native builds.
|
|
131
|
+
- **Document-OS apps (Anytype, Logseq, SilverBullet, Capacities):** full applications with their own UI and storage. `sense` is headless: your files stay files, there's no app to run.
|
|
132
|
+
|
|
133
|
+
Dependencies: [yaml](https://github.com/eemeli/yaml), [remove-markdown](https://github.com/zuchka/remove-markdown), [fast-glob](https://github.com/mrmlnc/fast-glob), [@huggingface/tokenizers](https://github.com/huggingface/tokenizers.js) (pure JS), and Node's built-in SQLite. No native builds.
|
|
190
134
|
|
|
191
135
|
## License
|
|
192
136
|
|
package/package.json
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "sensemaking",
|
|
3
|
-
"version": "0.9.
|
|
4
|
-
"description": "Query and search
|
|
3
|
+
"version": "0.9.5",
|
|
4
|
+
"description": "Query and search your markdown notes with context-aware progressive disclosure: SQL over frontmatter, links, and text, plus semantic search and link-graph ranking. No server, no build step",
|
|
5
5
|
"keywords": [
|
|
6
6
|
"markdown",
|
|
7
7
|
"frontmatter",
|
|
@@ -11,10 +11,19 @@
|
|
|
11
11
|
"query",
|
|
12
12
|
"search",
|
|
13
13
|
"semantic-search",
|
|
14
|
+
"hybrid-search",
|
|
14
15
|
"knowledge-base",
|
|
15
16
|
"rag",
|
|
16
|
-
"
|
|
17
|
+
"markdown-database",
|
|
18
|
+
"document-os",
|
|
17
19
|
"notes",
|
|
20
|
+
"obsidian",
|
|
21
|
+
"markdowndb",
|
|
22
|
+
"iwe",
|
|
23
|
+
"silverbullet",
|
|
24
|
+
"logseq",
|
|
25
|
+
"anytype",
|
|
26
|
+
"zk",
|
|
18
27
|
"agent",
|
|
19
28
|
"agent-memory",
|
|
20
29
|
"memory-consolidation",
|
package/skills/sense/EXAMPLES.md
CHANGED
|
@@ -1,7 +1,6 @@
|
|
|
1
1
|
# sense: worked examples
|
|
2
2
|
|
|
3
|
-
Outputs below are illustrative. Every result is a reference; reading happens afterward, through
|
|
4
|
-
the filesystem, on the paths that earned it.
|
|
3
|
+
Outputs below are illustrative. Every result is a reference; reading happens afterward, through the filesystem, on the paths that earned it.
|
|
5
4
|
|
|
6
5
|
## A. "Do the notes say anything about X?"
|
|
7
6
|
|
|
@@ -18,9 +17,7 @@ sense search "pricing OR billing OR invoicing" --k 10 --format json
|
|
|
18
17
|
]
|
|
19
18
|
```
|
|
20
19
|
|
|
21
|
-
Tens of tokens per row; often the `summary` answers the question with no read at all. A row with
|
|
22
|
-
`via: "link"` never contained the terms — it is linked from notes that did. `lines`, when
|
|
23
|
-
set, is the section that earned the row, a direct `Read` range.
|
|
20
|
+
Tens of tokens per row; often the `summary` answers the question with no read at all. A row with `via: "link"` never contained the terms. It is linked from notes that did. `lines`, when set, is the section that earned the row, a direct `Read` range.
|
|
24
21
|
|
|
25
22
|
## B. Known-field filtering (no search)
|
|
26
23
|
|
|
@@ -28,11 +25,9 @@ set, is the section that earned the row, a direct `Read` range.
|
|
|
28
25
|
sense query "SELECT path, title, status FROM frontmatter WHERE status = 'active' AND has(tags, ?)" pricing --format json
|
|
29
26
|
```
|
|
30
27
|
|
|
31
|
-
Plain SQL on discovered columns. Combine with search by joining `content` and adding
|
|
32
|
-
`AND content MATCH ?` — filter and rank in one query.
|
|
28
|
+
Plain SQL on discovered columns. Combine with search by joining `content` and adding `AND content MATCH ?`: filter and rank in one query.
|
|
33
29
|
|
|
34
|
-
Per-member counts on an array field (GROUP BY on the raw column would split `["a","b"]` from
|
|
35
|
-
`["b","a"]`):
|
|
30
|
+
Per-member counts on an array field (GROUP BY on the raw column would split `["a","b"]` from `["b","a"]`):
|
|
36
31
|
|
|
37
32
|
```
|
|
38
33
|
sense query "SELECT j.value AS tag, COUNT(*) n FROM frontmatter, json_each(frontmatter.tags) j GROUP BY j.value ORDER BY n DESC"
|
|
@@ -55,8 +50,7 @@ links out (7): notes/pricing-model.md, ...
|
|
|
55
50
|
backlinks (2): notes/_index.md, notes/roadmap.md
|
|
56
51
|
```
|
|
57
52
|
|
|
58
|
-
The note is ~4,400 tokens; the peek is ~500. If only one section matters, `Read` its line range
|
|
59
|
-
(~400 tokens) — a tenth of the file.
|
|
53
|
+
The note is ~4,400 tokens; the peek is ~500. If only one section matters, `Read` its line range (~400 tokens), a tenth of the file.
|
|
60
54
|
|
|
61
55
|
## D. The graph
|
|
62
56
|
|
|
@@ -89,10 +83,7 @@ sense search "children dying from poor nutrition" --k 3 --format json
|
|
|
89
83
|
]
|
|
90
84
|
```
|
|
91
85
|
|
|
92
|
-
A `via: "vector"` row never contained the terms
|
|
93
|
-
the cosine against the chunk `lines` names, a direct `Read` range. Vector rows appear whenever
|
|
94
|
-
the scope's preset has semantic on (the default); a result of only vector rows means the words
|
|
95
|
-
themselves are nowhere in the scope.
|
|
86
|
+
A `via: "vector"` row never contained the terms. It is semantically near them; `similarity` is the cosine against the chunk `lines` names, a direct `Read` range. Vector rows appear whenever the scope's preset has semantic on (the default); a result of only vector rows means the words themselves are nowhere in the scope.
|
|
96
87
|
|
|
97
88
|
## Consequences
|
|
98
89
|
|
package/skills/sense/SKILL.md
CHANGED
|
@@ -1,48 +1,23 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: sense
|
|
3
|
-
description: Query a markdown tree with the sense CLI
|
|
3
|
+
description: Query a markdown tree with the sense CLI: filter notes by frontmatter, full-text search the prose, follow wikilinks/backlinks, and read note outlines. Use when the user wants to query, filter, count, search, or report on a folder of markdown notes, when you need to find which notes discuss a topic before reading them, when you want a note's backlinks or structure, when a directory has a sense.config.json, or when asked to add a named query to one.
|
|
4
4
|
---
|
|
5
5
|
|
|
6
6
|
# sense
|
|
7
7
|
|
|
8
|
-
SQL over a markdown tree, kept fresh by a filesystem check on every query. Every file becomes
|
|
9
|
-
rows in `frontmatter` (one column per key, plus `path`/`_mtime`/`_size`/`_rank`), `content`
|
|
10
|
-
(FTS5: `title`, `summary`, `text`), `links` (`src`, `target`, `dst` — `NULL` dst = dead link),
|
|
11
|
-
and `sections` (heading outline with line ranges and token estimates). Features add their own
|
|
12
|
-
storage; `map` and `status` report which are on.
|
|
8
|
+
SQL over a markdown tree, kept fresh by a filesystem check on every query. Every file becomes rows in `frontmatter` (one column per key, plus `path`/`_mtime`/`_size`/`_rank`), `content` (FTS5: `title`, `summary`, `text`), `links` (`src`, `target`, `dst`; `NULL` dst = dead link), and `sections` (heading outline with line ranges and token estimates). Features add their own storage; `map` and `status` report which are on.
|
|
13
9
|
|
|
14
10
|
## What each tool is for
|
|
15
11
|
|
|
16
|
-
Every result is a reference (path, metadata, excerpt), never file contents; prose enters
|
|
17
|
-
|
|
18
|
-
|
|
19
|
-
question
|
|
20
|
-
|
|
21
|
-
-
|
|
22
|
-
|
|
23
|
-
|
|
24
|
-
-
|
|
25
|
-
has: word match (bare words AND-join — one absent word = zero lexical rows; write
|
|
26
|
-
`a OR b OR c` for any-word), link-graph expansion, and vector similarity, fused into one
|
|
27
|
-
ranked list. Read `via` per row: `match` rows contained your words; `vector`-only rows
|
|
28
|
-
did not — they are the "these words aren't in the tree; this is what's near in meaning"
|
|
29
|
-
signal. Vector rows are conceptual similarity, not typo-tolerance; false positives are
|
|
30
|
-
expected, labeled, and bounded by `--k`. `--lexical` skips vectors for one command when
|
|
31
|
-
word-presence is the question.
|
|
32
|
-
- `map` answers "what is this tree" — fields, hub notes, recent changes — when the tree is
|
|
33
|
-
unfamiliar.
|
|
34
|
-
- `peek <path>` prices a file before you pay for it: outline with `[L143-162, ~380t]`
|
|
35
|
-
ranges, links both ways. Every list shows its first 20 with the true total; the
|
|
36
|
-
`sections` and `links` tables hold the rest, so a peek costs a few hundred tokens on any
|
|
37
|
-
note — heading-dense monsters included.
|
|
38
|
-
- When you know the file and need its contents, `Read` it — sense adds nothing there. On
|
|
39
|
-
large files peek's ranges let you read just one section; small files are often cheaper
|
|
40
|
-
whole.
|
|
41
|
-
|
|
42
|
-
Output defaults to a table, built for humans; `--format json` returns the same rows
|
|
43
|
-
machine-parseable. That also makes a saved query usable as a CI/hook gate with zero added
|
|
44
|
-
mechanism: `[ "$(sense <query> --format json)" = "[]" ]` is true exactly when the query
|
|
45
|
-
returned no rows.
|
|
12
|
+
Every result is a reference (path, metadata, excerpt), never file contents; prose enters context only when you Read it. Costs: `map` is fixed-size, a `search` row is tens of tokens, and a `peek` stays flat however large the note is. Which tool fits is a property of the question:
|
|
13
|
+
|
|
14
|
+
- A deterministic, factual answer over known fields (counts, filters, "which notes have X") is SQL: `sense query`, a named query, or `search --where`. Enumerates every match; same result regardless of phrasing.
|
|
15
|
+
- Locating notes about something is `search`, one text through every engine the scope has: word match (bare words AND-join, one absent word = zero lexical rows; write `a OR b OR c` for any-word), link-graph expansion, and vector similarity, fused into one ranked list. Read `via` per row: `match` rows contained your words; `vector`-only rows did not. A `vector`-only row means the search words don't appear in that note; it showed up because the model judged it semantically related. Vector rows are conceptual similarity, not typo-tolerance; false positives are expected, labeled, and bounded by `--k`. `--lexical` skips vectors for one command when word-presence is the question.
|
|
16
|
+
- `map` answers "what is this tree" (fields, hub notes, recent changes) when the tree is unfamiliar.
|
|
17
|
+
- `peek <path>` prices a file before you pay for it: outline with `[L143-162, ~380t]` ranges, links both ways. Every list shows its first 20 with the true total; the `sections` and `links` tables hold the rest, so a peek costs a few hundred tokens on any note, heading-dense monsters included.
|
|
18
|
+
- When you know the file and need its contents, `Read` it. sense adds nothing there. On large files peek's ranges let you read just one section; small files are often cheaper whole.
|
|
19
|
+
|
|
20
|
+
Output defaults to a table, built for humans; `--format json` returns the same rows machine-parseable. That also makes a saved query usable as a CI/hook gate with zero added mechanism: `[ "$(sense <query> --format json)" = "[]" ]` is true exactly when the query returned no rows.
|
|
46
21
|
|
|
47
22
|
## Commands
|
|
48
23
|
|
|
@@ -56,54 +31,15 @@ sense <name> [params...] # named query or saved search fr
|
|
|
56
31
|
sense --list | status | rebuild | check
|
|
57
32
|
```
|
|
58
33
|
|
|
59
|
-
- Terms pass verbatim to FTS5 MATCH. Bare words AND-join
|
|
60
|
-
|
|
61
|
-
|
|
62
|
-
|
|
63
|
-
-
|
|
64
|
-
|
|
65
|
-
|
|
66
|
-
|
|
67
|
-
|
|
68
|
-
both ways.
|
|
69
|
-
- A frontmatter query enumerates its matches deterministically; search ranks by term overlap,
|
|
70
|
-
so results shift as phrasing shifts. Trade-off: a query needs a known field, search doesn't.
|
|
71
|
-
- The `via` column says what produced each row — `match` (words hit), `link` (connected to
|
|
72
|
-
notes that hit), `vector` (near in meaning), and combinations. The `lines` column, when
|
|
73
|
-
set, points at the section that earned the row — the best-matching chunk on vector rows,
|
|
74
|
-
the term cluster's section on large lexical notes — and is a direct `Read` range; null
|
|
75
|
-
means the whole note is the reference.
|
|
76
|
-
- Scope comes from presets: bare `search` uses the config's `default` preset; `--preset
|
|
77
|
-
<name>` picks another (unknown names error, listing what's declared); `--include <glob>`
|
|
78
|
-
is an ad-hoc scope that replaces the preset's globs for one command. `--where` takes any
|
|
79
|
-
SQL condition against frontmatter alias `f` — not only field equality:
|
|
80
|
-
`"f.status = 'active' AND has(f.tags, 'x')"`, `"datetime(f.created) >= datetime(?)"` —
|
|
81
|
-
and filters within the scope. `sense status` shows every preset with its coverage.
|
|
82
|
-
- `score` is a rank-fusion value: it ranks rows within one result set and is not comparable
|
|
83
|
-
across queries, not a relevance magnitude — it encodes how many signals fired and at what
|
|
84
|
-
rank, so a perfect lexical hit and a weak vector-only hit can read the same number. With
|
|
85
|
-
vectors active, rows carry `similarity`: the cosine (-1 to 1) of the query against that
|
|
86
|
-
file's best-matching chunk — the same chunk the `lines` range points at. It orders vector
|
|
87
|
-
evidence within a result set; the range it spans depends on the corpus and the embedding
|
|
88
|
-
model, and compresses on small trees, where even a nonsense query has a moderately near
|
|
89
|
-
neighbour somewhere. Compare similarities within a result set rather than against a fixed
|
|
90
|
-
cutoff carried between trees.
|
|
91
|
-
- Absence evidence lives in the labels: `search --lexical` (or a semantic-off scope)
|
|
92
|
-
returns 0 rows when the words are nowhere in the tree. Default `search` always returns
|
|
93
|
-
up to `k` rows — nearest-neighbour search has a nearest neighbour for any input — so a
|
|
94
|
-
result of only `via: vector` rows IS the absence signal for the words themselves;
|
|
95
|
-
`similarity` and the snippet are the evidence for judging whether a vector row is a real
|
|
96
|
-
conceptual hit.
|
|
97
|
-
- Besides SQL strings, a config entry can save a whole search:
|
|
98
|
-
`"hot": { "search": "pricing OR billing", "preset": "raw", "k": 20 }` runs as
|
|
99
|
-
`sense hot` — the scenario's settings ride along with the name, so repeat runs need no
|
|
100
|
-
flags. An invocation-level `--preset`, `--k`, `--where`, or `--lexical` overrides the
|
|
101
|
-
saved value; `--list` marks these entries `(search)`.
|
|
102
|
-
- `sense check` prepares every saved query and probes every saved search lexically with
|
|
103
|
-
k=1, so a typo'd column, stale SQL, bad FTS5 syntax, or unknown preset fails at check
|
|
104
|
-
time instead of silently mid-task. It reports row counts; whether an empty result is
|
|
105
|
-
good or bad is the reader's judgment — a dead-link query returning rows means broken
|
|
106
|
-
citations to fix, and the agent reads that directly.
|
|
34
|
+
- Terms pass verbatim to FTS5 MATCH. Bare words AND-join (one absent word means zero rows), so write `OR` yourself when you want any-word matching; double-quote punctuated terms (`"customer-facing"`, `"founder's"`); invalid syntax is an error, not a rewrite. The same rules apply to search commands you write into subagent briefs.
|
|
35
|
+
- When a search misses, the recall levers are: OR-in synonyms and concrete instances (the index only knows the words in the files; a note about a specific tool rarely names its category), raise `--k` (a row costs tens of tokens), and widen the scope (`--preset`, or `--include` for an ad-hoc glob). Vector rows already cover the meaning-over-words gap by default. Each widening adds candidates and dilutes ranking, so the noise trade-off runs both ways.
|
|
36
|
+
- A frontmatter query enumerates its matches deterministically; search ranks by term overlap, so results shift as phrasing shifts. Trade-off: a query needs a known field, search doesn't.
|
|
37
|
+
- The `via` column says what produced each row: `match` (words hit), `link` (connected to notes that hit), `vector` (near in meaning), and combinations. The `lines` column, when set, points at the section that earned the row (the best-matching chunk on vector rows, the term cluster's section on large lexical notes) and is a direct `Read` range; null means the whole note is the reference.
|
|
38
|
+
- Scope comes from presets: bare `search` uses the config's `default` preset; `--preset <name>` picks another (unknown names error, listing what's declared); `--include <glob>` is an ad-hoc scope that replaces the preset's globs for one command. `--where` takes any SQL condition against frontmatter alias `f`, not only field equality: `"f.status = 'active' AND has(f.tags, 'x')"`, `"datetime(f.created) >= datetime(?)"`, and filters within the scope. `sense status` shows every preset with its coverage.
|
|
39
|
+
- `score` is a rank-fusion value: it ranks rows within one result set and is not comparable across queries, not a relevance magnitude. It encodes how many signals fired and at what rank, so a perfect lexical hit and a weak vector-only hit can read the same number. With vectors active, rows carry `similarity`: the cosine (-1 to 1) of the query against that file's best-matching chunk (the same chunk the `lines` range points at). It orders vector evidence within a result set; the range it spans depends on the corpus and the embedding model, and compresses on small trees, where even a nonsense query has a moderately near neighbour somewhere. Compare similarities within a result set rather than against a fixed cutoff carried between trees.
|
|
40
|
+
- Absence evidence lives in the labels: `search --lexical` (or a semantic-off scope) returns 0 rows when the words are nowhere in the tree. Default `search` always returns up to `k` rows (nearest-neighbour search has a nearest neighbour for any input), so a result of only `via: vector` rows IS the absence signal for the words themselves; `similarity` and the snippet are the evidence for judging whether a vector row is a real conceptual hit.
|
|
41
|
+
- Besides SQL strings, a config entry can save a whole search: `"hot": { "search": "pricing OR billing", "preset": "raw", "k": 20 }` runs as `sense hot`. The scenario's settings ride along with the name, so repeat runs need no flags. An invocation-level `--preset`, `--k`, `--where`, or `--lexical` overrides the saved value; `--list` marks these entries `(search)`.
|
|
42
|
+
- `sense check` prepares every saved query and probes every saved search lexically with k=1, so a typo'd column, stale SQL, bad FTS5 syntax, or unknown preset fails at check time instead of silently mid-task. It reports row counts; whether an empty result is good or bad is the reader's judgment: a dead-link query returning rows means broken citations to fix, and the agent reads that directly.
|
|
107
43
|
|
|
108
44
|
## SQL
|
|
109
45
|
|
|
@@ -119,55 +55,22 @@ sense query "SELECT heading, start_line, tokens FROM sections WHERE path = ?" a.
|
|
|
119
55
|
sense query "SELECT j.value, COUNT(*) n FROM frontmatter, json_each(frontmatter.tags) j GROUP BY j.value ORDER BY n DESC" # count per array member
|
|
120
56
|
```
|
|
121
57
|
|
|
122
|
-
- `content MATCH` takes FTS5 syntax: `a OR b`, `"phrase"`, `pref*`, `NEAR(a b, 5)`,
|
|
123
|
-
|
|
124
|
-
|
|
125
|
-
|
|
126
|
-
-
|
|
127
|
-
|
|
128
|
-
|
|
129
|
-
1 MB note. `search` bounds this itself (docs past 16 KB get an equivalent excerpt another
|
|
130
|
-
way); in hand-written SQL, guard it: `CASE WHEN length(text) <= 16384 THEN snippet(...)
|
|
131
|
-
END`, or select `title`/`summary` instead of an excerpt.
|
|
132
|
-
- Select `content.title`/`content.summary` (always exist, empty when absent) rather than
|
|
133
|
-
`f.title`/`f.summary` (discovered columns — error on trees that never declare them).
|
|
134
|
-
- Frontmatter values keep their YAML type: strings are TEXT, whole numbers and booleans are
|
|
135
|
-
INTEGER (`true` stores as 1, so `WHERE flag = 1` matches and `WHERE flag = 'true'` matches
|
|
136
|
-
nothing), fractions are REAL, lists and maps are JSON text. `map` prints the observed type
|
|
137
|
-
per field, and a field showing two types (`integer,text`) has drifted across notes.
|
|
138
|
-
- `has(field, value)`: array membership on JSON-array fields, substring on strings, false on NULL
|
|
139
|
-
— the `includes()` convention. Substring means `has(f.status, 'active')` also matches
|
|
140
|
-
`inactive`; exact scalar match is `f.status = ?`, deliberate substring is `LIKE`, exact array
|
|
141
|
-
membership is `EXISTS (SELECT 1 FROM json_each(f.tags) WHERE value = ?)`.
|
|
142
|
-
To aggregate per member instead, use `json_each(frontmatter.<field>)` (above) -- GROUP BY on the
|
|
143
|
-
raw column splits `["a","b"]` and `["b","a"]` into separate buckets.
|
|
144
|
-
- Date fields are stored as written. Compare through `datetime()`, which normalizes ISO 8601
|
|
145
|
-
timezone offsets to UTC: `WHERE datetime(created) >= datetime(?)`. Bare string comparison
|
|
146
|
-
is only safe when every note uses the same offset.
|
|
147
|
-
- To bound what a query puts into context: `snippet()` excerpts just the matching text,
|
|
148
|
-
`LIMIT` caps row counts, and selecting `path`/`title`/`summary` keeps rows small.
|
|
149
|
-
`SELECT text FROM content` returns the tree's entire prose (sense warns past 50 KB).
|
|
150
|
-
Aggregates (`COUNT`, `GROUP BY`) are already bounded. `SELECT * FROM frontmatter` is always
|
|
151
|
-
safe — prose is not a frontmatter column.
|
|
58
|
+
- `content MATCH` takes FTS5 syntax: `a OR b`, `"phrase"`, `pref*`, `NEAR(a b, 5)`, `summary: term`. Stemmed; markdown stripped at index time. Double-quote any term with punctuation. Bare `customer-facing` errors (`-` reads as a column filter), bare apostrophes are syntax errors: write `"customer-facing"`, `"founder's"`.
|
|
59
|
+
- Rank with `ORDER BY bm25(content, 10.0, 5.0, 1.0)` (title > summary > body); excerpt with `snippet(content, -1, '«', '»', '…', 10)`. snippet() re-tokenizes each matched doc and its cost grows superlinearly with doc size, measured ~10 s per query on a tree holding one 1 MB note. `search` bounds this itself (docs past 16 KB get an equivalent excerpt another way); in hand-written SQL, guard it: `CASE WHEN length(text) <= 16384 THEN snippet(...) END`, or select `title`/`summary` instead of an excerpt.
|
|
60
|
+
- Select `content.title`/`content.summary` (always exist, empty when absent) rather than `f.title`/`f.summary` (discovered columns; error on trees that never declare them).
|
|
61
|
+
- Frontmatter values keep their YAML type: strings are TEXT, whole numbers and booleans are INTEGER (`true` stores as 1, so `WHERE flag = 1` matches and `WHERE flag = 'true'` matches nothing), fractions are REAL, lists and maps are JSON text. `map` prints the observed type per field, and a field showing two types (`integer,text`) has drifted across notes.
|
|
62
|
+
- `has(field, value)`: array membership on JSON-array fields, substring on strings, false on NULL. This is the `includes()` convention. Substring means `has(f.status, 'active')` also matches `inactive`; exact scalar match is `f.status = ?`, deliberate substring is `LIKE`, exact array membership is `EXISTS (SELECT 1 FROM json_each(f.tags) WHERE value = ?)`. To aggregate per member instead, use `json_each(frontmatter.<field>)` (above) -- GROUP BY on the raw column splits `["a","b"]` and `["b","a"]` into separate buckets.
|
|
63
|
+
- Date fields are stored as written. Compare through `datetime()`, which normalizes ISO 8601 timezone offsets to UTC: `WHERE datetime(created) >= datetime(?)`. Bare string comparison is only safe when every note uses the same offset.
|
|
64
|
+
- To bound what a query puts into context: `snippet()` excerpts just the matching text, `LIMIT` caps row counts, and selecting `path`/`title`/`summary` keeps rows small. `SELECT text FROM content` returns the tree's entire prose (sense warns past 50 KB). Aggregates (`COUNT`, `GROUP BY`) are already bounded. `SELECT * FROM frontmatter` is always safe: prose is not a frontmatter column.
|
|
152
65
|
|
|
153
66
|
Worked traces: [EXAMPLES.md](EXAMPLES.md).
|
|
154
67
|
|
|
155
68
|
## Setup and upkeep
|
|
156
69
|
|
|
157
|
-
- Missing CLI: `npm install -g sensemaking`. Missing config: `sense init` at the tree root.
|
|
158
|
-
|
|
159
|
-
tree (presets, frontmatter conventions, note design) is the `sense-setup` skill.
|
|
160
|
-
- `map` and `status` report each preset's coverage (files matched, embedded count) —
|
|
161
|
-
indexing derives from presets, so the coverage numbers are how you see what a config
|
|
162
|
-
actually indexes and embeds. A scope with fewer signals just uses fewer (a semantic-off
|
|
163
|
-
preset searches lexically); a saved search naming an unknown preset errors at `check`.
|
|
70
|
+
- Missing CLI: `npm install -g sensemaking`. Missing config: `sense init` at the tree root. Discovery walks up from cwd; `--config <path>` overrides. Setting up or restructuring a tree (presets, frontmatter conventions, note design) is the `sense-setup` skill.
|
|
71
|
+
- `map` and `status` report each preset's coverage (files matched, embedded count). Indexing derives from presets, so the coverage numbers are how you see what a config actually indexes and embeds. A scope with fewer signals just uses fewer (a semantic-off preset searches lexically); a saved search naming an unknown preset errors at `check`.
|
|
164
72
|
- Save a query into `sense.config.json` only when it will be reused; run ad-hoc otherwise.
|
|
165
|
-
- A one-line `summary:` per note is optional and pays twice: it appears in result rows and is a
|
|
166
|
-
|
|
167
|
-
|
|
168
|
-
|
|
169
|
-
- Reserved frontmatter keys (dropped with a warning): `path`, `_mtime`, `_size`, `_rank`,
|
|
170
|
-
`content`, `links`, `sections`.
|
|
171
|
-
- Exit codes: `0` ok, `1` error (SQLite message verbatim), `2` usage (unknown query, wrong
|
|
172
|
-
param count).
|
|
173
|
-
- Doubted cache: `sense rebuild`. Rarely needed — every query reconciles first.
|
|
73
|
+
- A one-line `summary:` per note is optional and pays twice: it appears in result rows and is a weighted search field. Date comparisons work for dates written as ISO 8601 (`2026-08-12`, or with time and offset), the only format `datetime()` parses. Field names in examples (`status`, `tags`, `created`) are illustrative; your tree defines its own.
|
|
74
|
+
- Reserved frontmatter keys (dropped with a warning): `path`, `_mtime`, `_size`, `_rank`, `content`, `links`, `sections`.
|
|
75
|
+
- Exit codes: `0` ok, `1` error (SQLite message verbatim), `2` usage (unknown query, wrong param count).
|
|
76
|
+
- Doubted cache: `sense rebuild`. Rarely needed; every query reconciles first.
|
|
@@ -1,13 +1,10 @@
|
|
|
1
1
|
# sense-setup: worked configurations
|
|
2
2
|
|
|
3
|
-
Four tree shapes, each with its config and the commands an agent actually runs. Field
|
|
4
|
-
names and folder names are illustrative — your tree defines its own.
|
|
3
|
+
Four tree shapes, each with its config and the commands an agent actually runs. Field names and folder names are illustrative; your tree defines its own.
|
|
5
4
|
|
|
6
5
|
## A. Compiled wiki over immutable sources (the llm-wiki pattern)
|
|
7
6
|
|
|
8
|
-
`raw/` holds ingested sources
|
|
9
|
-
pages — linked, curated. The human drops sources and asks questions; the agent compiles
|
|
10
|
-
and cites.
|
|
7
|
+
`raw/` holds ingested sources: big, never hand-edited. `wiki/` holds agent-compiled pages: linked, curated. The human drops sources and asks questions; the agent compiles and cites.
|
|
11
8
|
|
|
12
9
|
```json
|
|
13
10
|
{
|
|
@@ -31,17 +28,11 @@ sense search "rotary embeddings" --preset raw # cite from sources; lexical, k=
|
|
|
31
28
|
sense dead-links # rows are broken citations to fix
|
|
32
29
|
```
|
|
33
30
|
|
|
34
|
-
What the shape buys: bare search never ranks raw noise above compiled pages; raw pays no
|
|
35
|
-
vector/link cost; the compile queue, stub list, and citation integrity are one saved
|
|
36
|
-
query each. The maintenance loop is `uncompiled` → write the wiki page citing its sources
|
|
37
|
-
→ `dead-links` stays empty.
|
|
31
|
+
What the shape buys: bare search never ranks raw noise above compiled pages; raw pays no vector/link cost; the compile queue, stub list, and citation integrity are one saved query each. The maintenance loop is `uncompiled` → write the wiki page citing its sources → `dead-links` stays empty.
|
|
38
32
|
|
|
39
33
|
## B. Nightly agent memory, consolidated (the dreaming pattern)
|
|
40
34
|
|
|
41
|
-
`memory/` accumulates small notes written at session end, each with `project`,
|
|
42
|
-
`created`, and `kind` (observation / steer / decision) frontmatter. A consolidation agent
|
|
43
|
-
runs periodically: prune, merge, surface contradictions for the human. Retired notes move
|
|
44
|
-
to `archive/` — still queryable, no longer embedded or ranked.
|
|
35
|
+
`memory/` accumulates small notes written at session end, each with `project`, `created`, and `kind` (observation / steer / decision) frontmatter. A consolidation agent runs periodically: prune, merge, surface contradictions for the human. Retired notes move to `archive/`: still queryable, no longer embedded or ranked.
|
|
45
36
|
|
|
46
37
|
```json
|
|
47
38
|
{
|
|
@@ -69,17 +60,11 @@ sense retirement # old + uncited → move to arch
|
|
|
69
60
|
sense unfiled # rows are notes missing a project — file them
|
|
70
61
|
```
|
|
71
62
|
|
|
72
|
-
Presets are structural (live vs archived); per-project filtering is metadata (`project = ?`)
|
|
73
|
-
— one tree serves every project. Semantic search over the memory preset is the
|
|
74
|
-
near-duplicate detector: search a new note's own summary and read `similarity` within the
|
|
75
|
-
results. Whether an old steer was overridden is a question for the human — found by
|
|
76
|
-
search, never decided by it.
|
|
63
|
+
Presets are structural (live vs archived); per-project filtering is metadata (`project = ?`). One tree serves every project. Semantic search over the memory preset is the near-duplicate detector: search a new note's own summary and read `similarity` within the results. Whether an old steer was overridden is a question for the human, found by search, never decided by it.
|
|
77
64
|
|
|
78
65
|
## C. Evidence corpus: claims trace to sources
|
|
79
66
|
|
|
80
|
-
`sources/` (immutable imports), `notes/` (one reading note per source), `reviews/`
|
|
81
|
-
(synthesis whose claims must cite notes). The same shape fits incident reports and
|
|
82
|
-
postmortems, user research and findings, due diligence and memos.
|
|
67
|
+
`sources/` (immutable imports), `notes/` (one reading note per source), `reviews/` (synthesis whose claims must cite notes). The same shape fits incident reports and postmortems, user research and findings, due diligence and memos.
|
|
83
68
|
|
|
84
69
|
```json
|
|
85
70
|
{
|
|
@@ -105,8 +90,7 @@ sense unread # the reading queue: sources no
|
|
|
105
90
|
|
|
106
91
|
## D. A plain vault: zero configuration
|
|
107
92
|
|
|
108
|
-
Someone else's Obsidian vault, heterogeneous
|
|
109
|
-
`sense init` starter untouched. The workflow is discovery:
|
|
93
|
+
Someone else's Obsidian vault, heterogeneous with no structure worth declaring. The `sense init` starter is left untouched. The workflow is discovery:
|
|
110
94
|
|
|
111
95
|
```
|
|
112
96
|
sense map # fields in use, hub notes, recent changes
|
|
@@ -115,7 +99,4 @@ sense query "SELECT j.value AS tag, COUNT(*) n FROM frontmatter, json_each(front
|
|
|
115
99
|
sense peek "Plugins/dataview.md" # outline + links before reading
|
|
116
100
|
```
|
|
117
101
|
|
|
118
|
-
Presets earn their place only when a tree has parts deserving different treatment; a tree
|
|
119
|
-
that is one kind of thing needs none of the vocabulary above. On a big vault, raise
|
|
120
|
-
`default`'s `k` and read `lines` ranges instead of whole files — or start from the
|
|
121
|
-
starter's `large` preset.
|
|
102
|
+
Presets earn their place only when a tree has parts deserving different treatment; a tree that is one kind of thing needs none of the vocabulary above. On a big vault, raise `default`'s `k` and read `lines` ranges instead of whole files, or start from the starter's `large` preset.
|
|
@@ -1,32 +1,21 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: sense-setup
|
|
3
|
-
description: Set up the sense CLI on a markdown tree and make the tree-design decisions that shape it
|
|
3
|
+
description: Set up the sense CLI on a markdown tree and make the tree-design decisions that shape it: sense init, presets (which files, which settings, vectors on or off), and the trade-offs of frontmatter conventions, summaries, folder layout, and note size. Use when creating or restructuring a markdown knowledge base, running sense init, editing sense.config.json, configuring search scope or vectors, or deciding how notes should be written for an agent to query later.
|
|
4
4
|
---
|
|
5
5
|
|
|
6
6
|
# sense: setup and tree design
|
|
7
7
|
|
|
8
|
-
Querying an existing tree is the `sense` skill. This one covers making a tree:
|
|
9
|
-
installing, writing presets, and the design decisions a tree owner faces.
|
|
10
|
-
Worked configurations for common tree shapes: [EXAMPLES.md](EXAMPLES.md).
|
|
8
|
+
Querying an existing tree is the `sense` skill. This one covers making a tree: installing, writing presets, and the design decisions a tree owner faces. Worked configurations for common tree shapes: [EXAMPLES.md](EXAMPLES.md).
|
|
11
9
|
|
|
12
10
|
## Setup
|
|
13
11
|
|
|
14
|
-
- `npm install -g sensemaking`, then `sense init` at the tree root writes
|
|
15
|
-
`sense.config.json` — two presets (`default`, and `large` showing what a big
|
|
16
|
-
vault tunes), everything on. Config discovery walks up from cwd;
|
|
17
|
-
`--config <path>` overrides.
|
|
12
|
+
- `npm install -g sensemaking`, then `sense init` at the tree root writes `sense.config.json`: two presets (`default`, and `large` showing what a big vault tunes), everything on. Config discovery walks up from cwd; `--config <path>` overrides.
|
|
18
13
|
- Globs resolve relative to the config file, never the cwd.
|
|
19
|
-
- `sense status` and `sense map` show each preset's coverage (files matched,
|
|
20
|
-
embedded count), so what a config actually indexes is always visible in
|
|
21
|
-
output. A config edit that changes coverage rebuilds the cache and names the
|
|
22
|
-
preset that caused it on stderr.
|
|
14
|
+
- `sense status` and `sense map` show each preset's coverage (files matched, embedded count), so what a config actually indexes is always visible in output. A config edit that changes coverage rebuilds the cache and names the preset that caused it on stderr.
|
|
23
15
|
|
|
24
16
|
## Presets
|
|
25
17
|
|
|
26
|
-
A preset is a named, self-contained bundle of settings. `default` (required) is
|
|
27
|
-
what bare commands use; every other preset is addressed by name
|
|
28
|
-
(`sense search "..." --preset raw`, or `"preset": "raw"` in a saved search).
|
|
29
|
-
No inheritance: what a preset states is all it does.
|
|
18
|
+
A preset is a named, self-contained bundle of settings. `default` (required) is what bare commands use; every other preset is addressed by name (`sense search "..." --preset raw`, or `"preset": "raw"` in a saved search). No inheritance: what a preset states is all it does.
|
|
30
19
|
|
|
31
20
|
| field | means | default |
|
|
32
21
|
|---|---|---|
|
|
@@ -35,76 +24,26 @@ No inheritance: what a preset states is all it does.
|
|
|
35
24
|
| `semantic` | vectors for this preset's files and searches | on; only ever written as `false` |
|
|
36
25
|
| `where` | a standing SQL filter on frontmatter | none |
|
|
37
26
|
|
|
38
|
-
**Indexing derives from presets.** A file is indexed if any preset includes it;
|
|
39
|
-
it is embedded if any covering preset has semantic on. Consequences worth
|
|
40
|
-
designing around:
|
|
27
|
+
**Indexing derives from presets.** A file is indexed if any preset includes it; it is embedded if any covering preset has semantic on. Consequences worth designing around:
|
|
41
28
|
|
|
42
|
-
- A layer of the tree covered only by a `semantic: false` preset (raw sources,
|
|
43
|
-
archives, generated output) is fully searchable lexically and by SQL but
|
|
44
|
-
costs no vector work — the main scale lever.
|
|
29
|
+
- A layer of the tree covered only by a `semantic: false` preset (raw sources, archives, generated output) is fully searchable lexically and by SQL but costs no vector work. This is the main scale lever.
|
|
45
30
|
- Files no preset includes are not indexed at all.
|
|
46
31
|
- Presets may overlap; they are views, not partitions.
|
|
47
|
-
- The first semantic search embeds everything covered (progress on stderr;
|
|
48
|
-
|
|
49
|
-
the built-in static model unless a top-level
|
|
50
|
-
`"embed": { "model", "type": "static"|"api", "url", "key" }` block points at
|
|
51
|
-
a Model2Vec model, local path, or OpenAI-compatible endpoint. `static`
|
|
52
|
-
handles paraphrase and reworded concepts; tight domain jargon ("heart
|
|
53
|
-
attack" for "myocardial infarction") is where an `api` transformer model
|
|
54
|
-
tends to do better — measured in BENCHMARKING.md, "Retrieval quality".
|
|
55
|
-
- Global `features` (`links`, `sections`, `rank`) still toggle tree-wide;
|
|
56
|
-
most trees never touch them.
|
|
32
|
+
- The first semantic search embeds everything covered (progress on stderr; minutes on tens of thousands of notes, seconds on small trees). Vectors use the built-in static model unless a top-level `"embed": { "model", "type": "static"|"api", "url", "key" }` block points at a Model2Vec model, local path, or OpenAI-compatible endpoint. `static` handles paraphrase and reworded concepts; tight domain jargon ("heart attack" for "myocardial infarction") is where an `api` transformer model tends to do better, measured in BENCHMARKING.md, "Retrieval quality".
|
|
33
|
+
- Global `features` (`links`, `sections`, `rank`) still toggle tree-wide; most trees never touch them.
|
|
57
34
|
|
|
58
|
-
**Large vaults**: everything except the vector build is measured linear to
|
|
59
|
-
100k notes with no tuning (BENCHMARKING.md). The knobs that matter are `k`
|
|
60
|
-
(more, smaller results — rows carry `lines` section ranges, so agents read
|
|
61
|
-
sections, not files) and `semantic: false` on the layers that don't earn
|
|
62
|
-
vectors.
|
|
35
|
+
**Large vaults**: everything except the vector build is measured linear to 100k notes with no tuning (BENCHMARKING.md). The knobs that matter are `k` (more, smaller results; rows carry `lines` section ranges, so agents read sections, not files) and `semantic: false` on the layers that don't earn vectors.
|
|
63
36
|
|
|
64
37
|
## Tree design decisions
|
|
65
38
|
|
|
66
|
-
These belong to the tree's owner. sense works with any of them and reads no
|
|
67
|
-
|
|
68
|
-
|
|
69
|
-
- **
|
|
70
|
-
|
|
71
|
-
|
|
72
|
-
|
|
73
|
-
|
|
74
|
-
|
|
75
|
-
|
|
76
|
-
|
|
77
|
-
`map` prints the observed type per field.
|
|
78
|
-
- **Presets are path-shaped; frontmatter is state-shaped.** A preset's coverage
|
|
79
|
-
must be computable from the path alone (it decides indexing, baked into the
|
|
80
|
-
cache). Volatile state (`status`, `project`, dates) lives in frontmatter and
|
|
81
|
-
filters at query time (`where`, `has()`, `datetime()`). A state worth
|
|
82
|
-
different *indexing* (retired memory, superseded sources) is a state worth
|
|
83
|
-
moving the file — the archive-folder pattern in EXAMPLES.md.
|
|
84
|
-
- **What a note omits is also a filter.** A layer that deliberately carries none
|
|
85
|
-
of the fields the saved views filter on is excluded from all of them without
|
|
86
|
-
any view naming the layer. Sparse fields cut both ways: less of the tree
|
|
87
|
-
filters when you want breadth, and exactly this separation when layers
|
|
88
|
-
differ in authority.
|
|
89
|
-
- **Dates.** `datetime()` comparisons work for dates written as ISO 8601 —
|
|
90
|
-
the only format it parses. A tree that mixes date formats can store them,
|
|
91
|
-
but can't compare them in SQL.
|
|
92
|
-
- **Summaries.** A one-line `summary:` is optional and pays twice: it shows in
|
|
93
|
-
every result row (often answering a question with no file read) and is a
|
|
94
|
-
weighted search field ranked above body text. The cost is writing and
|
|
95
|
-
maintaining the line as notes change.
|
|
96
|
-
- **Folder shape.** Globs find the files, paths are queryable text, links
|
|
97
|
-
resolve by basename at any depth — but presets make folders meaningful:
|
|
98
|
-
a folder is the natural unit that gets its own coverage and settings.
|
|
99
|
-
- **Note size.** Many small notes: precise search hits, whole-file reads stay
|
|
100
|
-
cheap, more links to maintain. Fewer large notes: `sections`, `peek`, and
|
|
101
|
-
the `lines` column carry the cost down to line-range reads. Both work.
|
|
102
|
-
- **Recurring questions.** Save a scenario an agent will repeat: a SQL string
|
|
103
|
-
for filters and reports, or a saved search
|
|
104
|
-
(`"hot": { "search": "...", "preset": "raw", "k": 5 }`). Either runs as
|
|
105
|
-
`sense <name>`, and `sense check` validates both kinds against the real
|
|
106
|
-
tree, so a broken saved scenario fails at check time, not mid-task.
|
|
107
|
-
- **Where decisions live.** Choices that should outlive one conversation can
|
|
108
|
-
be recorded in the agent's own instruction or skill files, or in a note in
|
|
109
|
-
the tree itself; a one-off search over an existing corpus needs none of
|
|
110
|
-
that.
|
|
39
|
+
These belong to the tree's owner. sense works with any of them and reads no instruction files of its own; each choice only changes what queries can do.
|
|
40
|
+
|
|
41
|
+
- **Frontmatter fields.** Columns are discovered per tree: whatever keys notes declare become queryable. Consistent fields across notes make SQL filters and named queries possible (`WHERE status = 'active'`). SQLite's compiled column limit (2,000; sqlite.org/limits.html) bounds distinct keys per tree. The crawl stops with an error naming the count and the levers. Reserved keys (dropped with a warning): `path`, `_mtime`, `_size`, `_rank`, `content`, `links`, `sections`. Values keep their YAML type: strings TEXT, whole numbers and booleans INTEGER (`true` is 1), fractions REAL, lists and maps JSON text; `map` prints the observed type per field.
|
|
42
|
+
- **Presets are path-shaped; frontmatter is state-shaped.** A preset's coverage must be computable from the path alone (it decides indexing, baked into the cache). Volatile state (`status`, `project`, dates) lives in frontmatter and filters at query time (`where`, `has()`, `datetime()`). A state worth different *indexing* (retired memory, superseded sources) is a state worth moving the file: the archive-folder pattern in EXAMPLES.md.
|
|
43
|
+
- **What a note omits is also a filter.** A layer that deliberately carries none of the fields the saved views filter on is excluded from all of them without any view naming the layer. Sparse fields cut both ways: less of the tree filters when you want breadth, and exactly this separation when layers differ in authority.
|
|
44
|
+
- **Dates.** `datetime()` comparisons work for dates written as ISO 8601, the only format it parses. A tree that mixes date formats can store them, but can't compare them in SQL.
|
|
45
|
+
- **Summaries.** A one-line `summary:` is optional and pays twice: it shows in every result row (often answering a question with no file read) and is a weighted search field ranked above body text. The cost is writing and maintaining the line as notes change.
|
|
46
|
+
- **Folder shape.** Globs find the files, paths are queryable text, links resolve by basename at any depth, but presets make folders meaningful: a folder is the natural unit that gets its own coverage and settings.
|
|
47
|
+
- **Note size.** Many small notes: precise search hits, whole-file reads stay cheap, more links to maintain. Fewer large notes: `sections`, `peek`, and the `lines` column carry the cost down to line-range reads. Both work.
|
|
48
|
+
- **Recurring questions.** Save a scenario an agent will repeat: a SQL string for filters and reports, or a saved search (`"hot": { "search": "...", "preset": "raw", "k": 5 }`). Either runs as `sense <name>`, and `sense check` validates both kinds against the real tree, so a broken saved scenario fails at check time, not mid-task.
|
|
49
|
+
- **Where decisions live.** Choices that should outlive one conversation can be recorded in the agent's own instruction or skill files, or in a note in the tree itself; a one-off search over an existing corpus needs none of that.
|