sensemaking 0.9.3 → 0.9.5

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -1,48 +1,23 @@
1
1
  ---
2
2
  name: sense
3
- description: Query a markdown tree with the sense CLI — filter notes by frontmatter, full-text search the prose, follow wikilinks/backlinks, and read note outlines. Use when the user wants to query, filter, count, search, or report on a folder of markdown notes, when you need to find which notes discuss a topic before reading them, when you want a note's backlinks or structure, when a directory has a sense.config.json, or when asked to add a named query to one.
3
+ description: Query a markdown tree with the sense CLI: filter notes by frontmatter, full-text search the prose, follow wikilinks/backlinks, and read note outlines. Use when the user wants to query, filter, count, search, or report on a folder of markdown notes, when you need to find which notes discuss a topic before reading them, when you want a note's backlinks or structure, when a directory has a sense.config.json, or when asked to add a named query to one.
4
4
  ---
5
5
 
6
6
  # sense
7
7
 
8
- SQL over a markdown tree, kept fresh by a filesystem check on every query. Every file becomes
9
- rows in `frontmatter` (one column per key, plus `path`/`_mtime`/`_size`/`_rank`), `content`
10
- (FTS5: `title`, `summary`, `text`), `links` (`src`, `target`, `dst` — `NULL` dst = dead link),
11
- and `sections` (heading outline with line ranges and token estimates). Features add their own
12
- storage; `map` and `status` report which are on.
8
+ SQL over a markdown tree, kept fresh by a filesystem check on every query. Every file becomes rows in `frontmatter` (one column per key, plus `path`/`_mtime`/`_size`/`_rank`), `content` (FTS5: `title`, `summary`, `text`), `links` (`src`, `target`, `dst`; `NULL` dst = dead link), and `sections` (heading outline with line ranges and token estimates). Features add their own storage; `map` and `status` report which are on.
13
9
 
14
10
  ## What each tool is for
15
11
 
16
- Every result is a reference (path, metadata, excerpt), never file contents; prose enters
17
- context only when you Read it. Costs: `map` is fixed-size, a `search` row is tens of tokens,
18
- and a `peek` stays flat however large the note is. Which tool fits is a property of the
19
- question:
20
-
21
- - A deterministic, factual answer over known fields — counts, filters, "which notes have
22
- X" — is SQL: `sense query`, a named query, or `search --where`. Enumerates every match;
23
- same result regardless of phrasing.
24
- - Locating notes about something is `search` — one text through every engine the scope
25
- has: word match (bare words AND-join — one absent word = zero lexical rows; write
26
- `a OR b OR c` for any-word), link-graph expansion, and vector similarity, fused into one
27
- ranked list. Read `via` per row: `match` rows contained your words; `vector`-only rows
28
- did not — they are the "these words aren't in the tree; this is what's near in meaning"
29
- signal. Vector rows are conceptual similarity, not typo-tolerance; false positives are
30
- expected, labeled, and bounded by `--k`. `--lexical` skips vectors for one command when
31
- word-presence is the question.
32
- - `map` answers "what is this tree" — fields, hub notes, recent changes — when the tree is
33
- unfamiliar.
34
- - `peek <path>` prices a file before you pay for it: outline with `[L143-162, ~380t]`
35
- ranges, links both ways. Every list shows its first 20 with the true total; the
36
- `sections` and `links` tables hold the rest, so a peek costs a few hundred tokens on any
37
- note — heading-dense monsters included.
38
- - When you know the file and need its contents, `Read` it — sense adds nothing there. On
39
- large files peek's ranges let you read just one section; small files are often cheaper
40
- whole.
41
-
42
- Output defaults to a table, built for humans; `--format json` returns the same rows
43
- machine-parseable. That also makes a saved query usable as a CI/hook gate with zero added
44
- mechanism: `[ "$(sense <query> --format json)" = "[]" ]` is true exactly when the query
45
- returned no rows.
12
+ Every result is a reference (path, metadata, excerpt), never file contents; prose enters context only when you Read it. Costs: `map` is fixed-size, a `search` row is tens of tokens, and a `peek` stays flat however large the note is. Which tool fits is a property of the question:
13
+
14
+ - A deterministic, factual answer over known fields (counts, filters, "which notes have X") is SQL: `sense query`, a named query, or `search --where`. Enumerates every match; same result regardless of phrasing.
15
+ - Locating notes about something is `search`, one text through every engine the scope has: word match (bare words AND-join, one absent word = zero lexical rows; write `a OR b OR c` for any-word), link-graph expansion, and vector similarity, fused into one ranked list. Read `via` per row: `match` rows contained your words; `vector`-only rows did not. A `vector`-only row means the search words don't appear in that note; it showed up because the model judged it semantically related. Vector rows are conceptual similarity, not typo-tolerance; false positives are expected, labeled, and bounded by `--k`. `--lexical` skips vectors for one command when word-presence is the question.
16
+ - `map` answers "what is this tree" (fields, hub notes, recent changes) when the tree is unfamiliar.
17
+ - `peek <path>` prices a file before you pay for it: outline with `[L143-162, ~380t]` ranges, links both ways. Every list shows its first 20 with the true total; the `sections` and `links` tables hold the rest, so a peek costs a few hundred tokens on any note, heading-dense monsters included.
18
+ - When you know the file and need its contents, `Read` it. sense adds nothing there. On large files peek's ranges let you read just one section; small files are often cheaper whole.
19
+
20
+ Output defaults to a table, built for humans; `--format json` returns the same rows machine-parseable. That also makes a saved query usable as a CI/hook gate with zero added mechanism: `[ "$(sense <query> --format json)" = "[]" ]` is true exactly when the query returned no rows.
46
21
 
47
22
  ## Commands
48
23
 
@@ -56,54 +31,15 @@ sense <name> [params...] # named query or saved search fr
56
31
  sense --list | status | rebuild | check
57
32
  ```
58
33
 
59
- - Terms pass verbatim to FTS5 MATCH. Bare words AND-join — one absent word means zero rows —
60
- so write `OR` yourself when you want any-word matching; double-quote punctuated terms
61
- (`"customer-facing"`, `"founder's"`); invalid syntax is an error, not a rewrite. The same
62
- rules apply to search commands you write into subagent briefs.
63
- - When a search misses, the recall levers are: OR-in synonyms and concrete instances (the
64
- index only knows the words in the files — a note about a specific tool rarely names its
65
- category), raise `--k` (a row costs tens of tokens), and widen the scope (`--preset`, or
66
- `--include` for an ad-hoc glob). Vector rows already cover the meaning-over-words gap by
67
- default. Each widening adds candidates and dilutes ranking, so the noise trade-off runs
68
- both ways.
69
- - A frontmatter query enumerates its matches deterministically; search ranks by term overlap,
70
- so results shift as phrasing shifts. Trade-off: a query needs a known field, search doesn't.
71
- - The `via` column says what produced each row — `match` (words hit), `link` (connected to
72
- notes that hit), `vector` (near in meaning), and combinations. The `lines` column, when
73
- set, points at the section that earned the row — the best-matching chunk on vector rows,
74
- the term cluster's section on large lexical notes — and is a direct `Read` range; null
75
- means the whole note is the reference.
76
- - Scope comes from presets: bare `search` uses the config's `default` preset; `--preset
77
- <name>` picks another (unknown names error, listing what's declared); `--include <glob>`
78
- is an ad-hoc scope that replaces the preset's globs for one command. `--where` takes any
79
- SQL condition against frontmatter alias `f` — not only field equality:
80
- `"f.status = 'active' AND has(f.tags, 'x')"`, `"datetime(f.created) >= datetime(?)"` —
81
- and filters within the scope. `sense status` shows every preset with its coverage.
82
- - `score` is a rank-fusion value: it ranks rows within one result set and is not comparable
83
- across queries, not a relevance magnitude — it encodes how many signals fired and at what
84
- rank, so a perfect lexical hit and a weak vector-only hit can read the same number. With
85
- vectors active, rows carry `similarity`: the cosine (-1 to 1) of the query against that
86
- file's best-matching chunk — the same chunk the `lines` range points at. It orders vector
87
- evidence within a result set; the range it spans depends on the corpus and the embedding
88
- model, and compresses on small trees, where even a nonsense query has a moderately near
89
- neighbour somewhere. Compare similarities within a result set rather than against a fixed
90
- cutoff carried between trees.
91
- - Absence evidence lives in the labels: `search --lexical` (or a semantic-off scope)
92
- returns 0 rows when the words are nowhere in the tree. Default `search` always returns
93
- up to `k` rows — nearest-neighbour search has a nearest neighbour for any input — so a
94
- result of only `via: vector` rows IS the absence signal for the words themselves;
95
- `similarity` and the snippet are the evidence for judging whether a vector row is a real
96
- conceptual hit.
97
- - Besides SQL strings, a config entry can save a whole search:
98
- `"hot": { "search": "pricing OR billing", "preset": "raw", "k": 20 }` runs as
99
- `sense hot` — the scenario's settings ride along with the name, so repeat runs need no
100
- flags. An invocation-level `--preset`, `--k`, `--where`, or `--lexical` overrides the
101
- saved value; `--list` marks these entries `(search)`.
102
- - `sense check` prepares every saved query and probes every saved search lexically with
103
- k=1, so a typo'd column, stale SQL, bad FTS5 syntax, or unknown preset fails at check
104
- time instead of silently mid-task. It reports row counts; whether an empty result is
105
- good or bad is the reader's judgment — a dead-link query returning rows means broken
106
- citations to fix, and the agent reads that directly.
34
+ - Terms pass verbatim to FTS5 MATCH. Bare words AND-join (one absent word means zero rows), so write `OR` yourself when you want any-word matching; double-quote punctuated terms (`"customer-facing"`, `"founder's"`); invalid syntax is an error, not a rewrite. The same rules apply to search commands you write into subagent briefs.
35
+ - When a search misses, the recall levers are: OR-in synonyms and concrete instances (the index only knows the words in the files; a note about a specific tool rarely names its category), raise `--k` (a row costs tens of tokens), and widen the scope (`--preset`, or `--include` for an ad-hoc glob). Vector rows already cover the meaning-over-words gap by default. Each widening adds candidates and dilutes ranking, so the noise trade-off runs both ways.
36
+ - A frontmatter query enumerates its matches deterministically; search ranks by term overlap, so results shift as phrasing shifts. Trade-off: a query needs a known field, search doesn't.
37
+ - The `via` column says what produced each row: `match` (words hit), `link` (connected to notes that hit), `vector` (near in meaning), and combinations. The `lines` column, when set, points at the section that earned the row (the best-matching chunk on vector rows, the term cluster's section on large lexical notes) and is a direct `Read` range; null means the whole note is the reference.
38
+ - Scope comes from presets: bare `search` uses the config's `default` preset; `--preset <name>` picks another (unknown names error, listing what's declared); `--include <glob>` is an ad-hoc scope that replaces the preset's globs for one command. `--where` takes any SQL condition against frontmatter alias `f`, not only field equality: `"f.status = 'active' AND has(f.tags, 'x')"`, `"datetime(f.created) >= datetime(?)"`, and filters within the scope. `sense status` shows every preset with its coverage.
39
+ - `score` is a rank-fusion value: it ranks rows within one result set and is not comparable across queries, not a relevance magnitude. It encodes how many signals fired and at what rank, so a perfect lexical hit and a weak vector-only hit can read the same number. With vectors active, rows carry `similarity`: the cosine (-1 to 1) of the query against that file's best-matching chunk (the same chunk the `lines` range points at). It orders vector evidence within a result set; the range it spans depends on the corpus and the embedding model, and compresses on small trees, where even a nonsense query has a moderately near neighbour somewhere. Compare similarities within a result set rather than against a fixed cutoff carried between trees.
40
+ - Absence evidence lives in the labels: `search --lexical` (or a semantic-off scope) returns 0 rows when the words are nowhere in the tree. Default `search` always returns up to `k` rows (nearest-neighbour search has a nearest neighbour for any input), so a result of only `via: vector` rows IS the absence signal for the words themselves; `similarity` and the snippet are the evidence for judging whether a vector row is a real conceptual hit.
41
+ - Besides SQL strings, a config entry can save a whole search: `"hot": { "search": "pricing OR billing", "preset": "raw", "k": 20 }` runs as `sense hot`. The scenario's settings ride along with the name, so repeat runs need no flags. An invocation-level `--preset`, `--k`, `--where`, or `--lexical` overrides the saved value; `--list` marks these entries `(search)`.
42
+ - `sense check` prepares every saved query and probes every saved search lexically with k=1, so a typo'd column, stale SQL, bad FTS5 syntax, or unknown preset fails at check time instead of silently mid-task. It reports row counts; whether an empty result is good or bad is the reader's judgment: a dead-link query returning rows means broken citations to fix, and the agent reads that directly.
107
43
 
108
44
  ## SQL
109
45
 
@@ -119,55 +55,22 @@ sense query "SELECT heading, start_line, tokens FROM sections WHERE path = ?" a.
119
55
  sense query "SELECT j.value, COUNT(*) n FROM frontmatter, json_each(frontmatter.tags) j GROUP BY j.value ORDER BY n DESC" # count per array member
120
56
  ```
121
57
 
122
- - `content MATCH` takes FTS5 syntax: `a OR b`, `"phrase"`, `pref*`, `NEAR(a b, 5)`,
123
- `summary: term`. Stemmed; markdown stripped at index time. Double-quote any term with
124
- punctuation — bare `customer-facing` errors (`-` reads as a column filter), bare
125
- apostrophes are syntax errors: write `"customer-facing"`, `"founder's"`.
126
- - Rank with `ORDER BY bm25(content, 10.0, 5.0, 1.0)` (title > summary > body); excerpt with
127
- `snippet(content, -1, '«', '»', '…', 10)`. snippet() re-tokenizes each matched doc and its
128
- cost grows superlinearly with doc size — measured ~10 s per query on a tree holding one
129
- 1 MB note. `search` bounds this itself (docs past 16 KB get an equivalent excerpt another
130
- way); in hand-written SQL, guard it: `CASE WHEN length(text) <= 16384 THEN snippet(...)
131
- END`, or select `title`/`summary` instead of an excerpt.
132
- - Select `content.title`/`content.summary` (always exist, empty when absent) rather than
133
- `f.title`/`f.summary` (discovered columns — error on trees that never declare them).
134
- - Frontmatter values keep their YAML type: strings are TEXT, whole numbers and booleans are
135
- INTEGER (`true` stores as 1, so `WHERE flag = 1` matches and `WHERE flag = 'true'` matches
136
- nothing), fractions are REAL, lists and maps are JSON text. `map` prints the observed type
137
- per field, and a field showing two types (`integer,text`) has drifted across notes.
138
- - `has(field, value)`: array membership on JSON-array fields, substring on strings, false on NULL
139
- — the `includes()` convention. Substring means `has(f.status, 'active')` also matches
140
- `inactive`; exact scalar match is `f.status = ?`, deliberate substring is `LIKE`, exact array
141
- membership is `EXISTS (SELECT 1 FROM json_each(f.tags) WHERE value = ?)`.
142
- To aggregate per member instead, use `json_each(frontmatter.<field>)` (above) -- GROUP BY on the
143
- raw column splits `["a","b"]` and `["b","a"]` into separate buckets.
144
- - Date fields are stored as written. Compare through `datetime()`, which normalizes ISO 8601
145
- timezone offsets to UTC: `WHERE datetime(created) >= datetime(?)`. Bare string comparison
146
- is only safe when every note uses the same offset.
147
- - To bound what a query puts into context: `snippet()` excerpts just the matching text,
148
- `LIMIT` caps row counts, and selecting `path`/`title`/`summary` keeps rows small.
149
- `SELECT text FROM content` returns the tree's entire prose (sense warns past 50 KB).
150
- Aggregates (`COUNT`, `GROUP BY`) are already bounded. `SELECT * FROM frontmatter` is always
151
- safe — prose is not a frontmatter column.
58
+ - `content MATCH` takes FTS5 syntax: `a OR b`, `"phrase"`, `pref*`, `NEAR(a b, 5)`, `summary: term`. Stemmed; markdown stripped at index time. Double-quote any term with punctuation. Bare `customer-facing` errors (`-` reads as a column filter), bare apostrophes are syntax errors: write `"customer-facing"`, `"founder's"`.
59
+ - Rank with `ORDER BY bm25(content, 10.0, 5.0, 1.0)` (title > summary > body); excerpt with `snippet(content, -1, '«', '»', '…', 10)`. snippet() re-tokenizes each matched doc and its cost grows superlinearly with doc size, measured ~10 s per query on a tree holding one 1 MB note. `search` bounds this itself (docs past 16 KB get an equivalent excerpt another way); in hand-written SQL, guard it: `CASE WHEN length(text) <= 16384 THEN snippet(...) END`, or select `title`/`summary` instead of an excerpt.
60
+ - Select `content.title`/`content.summary` (always exist, empty when absent) rather than `f.title`/`f.summary` (discovered columns; error on trees that never declare them).
61
+ - Frontmatter values keep their YAML type: strings are TEXT, whole numbers and booleans are INTEGER (`true` stores as 1, so `WHERE flag = 1` matches and `WHERE flag = 'true'` matches nothing), fractions are REAL, lists and maps are JSON text. `map` prints the observed type per field, and a field showing two types (`integer,text`) has drifted across notes.
62
+ - `has(field, value)`: array membership on JSON-array fields, substring on strings, false on NULL. This is the `includes()` convention. Substring means `has(f.status, 'active')` also matches `inactive`; exact scalar match is `f.status = ?`, deliberate substring is `LIKE`, exact array membership is `EXISTS (SELECT 1 FROM json_each(f.tags) WHERE value = ?)`. To aggregate per member instead, use `json_each(frontmatter.<field>)` (above) -- GROUP BY on the raw column splits `["a","b"]` and `["b","a"]` into separate buckets.
63
+ - Date fields are stored as written. Compare through `datetime()`, which normalizes ISO 8601 timezone offsets to UTC: `WHERE datetime(created) >= datetime(?)`. Bare string comparison is only safe when every note uses the same offset.
64
+ - To bound what a query puts into context: `snippet()` excerpts just the matching text, `LIMIT` caps row counts, and selecting `path`/`title`/`summary` keeps rows small. `SELECT text FROM content` returns the tree's entire prose (sense warns past 50 KB). Aggregates (`COUNT`, `GROUP BY`) are already bounded. `SELECT * FROM frontmatter` is always safe: prose is not a frontmatter column.
152
65
 
153
66
  Worked traces: [EXAMPLES.md](EXAMPLES.md).
154
67
 
155
68
  ## Setup and upkeep
156
69
 
157
- - Missing CLI: `npm install -g sensemaking`. Missing config: `sense init` at the tree root.
158
- Discovery walks up from cwd; `--config <path>` overrides. Setting up or restructuring a
159
- tree (presets, frontmatter conventions, note design) is the `sense-setup` skill.
160
- - `map` and `status` report each preset's coverage (files matched, embedded count) —
161
- indexing derives from presets, so the coverage numbers are how you see what a config
162
- actually indexes and embeds. A scope with fewer signals just uses fewer (a semantic-off
163
- preset searches lexically); a saved search naming an unknown preset errors at `check`.
70
+ - Missing CLI: `npm install -g sensemaking`. Missing config: `sense init` at the tree root. Discovery walks up from cwd; `--config <path>` overrides. Setting up or restructuring a tree (presets, frontmatter conventions, note design) is the `sense-setup` skill.
71
+ - `map` and `status` report each preset's coverage (files matched, embedded count). Indexing derives from presets, so the coverage numbers are how you see what a config actually indexes and embeds. A scope with fewer signals just uses fewer (a semantic-off preset searches lexically); a saved search naming an unknown preset errors at `check`.
164
72
  - Save a query into `sense.config.json` only when it will be reused; run ad-hoc otherwise.
165
- - A one-line `summary:` per note is optional and pays twice: it appears in result rows and is a
166
- weighted search field. Date comparisons work for dates written as ISO 8601 (`2026-08-12`, or
167
- with time and offset) — the only format `datetime()` parses. Field names in examples
168
- (`status`, `tags`, `created`) are illustrative; your tree defines its own.
169
- - Reserved frontmatter keys (dropped with a warning): `path`, `_mtime`, `_size`, `_rank`,
170
- `content`, `links`, `sections`.
171
- - Exit codes: `0` ok, `1` error (SQLite message verbatim), `2` usage (unknown query, wrong
172
- param count).
173
- - Doubted cache: `sense rebuild`. Rarely needed — every query reconciles first.
73
+ - A one-line `summary:` per note is optional and pays twice: it appears in result rows and is a weighted search field. Date comparisons work for dates written as ISO 8601 (`2026-08-12`, or with time and offset), the only format `datetime()` parses. Field names in examples (`status`, `tags`, `created`) are illustrative; your tree defines its own.
74
+ - Reserved frontmatter keys (dropped with a warning): `path`, `_mtime`, `_size`, `_rank`, `content`, `links`, `sections`.
75
+ - Exit codes: `0` ok, `1` error (SQLite message verbatim), `2` usage (unknown query, wrong param count).
76
+ - Doubted cache: `sense rebuild`. Rarely needed; every query reconciles first.
@@ -1,13 +1,10 @@
1
1
  # sense-setup: worked configurations
2
2
 
3
- Four tree shapes, each with its config and the commands an agent actually runs. Field
4
- names and folder names are illustrative — your tree defines its own.
3
+ Four tree shapes, each with its config and the commands an agent actually runs. Field names and folder names are illustrative; your tree defines its own.
5
4
 
6
5
  ## A. Compiled wiki over immutable sources (the llm-wiki pattern)
7
6
 
8
- `raw/` holds ingested sources — big, never hand-edited. `wiki/` holds agent-compiled
9
- pages — linked, curated. The human drops sources and asks questions; the agent compiles
10
- and cites.
7
+ `raw/` holds ingested sources: big, never hand-edited. `wiki/` holds agent-compiled pages: linked, curated. The human drops sources and asks questions; the agent compiles and cites.
11
8
 
12
9
  ```json
13
10
  {
@@ -31,17 +28,11 @@ sense search "rotary embeddings" --preset raw # cite from sources; lexical, k=
31
28
  sense dead-links # rows are broken citations to fix
32
29
  ```
33
30
 
34
- What the shape buys: bare search never ranks raw noise above compiled pages; raw pays no
35
- vector/link cost; the compile queue, stub list, and citation integrity are one saved
36
- query each. The maintenance loop is `uncompiled` → write the wiki page citing its sources
37
- → `dead-links` stays empty.
31
+ What the shape buys: bare search never ranks raw noise above compiled pages; raw pays no vector/link cost; the compile queue, stub list, and citation integrity are one saved query each. The maintenance loop is `uncompiled` → write the wiki page citing its sources → `dead-links` stays empty.
38
32
 
39
33
  ## B. Nightly agent memory, consolidated (the dreaming pattern)
40
34
 
41
- `memory/` accumulates small notes written at session end, each with `project`,
42
- `created`, and `kind` (observation / steer / decision) frontmatter. A consolidation agent
43
- runs periodically: prune, merge, surface contradictions for the human. Retired notes move
44
- to `archive/` — still queryable, no longer embedded or ranked.
35
+ `memory/` accumulates small notes written at session end, each with `project`, `created`, and `kind` (observation / steer / decision) frontmatter. A consolidation agent runs periodically: prune, merge, surface contradictions for the human. Retired notes move to `archive/`: still queryable, no longer embedded or ranked.
45
36
 
46
37
  ```json
47
38
  {
@@ -69,17 +60,11 @@ sense retirement # old + uncited → move to arch
69
60
  sense unfiled # rows are notes missing a project — file them
70
61
  ```
71
62
 
72
- Presets are structural (live vs archived); per-project filtering is metadata (`project = ?`)
73
- — one tree serves every project. Semantic search over the memory preset is the
74
- near-duplicate detector: search a new note's own summary and read `similarity` within the
75
- results. Whether an old steer was overridden is a question for the human — found by
76
- search, never decided by it.
63
+ Presets are structural (live vs archived); per-project filtering is metadata (`project = ?`). One tree serves every project. Semantic search over the memory preset is the near-duplicate detector: search a new note's own summary and read `similarity` within the results. Whether an old steer was overridden is a question for the human, found by search, never decided by it.
77
64
 
78
65
  ## C. Evidence corpus: claims trace to sources
79
66
 
80
- `sources/` (immutable imports), `notes/` (one reading note per source), `reviews/`
81
- (synthesis whose claims must cite notes). The same shape fits incident reports and
82
- postmortems, user research and findings, due diligence and memos.
67
+ `sources/` (immutable imports), `notes/` (one reading note per source), `reviews/` (synthesis whose claims must cite notes). The same shape fits incident reports and postmortems, user research and findings, due diligence and memos.
83
68
 
84
69
  ```json
85
70
  {
@@ -105,8 +90,7 @@ sense unread # the reading queue: sources no
105
90
 
106
91
  ## D. A plain vault: zero configuration
107
92
 
108
- Someone else's Obsidian vault, heterogeneous, no structure worth declaring — the
109
- `sense init` starter untouched. The workflow is discovery:
93
+ Someone else's Obsidian vault, heterogeneous with no structure worth declaring. The `sense init` starter is left untouched. The workflow is discovery:
110
94
 
111
95
  ```
112
96
  sense map # fields in use, hub notes, recent changes
@@ -115,7 +99,4 @@ sense query "SELECT j.value AS tag, COUNT(*) n FROM frontmatter, json_each(front
115
99
  sense peek "Plugins/dataview.md" # outline + links before reading
116
100
  ```
117
101
 
118
- Presets earn their place only when a tree has parts deserving different treatment; a tree
119
- that is one kind of thing needs none of the vocabulary above. On a big vault, raise
120
- `default`'s `k` and read `lines` ranges instead of whole files — or start from the
121
- starter's `large` preset.
102
+ Presets earn their place only when a tree has parts deserving different treatment; a tree that is one kind of thing needs none of the vocabulary above. On a big vault, raise `default`'s `k` and read `lines` ranges instead of whole files, or start from the starter's `large` preset.
@@ -1,32 +1,21 @@
1
1
  ---
2
2
  name: sense-setup
3
- description: Set up the sense CLI on a markdown tree and make the tree-design decisions that shape it — sense init, presets (which files, which settings, vectors on or off), and the trade-offs of frontmatter conventions, summaries, folder layout, and note size. Use when creating or restructuring a markdown knowledge base, running sense init, editing sense.config.json, configuring search scope or vectors, or deciding how notes should be written for an agent to query later.
3
+ description: Set up the sense CLI on a markdown tree and make the tree-design decisions that shape it: sense init, presets (which files, which settings, vectors on or off), and the trade-offs of frontmatter conventions, summaries, folder layout, and note size. Use when creating or restructuring a markdown knowledge base, running sense init, editing sense.config.json, configuring search scope or vectors, or deciding how notes should be written for an agent to query later.
4
4
  ---
5
5
 
6
6
  # sense: setup and tree design
7
7
 
8
- Querying an existing tree is the `sense` skill. This one covers making a tree:
9
- installing, writing presets, and the design decisions a tree owner faces.
10
- Worked configurations for common tree shapes: [EXAMPLES.md](EXAMPLES.md).
8
+ Querying an existing tree is the `sense` skill. This one covers making a tree: installing, writing presets, and the design decisions a tree owner faces. Worked configurations for common tree shapes: [EXAMPLES.md](EXAMPLES.md).
11
9
 
12
10
  ## Setup
13
11
 
14
- - `npm install -g sensemaking`, then `sense init` at the tree root writes
15
- `sense.config.json` — two presets (`default`, and `large` showing what a big
16
- vault tunes), everything on. Config discovery walks up from cwd;
17
- `--config <path>` overrides.
12
+ - `npm install -g sensemaking`, then `sense init` at the tree root writes `sense.config.json`: two presets (`default`, and `large` showing what a big vault tunes), everything on. Config discovery walks up from cwd; `--config <path>` overrides.
18
13
  - Globs resolve relative to the config file, never the cwd.
19
- - `sense status` and `sense map` show each preset's coverage (files matched,
20
- embedded count), so what a config actually indexes is always visible in
21
- output. A config edit that changes coverage rebuilds the cache and names the
22
- preset that caused it on stderr.
14
+ - `sense status` and `sense map` show each preset's coverage (files matched, embedded count), so what a config actually indexes is always visible in output. A config edit that changes coverage rebuilds the cache and names the preset that caused it on stderr.
23
15
 
24
16
  ## Presets
25
17
 
26
- A preset is a named, self-contained bundle of settings. `default` (required) is
27
- what bare commands use; every other preset is addressed by name
28
- (`sense search "..." --preset raw`, or `"preset": "raw"` in a saved search).
29
- No inheritance: what a preset states is all it does.
18
+ A preset is a named, self-contained bundle of settings. `default` (required) is what bare commands use; every other preset is addressed by name (`sense search "..." --preset raw`, or `"preset": "raw"` in a saved search). No inheritance: what a preset states is all it does.
30
19
 
31
20
  | field | means | default |
32
21
  |---|---|---|
@@ -35,76 +24,26 @@ No inheritance: what a preset states is all it does.
35
24
  | `semantic` | vectors for this preset's files and searches | on; only ever written as `false` |
36
25
  | `where` | a standing SQL filter on frontmatter | none |
37
26
 
38
- **Indexing derives from presets.** A file is indexed if any preset includes it;
39
- it is embedded if any covering preset has semantic on. Consequences worth
40
- designing around:
27
+ **Indexing derives from presets.** A file is indexed if any preset includes it; it is embedded if any covering preset has semantic on. Consequences worth designing around:
41
28
 
42
- - A layer of the tree covered only by a `semantic: false` preset (raw sources,
43
- archives, generated output) is fully searchable lexically and by SQL but
44
- costs no vector work — the main scale lever.
29
+ - A layer of the tree covered only by a `semantic: false` preset (raw sources, archives, generated output) is fully searchable lexically and by SQL but costs no vector work. This is the main scale lever.
45
30
  - Files no preset includes are not indexed at all.
46
31
  - Presets may overlap; they are views, not partitions.
47
- - The first semantic search embeds everything covered (progress on stderr;
48
- minutes on tens of thousands of notes, seconds on small trees). Vectors use
49
- the built-in static model unless a top-level
50
- `"embed": { "model", "type": "static"|"api", "url", "key" }` block points at
51
- a Model2Vec model, local path, or OpenAI-compatible endpoint. `static`
52
- handles paraphrase and reworded concepts; tight domain jargon ("heart
53
- attack" for "myocardial infarction") is where an `api` transformer model
54
- tends to do better — measured in BENCHMARKING.md, "Retrieval quality".
55
- - Global `features` (`links`, `sections`, `rank`) still toggle tree-wide;
56
- most trees never touch them.
32
+ - The first semantic search embeds everything covered (progress on stderr; minutes on tens of thousands of notes, seconds on small trees). Vectors use the built-in static model unless a top-level `"embed": { "model", "type": "static"|"api", "url", "key" }` block points at a Model2Vec model, local path, or OpenAI-compatible endpoint. `static` handles paraphrase and reworded concepts; tight domain jargon ("heart attack" for "myocardial infarction") is where an `api` transformer model tends to do better, measured in BENCHMARKING.md, "Retrieval quality".
33
+ - Global `features` (`links`, `sections`, `rank`) still toggle tree-wide; most trees never touch them.
57
34
 
58
- **Large vaults**: everything except the vector build is measured linear to
59
- 100k notes with no tuning (BENCHMARKING.md). The knobs that matter are `k`
60
- (more, smaller results — rows carry `lines` section ranges, so agents read
61
- sections, not files) and `semantic: false` on the layers that don't earn
62
- vectors.
35
+ **Large vaults**: everything except the vector build is measured linear to 100k notes with no tuning (BENCHMARKING.md). The knobs that matter are `k` (more, smaller results; rows carry `lines` section ranges, so agents read sections, not files) and `semantic: false` on the layers that don't earn vectors.
63
36
 
64
37
  ## Tree design decisions
65
38
 
66
- These belong to the tree's owner. sense works with any of them and reads no
67
- instruction files of its own; each choice only changes what queries can do.
68
-
69
- - **Frontmatter fields.** Columns are discovered per tree — whatever keys notes
70
- declare become queryable. Consistent fields across notes make SQL filters
71
- and named queries possible (`WHERE status = 'active'`). SQLite's compiled
72
- column limit (2,000; sqlite.org/limits.html) bounds distinct keys per tree —
73
- the crawl stops with an error naming the count and the levers. Reserved keys
74
- (dropped with a warning): `path`, `_mtime`, `_size`, `_rank`, `content`,
75
- `links`, `sections`. Values keep their YAML type: strings TEXT, whole numbers
76
- and booleans INTEGER (`true` is 1), fractions REAL, lists and maps JSON text;
77
- `map` prints the observed type per field.
78
- - **Presets are path-shaped; frontmatter is state-shaped.** A preset's coverage
79
- must be computable from the path alone (it decides indexing, baked into the
80
- cache). Volatile state (`status`, `project`, dates) lives in frontmatter and
81
- filters at query time (`where`, `has()`, `datetime()`). A state worth
82
- different *indexing* (retired memory, superseded sources) is a state worth
83
- moving the file — the archive-folder pattern in EXAMPLES.md.
84
- - **What a note omits is also a filter.** A layer that deliberately carries none
85
- of the fields the saved views filter on is excluded from all of them without
86
- any view naming the layer. Sparse fields cut both ways: less of the tree
87
- filters when you want breadth, and exactly this separation when layers
88
- differ in authority.
89
- - **Dates.** `datetime()` comparisons work for dates written as ISO 8601 —
90
- the only format it parses. A tree that mixes date formats can store them,
91
- but can't compare them in SQL.
92
- - **Summaries.** A one-line `summary:` is optional and pays twice: it shows in
93
- every result row (often answering a question with no file read) and is a
94
- weighted search field ranked above body text. The cost is writing and
95
- maintaining the line as notes change.
96
- - **Folder shape.** Globs find the files, paths are queryable text, links
97
- resolve by basename at any depth — but presets make folders meaningful:
98
- a folder is the natural unit that gets its own coverage and settings.
99
- - **Note size.** Many small notes: precise search hits, whole-file reads stay
100
- cheap, more links to maintain. Fewer large notes: `sections`, `peek`, and
101
- the `lines` column carry the cost down to line-range reads. Both work.
102
- - **Recurring questions.** Save a scenario an agent will repeat: a SQL string
103
- for filters and reports, or a saved search
104
- (`"hot": { "search": "...", "preset": "raw", "k": 5 }`). Either runs as
105
- `sense <name>`, and `sense check` validates both kinds against the real
106
- tree, so a broken saved scenario fails at check time, not mid-task.
107
- - **Where decisions live.** Choices that should outlive one conversation can
108
- be recorded in the agent's own instruction or skill files, or in a note in
109
- the tree itself; a one-off search over an existing corpus needs none of
110
- that.
39
+ These belong to the tree's owner. sense works with any of them and reads no instruction files of its own; each choice only changes what queries can do.
40
+
41
+ - **Frontmatter fields.** Columns are discovered per tree: whatever keys notes declare become queryable. Consistent fields across notes make SQL filters and named queries possible (`WHERE status = 'active'`). SQLite's compiled column limit (2,000; sqlite.org/limits.html) bounds distinct keys per tree. The crawl stops with an error naming the count and the levers. Reserved keys (dropped with a warning): `path`, `_mtime`, `_size`, `_rank`, `content`, `links`, `sections`. Values keep their YAML type: strings TEXT, whole numbers and booleans INTEGER (`true` is 1), fractions REAL, lists and maps JSON text; `map` prints the observed type per field.
42
+ - **Presets are path-shaped; frontmatter is state-shaped.** A preset's coverage must be computable from the path alone (it decides indexing, baked into the cache). Volatile state (`status`, `project`, dates) lives in frontmatter and filters at query time (`where`, `has()`, `datetime()`). A state worth different *indexing* (retired memory, superseded sources) is a state worth moving the file: the archive-folder pattern in EXAMPLES.md.
43
+ - **What a note omits is also a filter.** A layer that deliberately carries none of the fields the saved views filter on is excluded from all of them without any view naming the layer. Sparse fields cut both ways: less of the tree filters when you want breadth, and exactly this separation when layers differ in authority.
44
+ - **Dates.** `datetime()` comparisons work for dates written as ISO 8601, the only format it parses. A tree that mixes date formats can store them, but can't compare them in SQL.
45
+ - **Summaries.** A one-line `summary:` is optional and pays twice: it shows in every result row (often answering a question with no file read) and is a weighted search field ranked above body text. The cost is writing and maintaining the line as notes change.
46
+ - **Folder shape.** Globs find the files, paths are queryable text, links resolve by basename at any depth, but presets make folders meaningful: a folder is the natural unit that gets its own coverage and settings.
47
+ - **Note size.** Many small notes: precise search hits, whole-file reads stay cheap, more links to maintain. Fewer large notes: `sections`, `peek`, and the `lines` column carry the cost down to line-range reads. Both work.
48
+ - **Recurring questions.** Save a scenario an agent will repeat: a SQL string for filters and reports, or a saved search (`"hot": { "search": "...", "preset": "raw", "k": 5 }`). Either runs as `sense <name>`, and `sense check` validates both kinds against the real tree, so a broken saved scenario fails at check time, not mid-task.
49
+ - **Where decisions live.** Choices that should outlive one conversation can be recorded in the agent's own instruction or skill files, or in a note in the tree itself; a one-off search over an existing corpus needs none of that.
@@ -1,5 +0,0 @@
1
- export declare class ExitError extends Error {
2
- readonly code: number;
3
- constructor(code: number);
4
- }
5
- export declare function usageError(message: string, usage?: string): never;
@@ -1,5 +0,0 @@
1
- export declare class ExitError extends Error {
2
- readonly code: number;
3
- constructor(code: number);
4
- }
5
- export declare function usageError(message: string, usage?: string): never;
@@ -1,147 +0,0 @@
1
- // Commands never end the process themselves: they throw this, and cli.ts -- the one place
2
- // that owns the exit -- drains stdio through exit-compat and returns. Neither alternative
3
- // works here: a direct process.exit() races a pending write, and a bare return would let a
4
- // command body run on after --help had already printed its usage.
5
- "use strict";
6
- Object.defineProperty(exports, "__esModule", {
7
- value: true
8
- });
9
- function _export(target, all) {
10
- for(var name in all)Object.defineProperty(target, name, {
11
- enumerable: true,
12
- get: Object.getOwnPropertyDescriptor(all, name).get
13
- });
14
- }
15
- _export(exports, {
16
- get ExitError () {
17
- return ExitError;
18
- },
19
- get usageError () {
20
- return usageError;
21
- }
22
- });
23
- function _assert_this_initialized(self) {
24
- if (self === void 0) {
25
- throw new ReferenceError("this hasn't been initialised - super() hasn't been called");
26
- }
27
- return self;
28
- }
29
- function _call_super(_this, derived, args) {
30
- derived = _get_prototype_of(derived);
31
- return _possible_constructor_return(_this, _is_native_reflect_construct() ? Reflect.construct(derived, args || [], _get_prototype_of(_this).constructor) : derived.apply(_this, args));
32
- }
33
- function _class_call_check(instance, Constructor) {
34
- if (!(instance instanceof Constructor)) {
35
- throw new TypeError("Cannot call a class as a function");
36
- }
37
- }
38
- function _construct(Parent, args, Class) {
39
- if (_is_native_reflect_construct()) {
40
- _construct = Reflect.construct;
41
- } else {
42
- _construct = function construct(Parent, args, Class) {
43
- var a = [
44
- null
45
- ];
46
- a.push.apply(a, args);
47
- var Constructor = Function.bind.apply(Parent, a);
48
- var instance = new Constructor();
49
- if (Class) _set_prototype_of(instance, Class.prototype);
50
- return instance;
51
- };
52
- }
53
- return _construct.apply(null, arguments);
54
- }
55
- function _get_prototype_of(o) {
56
- _get_prototype_of = Object.setPrototypeOf ? Object.getPrototypeOf : function getPrototypeOf(o) {
57
- return o.__proto__ || Object.getPrototypeOf(o);
58
- };
59
- return _get_prototype_of(o);
60
- }
61
- function _inherits(subClass, superClass) {
62
- if (typeof superClass !== "function" && superClass !== null) {
63
- throw new TypeError("Super expression must either be null or a function");
64
- }
65
- subClass.prototype = Object.create(superClass && superClass.prototype, {
66
- constructor: {
67
- value: subClass,
68
- writable: true,
69
- configurable: true
70
- }
71
- });
72
- if (superClass) _set_prototype_of(subClass, superClass);
73
- }
74
- function _is_native_function(fn) {
75
- return Function.toString.call(fn).indexOf("[native code]") !== -1;
76
- }
77
- function _possible_constructor_return(self, call) {
78
- if (call && (_type_of(call) === "object" || typeof call === "function")) {
79
- return call;
80
- }
81
- return _assert_this_initialized(self);
82
- }
83
- function _set_prototype_of(o, p) {
84
- _set_prototype_of = Object.setPrototypeOf || function setPrototypeOf(o, p) {
85
- o.__proto__ = p;
86
- return o;
87
- };
88
- return _set_prototype_of(o, p);
89
- }
90
- function _type_of(obj) {
91
- "@swc/helpers - typeof";
92
- return obj && typeof Symbol !== "undefined" && obj.constructor === Symbol ? "symbol" : typeof obj;
93
- }
94
- function _wrap_native_super(Class) {
95
- var _cache = typeof Map === "function" ? new Map() : undefined;
96
- _wrap_native_super = function wrapNativeSuper(Class) {
97
- if (Class === null || !_is_native_function(Class)) return Class;
98
- if (typeof Class !== "function") {
99
- throw new TypeError("Super expression must either be null or a function");
100
- }
101
- if (typeof _cache !== "undefined") {
102
- if (_cache.has(Class)) return _cache.get(Class);
103
- _cache.set(Class, Wrapper);
104
- }
105
- function Wrapper() {
106
- return _construct(Class, arguments, _get_prototype_of(this).constructor);
107
- }
108
- Wrapper.prototype = Object.create(Class.prototype, {
109
- constructor: {
110
- value: Wrapper,
111
- enumerable: false,
112
- writable: true,
113
- configurable: true
114
- }
115
- });
116
- return _set_prototype_of(Wrapper, Class);
117
- };
118
- return _wrap_native_super(Class);
119
- }
120
- function _is_native_reflect_construct() {
121
- try {
122
- var result = !Boolean.prototype.valueOf.call(Reflect.construct(Boolean, [], function() {}));
123
- } catch (_) {}
124
- return (_is_native_reflect_construct = function() {
125
- return !!result;
126
- })();
127
- }
128
- var ExitError = /*#__PURE__*/ function(Error1) {
129
- "use strict";
130
- _inherits(ExitError, Error1);
131
- function ExitError(code) {
132
- _class_call_check(this, ExitError);
133
- var _this;
134
- _this = _call_super(this, ExitError, [
135
- "exit ".concat(code)
136
- ]);
137
- _this.code = code;
138
- return _this;
139
- }
140
- return ExitError;
141
- }(_wrap_native_super(Error));
142
- function usageError(message, usage) {
143
- console.error(message);
144
- if (usage !== undefined) console.error(usage);
145
- throw new ExitError(2);
146
- }
147
- /* CJS INTEROP */ if (exports.__esModule && exports.default) { try { Object.defineProperty(exports.default, '__esModule', { value: true }); for (var key in exports) { exports.default[key] = exports[key]; } } catch (_) {}; module.exports = exports.default; }
@@ -1 +0,0 @@
1
- {"version":3,"sources":["/Users/kevin/Dev/OpenSource/ai/sensemaking/src/cli/exit.ts"],"sourcesContent":["// Commands never end the process themselves: they throw this, and cli.ts -- the one place\n// that owns the exit -- drains stdio through exit-compat and returns. Neither alternative\n// works here: a direct process.exit() races a pending write, and a bare return would let a\n// command body run on after --help had already printed its usage.\nexport class ExitError extends Error {\n readonly code: number;\n constructor(code: number) {\n super(`exit ${code}`);\n this.code = code;\n }\n}\n\nexport function usageError(message: string, usage?: string): never {\n console.error(message);\n if (usage !== undefined) console.error(usage);\n throw new ExitError(2);\n}\n"],"names":["ExitError","usageError","code","Error","message","usage","console","error","undefined"],"mappings":"AAAA,0FAA0F;AAC1F,0FAA0F;AAC1F,2FAA2F;AAC3F,kEAAkE;;;;;;;;;;;;QACrDA;eAAAA;;QAQGC;eAAAA;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;AART,IAAA,AAAMD,0BAAN;;cAAMA;aAAAA,UAECE,IAAY;gCAFbF;;gBAGT,kBAHSA;YAGF,QAAY,OAALE;;QACd,MAAKA,IAAI,GAAGA;;;WAJHF;qBAAkBG;AAQxB,SAASF,WAAWG,OAAe,EAAEC,KAAc;IACxDC,QAAQC,KAAK,CAACH;IACd,IAAIC,UAAUG,WAAWF,QAAQC,KAAK,CAACF;IACvC,MAAM,IAAIL,UAAU;AACtB"}
@@ -1,5 +0,0 @@
1
- export declare class ExitError extends Error {
2
- readonly code: number;
3
- constructor(code: number);
4
- }
5
- export declare function usageError(message: string, usage?: string): never;
@@ -1,15 +0,0 @@
1
- // Commands never end the process themselves: they throw this, and cli.ts -- the one place
2
- // that owns the exit -- drains stdio through exit-compat and returns. Neither alternative
3
- // works here: a direct process.exit() races a pending write, and a bare return would let a
4
- // command body run on after --help had already printed its usage.
5
- export class ExitError extends Error {
6
- constructor(code){
7
- super(`exit ${code}`);
8
- this.code = code;
9
- }
10
- }
11
- export function usageError(message, usage) {
12
- console.error(message);
13
- if (usage !== undefined) console.error(usage);
14
- throw new ExitError(2);
15
- }
@@ -1 +0,0 @@
1
- {"version":3,"sources":["/Users/kevin/Dev/OpenSource/ai/sensemaking/src/cli/exit.ts"],"sourcesContent":["// Commands never end the process themselves: they throw this, and cli.ts -- the one place\n// that owns the exit -- drains stdio through exit-compat and returns. Neither alternative\n// works here: a direct process.exit() races a pending write, and a bare return would let a\n// command body run on after --help had already printed its usage.\nexport class ExitError extends Error {\n readonly code: number;\n constructor(code: number) {\n super(`exit ${code}`);\n this.code = code;\n }\n}\n\nexport function usageError(message: string, usage?: string): never {\n console.error(message);\n if (usage !== undefined) console.error(usage);\n throw new ExitError(2);\n}\n"],"names":["ExitError","Error","code","usageError","message","usage","console","error","undefined"],"mappings":"AAAA,0FAA0F;AAC1F,0FAA0F;AAC1F,2FAA2F;AAC3F,kEAAkE;AAClE,OAAO,MAAMA,kBAAkBC;IAE7B,YAAYC,IAAY,CAAE;QACxB,KAAK,CAAC,CAAC,KAAK,EAAEA,MAAM;QACpB,IAAI,CAACA,IAAI,GAAGA;IACd;AACF;AAEA,OAAO,SAASC,WAAWC,OAAe,EAAEC,KAAc;IACxDC,QAAQC,KAAK,CAACH;IACd,IAAIC,UAAUG,WAAWF,QAAQC,KAAK,CAACF;IACvC,MAAM,IAAIL,UAAU;AACtB"}