sensemaking 0.24.6 → 0.25.0

package/schema.json

@@ -12,7 +12,7 @@
     },
     "version": {
       "type": "integer",
-      "enum": [1, 2, 3, 4, 5],
+      "enum": [1, 2, 3, 4, 5, 6],
       "description": "Config format version. Older versions are auto-migrated (and the file rewritten) on load; a version newer than this sense build supports makes it exit with an error rather than misinterpret the file. Omit to default to 1."
     },
     "root": {
@@ -20,6 +20,11 @@
       "minLength": 1,
       "description": "Optional markdown tree root. Relative paths resolve against the directory containing sense.config.json; omitted means that directory, preserving colocated-config behavior. Preset globs and stored paths are relative to this root, while .sense state remains beside the config."
     },
+    "build": {
+      "type": "boolean",
+      "default": true,
+      "description": "CLI query-time incremental build default. True prepares the capabilities each query needs; false reads the existing index without scanning. --no-build overrides true for one query. Explicit sense build and sense watch always prepare the index. Library open(config, options) uses its own build option."
+    },
     "presets": {
       "type": "object",
       "description": "File selection, index-time, and named search defaults. A preset is self-contained: no inheritance between presets, and no ordering -- presets may overlap freely (they are views, not partitions), and a preset whose globs match nothing is valid. A file is indexed iff any preset's include/exclude covers it (union across presets); `default` is used when a command names no preset, and must be declared. A file is embedded when the top-level `embed` block names a model and some covering preset's `signals` include `vectors`. `sense init` generates `default` plus one `large` example.",
@@ -113,7 +118,7 @@
     },
     "queries": {
       "type": "object",
-      "description": "Saved queries runnable as `sense <name> [params...]`, each naming the verb it runs, one to one with the two commands: `{ sql }` runs like `sense sql`, `{ search }` like `sense search`. A bare string is rejected -- it silently meant SQL. `{ sql }`: `?` placeholders bind to CLI positional args in order; deterministic and enumerating, including raw FTS5 `MATCH` for word search under your own SQL -- \"0 rows = not in the tree\" lives here. Tables: `frontmatter` (one row per file, one column per discovered frontmatter key, plus `path`/`_mtime`/`_size`/`_rank`/`_parse_error`, the last being NULL when the frontmatter parsed and the YAML message when it did not, in which case no other column is populated), `content` (FTS5: `title`, `summary`, `text`, `path`, plus machine-written `title_seg`/`summary_seg`/`text_seg` sidecars holding the grapheme phrases for Chinese, Japanese, Thai, Khmer, Lao, and Burmese text -- present for matching, not reading; a hand-written `MATCH` reaches them through `segment()`), `links` (`src`, `target`, `dst`), `sections` (`path`, `idx`, `level`, `heading`, `start_line`, `end_line`, `tokens`), and `preset_files` (`preset`, `path`) -- which presets cover which files. `has(field, value)`: array membership on a JSON-array field, substring match on a string (so has(f.status, 'active') also matches 'inactive'), false on NULL. `segment(terms)`: rewrites a run of unspaced-script text in `terms` into the ordered grapheme phrase the sidecar columns need; text with no such run passes through unchanged, so it is safe to add to any query. Exact matches: `=` for scalars, `EXISTS (SELECT 1 FROM json_each(f.tags) WHERE value = ?)` for array members. Canonical query: `SELECT f.path, content.title, content.summary, CASE WHEN length(content.text) <= 16384 THEN snippet(content, 2, '«', '»', '…', 10) END AS hit FROM frontmatter f JOIN content ON content.path = f.path WHERE content MATCH ? ORDER BY bm25(content, 10.0, 5.0, 1.0, 0, 10.0, 5.0, 1.0) LIMIT 10`. snippet() names column 2 (`text`) explicitly rather than -1 (best column), since -1 could surface a sidecar as the excerpt. bm25's full form repeats the three weights onto the sidecars so a title hit reached through `title_seg` ranks like one reached through `title`; the plain `bm25(content, 10.0, 5.0, 1.0)` still runs (FTS5 defaults unnamed columns to 1.0) but ranks a sidecar match at body weight. The CASE bounds snippet(), which re-tokenizes each matched document and costs seconds per query once a tree holds a megabyte-scale note; `search` applies the same bound internally. Saved search object: one text driving every engine the scoped preset has -- word match, links, vectors -- fused into one ranked list, `via` labeling which engine produced each row; `search` text must be non-empty (a saved query saves a question -- a scope without one is just flags). `preset` names one declared preset (defaults to `default`); `include` is an ad hoc glob scope that replaces the preset's include/exclude entirely, same as `search --include`; `where` and `k` behave like the `search` command's flags. `sense <name>` behaves like `sense search <search> [--preset] [--include] [--where] [--k]` with zero flags; it takes no positional parameters. Running an entry validates it: a typo'd column or unknown preset errors and exits nonzero, and a parameterised entry validates with any argument, since SQL is prepared before parameters bind. Nothing asserts on the result itself; a returned row set is the reader's judgment. Reserved frontmatter keys: `path`, `_mtime`, `_size`, `_rank`, `_parse_error`, `content`, `links`, `sections`. Reserved names (unreachable as saved queries, any shape): `init`, `sql`, `search`, `map`, `peek`, `path`, `related`, `download`, `watch`, `status`.",
+      "description": "Saved queries runnable as `sense <name> [params...]`, each naming the verb it runs, one to one with the two commands: `{ sql }` runs like `sense sql`, `{ search }` like `sense search`. A bare string is rejected -- it silently meant SQL. `{ sql }`: `?` placeholders bind to CLI positional args in order; deterministic and enumerating, including raw FTS5 `MATCH` for word search under your own SQL -- \"0 rows = not in the tree\" lives here. Tables: `frontmatter` (one row per file, one column per discovered frontmatter key, plus `path`/`_mtime`/`_size`/`_rank`/`_parse_error`, the last being NULL when the frontmatter parsed and the YAML message when it did not, in which case no other column is populated), `content` (FTS5: `title`, `summary`, `text`, `path`, plus machine-written `title_seg`/`summary_seg`/`text_seg` sidecars holding the grapheme phrases for Chinese, Japanese, Thai, Khmer, Lao, and Burmese text -- present for matching, not reading; a hand-written `MATCH` reaches them through `segment()`), `links` (`src`, `target`, `dst`), `sections` (`path`, `idx`, `level`, `heading`, `start_line`, `end_line`, `tokens`), and `preset_files` (`preset`, `path`) -- which presets cover which files. `has(field, value)`: array membership on a JSON-array field, substring match on a string (so has(f.status, 'active') also matches 'inactive'), false on NULL. `segment(terms)`: rewrites a run of unspaced-script text in `terms` into the ordered grapheme phrase the sidecar columns need; text with no such run passes through unchanged, so it is safe to add to any query. Exact matches: `=` for scalars, `EXISTS (SELECT 1 FROM json_each(f.tags) WHERE value = ?)` for array members. Canonical query: `SELECT f.path, content.title, content.summary, CASE WHEN length(content.text) <= 16384 THEN snippet(content, 2, '«', '»', '…', 10) END AS hit FROM frontmatter f JOIN content ON content.path = f.path WHERE content MATCH ? ORDER BY bm25(content, 10.0, 5.0, 1.0, 0, 10.0, 5.0, 1.0) LIMIT 10`. snippet() names column 2 (`text`) explicitly rather than -1 (best column), since -1 could surface a sidecar as the excerpt. bm25's full form repeats the three weights onto the sidecars so a title hit reached through `title_seg` ranks like one reached through `title`; the plain `bm25(content, 10.0, 5.0, 1.0)` still runs (FTS5 defaults unnamed columns to 1.0) but ranks a sidecar match at body weight. The CASE bounds snippet(), which re-tokenizes each matched document and costs seconds per query once a tree holds a megabyte-scale note; `search` applies the same bound internally. Saved search object: one text driving every engine the scoped preset has -- word match, links, vectors -- fused into one ranked list, `via` labeling which engine produced each row; `search` text must be non-empty (a saved query saves a question -- a scope without one is just flags). `preset` names one declared preset (defaults to `default`); `include` is an ad hoc glob scope that replaces the preset's include/exclude entirely, same as `search --include`; `where` and `k` behave like the `search` command's flags. `sense <name>` behaves like `sense search <search> [--preset] [--include] [--where] [--k]` with zero flags; it takes no positional parameters. Running an entry validates it: a typo'd column or unknown preset errors and exits nonzero, and a parameterised entry validates with any argument, since SQL is prepared before parameters bind. Nothing asserts on the result itself; a returned row set is the reader's judgment. Reserved frontmatter keys: `path`, `_mtime`, `_size`, `_rank`, `_parse_error`, `content`, `links`, `sections`. Reserved names (unreachable as saved queries, any shape): `init`, `build`, `sql`, `search`, `map`, `peek`, `path`, `related`, `download`, `watch`, `status`.",
       "additionalProperties": {
         "oneOf": [
           {

package/skills/sense/SKILL.md

@@ -5,7 +5,7 @@ description: "Query a markdown tree with the sense CLI: filter notes by frontmat
 
 # sense
 
-Use sense to locate evidence in a markdown tree before reading files. Every command reconciles changed files first. Results contain paths, metadata, excerpts, and line ranges. Read the returned files or ranges when the task needs their prose.
+Use sense to locate evidence in a markdown tree before reading files. With the config default `"build": true`, CLI queries incrementally scan the configured tree for the capabilities they need; vector preparation is limited to the eligible scope. Config `"build": false` or a one-query `--no-build` reads the last completed indexed generation, checks requested readiness and does not scan live files. Explicit build and watch prepare it regardless of that default. Results contain paths, metadata, excerpts, and line ranges. Read the returned files or ranges when the task needs their prose.
 
 Setup, store selection, presets, and note design belong to the `sense-setup` skill. Translating an Obsidian Bases file belongs to `sense-bases`.
 
@@ -133,5 +133,5 @@ Worked command traces are in [EXAMPLES.md](EXAMPLES.md).
 
 - Install a missing CLI with `npm install -g sensemaking`.
 - `sense status` prints the cache path and watcher state.
-- Delete the cache directory printed by `sense status` only when the derived index is in doubt. The next command rebuilds it.
-- Use `sense watch` when another process should keep the index warm during frequent edits. Queries remain responsible for their own freshness check.
+- Use `sense build --force` to recreate a derived index whose state is in doubt; source files, configuration, and the watcher ownership claim remain in place.
+- Use `sense watch` when another process should keep all configured capabilities prepared during frequent edits. A default query scans the configured tree for its needed capabilities and limits vector preparation to eligible paths; `--no-build` reads the completed generation only.

package/skills/sense-bases/SKILL.md

@@ -144,5 +144,4 @@ via `GROUP_CONCAT`, all from one bound path.
 - `columnSize`, card/table/list/map chrome: pixel layout over the same rows.
 - Obsidian choosing `this` from the active pane: there is no pane; the path parameter is the
   resolution.
-- Live re-evaluation as files change: every sense query reconciles against the filesystem
-  first, so the freshness is already there; nothing re-renders on its own.
+- Live re-evaluation as files change: default CLI queries incrementally prepare the needed capabilities; `--no-build` reads the last completed generation and requires its requested readiness without scanning live files. Nothing re-renders on its own.

package/skills/sense-setup/SKILL.md

@@ -86,7 +86,7 @@ Vectors require two choices. The top-level `embed` block names the model and pro
 }
 ```
 
-The first vector search downloads a named static model and embeds the covered notes. `sense download` fetches the model earlier when CI, offline work, or timing makes that useful. A config change that alters the model, vector coverage, or chunking can rebuild vectors.
+The config's `"build": true` default lets CLI queries prepare their needed capabilities. Set `"build": false` for manual-build or watch workflows. The first build-enabled vector search downloads a named static model and embeds its eligible scope. `sense build` and `sense watch` prepare all configured capabilities regardless of this setting; `sense download` fetches the model earlier when CI, offline work, or timing makes that useful. A config change that alters the model, vector coverage, or chunking can rebuild vectors. Config `"build": false` or `--no-build` requires the requested vector scope to be ready and does not read newer live files.
 
 Read [embedding setup](references/embeddings.md) when choosing a provider or model, supporting a non-English tree, changing chunk size, or tuning signal weights.
 

package/skills/sense-setup/references/embeddings.md

@@ -20,7 +20,7 @@ For a static model, inspect current Model2Vec models and their cards rather than
 
 ## Download and rebuild behavior
 
-Naming a remote static model in the config authorizes its download. `sense download` performs that fetch before the first query. The first vector-participating search then embeds the indexed notes.
+Naming a remote static model in the config authorizes its download. `sense download` performs that fetch before a query. A default vector query embeds its eligible scope; `sense build` and `sense watch` prepare all configured vector capabilities, while `--no-build` requires the requested scope to be ready.
 
 Changing the model changes the vector space and rebuilds embeddings. Changing preset coverage can also add, remove, or rebuild vector rows. Settle the broad scope before embedding a large tree.
 

package/skills/sense-setup/references/store-benchmarks.md

@@ -1,16 +1,16 @@
-<!-- sense-store-benchmark release=0.24.6 -->
+<!-- sense-store-benchmark release=0.25.0 -->
 # Current store benchmark summary
 
-The release assessment generated this file for store selection. Release `0.24.6` passed on 2026-09-16, measured on Apple M4 Pro with Node v26.7.0.
+The release assessment generated this file for store selection. Release `0.25.0` passed on 2026-09-20, measured on Apple M4 Pro with Node v26.8.2.
 
 The timing rows ran on the same 6,566-note tree. They include CLI startup and each store's complete selected path. Ranked candidates and downstream work can differ by store, so these are current operating measurements rather than an isolated database-engine contest.
 
 | Store | Cold index | Warm count | Lexical search | Semantic search | Portable semantic nDCG@10 |
 |---|---|---|---|---|---|
-| sqlite | 1,471 ms | 143 ms | 192 ms | 318 ms | 0.3306 |
-| duckdb | 2,149 ms | 162 ms | 279 ms | 399 ms | 0.3284 |
-| turso | 2,727 ms | 138 ms | 237 ms | 584 ms | 0.3307 |
+| sqlite | 1,197 ms | 126 ms | 190 ms | 338 ms | 0.3306 |
+| duckdb | 1,801 ms | 164 ms | 297 ms | 415 ms | 0.3284 |
+| turso | 2,056 ms | 135 ms | 232 ms | 640 ms | 0.3307 |
 
-Cold index is the first `status` that builds the cache. Warm count is a no-change `COUNT(*)` query. Lexical and semantic search are steady-state `sense search` commands. Lower timing is faster. Higher nDCG@10 is better; that quality column uses the same NFCorpus queries, judgments, result count, and model on every store.
+Cold index measures core index creation and a count query, excluding document embeddings. Warm count is a no-change `COUNT(*)` query. Lexical and semantic search are steady-state `sense search` commands. Lower timing is faster. Higher nDCG@10 is better; that quality column uses the same NFCorpus queries, judgments, result count, and model on every store.
 
 Choose from the intended workflow, capabilities, and SQL compatibility. These numbers describe the current Sense implementations, not a permanent ranking of the engines. Treat small timing or relevance differences as diagnostic unless a representative workload for the target tree reproduces them.