sensemaking 0.2.1 → 0.3.0

package/schema.json

@@ -1,7 +1,7 @@
 {
   "$schema": "http://json-schema.org/draft-07/schema#",
   "title": "sense.config.json",
-  "description": "Config for sense -- SQL over the frontmatter of a markdown tree.",
+  "description": "Config for sense -- SQL over the frontmatter, content, links, and structure of a markdown tree.",
   "type": "object",
   "additionalProperties": false,
   "required": ["scan", "queries"],
@@ -12,8 +12,8 @@
     },
     "version": {
       "type": "integer",
-      "enum": [1],
-      "description": "Config format version. Bumped only on breaking changes to this file's shape. Omit to default to 1; a version newer than this sense build supports makes it exit with an error rather than misinterpret the file."
+      "enum": [1, 2],
+      "description": "Config format version. Older versions are auto-migrated (and the file rewritten) on load; a version newer than this sense build supports makes it exit with an error rather than misinterpret the file. Omit to default to 1."
     },
     "scan": {
       "type": "object",
@@ -29,9 +29,28 @@
         }
       }
     },
+    "features": {
+      "type": "object",
+      "additionalProperties": false,
+      "description": "Independently toggleable features. Absent block or key means enabled; set false to disable. Toggling a feature rebuilds the cache. Verbs degrade rather than fail: `find` without links is BM25-only, `map` without rank omits hubs, `peek` without sections omits the outline.",
+      "properties": {
+        "links": {
+          "type": "boolean",
+          "description": "Extract wikilinks and relative markdown links into the `links` table (src, target as written, dst resolved path or NULL for dead links). Powers backlinks, graph expansion in `find`, and `peek`'s link lists."
+        },
+        "sections": {
+          "type": "boolean",
+          "description": "Extract headings into the `sections` table (path, idx, level, heading, start_line, end_line, tokens estimate). Powers `peek`'s outline so agents can read line ranges instead of whole files."
+        },
+        "rank": {
+          "type": "boolean",
+          "description": "PageRank over resolved links into `frontmatter._rank` at reconcile -- a static importance prior. Powers `map`'s hub list; usable in any ORDER BY. Requires `links`."
+        }
+      }
+    },
     "queries": {
       "type": "object",
-      "description": "Named SQL queries runnable as `sense <name> [params...]`. Tables: `frontmatter` (one row per file, one column per discovered frontmatter key, plus `path`/`_mtime`/`_size`) and `content`, an FTS5 index (`title`, `summary`, `text`, `path`) for content search. `?` placeholders bind to CLI positional args in order. `has(field, value)`: array membership on a JSON-array field, substring match on a string, false on NULL. Canonical query: `SELECT f.path, content.title, content.summary, snippet(content, -1, '«', '»', '…', 10) AS hit FROM frontmatter f JOIN content ON content.path = f.path WHERE content MATCH ? ORDER BY bm25(content, 10.0, 5.0, 1.0) LIMIT 10`. Reserved frontmatter keys: `path`, `_mtime`, `_size`, `content`. Reserved query names (unreachable as subcommands): `init`, `query`, `watch`, `status`, `rebuild`.",
+      "description": "Named SQL queries runnable as `sense <name> [params...]`. Tables: `frontmatter` (one row per file, one column per discovered frontmatter key, plus `path`/`_mtime`/`_size`/`_rank`), `content` (FTS5: `title`, `summary`, `text`, `path`), `links` (`src`, `target`, `dst`), and `sections` (`path`, `idx`, `level`, `heading`, `start_line`, `end_line`, `tokens`). `?` placeholders bind to CLI positional args in order. `has(field, value)`: array membership on a JSON-array field, substring match on a string, false on NULL. Canonical query: `SELECT f.path, content.title, content.summary, snippet(content, -1, '«', '»', '…', 10) AS hit FROM frontmatter f JOIN content ON content.path = f.path WHERE content MATCH ? ORDER BY bm25(content, 10.0, 5.0, 1.0) LIMIT 10`. Reserved frontmatter keys: `path`, `_mtime`, `_size`, `_rank`, `content`, `links`, `sections`. Reserved query names (unreachable as subcommands): `init`, `query`, `find`, `map`, `peek`, `watch`, `status`, `rebuild`.",
       "additionalProperties": { "type": "string" }
     }
   }

package/skills/sense/EXAMPLES.md

@@ -1,115 +1,80 @@
 # sense: worked examples
 
-Every query returns *references with evidence* — enough to decide which files to open, never the
-files themselves. Reading happens afterward, through the filesystem, on the paths that earned it.
+Outputs below are illustrative. Every result is a reference; reading happens afterward, through
+the filesystem, on the paths that earned it.
 
-Token counts below are from a real 26-note vault (~62 KB of markdown): a search costs ~1–2% of
-reading the files it points at.
-
-## A. "Does the vault say anything about X?" (discovery search)
-
-The question an agent should ask *before* reading anything — cheap enough to run speculatively.
+## A. "Does the vault say anything about X?"
 
 ```
-sense query "SELECT f.path, content.title, content.summary, snippet(content, -1, '«', '»', '…', 10) AS hit
-  FROM frontmatter f JOIN content ON content.path = f.path
-  WHERE content MATCH ?
-  ORDER BY bm25(content, 10.0, 5.0, 1.0) LIMIT 10" "compensation OR salary" --format json
+sense find "pricing OR billing OR invoicing" --k 10 --format json
 ```
 
 ```json
 [
-  {
-    "path": "knowledge/compensation-floor.md",
-    "title": "Compensation floor",
-    "summary": "The comp floor and how to apply it when screening",
-    "hit": "«Compensation» floor Kevin's «compensation» preference for the later job…"
-  },
-  {
-    "path": "methodology/jobs-table-schema.md",
-    "title": "Jobs table schema",
-    "summary": "",
-    "hit": "…B Company employer C «Salary» as posted; blank if not…"
-  }
+  { "path": "notes/pricing-model.md", "title": "Pricing model",
+    "summary": "Tiered per-seat pricing; floor and discount rules", "hit": "«Pricing» floor for annual…", "via": "match+link", "score": 0.0333 },
+  { "path": "notes/renewal-playbook.md", "title": "Renewal playbook",
+    "summary": "Renewal sequence and owners", "hit": "…«billing» contact confirms the PO…", "via": "match", "score": 0.0313 }
 ]
 ```
 
-~55 tokens per row. Decide from `title`/`summary`/`hit`; often the row itself answers the
-question. If not, `Read knowledge/compensation-floor.md` (~800 tokens) — one file, not the vault
-(~16,000 tokens).
-
-Note the `content.title`/`content.summary` spelling: those columns always exist on `content`
-(empty string when a note lacks the key), so this query works on any vault. `f.title`/`f.summary`
-only work once some note actually declares the key — `frontmatter` columns are discovered, not fixed.
+~30 tokens per row; often the `summary` answers the question with no read at all. A row with
+`via: "link"` never contained the terms — it is linked from notes that did, usually a signal
+worth following, not noise.
 
-## B. Filtering on frontmatter you already know (no content search)
-
-When the fields are known — typically because an agent wrote the notes to a schema — plain SQL on
-`frontmatter` is the whole query. No join needed when you aren't searching prose:
+## B. Known-field filtering (no search)
 
 ```
-sense query "SELECT path, title, summary FROM frontmatter
-  WHERE status = 'active' AND has(track, ?) ORDER BY updated DESC" within-tech --format json
+sense query "SELECT path, title, status FROM frontmatter WHERE status = 'active' AND has(tags, ?)" pricing --format json
 ```
 
-`has()` does array membership on JSON-array fields (`track: [a, b]`), substring on strings, false
-on missing keys. A note missing `summary` yields NULL — costs nothing, breaks nothing. (Only if
-*no* note in the vault declares a key does its column not exist at all — pattern D's pragma shows
-what's there.)
-
-## C. Filter + search + rank in one query
+Plain SQL on discovered columns. Combine with search by joining `content` and adding
+`AND content MATCH ?` — filter and rank in one query.
 
-The case neither grep nor a frontmatter-only tool can do: a hard constraint AND a relevance
-ranking, composed.
+## C. Structure before reading
 
 ```
-sense query "SELECT f.path, content.title, content.summary, snippet(content, -1, '«', '»', '…', 10) AS hit
-  FROM frontmatter f JOIN content ON content.path = f.path
-  WHERE f.status = 'active' AND has(f.track, 'within-tech') AND content MATCH ?
-  ORDER BY bm25(content, 10.0, 5.0, 1.0) LIMIT 10" remote --format json
+sense peek notes/architecture-decisions.md
 ```
 
-The frontmatter conditions are hard filters (a non-active note can never appear); the `MATCH` +
-`bm25()` ranks whatever survives. The join exists **only** for `MATCH`/`bm25()`/`snippet()` —
-never use it to fetch text.
-
-## D. Cold start: a vault you didn't build
-
 ```
-sense --list                                                                # named queries, if any
-sense query "SELECT name FROM pragma_table_info('frontmatter') ORDER BY name"  # discover the fields
-sense query "SELECT DISTINCT type FROM frontmatter"                         # discover a field's values
-sense query "SELECT count(*) AS n FROM frontmatter"                         # corpus size
+notes/architecture-decisions.md  (~4400 tokens)
+  title: Architecture decisions
+sections:
+  ## Storage layer — why SQLite  [L41-88, ~610t]
+  ## Queue — rejected options  [L89-120, ~380t]
+  ...
+links out (7): notes/pricing-model.md, ...
+backlinks (2): notes/_index.md, notes/roadmap.md
 ```
 
-Each costs a few dozen tokens and turns an unknown corpus into a queryable schema. From there,
-patterns A–C apply.
+The note is ~4,400 tokens; the peek is ~500. If only one section matters, `Read` its line range
+(~400 tokens) — a tenth of the file.
 
-## E. Search syntax worth knowing
+## D. The graph
 
 ```
-"exact phrase"          # phrase match
-compensat*              # prefix
-summary: onboarding     # only match in the summary field
-title: retro OR text: retrospective
-NEAR(salary negotiate, 8)
+sense query "SELECT src FROM links WHERE dst = ?" notes/pricing-model.md    # who cites this
+sense query "SELECT src, target FROM links WHERE dst IS NULL"               # dead links
+sense query "SELECT path, round(_rank*100,2) r FROM frontmatter ORDER BY _rank DESC LIMIT 5"  # load-bearing notes
 ```
 
-Stemming is on: `negotiate` finds "negotiating". Markdown is stripped from the index, so search
-for the words, not the syntax around them.
+`sense map` prints the hub list without SQL; use these when you need it filtered or joined.
 
-## F. Anti-patterns
+## E. Cold start on an unknown vault
 
 ```
-sense query "SELECT text FROM content"          # dumps every note into context
+sense map                          # fields, hubs, recent — read this first
+sense --list                       # named queries someone already saved
+sense query "SELECT DISTINCT type FROM frontmatter"    # what a field's values are
 ```
-This is the one query shape that defeats the tool's purpose. `sense` warns on stderr when a
-result exceeds 50 KB (stdout stays clean for --format json), but the fix is upstream: select
-`path` + `snippet()`, keep a `LIMIT`, and `Read` the files that deserve it.
-
-Other traps:
-- **No `LIMIT`** on a search — fine on a tiny vault, a context bomb on a big one. Habit: always.
-- **Fetching content through SQL** because the join is there. The join ranks; the filesystem
-  retrieves. An agent already has `Read` and `grep` for the path the query returned.
-- **Adding a named query for a one-off question.** `sense query "<sql>"` exists so the config
-  only accumulates queries that are genuinely reused.
+
+## Anti-patterns
+
+| don't | because | instead |
+|---|---|---|
+| `SELECT text FROM content` | dumps the vault into context | `snippet(content, -1, '«', '»', '…', 10)` |
+| `sense find "pricing"` | one word misses synonyms | `"pricing OR billing OR invoicing"` |
+| `Read` a 4,000-token file for one section | 10× the tokens needed | `peek` first, `Read` the line range |
+| queries without `LIMIT` | unbounded output | `LIMIT 10`, widen only if all rows look wrong |
+| save every query to config | config churn for one-offs | ad-hoc `sense query`; save only reusable views |

package/skills/sense/SKILL.md

@@ -1,112 +1,78 @@
 ---
 name: sense
-description: Query a markdown tree with SQL via the sense CLI — filter notes by YAML frontmatter, then full-text search inside them. Use when the user wants to query, filter, count, search, or report on a folder of markdown notes, when you need to find which notes discuss a topic before reading them, when a directory has a sense.config.json, when asked what named queries exist, or when asked to add a new query to one.
+description: Query a markdown tree with the sense CLI — filter notes by frontmatter, full-text search the prose, follow wikilinks/backlinks, and read note outlines. Use when the user wants to query, filter, count, search, or report on a folder of markdown notes, when you need to find which notes discuss a topic before reading them, when you want a note's backlinks or structure, when a directory has a sense.config.json, or when asked to add a named query to one.
 ---
 
 # sense
 
-`sense` runs SQL over a tree of markdown files. Two tables, two facets of the same document:
+SQL over a markdown tree, kept fresh by a filesystem check on every query. Four tables per file:
+`frontmatter` (one column per key, plus `path`/`_mtime`/`_size`/`_rank`), `content` (FTS5:
+`title`, `summary`, `text`), `links` (`src`, `target`, `dst` — `NULL` dst = dead link),
+`sections` (heading outline with line ranges and token estimates).
 
-| table         | one row per file holds…                                               | for…              |
-|---------------|------------------------------------------------------------------------|-------------------|
-| `frontmatter` | one column per frontmatter key (+ reserved `path`, `_mtime`, `_size`)  | filtering         |
-| `content`     | FTS5 index: `title`, `summary`, `text` (+ `path` to join on)           | searching/ranking |
+## The descent
 
-No running app required — every query re-checks the filesystem first, so results are never stale.
+Spend tokens in this order (progressive disclosure: metadata first, payloads just in time); go
+only as deep as the question needs.
 
-## The flow: query → decide → Read
+1. `sense map` — orient once: fields, hub notes, recent changes. Fixed-size output.
+2. `sense find "<terms>"` — locate: ranked references with excerpts, ~30 tokens/row.
+3. `sense peek <path>` — structure before reading: outline with `[L143-162, ~380t]` ranges,
+   links both ways. ~17% the cost of reading the file.
+4. `Read` the line range peek gave you — not the whole file.
 
-A query result is *references with evidence*, never file contents: `path`, `title`, `summary` (if
-present), `hit` (matching excerpt) — enough to decide whether a file is worth opening at ~2% of
-the token cost of reading it. Then `Read` the one or two paths that matter.
+Every result is a reference, never file contents. Prefer `--format json` when consuming output.
 
-```
-sense query "SELECT f.path, content.title, content.summary, snippet(content, -1, '«', '»', '…', 10) AS hit
-  FROM frontmatter f JOIN content ON content.path = f.path
-  WHERE content MATCH ?
-  ORDER BY bm25(content, 10.0, 5.0, 1.0) LIMIT 10" "compensation OR equity" --format json
-```
-
-Select `title`/`summary` from `content`, where they always exist (empty when a note lacks the
-key) — on `frontmatter` they're discovered columns, present only if some note declares them, so
-`f.summary` errors on a vault with no summaries yet. Add frontmatter conditions
-(`f.status = 'active' AND has(f.track, ?)`) to the same WHERE — the filter and the search compose
-in one query. Worked traces for the common cases (discovery,
-known-field filtering, cold start, anti-patterns): [EXAMPLES.md](EXAMPLES.md).
-
-## Before anything else
-
-Confirm `sense` is installed (`sense --list` or `which sense`; if missing,
-`npm install -g sensemaking`) and that a config exists. Discovery walks **up** from cwd looking for
-`sense.config.json`, same as git looks for `.git` — you don't need to `cd` to the project root, and
-`--config <path>` bypasses discovery entirely. If no config exists yet, `sense init` at the tree's
-root writes a minimal one (globs only, no queries — ad-hoc `sense query` works immediately).
-
-## Cold start: discover what's queryable
+## Verbs
 
 ```
-sense --list                                                              # named queries, if any
-sense query "SELECT name FROM pragma_table_info('frontmatter') ORDER BY name"  # what frontmatter fields exist
-sense query "SELECT DISTINCT status FROM frontmatter"                      # what values a field takes
+sense find "pricing OR billing OR invoicing" --where "f.status = 'active'" --k 10
+sense peek notes/pricing-model.md               # a unique basename also works
+sense map
+sense query "<sql>" [params...]                 # ad-hoc SQL; ? binds positional args, count-checked
+sense <name> [params...]                        # named query from sense.config.json
+sense --list | status | rebuild
 ```
 
-If you wrote the notes, you already know the fields; the pragma is for vaults you didn't build.
+- **Expand terms before searching.** Write `pricing OR billing OR invoicing`, not one word — you
+  know the synonyms; the index only knows the words in the files.
+- **Over-fetch, then choose.** `--k 20`, read the rows, open the 2–3 that matter.
+- `find` fuses BM25 with link-graph expansion; the `via` column says what produced each row —
+  `match` (terms hit), `link` (connected to notes that hit), `match+link` (both).
+- `--where` takes a frontmatter condition against alias `f`, e.g. `"f.status = 'active' AND has(f.tags, 'x')"`.
 
-## Run a query
+## SQL, when the verbs aren't enough
 
 ```
-sense <name> [params...] --format json     # named query from sense.config.json
-sense query "<sql>" [params...]            # ad-hoc SQL, no config edit
+sense query "SELECT name FROM pragma_table_info('frontmatter')"          # what fields exist
+sense query "SELECT DISTINCT status FROM frontmatter"                    # what values a field takes
+sense query "SELECT src FROM links WHERE dst = ?" notes/pricing-model.md  # backlinks
+sense query "SELECT src, target FROM links WHERE dst IS NULL"            # dead links
+sense query "SELECT path FROM frontmatter WHERE path NOT IN (SELECT dst FROM links WHERE dst IS NOT NULL)"  # orphans
+sense query "SELECT heading, start_line, tokens FROM sections WHERE path = ?" a.md   # budget a read
 ```
 
-**Prefer `--format json` when consuming output as an agent.** Positional args bind to `?`
-placeholders in order, count-checked strictly (wrong count = exit 2, never a silent empty result).
-Save a query into `sense.config.json` (plain JSON, edit directly) only when it's meant to be
-reused as a named view — never add-then-remove one for a one-off question.
-
-## Search syntax and ranking
-
-- `content MATCH ?` takes FTS5 syntax: `a OR b`, `a AND b`, `"exact phrase"`, `pref*`,
-  `NEAR(a b, 5)`, column-scoped `summary: onboarding`.
-- `bm25(content, 10.0, 5.0, 1.0)` — weights follow column order (title, summary, text), so a
-  title hit outranks a passing mention. Lower is better; plain `ORDER BY bm25(…)` sorts best-first.
-- `snippet(content, -1, '«', '»', '…', 10)` — bounded excerpt; `-1` picks whichever column
-  matched; the last argument is the excerpt budget in tokens.
-- Stemming is on (`negotiate` matches "negotiating"); markdown syntax is stripped at index time,
-  so snippets are clean prose and `**bold**` matches `bold`.
-
-## Keep results small
-
-- Select `path, title, summary` + a `snippet()` — never `SELECT text FROM content`, which dumps
-  the whole tree into context (`sense` warns on stderr past 50 KB).
-- Always `LIMIT`. Ten rows is plenty; widen only if they all look wrong.
-- `SELECT * FROM frontmatter` is safe — prose is deliberately not a `frontmatter` column.
-
-## When writing notes, not just querying them
-
-Give every note a one-line `summary:` in its frontmatter — what's on the page and when it's worth
-opening, like a skill's `description:`. It pays twice: it appears in result rows (often answering
-the question with no file read at all), and it's a weighted search field. Keep it to one line.
-
-Reserved frontmatter key names (dropped with a warning): `path`, `_mtime`, `_size`, `content`.
-
-## `has()` semantics
-
-The one custom SQL function, for frontmatter fields that are arrays or free text:
-
-| field type               | `has(field, value)` means |
-|--------------------------|---------------------------|
-| JSON array (e.g. `tags`) | array membership          |
-| string                   | substring match           |
-| NULL (key absent)        | always false              |
-
-## Exit codes
-
-`0` success · `1` a real error (bad config, SQL error — the SQLite message is printed verbatim) ·
-`2` usage error (missing query name, unknown query name, wrong parameter count).
-
-## When results look stale
-
-`sense rebuild` deletes the local `.sense/` cache and re-crawls from scratch. Rarely needed since
-every query reconciles on open — use it if you doubt the cache. `sense status` shows doc count, db
-path, and whether a background `sense watch` (optional pre-warmer) is running.
+- `content MATCH` takes FTS5 syntax: `a OR b`, `"phrase"`, `pref*`, `NEAR(a b, 5)`,
+  `summary: term`. Stemmed; markdown stripped at index time.
+- Rank with `ORDER BY bm25(content, 10.0, 5.0, 1.0)` (title > summary > body); excerpt with
+  `snippet(content, -1, '«', '»', '…', 10)`.
+- Select `content.title`/`content.summary` (always exist, empty when absent) rather than
+  `f.title`/`f.summary` (discovered columns — error on vaults that never declare them).
+- `has(field, value)`: array membership on JSON-array fields, substring on strings, false on NULL.
+- Never `SELECT text FROM content` — that is the whole vault into context. `SELECT * FROM
+  frontmatter` is safe; prose is not a frontmatter column. Always `LIMIT`.
+
+Worked traces: [EXAMPLES.md](EXAMPLES.md).
+
+## Setup and upkeep
+
+- Missing CLI: `npm install -g sensemaking`. Missing config: `sense init` at the tree root.
+  Discovery walks up from cwd; `--config <path>` overrides.
+- Save a query into `sense.config.json` only when it will be reused; run ad-hoc otherwise.
+- When writing notes, give each a one-line `summary:` — it appears in every result row and is a
+  weighted search field.
+- Reserved frontmatter keys (dropped with a warning): `path`, `_mtime`, `_size`, `_rank`,
+  `content`, `links`, `sections`.
+- Exit codes: `0` ok, `1` error (SQLite message verbatim), `2` usage (unknown query, wrong
+  param count).
+- Doubted cache: `sense rebuild`. Rarely needed — every query reconciles first.