sensemaking 0.1.0 → 0.2.1

package/schema.json

@@ -19,7 +19,7 @@
       "type": "object",
       "additionalProperties": false,
       "required": ["include"],
-      "description": "Which files become rows in the `docs` table.",
+      "description": "Which files become rows in the `frontmatter` table.",
       "properties": {
         "include": {
           "type": "array",
@@ -31,7 +31,7 @@
     },
     "queries": {
       "type": "object",
-      "description": "Named SQL queries runnable as `sense <name> [params...]`. Each value is a SQL SELECT against the `docs` table (one row per file, one column per discovered frontmatter key, plus reserved `path`/`_mtime`/`_size`). `?` placeholders bind to CLI positional arguments in order. The custom `has(field, value)` function does array-membership on a JSON-array field, substring match on a string field, and is always false on NULL. The names `init`, `query`, `watch`, `status`, and `rebuild` are reserved subcommands — a query with one of those names is unreachable from the CLI.",
+      "description": "Named SQL queries runnable as `sense <name> [params...]`. Tables: `frontmatter` (one row per file, one column per discovered frontmatter key, plus `path`/`_mtime`/`_size`) and `content`, an FTS5 index (`title`, `summary`, `text`, `path`) for content search. `?` placeholders bind to CLI positional args in order. `has(field, value)`: array membership on a JSON-array field, substring match on a string, false on NULL. Canonical query: `SELECT f.path, content.title, content.summary, snippet(content, -1, '«', '»', '…', 10) AS hit FROM frontmatter f JOIN content ON content.path = f.path WHERE content MATCH ? ORDER BY bm25(content, 10.0, 5.0, 1.0) LIMIT 10`. Reserved frontmatter keys: `path`, `_mtime`, `_size`, `content`. Reserved query names (unreachable as subcommands): `init`, `query`, `watch`, `status`, `rebuild`.",
       "additionalProperties": { "type": "string" }
     }
   }

package/skills/sense/EXAMPLES.md

@@ -0,0 +1,115 @@
+# sense: worked examples
+
+Every query returns *references with evidence* — enough to decide which files to open, never the
+files themselves. Reading happens afterward, through the filesystem, on the paths that earned it.
+
+Token counts below are from a real 26-note vault (~62 KB of markdown): a search costs ~1–2% of
+reading the files it points at.
+
+## A. "Does the vault say anything about X?" (discovery search)
+
+The question an agent should ask *before* reading anything — cheap enough to run speculatively.
+
+```
+sense query "SELECT f.path, content.title, content.summary, snippet(content, -1, '«', '»', '…', 10) AS hit
+  FROM frontmatter f JOIN content ON content.path = f.path
+  WHERE content MATCH ?
+  ORDER BY bm25(content, 10.0, 5.0, 1.0) LIMIT 10" "compensation OR salary" --format json
+```
+
+```json
+[
+  {
+    "path": "knowledge/compensation-floor.md",
+    "title": "Compensation floor",
+    "summary": "The comp floor and how to apply it when screening",
+    "hit": "«Compensation» floor Kevin's «compensation» preference for the later job…"
+  },
+  {
+    "path": "methodology/jobs-table-schema.md",
+    "title": "Jobs table schema",
+    "summary": "",
+    "hit": "…B Company employer C «Salary» as posted; blank if not…"
+  }
+]
+```
+
+~55 tokens per row. Decide from `title`/`summary`/`hit`; often the row itself answers the
+question. If not, `Read knowledge/compensation-floor.md` (~800 tokens) — one file, not the vault
+(~16,000 tokens).
+
+Note the `content.title`/`content.summary` spelling: those columns always exist on `content`
+(empty string when a note lacks the key), so this query works on any vault. `f.title`/`f.summary`
+only work once some note actually declares the key — `frontmatter` columns are discovered, not fixed.
+
+## B. Filtering on frontmatter you already know (no content search)
+
+When the fields are known — typically because an agent wrote the notes to a schema — plain SQL on
+`frontmatter` is the whole query. No join needed when you aren't searching prose:
+
+```
+sense query "SELECT path, title, summary FROM frontmatter
+  WHERE status = 'active' AND has(track, ?) ORDER BY updated DESC" within-tech --format json
+```
+
+`has()` does array membership on JSON-array fields (`track: [a, b]`), substring on strings, false
+on missing keys. A note missing `summary` yields NULL — costs nothing, breaks nothing. (Only if
+*no* note in the vault declares a key does its column not exist at all — pattern D's pragma shows
+what's there.)
+
+## C. Filter + search + rank in one query
+
+The case neither grep nor a frontmatter-only tool can do: a hard constraint AND a relevance
+ranking, composed.
+
+```
+sense query "SELECT f.path, content.title, content.summary, snippet(content, -1, '«', '»', '…', 10) AS hit
+  FROM frontmatter f JOIN content ON content.path = f.path
+  WHERE f.status = 'active' AND has(f.track, 'within-tech') AND content MATCH ?
+  ORDER BY bm25(content, 10.0, 5.0, 1.0) LIMIT 10" remote --format json
+```
+
+The frontmatter conditions are hard filters (a non-active note can never appear); the `MATCH` +
+`bm25()` ranks whatever survives. The join exists **only** for `MATCH`/`bm25()`/`snippet()` —
+never use it to fetch text.
+
+## D. Cold start: a vault you didn't build
+
+```
+sense --list                                                                # named queries, if any
+sense query "SELECT name FROM pragma_table_info('frontmatter') ORDER BY name"  # discover the fields
+sense query "SELECT DISTINCT type FROM frontmatter"                         # discover a field's values
+sense query "SELECT count(*) AS n FROM frontmatter"                         # corpus size
+```
+
+Each costs a few dozen tokens and turns an unknown corpus into a queryable schema. From there,
+patterns A–C apply.
+
+## E. Search syntax worth knowing
+
+```
+"exact phrase"          # phrase match
+compensat*              # prefix
+summary: onboarding     # only match in the summary field
+title: retro OR text: retrospective
+NEAR(salary negotiate, 8)
+```
+
+Stemming is on: `negotiate` finds "negotiating". Markdown is stripped from the index, so search
+for the words, not the syntax around them.
+
+## F. Anti-patterns
+
+```
+sense query "SELECT text FROM content"          # dumps every note into context
+```
+This is the one query shape that defeats the tool's purpose. `sense` warns on stderr when a
+result exceeds 50 KB (stdout stays clean for --format json), but the fix is upstream: select
+`path` + `snippet()`, keep a `LIMIT`, and `Read` the files that deserve it.
+
+Other traps:
+- **No `LIMIT`** on a search — fine on a tiny vault, a context bomb on a big one. Habit: always.
+- **Fetching content through SQL** because the join is there. The join ranks; the filesystem
+  retrieves. An agent already has `Read` and `grep` for the path the query returned.
+- **Adding a named query for a one-off question.** `sense query "<sql>"` exists so the config
+  only accumulates queries that are genuinely reused.

package/skills/sense/SKILL.md

@@ -1,85 +1,104 @@
 ---
 name: sense
-description: Query a markdown tree's YAML frontmatter with SQL via the sense CLI. Use when the user wants to query, filter, count, or report on a folder of markdown notes by their frontmatter fields, when a directory has a sense.config.json, when asked what named queries exist, or when asked to add a new query to one.
+description: Query a markdown tree with SQL via the sense CLI — filter notes by YAML frontmatter, then full-text search inside them. Use when the user wants to query, filter, count, search, or report on a folder of markdown notes, when you need to find which notes discuss a topic before reading them, when a directory has a sense.config.json, when asked what named queries exist, or when asked to add a new query to one.
 ---
 
 # sense
 
-`sense` runs SQL over the frontmatter of a tree of markdown files. Each file is a row in a
-`docs` table (one column per frontmatter key); the file's own content is never queried, only its
-frontmatter. A running app is **never** required — files on disk are the only source of truth.
+`sense` runs SQL over a tree of markdown files. Two tables, two facets of the same document:
 
-## Before anything else
+| table         | one row per file holds…                                               | for…              |
+|---------------|------------------------------------------------------------------------|-------------------|
+| `frontmatter` | one column per frontmatter key (+ reserved `path`, `_mtime`, `_size`)  | filtering         |
+| `content`     | FTS5 index: `title`, `summary`, `text` (+ `path` to join on)           | searching/ranking |
+
+No running app required — every query re-checks the filesystem first, so results are never stale.
 
-Confirm `sense` is installed (`sense --list` or `which sense`; if missing, `npm install -g sense`)
-and that a config exists. Discovery walks **up** from cwd looking for `sense.config.json`, same as
-git looks for `.git` — you don't need to `cd` to the project root, and `--config <path>` bypasses
-discovery entirely. If no config exists yet in the tree you're being asked to query, run
-`sense init` at its root to write a starter config, then tailor the queries.
+## The flow: query → decide → Read
 
-## See what's queryable
+A query result is *references with evidence*, never file contents: `path`, `title`, `summary` (if
+present), `hit` (matching excerpt) — enough to decide whether a file is worth opening at ~2% of
+the token cost of reading it. Then `Read` the one or two paths that matter.
 
 ```
-sense --list
+sense query "SELECT f.path, content.title, content.summary, snippet(content, -1, '«', '»', '…', 10) AS hit
+  FROM frontmatter f JOIN content ON content.path = f.path
+  WHERE content MATCH ?
+  ORDER BY bm25(content, 10.0, 5.0, 1.0) LIMIT 10" "compensation OR equity" --format json
 ```
 
-Prints the named queries defined in `sense.config.json`, sorted. Read that file directly if you
-need to see the SQL, not just the names.
+Select `title`/`summary` from `content`, where they always exist (empty when a note lacks the
+key) — on `frontmatter` they're discovered columns, present only if some note declares them, so
+`f.summary` errors on a vault with no summaries yet. Add frontmatter conditions
+(`f.status = 'active' AND has(f.track, ?)`) to the same WHERE — the filter and the search compose
+in one query. Worked traces for the common cases (discovery,
+known-field filtering, cold start, anti-patterns): [EXAMPLES.md](EXAMPLES.md).
 
-## Run a query
+## Before anything else
+
+Confirm `sense` is installed (`sense --list` or `which sense`; if missing,
+`npm install -g sensemaking`) and that a config exists. Discovery walks **up** from cwd looking for
+`sense.config.json`, same as git looks for `.git` — you don't need to `cd` to the project root, and
+`--config <path>` bypasses discovery entirely. If no config exists yet, `sense init` at the tree's
+root writes a minimal one (globs only, no queries — ad-hoc `sense query` works immediately).
+
+## Cold start: discover what's queryable
 
 ```
-sense <name> [params...] --format json
+sense --list                                                              # named queries, if any
+sense query "SELECT name FROM pragma_table_info('frontmatter') ORDER BY name"  # what frontmatter fields exist
+sense query "SELECT DISTINCT status FROM frontmatter"                      # what values a field takes
 ```
 
-**Prefer `--format json` when consuming output as an agent** — it's structured and avoids parsing
-a padded text table. `--format table` (the default) is for humans at a terminal.
+If you wrote the notes, you already know the fields; the pragma is for vaults you didn't build.
 
-Named queries may contain `?` placeholders; positional arguments after the name bind to them in
-order. The parameter count is checked strictly — passing the wrong number of arguments is a usage
-error (exit 2), never a silent empty result.
+## Run a query
 
 ```
-sense by-tag urgent --format json
+sense <name> [params...] --format json     # named query from sense.config.json
+sense query "<sql>" [params...]            # ad-hoc SQL, no config edit
 ```
 
-## One-off questions: `query`, not config edits
+**Prefer `--format json` when consuming output as an agent.** Positional args bind to `?`
+placeholders in order, count-checked strictly (wrong count = exit 2, never a silent empty result).
+Save a query into `sense.config.json` (plain JSON, edit directly) only when it's meant to be
+reused as a named view — never add-then-remove one for a one-off question.
 
-For an ad-hoc question, run SQL directly — do NOT add a temporary query to
-`sense.config.json` and remove it afterward:
+## Search syntax and ranking
 
-```
-sense query "SELECT path FROM docs WHERE has(phase, 'screen') AND NOT has(phase, 'explore')" --format json
-sense query "SELECT path FROM docs WHERE has(tags, ?)" urgent --format json
-```
+- `content MATCH ?` takes FTS5 syntax: `a OR b`, `a AND b`, `"exact phrase"`, `pref*`,
+  `NEAR(a b, 5)`, column-scoped `summary: onboarding`.
+- `bm25(content, 10.0, 5.0, 1.0)` — weights follow column order (title, summary, text), so a
+  title hit outranks a passing mention. Lower is better; plain `ORDER BY bm25(…)` sorts best-first.
+- `snippet(content, -1, '«', '»', '…', 10)` — bounded excerpt; `-1` picks whichever column
+  matched; the last argument is the excerpt budget in tokens.
+- Stemming is on (`negotiate` matches "negotiating"); markdown syntax is stripped at index time,
+  so snippets are clean prose and `**bold**` matches `bold`.
 
-Save a query into `sense.config.json` only when it's meant to be reused as a named view.
+## Keep results small
 
-## `has()` semantics
+- Select `path, title, summary` + a `snippet()` — never `SELECT text FROM content`, which dumps
+  the whole tree into context (`sense` warns on stderr past 50 KB).
+- Always `LIMIT`. Ten rows is plenty; widen only if they all look wrong.
+- `SELECT * FROM frontmatter` is safe — prose is deliberately not a `frontmatter` column.
 
-The one custom SQL function, for frontmatter fields that are arrays or free text:
+## When writing notes, not just querying them
 
-| field type              | `has(field, value)` means      |
-|--------------------------|---------------------------------|
-| JSON array (e.g. `tags`) | array membership                |
-| string                   | substring match                 |
-| NULL (key absent)        | always false                    |
+Give every note a one-line `summary:` in its frontmatter — what's on the page and when it's worth
+opening, like a skill's `description:`. It pays twice: it appears in result rows (often answering
+the question with no file read at all), and it's a weighted search field. Keep it to one line.
 
-## Adding a named query
+Reserved frontmatter key names (dropped with a warning): `path`, `_mtime`, `_size`, `content`.
 
-Edit `sense.config.json` directly — it's plain JSON, not something sense itself writes:
+## `has()` semantics
 
-```json
-{
-  "scan": { "include": ["**/*.md"] },
-  "queries": {
-    "my-query": "SELECT path, title FROM docs WHERE has(tags, ?) ORDER BY path"
-  }
-}
-```
-`scan.include` globs and query results are both relative to the config file's own directory,
-regardless of your cwd. Reserved columns (don't use as frontmatter key names): `path`, `_mtime`,
-`_size`.
+The one custom SQL function, for frontmatter fields that are arrays or free text:
+
+| field type               | `has(field, value)` means |
+|--------------------------|---------------------------|
+| JSON array (e.g. `tags`) | array membership          |
+| string                   | substring match           |
+| NULL (key absent)        | always false              |
 
 ## Exit codes
 
@@ -88,8 +107,6 @@ regardless of your cwd. Reserved columns (don't use as frontmatter key names): `
 
 ## When results look stale
 
-`sense rebuild` deletes the local `.sense/` cache and re-crawls every file from scratch. Every
-query already reconciles the cache against the filesystem on open, so this is rarely needed — use
-it if you doubt the cache rather than trying to debug it. `sense status` shows the doc count, db
-path, and whether a background `sense watch` process is running (it's an optional pre-warmer, not
-a correctness requirement).
+`sense rebuild` deletes the local `.sense/` cache and re-crawls from scratch. Rarely needed since
+every query reconciles on open — use it if you doubt the cache. `sense status` shows doc count, db
+path, and whether a background `sense watch` (optional pre-warmer) is running.