sensemaking 0.27.1 → 0.28.1

package/skills/sense-setup/references/embeddings.md

@@ -1,51 +0,0 @@
-# Embedding setup
-
-Read this guide when choosing an embedding provider or model, supporting a non-English tree, changing chunk size, or tuning signal weights.
-
-## Provider choice
-
-`static` loads a local Model2Vec model in JavaScript. The default model is `minishlab/potion-retrieval-32M`. A Hugging Face model id downloads into `~/.sense/models` on first vector use. A filesystem path points at a directory that already contains `model.safetensors` and `tokenizer.json`.
-
-`openai` calls an OpenAI-compatible `/embeddings` endpoint. Local Ollama and LM Studio servers use this protocol. A hosted endpoint sends note text to that service.
-
-`cohere` calls Cohere's native embedding endpoint. The `key` setting names the environment variable that contains the credential. Note text leaves the machine.
-
-## Model choice
-
-The default static model is English-only. For another language, choose a model whose published card declares that language. A static Model2Vec model must have compatible safetensors and tokenizer files with supported floating-point weights.
-
-Sense checks declared model languages against the indexed tree. A clear mismatch raises `EMBED_MODEL_MISMATCH`. A model card with no language declaration cannot be checked, so confirm it before indexing.
-
-For a static model, inspect current Model2Vec models and their cards rather than relying on a fixed list in this skill. For a local HTTP provider, inspect the model catalog exposed by the chosen runtime. Model availability and model cards change independently of sense releases.
-
-## Download and rebuild behavior
-
-Naming a remote static model in the config authorizes its download. `sense download` performs that fetch before a query. A default vector query embeds its eligible scope; `sense build` and `sense watch` prepare all configured vector capabilities, while `--no-build` requires the requested scope to be ready.
-
-Changing the provider, model, or effective endpoint invalidates stored vector values while preserving the indexed chunks. Changing `embed.chunkTokens` rebuilds affected embedding rows, and changing preset coverage can add, remove, or rebuild them. A build-enabled command performs this work; config `"build": false` or `--no-build` reports incompatible or unprepared state without rebuilding or repairing the Sense index. Settle the broad scope before embedding a large tree.
-
-## Chunk size
-
-`embed.chunkTokens` sets the estimated token ceiling for chunks. The default is 500. Sense splits at headings and then applies the ceiling to long sections. Lower it for an embedding model with a smaller useful context window. Raising it trades fewer vectors for less precise line ranges and more text per vector.
-
-## Signal weights
-
-A preset includes a signal by naming it:
-
-```json
-"signals": { "words": 1, "links": 1, "vectors": 1 }
-```
-
-The number is the signal's reciprocal-rank weight. It is not a boolean. Weight 1 gives equal contribution. A higher vector weight can help a strong encoder on some corpora and hurt another tree whose exact vocabulary carries the answer.
-
-Test representative questions from the actual tree before changing weights. Compare returned paths and evidence labels, not only the fused score.
-
-## Layers without vectors
-
-An `embed` block makes vectors available to the tree. A preset can still omit the vector signal:
-
-```json
-"signals": { "words": 1, "links": 1 }
-```
-
-Use that for layers where exact wording matters more than paraphrase, such as raw sources, archives, generated logs, or citation corpora. Those files can remain indexed without paying the embedding cost for that preset's scope.

package/skills/sense-setup/references/store-benchmarks.md

@@ -1,16 +0,0 @@
-<!-- sense-store-benchmark release=0.27.0 -->
-# Current store benchmark summary
-
-The release assessment generated this file for store selection. Release `0.27.0` passed on 2026-10-03, measured on Apple M1 Pro with Node v26.10.0.
-
-The timing rows ran on the same 6,566-note tree. They include CLI startup and each store's complete selected path. Ranked candidates and downstream work can differ by store, so these are current operating measurements rather than an isolated database-engine contest.
-
-| Store | Cold index | Warm count | Lexical search | Semantic search | Portable semantic nDCG@10 |
-|---|---|---|---|---|---|
-| sqlite | 2,254 ms | 260 ms | 350 ms | 576 ms | 0.3306 |
-| duckdb | 3,026 ms | 319 ms | 510 ms | 695 ms | 0.3282 |
-| turso | 3,273 ms | 279 ms | 412 ms | 1,081 ms | 0.3299 |
-
-Cold index measures core index creation and a count query, excluding document embeddings. Warm count is a no-change `COUNT(*)` query. Lexical and semantic search are steady-state `sense search` commands. Lower timing is faster. Higher nDCG@10 is better; that quality column uses the same NFCorpus queries, judgments, result count, and model on every store.
-
-Choose from the intended workflow, capabilities, and SQL compatibility. These numbers describe the current Sense implementations, not a permanent ranking of the engines. Treat small timing or relevance differences as diagnostic unless a representative workload for the target tree reproduces them.

package/skills/sense-setup/references/stores/duckdb.md

@@ -1,28 +0,0 @@
-# Choose DuckDB
-
-Choose DuckDB when the cache file is an analytical artifact that another DuckDB tool will query, when saved SQL needs DuckDB's types and functions, or when the cache belongs beside large local or cloud datasets.
-
-DuckDB is designed for analytical SQL and supports larger-than-memory work by spilling to disk. Its SQL closely follows PostgreSQL while keeping documented differences and extensions. See the official [DuckDB performance guide](https://duckdb.org/docs/current/guides/performance/how_to_tune_workloads) and [SQL introduction](https://duckdb.org/docs/current/sql/introduction).
-
-Sense's [store benchmark summary](../store-benchmarks.md) measures the current adapter paths on a named markdown tree. It does not measure DuckDB's general analytical ceiling. Use it when current Sense latency matters to the choice.
-
-## What the choice gives you
-
-- A standard `.sense/cache.duckdb` database that DuckDB tools can open.
-- Frontmatter in `VARIANT` columns, so lists, maps, booleans, numbers, and strings retain useful native shapes.
-- Embeddings in native fixed-width float arrays.
-- DuckDB SQL for downstream analysis.
-- Compatibility with external DuckDB workflows. MotherDuck supports hybrid queries over local and cloud data, although Sense does not connect, upload, or replicate the cache for you. See [MotherDuck's hybrid execution overview](https://motherduck.com/research/motherduck-duckdb-in-the-cloud-and-in-the-client/).
-
-## Current Sense adapter boundaries
-
-- DuckDB support is experimental.
-- The first open installs `@duckdb/node-api`, with a native download of about 110 MB.
-- Under DuckDB's [concurrency model](https://duckdb.org/docs/current/connect/concurrency), Sense opens an existing cache for no-build observation in read-only mode, so several independent reader processes can coexist. A build-enabled command or watcher opens it read-write and excludes every other process until it closes. Contenders wait within Sense's bounded lock budget, then fail with `STORE_BUSY`. This cross-process rule does not promise writer/reader coexistence or unrestricted sharing by independently configured DuckDB instances in one process.
-- `sense search` accepts bare words and quoted phrases. It rejects FTS5 boolean, prefix, `NEAR`, initial-token, grouping, and column-filter syntax.
-- SQLite's raw `MATCH`, `snippet()`, and `bm25()` queries do not port.
-- Mixed-type `VARIANT` fields can require explicit casts in predicates.
-
-Commands, public tables, `has()`, `basename()`, `segment()`, vectors, and `sense watch` remain available. Read the [DuckDB query guide](../../../sense/references/stores/duckdb.md) before translating saved SQL or advanced searches.
-
-Changing to DuckDB builds its separate cache. It does not migrate the SQLite or Turso cache.

package/skills/sense-setup/references/stores/sqlite.md

@@ -1,21 +0,0 @@
-# Choose SQLite
-
-SQLite is the default recommendation for a sense tree.
-
-Choose it when the tree has no engine-specific requirement, when saved searches use FTS5 operators, or when people and agents may run sense commands at the same time.
-
-## What the choice gives you
-
-- No optional store package. Sense uses Node's built-in SQLite.
-- The full documented word-search grammar, including boolean, prefix, `NEAR`, initial-token, grouping, and column-filter expressions.
-- Raw FTS5 SQL through `MATCH`, `bm25()`, and `snippet()`.
-- The sense SQL functions `has()`, `basename()`, and `segment()`.
-- WAL-backed concurrent sense commands.
-
-The cache is `.sense/cache.db`. It is derived data and can be rebuilt from the markdown files.
-
-## When another store fits the workflow
-
-Use DuckDB when downstream DuckDB tools need to query the cache, the tree needs DuckDB's analytical SQL and native types, or the cache belongs beside large datasets or in a local/cloud workflow. Use Turso when its embedded Rust engine or Tantivy index fits the intended integration. Compare current measurements in [the store benchmark summary](../store-benchmarks.md) when Sense performance affects the decision.
-
-If existing queries use FTS5 syntax, moving away from SQLite requires rewriting those queries. Read the selected engine's query guide under the `sense` skill before changing the config.

package/skills/sense-setup/references/stores/turso.md

@@ -1,25 +0,0 @@
-# Choose Turso
-
-Choose Turso when the tree benefits from the embedded Turso Database engine or its Tantivy text index, or when evaluating the engine as the Sense adapter evolves. The Sense adapter is experimental; the engine is a fast-moving Rust rewrite of SQLite whose trade-offs can change between releases.
-
-Upstream Turso offers async I/O and concurrent writes in its local engine, plus sync through a separate SDK. Sense currently opens one local cache through `@tursodatabase/database` and serializes Sense commands against that file. It does not configure sync, remote access, or encryption. Read [Turso's current SDK guide](https://docs.turso.tech/sdk/introduction) to distinguish engine capabilities from the adapter features Sense exposes.
-
-## What the choice gives you
-
-- A `.sense/cache.turso.db` file using the embedded Turso engine.
-- Tantivy-backed lexical ranking with shared stemming and n-gram support for unspaced scripts.
-- The asynchronous JavaScript database client used by sense's store adapter.
-- Native vector storage and search.
-
-## Current Sense adapter boundaries
-
-- Turso support is experimental.
-- The first open installs the optional native package.
-- Turso holds the cache file for the life of a connection. Concurrent sense commands wait for the current command or watcher cycle to close it.
-- `sense search` accepts bare words and quoted phrases. It rejects FTS5 boolean, prefix, `NEAR`, initial-token, grouping, and column-filter syntax.
-- SQLite's raw `MATCH`, `snippet()`, and `bm25()` queries do not port.
-- `has()` and `basename()` work through SQL rewriting. `segment()` is unavailable.
-
-Commands, public tables, vectors, and `sense watch` remain available. Read the [Turso query guide](../../../sense/references/stores/turso.md) before translating saved SQL or advanced searches.
-
-Changing to Turso builds its separate cache. It does not migrate the SQLite or DuckDB cache. Review the shipped [store benchmark summary](../store-benchmarks.md) if performance affects the choice.