@ataraxy-labs/sem 0.15.0 → 0.16.0

package/CHANGELOG.md

@@ -4,6 +4,106 @@ All notable changes to sem are documented in this file.
 
 ## [Unreleased]
 
+### Performance
+
+- **Token-efficient tool output**: the same answers at a fraction of the tokens the consuming model has to read (and pay for). (1) The context packer stops enumerating noise: related test entities are folded into per-role counts instead of packed as one-line "#[test]" stubs (unless the target itself is a test, when its test neighborhood is the question), bare attribute/comment signatures are skipped, and transitive tiers are capped at 25 entries per role with the remainder counted. What was dropped is stated explicitly in one line ("not packed: +64 direct dependents (64 tests) · sem_impact lists them"), so the signal survives at a fraction of the cost. Measured: `sem context` on a hot sem-core entity 7,272 to 4,113 tokens (-43%, 119 to 48 entries); on a hot weave entity 5,216 to 3,498 (-33%, 146 to 27 entries, the 119 test stubs now one line). (2) The `sem_entities` MCP tool renders compact per-line trees (name · type · lines, children indented, files as group headers) instead of pretty-printed JSON: measured 3.7x fewer tokens on a 115-entity file (5,028 to 1,361). Applies to the MCP path and query modes; CLI `--json` output is unchanged for scripts.
+
+- **Semantic commit index (storage engine layer 2)**: history is now stored as entity deltas. Each commit is semantic-diffed against its first parent exactly once and persisted as entity-change rows in the cache (`commits` + `entity_changes` tables, sha-keyed and branch-agnostic); every later history query is a SQLite lookup plus a diff of only the commits git gained since. Measured on the sem repo: `sem log` over 500 commits drops from 4.46s to **0.03s** on the second query (~150x), with the first query as the one-time indexing pass and each new commit costing one incremental diff. Applies to `sem log` repo analytics (hotspots + co-change pairs) and the MCP `sem_log` tool; per-entity traces are unchanged. Aggregation is shared code between the live git walk and the store (`aggregate_history_analytics`), tested to produce identical hotspot/co-change output, so the two paths cannot drift. Merge-heavy history gets better semantics: each commit is attributed its own first-parent diff instead of a diff against its arbitrary revwalk neighbor, and merge commits contribute no changes of their own (their first-parent diff restates the merged-in commits, which index individually). File-filtered queries index full-repo diffs once and filter at aggregation, so the first filtered query costs more than the old pathspec-scoped walk but every later query on any filter is instant. Cache schema v9; existing caches rebuild automatically on first use.
+
+- **CLI sidecar fast path**: `sem impact` now answers from the resident `sem mcp` server's warm graph via its unix socket before doing any local work — measured **4.5ms** end-to-end on a 158K-LOC repo, versus 22.4ms for the local cold path and 7.7ms for a ripgrep scan of the same repo: the full blast radius (callers, dependencies, depth-bounded transitive impact, affected tests) is now cheaper than a raw text match. Output is byte-identical to the local path (the sidecar ships serialized `EntityInfo`s that the CLI feeds to its existing printers; verified across all modes and `--json`). The fast path is an accelerator, never a requirement: bounded socket timeouts and silent fallback mean no resident server (or `SEM_NO_SIDECAR=1`, `--no-cache`, custom scopes, `.semignore`, `--entity-id`) just runs the normal local path. Server-side, the new sidecar `impact` op classifies affected tests only among the entities the impact BFS actually reached, instead of walking the whole corpus per call (6.8ms → 0.1ms on a 4.7K-entity graph).
+
+### Fixed
+
+- The context packer's "not packed" summary line pluralizes roles correctly ("transitive dependencies", not "dependencys").
+
+- Workspace version bumped to 0.16.0: `ContextResult` gained the public `omitted` field (a breaking change for struct-literal constructors, flagged by cargo-semver-checks), and 0.x semantics put breaking changes in the minor version.
+
+- The docs site deploys through workflow-based GitHub Pages (`.github/workflows/docs-pages.yml`: upload `/docs` verbatim, deploy) instead of the legacy branch-based Jekyll builder, which began failing repo-wide with zero-duration "Page build failed" errors on commits that didn't touch docs — including on direct build requests via the Pages API. The site is pure static HTML, so the legacy builder added nothing but a failure mode; deploys now also skip entirely on commits that don't change `docs/`.
+
+### Changed
+
+- The GitHub Action's PR-comment footer now tells the reader what to do next — "add it to your repo in 2 minutes", linking to the action's install snippet — instead of only naming the tool. Every entity-diff comment is seen by all of a repo's collaborators; the footer is the loop that turns viewers into installs.
+
+### Added
+
+- **`sem repos`** — where your code is stored, in one command. Two inventories side by side: the **cloud account** (authoritative `GET /v1/repos`: every indexed repo with status, entity/file counts, last-indexed time, indexed commit, and any indexing error rendered inline) and **local storage** (every entity cache under the sem cache root with size on disk, entity count, cache kind, and the repo it was built from). `--json` for scripts. Listing the account also reconciles this machine's `~/.sem/repos.json` mirror with server truth — stale entries (a repo registered mid-index stays "pending, 0 entities" forever otherwise) were silently mis-routing the local-vs-cloud decision for impact/context queries. Caches are now stamped with their repo root at save time (`repo_root` in `cache_metadata`); caches built before this show as unlabeled and self-label on their next rebuild.
+
+- Fish shell support, via the `tree-sitter-fish` grammar (gated behind the `lang-fish` feature in `grammar-all`). Extracts functions — including the config.fish pattern of definitions inside a top-level `if status is-interactive` block — and resolves fish call edges (a `command`'s name against repo functions, builtins excluded), so `sem impact` sees which fish functions call which. A `function` defined inside another function stays part of the outer entity's content, matching fish's runtime semantics (inner definitions become global, not lexical children). Previously `.fish` files fell back to generic line-based chunking with an unsupported-language warning. Thanks @thalys for the request (#433).
+
+### Documentation
+
+- **First-principles page on the docs site** (`docs/first-principles.html`, linked from every page's nav): four charts explaining why the recent latency work changes what an agent can afford to do, not just how fast it runs — the scan-vs-index crossover (a text scan pays per byte; residency removes the index's ~800ms hydrate floor, so the constant-time line wins at every repo size), the ~100ms human-perception threshold every new path now sits under, model turns as the real cost unit (3 → 2 → 1 inference turns per structural answer via one-call lookup, then prompt-time prefetch), and tokens per answer (the measured ~15% entity-tree-vs-JSON ratio). Measured numbers come from this changelog; model curves and turn timings are labeled illustrative on the page. Charts are dependency-free inline SVG with hover tooltips and a table view each.
+
+### Performance
+
+- **Content-store cache (storage engine layer 1)**: the entity cache no longer duplicates source text per entity. Each file's text is stored once (zstd) in a `file_contents` table, and any entity whose body is provably a byte slice of it (`content == file[start_byte..end_byte]`, verified at save time) stores NULL content and is re-sliced on load; unprovable entities (no spans, normalized endings) keep content inline. On a 139K-entity corpus (fresh cache both sides) this cut the cache 20% (269MB to 216MB; the content layer itself −58%, 80MB to 24MB inline + 10MB zstd), engaged for 77% of entities. Honest costs: warm full-content loads pay ~0.13s extra for decompress+slice on that corpus (0.38s to 0.51s); topology loads and the MCP server's in-memory hot path are unaffected, and cold build time is unchanged within noise (peak RSS ~−5%). Correctness gates: byte-identical graph vs the previous binary on the full corpus, byte-identical entity content round-trip (including multi-byte unicode and nested entities), and incremental saves keep the file store in sync with entity deletes. Cache schema v8 — existing caches rebuild automatically on first use.
+
+### Added
+
+- **Entity-addressed text search**: `sem_entities` takes a `text` parameter — an exact substring searched across entity bodies in the warm in-memory graph (no file reads). Hits come back addressed by the innermost enclosing entity (`file: entity (Lline): matched text`), ready to chain into `sem_context`/`sem_impact`, in ~20-30ms warm on an 85K-LOC repo. This retires the main remaining reason agents fell back to grep (strings, error messages, config keys); misses say honestly that comments between entities and non-code files are not covered.
+
+### Performance
+
+- Graph build: the scope resolver no longer allocates its debug resolution log (several owned strings per reference, discarded by every production path — only a bench consumed it), and edge dedup is index-based instead of cloning both entity IDs per edge into a hash set. Output is byte-identical (proven edge-for-edge on a 139K-entity build); ~1-3% fewer instructions retired. Groundwork toward #320/#322 — the remaining peak-memory work (entity content sharing, ID interning) is tracked there.
+
+### Added
+
+- **`sem hook prompt-submit`** (hidden plumbing): the prompt-time prefetch, compiled. Reads a Claude Code UserPromptSubmit event, extracts identifier-shaped tokens from the prompt (backticked, snake_case, CamelCase, qualified — never plain words), resolves them against the resident server's socket sidecar, and prints packed entity context for injection. **10ms end-to-end** (was ~40ms as a Python hook — interpreter startup and a git subprocess, both eliminated: repo root is found by walking to `.git` in-process). Silent on conversational prompts, slash commands, unknown names, or when no server is resident.
+
+### Added
+
+- The socket sidecar is unix-only (`cfg(unix)`): Windows builds skip it with a no-op and the prefetch hook falls back silently — the sidecar is an accelerator, never a requirement. (Fixes the Windows build break the sidecar introduced.)
+- **Socket sidecar**: `sem mcp` now exposes the warm in-memory graph on a per-repo unix socket (`~/.sem/sock/<repo-hash>.sock`, one JSON line in, one out). Short-lived local callers — the prompt-prefetch hook, future CLI fast paths — get one-call entity context in single-digit milliseconds instead of paying a fresh process plus SQLite hydrate (~800ms). Stale sockets from dead servers are detected and taken over; the sidecar is a silent accelerator, never a requirement.
+
+### Added
+
+- **One-call lookup**: `sem_context`'s `file_path` is now optional. With only an `entity_name`, the entity is resolved across the whole repo (unique match proceeds; ambiguity returns a compact candidate list with the files; no match returns near-name suggestions) and the body plus callers/callees comes back in a single round-trip — one agent call where grep needs two (search, then read). Measured 26ms wall on a prewarmed server, name-only, unfamiliar repo.
+
+### Performance
+
+- The sem MCP server is now **local-first and prewarmed**. Cloud-first routing on `sem_impact`/`sem_context` cost a network round-trip on every call before the local answer (and carried the same wrong-entity risk gated in the CLI); it is now behind `SEM_MCP_CLOUD=1` until the server resolves name+file strictly. The server also builds the CWD repo's graph in the background at startup, so the agent's first structural query answers from memory. Measured on an 85K-LOC repo: warm `sem_context` runs in under 1ms wall (faster than a ripgrep scan of the same repo), and the first call dropped from 129ms cold to ~0 with prewarm.
+
+### Removed
+
+- Team presence was pulled from the `--badge` package before it shipped as a feature (product call: not a feature for now). The statusline no longer shows teammates and the hook sends nothing anywhere; the dormant server endpoints remain unadvertised.
+
+### Added
+
+- The `--badge` statusline is now **live at trigger time**: a PreToolUse hook flips the badge to an animated spinner with the entity name the moment the agent calls sem (`⊕ sem ⠹ impact validateToken…`), and the completed state (count, latency, savings) lands when the call finishes. The render hot path never touches the network (renders measured at ~20ms).
+
+### Fixed
+
+- The `sem context` / `sem_context` budget packer no longer starves the target while neighbors feast. Previously a target too big for the budget collapsed to its first line (2 tokens) while a single large dependency could consume the entire budget with its full body. The target now degrades gracefully — full body → head-truncated body (docstring, fields, leading code, with an explicit `… truncated: N more lines` marker) using up to ~70% of the budget → bare signature — and no neighbor may cost more tokens than the target itself did (budget/10 floor), oversized neighbors degrading to signatures. On the same query (a large class, budget 2000) the target went from 2 tokens to 1,398 and the answer-relevant attributes are now in the payload.
+
+### Added
+
+- **Entity-level history analytics**: `sem log` with no entity now analyzes recent repo history in one pass and reports **hotspots** (the most-changed code entities, with commit counts, distinct authors, and the last commit that touched each) and **co-change pairs** (entities that repeatedly change in the same commits, with a confidence score — "these two never change apart"). Counts are per commit, code entities only (doc headings, config properties, and lockfile chunks are excluded so the signal is about code), and bulk commits touching >50 entities are excluded from pair-counting to keep quadratic noise out. Same via MCP: `sem_log` without `entity_name`. `--file` scopes to one file; `--json` returns everything. This is the time axis a snapshot dependency graph cannot see: which code churns, and which code moves together.
+
+### Changed
+
+- `sem_impact` MCP results now render as a **blast-radius tree** (`◉` header, one `├─▶` branch per file, real callers first, all-test files sunk to the bottom, nothing elided) — expanding the tool widget is the live graph, no separate viewer process needed. The bundled skill also instructs agents to draw the blast radius as a small ASCII tree directly in their reply when an impact result drives the answer.
+
+- `sem_impact` and `sem_context` MCP results now render as a compact entity tree instead of pretty-printed JSON: dependents/dependencies/transitive impact grouped one line per file, every entity name preserved, with the elapsed time and source in a footer. The same information lands in about 15% of the tokens, and the expanded tool widget in agent UIs reads at a glance (`⊕ entity · file`, `← 29 dependents · 10 files`, `⚡ 70 transitively affected`). Context entries keep their verbatim content under a per-entry header.
+
+## [0.15.1] - 2026-07-01
+
+### Added
+
+- `npx @ataraxy-labs/sem-skill --badge` (opt-in) installs a live sem badge in the Claude Code statusline: it shows how many structural queries ran this session, the last command **and the entity it analyzed**, its latency, a sparkline of recent latencies, and a rotating stat (distinct entities analyzed, top command) (`⊕ sem ×12  impact validateToken 9ms  ▁▂▃▅▂  · 7 entities analyzed`). It is fed by a PostToolUse hook that catches sem via **both** the MCP tools and the `sem` CLI (Bash), and falls back to recent activity so the badge never stalls on "idle". Non-destructive: it backs up settings and never overwrites an existing statusline (it prints how to add the badge yourself instead).
+- **GitHub Action** (`Ataraxy-Labs/sem/action`): entity-level semantic diff comments on pull requests. One sticky comment per PR showing which functions/classes/methods were added, modified, or deleted, updated in place on every push; cosmetic-only PRs (formatting/comments) are called out explicitly. Installs the prebuilt binary (~2s), needs no config or API keys, and never fails the build. sem's own PRs now dogfood it via `.github/workflows/pr-entity-diff.yml`.
+- The savings meter now lives in the **statusline itself** — no extra process. The `--badge` badge always shows the live estimated time + tokens this session's sem calls saved vs grep+read (`⊕ sem ×5 diff · ≈ 4m · ≈ 25k tokens saved`), and when idle it shows the lifetime total (`⊕ sem idle · ≈ 3h · ≈ 190k tokens saved`). The PostToolUse hook is the single writer of the persisted lifetime tally (`~/.claude/sem-savings.json`), so the counter grows from real usage whether or not the live viewer is open. Estimates stay anchored to the measured benchmark and labelled `≈`.
+- Live viewer for the `--badge` install: `~/.claude/sem-live.py` (run it in a spare terminal pane). It redraws an ASCII blast-radius graph each time sem runs — the analyzed entity, its direct callers (real ones surfaced, test fan-out collapsed), and the transitive count — plus a **savings meter**: a running, honestly-estimated tally of the grep+read round-trips, time, and tokens sem saved this session, and a lifetime counter persisted across sessions (`~/.claude/sem-savings.json`). Estimates are anchored to a measured benchmark and labelled `≈`. The badge hook now also records `--file` and cwd so the graph can be reconstructed.
+
+### Fixed
+
+- Repository discovery now tolerates Git worktrees that use the `extensions.relativeworktrees` config key, avoiding libgit2's unsupported-extension error when plain `git` can open the checkout.
+- Cloud-backed `sem impact` / `sem context` no longer answer queries they can't answer correctly. Two gates added: `--no-cache` now always computes fresh locally (previously the cloud snapshot was served anyway), and **file-hinted queries (`--file`) stay local** — the cloud resolves entities by name with a silent name-only fallback, so for same-named entities (e.g. ten `fn run` command handlers) it could return the *wrong entity's* graph, and a stale cloud index could drop dependents that exist locally. Local resolution disambiguates exactly; the cloud path returns once the server resolves name+file strictly and exposes its indexed commit for a freshness check.
+- Impact/dependency resolution now follows type-qualified associated calls (`Type::method()`) when the receiver is a known repo type, so a caller reached only through a static/associated path is no longer dropped from `sem impact`. Previously, e.g., a test helper calling `SemPlugin::detect_changes()` was invisible to the reverse-dependency graph, and its transitive callers were missing from the blast radius. Resolution stays precise: a bare module path (`foo::bar::baz()`) still does not bind to a same-name local function, and common associated names (`Type::new`, `::default`) are not guessed.
+
+### Performance
+
+- Faster graph hydrate on large repos. The public `EntityGraph` maps now use `rustc-hash` (FxHashMap) instead of std SipHash, matching the build's internal maps, and the SQLite cache sets read pragmas (`mmap_size`, `cache_size`, `temp_store=MEMORY`) on every connection. On a 200K-entity / 800K-edge graph this is about 9% faster to hydrate (0.42s to 0.39s, no overlap across repeats); negligible on small repos. Output is byte-identical.
+
 ## [0.15.0] - 2026-06-30
 
 ### Changed

package/README.md

@@ -51,6 +51,12 @@ Or via Homebrew:
 brew install sem-cli
 ```
 
+Or via winget on Windows:
+
+```powershell
+winget install AtaraxyLabs.sem
+```
+
 Or install the npm wrapper into `node_modules`:
 
 ```bash
@@ -205,6 +211,18 @@ sem log authenticateUser --limit 20
 sem log authenticateUser --json
 ```
 
+With no entity, `sem log` analyzes recent repo history at the entity level:
+**hotspots** (most-changed functions/classes, with author counts) and
+**co-change pairs** (entities that repeatedly change in the same commits —
+"if you touch one, don't forget the other"):
+
+```bash
+sem log                 # repo hotspots + co-change pairs (last 50 commits)
+sem log --limit 200     # deeper history
+sem log --file src/auth.ts   # scoped to one file
+sem log --json          # full data
+```
+
 ### sem entities
 
 List all entities under a file or directory path. No path is the same as `.`.
@@ -258,6 +276,29 @@ To disable and go back to normal git diff:
 sem unsetup
 ```
 
+## Entity-level diffs on every pull request
+
+Add the GitHub Action and every PR gets one sticky comment showing which
+functions, classes, and methods changed — updated in place on each push, and
+calling out cosmetic-only PRs (formatting/comments) explicitly:
+
+```yaml
+# .github/workflows/entity-diff.yml
+name: Entity diff
+on: pull_request
+permissions:
+  contents: read
+  pull-requests: write
+jobs:
+  entity-diff:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: Ataraxy-Labs/sem/action@v0.15.1
+```
+
+No config, no API keys, never fails your build. See [action/](action/) for details.
+
 ## Cloud acceleration (optional)
 
 sem builds an entity dependency graph to answer `impact`, `context`, and `entities`. On a small or medium repo that's instant. On a very large codebase, building the graph locally can take hundreds of milliseconds to a few seconds.
@@ -295,6 +336,7 @@ It is fully optional and transparent:
 | Swift | `.swift` | functions, classes, protocols, structs, enums, properties |
 | Elixir | `.ex` `.exs` | modules, functions, macros, guards, protocols |
 | Bash | `.sh` | functions |
+| Fish | `.fish` | functions |
 | Lua | `.lua` | functions (global, local, table, and method forms) |
 | HCL/Terraform | `.hcl` `.tf` `.tfvars` | blocks, attributes (qualified names for nested blocks) |
 | Kotlin | `.kt` `.kts` | classes, interfaces, objects, functions, properties, companion objects |

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@ataraxy-labs/sem",
-  "version": "0.15.0",
+  "version": "0.16.0",
   "description": "npm wrapper for the sem CLI. Downloads the matching release binary and exposes the sem command in node_modules/.bin.",
   "license": "MIT OR Apache-2.0",
   "type": "module",