@uehaj/sys1grep 0.5.0-next.0 → 0.5.0-next.2

package/README.md

@@ -4,6 +4,8 @@
 
 Background and design notes (Japanese): [Jevのキラーアプリ、「意味で探す grep」を作った](https://zenn.dev/uehaj/articles/jev-semgrep-grep-by-meaning) on Zenn.
 
+> **Renamed from semgrep (0.5.0).** Two reasons. The name collided with the static-analysis tool [Semgrep](https://semgrep.dev/), a trademark. And the tool was never meant for Jev alone: it targets System 1 models in general, and I'd like it to work with any models TypeSafe AI may release after Jev under other names :). So from 0.5.0 this is **sys1grep** (System 1 + grep): npm package `@uehaj/sys1grep` (was `@uehaj/semgrep`), commands `sys1grep` / `git sys1grep`, environment `SYS1GREP_*`, config `~/.config/sys1grep/.env`. The old `SEMGREP_*` names and `~/.config/semgrep/.env` still work for one minor release, each use printing a deprecation line; 1.0.0 drops them. Details: [CHANGELOG](CHANGELOG.md#renamed). The Zenn article's URL keeps its old slug.
+
 A grep that finds lines by **what they mean**, not by regular expressions.
 Matching is done by **Jev**, the System One model from [TypeSafe AI](https://typesafe.ai/).
 Jev does not generate text. It answers typed questions with probabilities, so for every line
@@ -14,9 +16,9 @@ and applies a threshold.
 ./sys1grep -n -e "customer is angry or frustrated" tickets.txt
 ```
 
-[![sys1grep demo: a Japanese meaning finds refund requests in six languages; "asking for a refund" vs "about a refund"; -Q finds the answer](docs/demo.svg)](https://uehaj.github.io/sys1grep/)
+[![sys1grep demo: a Japanese meaning finds refund requests in six languages; "asking for a refund" vs "about a refund"; -Q finds the answer](docs/demo.svg)](https://sys1grep.js.org/)
 
-<sub>▶ Click the demo, or open <a href="https://uehaj.github.io/sys1grep/">uehaj.github.io/sys1grep</a>, for the full demo on the landing page.</sub>
+<sub>▶ Click the demo, or open <a href="https://sys1grep.js.org/">uehaj.github.io/sys1grep</a>, for the full demo on the landing page.</sub>
 
 - Zero dependencies. One file, Node.js 20.16+ and `fetch`.
 - Fast. 30 lines go into one request, requests run 8 at a time. A 210-line file finishes in under a second.
@@ -134,8 +136,7 @@ as a plain regex, with no request at all. It prefilters its AND term: only the l
 ever ask that term's meanings, so a cheap regex in front of a meaning cuts both the bill and the wait. A line is sent
 only if some term's regexes all hold for it (a term with no regex holds for every line), so with
 `-e '/re/' -a A -e B` a line without `re` is still sent, asked `B` only. A query of regex terms alone
-sends nothing, except that `--sentence` (`=jev`, the default) still asks Jev where wrapped lines
-break; use `--sentence=rules` to stay offline. Anything that isn't shaped like `/…/flags` is still a meaning, so
+sends nothing, even with `--unit=sentence-by-jev`: the rules alone then decide where wrapped lines join. Anything that isn't shaped like `/…/flags` is still a meaning, so
 `-e '/etc 以下のファイルを変更している'` (no closing `/`) is unaffected; a meaning that really starts
 and ends with `/` can be written with a leading space to dodge the regex reading.
 
@@ -172,24 +173,50 @@ $ ./sys1grep -o -n -e '/[A-Z]+-\d+/' -a 'the ticket is still open' notes.txt   #
 Every line sent to Jev costs money and time, so the cheapest line is the one never sent. From the widest cut
 to the narrowest:
 
-- **Which files.** `-r` skips `.git`, `node_modules`, binary files, likely secrets and what git ignores;
-  [`git sys1grep`](#as-a-git-subcommand-git-sys1grep) searches tracked files only. `--include` / `--exclude`
-  (file-name globs) and `--changed-within` (`30m`, `7d`, `today`, `this-week`, a date) narrow them further.
-  A meaning that restricts itself to a language or a time of change narrows them by itself: see
-  [Scope from the meaning](#scope-from-the-meaning).
+- **Which files.** `-r` skips `.git`, `node_modules`, binary files, likely secrets, generated files (source maps,
+  minified JS/CSS, lock files) and what git ignores; [`git sys1grep`](#as-a-git-subcommand-git-sys1grep) searches
+  tracked files only. `--include` / `--exclude` (file-name globs) and `--changed-within` (`30m`, `7d`, `today`,
+  `this-week`, a date) narrow them further. A meaning that restricts itself to a language or a time of change
+  narrows them by itself: see [Scope from the meaning](#scope-from-the-meaning).
 - **Which lines.** A [regex term](#regex-terms) is matched locally, and only the lines it holds for are asked
-  its AND term's meanings. Blank lines are never sent.
+  its AND term's meanings. Blank lines are never sent. Only the first `-M`/`--max-columns` characters of a line
+  (default 2000, 8000 with `-z`) are sent: it is still searched and judged, just not past that cutoff.
 - **How many times.** [`--dedup`](#one-line-per-template---dedup) judges one line per template: lines that
-  differ only in ids, numbers, times or paths share one answer.
-- **Check before paying.** `--dry-run` sends nothing and prints the files, how many lines each would send and
-  every request with its questions. Its last line estimates the input tokens and, for TypeSafe itself, the price
-  (`~3178 input tokens, ~$0.000133`; within about 10%). `-i` shows the same totals on the terminal and sends only
-  after `y`.
+  differ only in ids, numbers, times or paths share one answer. `auto`/`always`/`never`, off by default for
+  now: a hint says when `--dedup=auto` would pay.
+- **Check before paying.** `--dry-run` sends nothing and prints the settings the search would run with, the
+  files, how many lines each would send and every request with its questions. Its last line estimates the input
+  tokens and, for TypeSafe itself, the price (`~3178 input tokens, ~$0.000133`; within about 10%). `-i` shows the
+  same totals on the terminal and sends only after `y`.
+  Before the bulk of requests, every target is sized (`--max-filesize`, default 10M); one over it is skipped
+  outright, like `rg`'s own `--max-filesize`, named on stderr (`-y` does not affect it). A `.gz` is sized
+  decompressed (zlib stops at the limit, so an oversized one is never inflated in full). stdin is sized once it
+  is read (it is already in memory), and skipped the same way if it is over; a `--cached` or `<tree>:` target
+  is a blob rather than a file, sized from its content (already read in one `git cat-file --batch` call, not a
+  fresh git process per blob); `-g`'s commits stay out of this, since each is already bounded by `-M` when it
+  is sent. The input about to be sent is priced too (`--max-cost`,
+  default 1 USD); over it, one question asks to continue on the terminal, `-y` answers it yes, and without a
+  terminal it is exit 2 (this applies with `-q` too — scripts pass `-y`).
 
 ```sh
 $ sys1grep --dry-run -r --include='*.log' --changed-within=today -e '/ERROR|FATAL/' -a 'a customer is affected' logs/
 ```
 
+`--verbose` (or `--dry-run`) also shows *where* a setting that was not typed on the command line came from
+(`SYS1GREP_OPTS`, an environment variable, `~/.config/sys1grep/.env`, or a preset's default), so a result that
+surprises you can be traced back to its source. The key's value never appears, only which option or variable
+supplied it:
+
+```sh
+$ SYS1GREP_OPTS='--level strict' sys1grep --verbose -e "the API key is read from a file" .
+sys1grep: endpoint api.typesafe.ai/v1/systemone (default), model jev-latest (default)
+sys1grep: key: SYS1GREP_API_KEY (~/.config/sys1grep/.env)
+sys1grep: SYS1GREP_OPTS: --level strict
+sys1grep: options: --level strict (SYS1GREP_OPTS) = -t 0.7 -T 0.3, --chunk 30, -j 8, scope on, -M 2000, --max-filesize 10M, --max-cost 1
+sys1grep: file ./a.py: 120 lines, 120 to send
+…
+```
+
 ## Install
 
 Two ways to use it: as a command-line tool (this section), or as a Claude Code skill
@@ -381,11 +408,18 @@ tests/tickets/sub/b.txt
 ```
 
 `-r` walks directories in sorted order and skips `.git`, `node_modules`, `.ssh`, `.aws`, `.gnupg`, `.kube`,
-`.docker`, binary files (a NUL byte in the first 8 KB, or a PDF; UTF-16 with a BOM is read as text) and files
+`.docker`, binary files (a NUL byte in the first 8 KB, or a PDF; UTF-16 with a BOM is read as text), files
 that usually hold secrets (`.env*`, `.netrc`, `.npmrc`, `.pypirc`, `.pgpass`, `.git-credentials`, `*.pem`,
-`*.key`, `*.p12`, `*.pfx`, `*.jks`, `*.keystore`, `id_rsa*` and friends; names compared without case). **Every line that is searched is sent to the TypeSafe API**, so point `-r` at a directory
+`*.key`, `*.p12`, `*.pfx`, `*.jks`, `*.keystore`, `id_rsa*` and friends; names compared without case) and
+generated files (`*.map`, `*.min.js`, `*.min.css`, `package-lock.json`, `yarn.lock`, `pnpm-lock.yaml`,
+`Cargo.lock`, `poetry.lock`, `composer.lock`, `Gemfile.lock`, `go.sum`). **Every line that is searched is sent
+to the TypeSafe API**, so point `-r` at a directory
 you mean to scan. Inside a git repository, `-r` also skips what git ignores (`.gitignore`, `.git/info/exclude`, the
 global excludes file), so build output and local files stay home; a tracked file is searched even if it matches.
+A file named `*.gz` (a rotated log: `app.log.1.gz`) is read decompressed, as `zgrep` does, and printed by its
+name on disk (`app.log.1.gz:12:...`); `--include='*.gz'` picks those, the binary sniff sees the decompressed
+bytes, a gzipped secret (`private.key.gz`) is skipped like the plain one, and a corrupt `.gz` is an unreadable file.
+Only gzip, only by the name.
 A file named explicitly on the command line is always searched, even if it matches the skip list or is ignored;
 so is a directory that git ignores, when you name it (`sys1grep -r -e ... dist`). `-l` prints each
 matching file once, in the order matches are found, and works with or without `-r`. `-c` prints the number of
@@ -453,6 +487,25 @@ applies even to tracked files. `--include`, `--exclude` and `--changed-within` f
 by a pathspec included. `--changed-within` reads the working tree's modification times, not git history: right after
 a clone or a checkout, every file it wrote counts as just changed. For help use `git sys1grep -h`: git takes `--help` itself and looks for a man page.
 
+`--cached`, `--untracked` and a `<tree>...` choose what is searched instead of the working tree, as `git grep` has
+them (only one of the three at a time):
+
+```sh
+$ git sys1grep --cached -e "a retry is attempted"       # staged, uncommitted changes included
+$ git sys1grep --untracked -e "a retry is attempted"    # tracked files plus untracked ones (.gitignore still applies)
+$ git sys1grep -n -e "a retry is attempted" main v0.3.1 -- '*.py'
+main:src/job.py:42:    retry(job, times=3)
+v0.3.1:src/job.py:40:    retry(job)
+```
+
+A `<tree>` (a branch, tag, commit or `@{u}`) is any argument before `--` that resolves as a revision; several may
+be given, searched in the order given, each line prefixed with the name as typed (`@{u}:path`, not the branch it
+resolves to). `--changed-within` needs the working tree: a blob (`--cached`, a `<tree>`) has no mtime of its own.
+A `<tree>`'s own pathspec takes a glob too, same as elsewhere in sys1grep (`git diff-tree` against the empty
+tree, not `git ls-tree`'s own literal/directory-prefix match); `--include` / `--exclude` (by name) still work.
+**An old `<tree>` can hold a secret an ordinary file once carried and was later removed from**: the skip list
+drops files by name only, not by what changed since.
+
 ### Everything that is *not* something
 
 ```sh
@@ -480,15 +533,15 @@ Matching records are printed NUL-terminated too, so pipe them through `tr '\0' '
 File names (`-l`) and counts (`-c`) stay on newlines, as they do in grep. With `-z`, `-n` numbers records,
 `-A` / `-B` / `-C` count neighbouring records, and `--chunk` counts records per request.
 
-### One sentence at a time (`--sentence`)
+### One sentence at a time (`--unit=sentence-by-*`)
 
-`--sentence` judges each sentence instead of each line. The output is still lines, as in grep: every line a
+`--unit=sentence-by-jev` (or `sentence-by-rule`, below) judges each sentence instead of each line. The output is still lines, as in grep: every line a
 matching sentence touches is printed, and on a terminal the sentence itself is in bold yellow (a regex match inside it, in grep's bold red).
 Wrapped lines are joined before splitting, so a sentence that runs over several lines is judged as one.
 [`tests/prose.txt`](tests/prose.txt) wraps an English paragraph and a Japanese one:
 
 ```sh
-$ ./sys1grep -n --sentence -e "the author admits they made a mistake" tests/prose.txt
+$ ./sys1grep -n --unit=sentence-by-jev -e "the author admits they made a mistake" tests/prose.txt
 1:I should have checked the input
 2:before shipping, and that was my
 3:mistake. Next time I will add a test
@@ -500,14 +553,14 @@ The sentence starts on line 1 and ends at `mistake.` on line 3; only that part i
 On a terminal, with both meanings and `-C 3` for context, the colors show where each sentence starts and ends
 inside a line: lines 3 and 9 are colored only up to the end of the matching sentence, and lines 4-7 are context (`-`):
 
-![--sentence -C 3 --color: the matching sentences in bold yellow, up to mistake. on line 3 and 返金してほしいです。 on line 9; lines 4 to 7 as context](docs/sentence.svg)
+![--unit=sentence-by-jev -C 3 --color: the matching sentences in bold yellow, up to mistake. on line 3 and 返金してほしいです。 on line 9; lines 4 to 7 as context](docs/sentence.svg)
 
 `-o` prints only the matching sentences, one per line, as `grep -o` prints only the matching part.
 `-n` then gives the line where the sentence starts. Japanese is joined without a space, as are Chinese,
 Thai, Lao, Khmer, Myanmar and Tibetan, which do not put spaces between words:
 
 ```sh
-$ ./sys1grep -n -o --sentence -e "the author admits they made a mistake" -e "customer is asking for a refund" tests/prose.txt
+$ ./sys1grep -n -o --unit=sentence-by-jev -e "the author admits they made a mistake" -e "customer is asking for a refund" tests/prose.txt
 1:I should have checked the input before shipping, and that was my mistake.
 8:先週買った掃除機が初日から動かないので返金してほしいです。
 ```
@@ -515,20 +568,20 @@ $ ./sys1grep -n -o --sentence -e "the author admits they made a mistake" -e "cus
 Without `-o`, `-c` counts lines and `-A` / `-B` / `-C` count lines, as usual. With `-o` they count sentences.
 With `-z`, each record is split on its own and matching records are printed whole.
 
-Jev finds a matching sentence inside a long line on its own, so `--sentence` is not needed for accuracy.
+Jev finds a matching sentence inside a long line on its own, so `--unit=sentence-by-jev` is not needed for accuracy.
 Use it to see which sentence matched, to get the sentences with `-o`, and when AND should hold within one
 sentence: the expression is evaluated per sentence. For the same reason `-v X` alone prints every line
-with at least one sentence that is not X; to find lines that are not X as a whole, leave `--sentence` off.
+with at least one sentence that is not X; to find lines that are not X as a whole, leave `--unit` at `line`.
 
 Japanese and Chinese entries often end without `。`: a chat message, a support ticket, a memo line. Joining
-them would glue separate entries into one "sentence". So by default (`--sentence`, the same as
-`--sentence=jev`) sys1grep asks Jev about each unpunctuated break next to a script written without word
+them would glue separate entries into one "sentence". So with
+`--unit=sentence-by-jev` sys1grep asks Jev about each unpunctuated break next to a script written without word
 spaces: "does this line break end a sentence or entry, or is it a wrap inside a sentence?" It sends 30 lines
 per request with one yes/no per break, and keeps the lines apart when the answer is 0.7 or more. On
 [`tests/corpus.txt`](tests/corpus.txt) this keeps the four one-line Japanese tickets apart, so
-`--sentence` finds the same refund requests (lines 14 and 18) as a line-by-line search, where the rules alone
+`--unit=sentence-by-jev` finds the same refund requests (lines 14 and 18) as a line-by-line search, where the rules alone
 merged the tickets and missed line 18. The extra requests cost about as much as one more meaning; use
-`--sentence=rules` to skip them. Breaks between English lines are never asked: joining them keeps a space,
+`--unit=sentence-by-rule` to skip them. Breaks between English lines are never asked: joining them keeps a space,
 and the full stop still ends the sentence.
 
 Where a newline cannot be inside a sentence, lines are not joined: at a blank line, next to brackets or
@@ -538,6 +591,88 @@ own. Sentences are cut by `Intl.Segmenter` ([Unicode UAX #29](https://unicode.or
 which splits at `.` `!` `?` `。` `！` `？` but also after abbreviations such as `Mr.`. Logs are not prose:
 consecutive log lines that start with a letter, such as `WARN ...` after `ERROR ...`, get joined.
 
+### One function at a time (`--unit=function`)
+
+In code the question is usually about a function ("retries on network failure"), and one line of it rarely
+says so. `--unit=function` judges each function and prints its lines, with `-n` giving file line numbers:
+
+```sh
+$ ./sys1grep -n --unit=function -e "retries on network failure" src/net.js
+src/net.js:3:async function fetchWithRetry(url) {
+src/net.js:4:  for (let i = 0; i < 5; i++) {
+src/net.js:5:    try { return await fetch(url); }
+src/net.js:6:    catch { await sleep(2 ** i * 100); }
+src/net.js:7:  }
+src/net.js:8:}
+```
+
+A function runs from a funcname line to the line before the next one, as `git grep -W` finds it; lines
+before the first funcname line are one unit, and blank lines at a function's end are not printed. The
+funcname lines are git's: the `diff=<driver>` attribute in `.gitattributes`, then `diff.<driver>.xfuncname`
+in git config. Without one, sys1grep has its own rule for JavaScript/TypeScript (a top-level `function`,
+`class`, or `const` / `let` / `var` bound to a function) and Python (`def` / `class`, nested ones too); a decorator line (`@retry`) starts
+the function it decorates. Otherwise it uses git's default: a line starting with a letter, `_` or `$`. git's own builtin drivers
+(`diff=python` and so on) are not read, so a driver without an `xfuncname` falls back the same way:
+
+```sh
+$ printf '*.go diff=golang\n' >> .gitattributes
+$ git config diff.golang.xfuncname '^(func|type)[[:space:]]'
+```
+
+`-M` defaults to 8000 characters here, as with `-z`; a longer function is judged on its first 8000.
+`--chunk` counts functions, `-A`/`-B`/`-C` and `-c` count lines, as with the sentence units. `-z`, `-g` and
+`-o` are refused.
+
+### From one function to another (`--step-to`)
+
+A question about code often comes in two steps: where does `--summarize` hand the lines to the tool, and then,
+what happens when that fails. The "that" in the second step is what the first one found. Multi-step matching
+answers both in one run: the expression before `--step-to` finds the start, sys1grep walks the calls from it,
+and the expression after `--step-to` picks the functions it reaches that match. Jev judges the two
+expressions; no score decides which call to follow.
+
+```sh
+$ sys1grep -e "--summarize hands the lines to the tool" --step-to "what happens when the tool fails to start or answer" sys1grep.mjs
+```
+
+A meaning may start with a dash, as the first one does, on any search and for `-e`, `-a`, `-v` and `-Q`: a value that starts with `-` is taken as the meaning unless it is an option (an option holds no space).
+
+Each path prints as a tree: the hop (the number of calls from the start), `file:line` and the function's name,
+`:` after an end and `-` after a function on the way, as grep marks a match and its context. A regex-only
+start and end send nothing, which makes a free reachability search. In Python 3.6's `json`, the functions
+that `load` reaches and that hold a `raise`:
+
+```
+$ cd /usr/lib64/python3.6/json && sys1grep -e '/^\s*def load\b/' --step-to '/\braise\b/' *.py
+sys1grep: walk: 1 at hop 0, 1 at hop 1, 3 at hop 2, 1 at hop 3, 1 at hop 4, 1 at hop 5; stopped: no new unit
+0 __init__.py-274-load
+  1 __init__.py:302:loads
+    2 decoder.py:334:decode
+      3 decoder.py:345:raw_decode
+        4 scanner.py-65-scan_once
+          5 scanner.py:28:_scan_once
+```
+
+- **The expressions.** The expression before `--step-to` is the start's, as in any search; every `-e` / `-a` /
+  `-v` / `-Q` after it is the end's. `--step-to MEANING` and `--step-to '/RE/'` are `--step-to -e ...`; the file
+  names follow. A step is a search, not a hop: one `--step-to` is two steps, whatever the hops between them.
+- **The edges.** Without `--edges` the unit is a function (`--unit=function`) and an edge is a call: `name(` in
+  a function's body, its comments, docstrings and strings left out, links to every function of that name. A name
+  comes from the funcname line (the `def` under a decorator). `--edges=FILE` walks any other relation instead,
+  one edge per line, `FROM_FILE:LINE<TAB>FROM_NAME<TAB>TO_FILE:LINE<TAB>TO_NAME`; a line stands for the unit that
+  holds it. `--reverse` walks the edges backwards, callee to caller.
+- **The walk.** Breadth first from every start at once. A function's hop is its shortest distance from any
+  start, and each function is reached once, so a cycle ends. `--hops=N`, `--hops=M..N` and `--hops=M..` pick the
+  hops an end may be at (default `0..`: a start that matches the end expression is an end at hop 0). A path that
+  reaches no end is left out. stderr says how many functions each hop reached and why the walk stopped: no new
+  function, or `--hops`.
+- **The cost.** Jev judges every function against the start expression in one batch, then the functions reached
+  within `--hops` against the end expression in another. `--max-cost` asks again before the second, counting the
+  first. `--dry-run` answers every question 0, so it walks from every function the start expression could hold
+  for and shows that bound.
+- **Not with** `-z`, `-g`, `-o`, `-c`, `-l`, `-A`/`-B`/`-C`, `--rank`, `--summarize` or `--dedup`. `-p` adds the
+  end expression's scores to each function judged against it; `-q` prints nothing and exits 0 when there is an end.
+
 ### One line per template (`--dedup`)
 
 Cost is proportional to the text sent, and machine-generated logs are mostly one skeleton with a different
@@ -545,6 +680,19 @@ id or number in it. `--dedup` masks ids, hashes, numbers, dates and times, paths
 the result, sends one line per group and reuses its answer for the rest. What is sent is that line's
 original text, and every line is printed as itself:
 
+`--dedup` takes `auto`, `always` or `never`; a bare `--dedup` is `--dedup=always`. **Default is `never`, for
+now** (#143): `auto` is held back until real-run stats say it should be the default. Every run still
+estimates locally, before anything is sent, what folding every kind (the best case) would save, and prints
+one stderr hint when that is worth at least twice the cost of `--dedup`'s own question:
+
+```
+$ sys1grep -e "a request failed" app.log
+sys1grep: 2040 units fold to at most 31 templates; --dedup=auto would save ~68 requests (~44k tokens)
+```
+
+`--dedup=auto` asks and folds only when that estimate says it pays; a file that barely repeats sends every
+line, asking nothing extra. `--verbose` / `--dry-run` print the decision and its numbers for every value.
+
 ```sh
 $ ./sys1grep --dedup -n -e "a request failed" app.log
 1:worker request 3fa9c1e27b failed: connection reset
@@ -561,7 +709,7 @@ could change a match, and those kinds are kept apart (one of the two requests ab
 Measured on real logs, with nothing kept: a 43,071-line system log folds into 550 templates (1.2% of the
 bytes), `install.log` to 21.1%, a Claude Code transcript (jsonl) only to 56.8%. It is for machine-generated
 logs; prose has no shared skeleton, and a meaning that reads a timestamp folds almost nothing. With `-z` or
-`--sentence` the records or sentences fold instead of lines.
+`--unit=sentence-by-*` the records or sentences fold instead of lines.
 
 After a search, the summary line says what the folding saved: `(2 folded by --dedup, ~235 input tokens /
 ~$0.000010 saved, 21%)` above. It estimates what the folded lines would have cost as requests of their own and subtracts
@@ -589,15 +737,133 @@ The expensive model reads only what Jev kept. Asking why `./.env` is no longer r
 `git log` (168 commits), Claude's input fell from 22,059 tokens to 1,356, the total cost with Jev's from
 $0.094 to $0.013, with the same answer (#69). It pays when the answer sits in a few lines.
 
-- The matching lines leave the machine a second time, to Anthropic.
+- The matching lines leave the machine a second time, to Anthropic (or whatever TOOL talks to).
 - `-n`, `-A/-B/-C`, `-p` and file names go in as they would print; colors never do. No match runs nothing (exit 1).
-- `SYS1GREP_SUMMARIZER` picks the TOOL of a bare `--summarize` (only `claude` so far), `SYS1GREP_SUMMARIZER_MODEL` its model.
+- `SYS1GREP_SUMMARIZER` picks the TOOL of a bare `--summarize`, `SYS1GREP_SUMMARIZER_MODEL` its model.
 - `-q`, `-l` and `-c` print no lines, so they cannot be combined with it.
+- In `SYS1GREP_OPTS` it summarizes every search, so every match goes to TOOL's provider too;
+  `--no-summarize` turns it off for one search.
+- The answer is plain text: an LLM that is not told writes Markdown, noise on a terminal, so plain is asked for.
+  `--format=markdown` or `=html` asks for those instead (`… --format=html … > summary.html`).
+  The answer prints as it comes, unchecked. In `SYS1GREP_OPTS` it is a standing preference, ignored without `--summarize` or `--rank`.
 - With `--dedup`, the TOOL gets what Jev got: each template's representative once, marked `(×N like it)`,
   not every line its answer was reused for.
 - More than 200 KB (about 50k tokens) is not sent at all: exit 2, with the size, before the TOOL is paid.
   It is never cut short, since a summary of the first part would read as a summary of all of it.
 
+`--summarize-prompt=TEXT` adds your own instruction after the fixed one (how long, what to focus on):
+
+```sh
+$ sys1grep -r -n --summarize --summarize-prompt="3 lines or fewer, just which file to fix" \
+    -e "the API key is read from a file" .
+```
+
+It needs `--summarize`; empty TEXT is the same as leaving it out. It comes after the `--format` sentence,
+so TEXT can override the format.
+
+Other TOOLs:
+
+```sh
+$ sys1grep --summarize=llm -e "..." FILE            # Simon Willison's llm, tools off (no -T given)
+$ sys1grep --summarize=pi -e "..." FILE             # pi --print --no-tools --no-session ...
+$ SYS1GREP_SUMMARIZER_MODEL=qwen3.5:9b sys1grep --summarize=ollama -e "..." FILE
+$ SYS1GREP_SUMMARIZER_MODEL=some-id sys1grep --summarize=lmstudio -e "..." FILE   # model id from GET /v1/models
+$ SYS1GREP_SUMMARIZER_MODEL=some-id sys1grep --summarize=http://localhost:8080/v1 -e "..." FILE  # llama.cpp, vLLM, LocalAI, a gateway
+```
+
+`ollama` and `lmstudio` talk to a local OpenAI-compatible server (`POST /v1/chat/completions`) by `fetch`, no
+CLI: the matching lines never leave the machine a second time. A plain `http(s)://` URL is any other
+OpenAI-compatible server. All three need `SYS1GREP_SUMMARIZER_MODEL`: none has a default model.
+`SYS1GREP_SUMMARIZER_API_KEY` goes as `Authorization: Bearer` to a URL TOOL only (never `SYS1GREP_API_KEY`,
+which is Jev's). `OLLAMA_HOST` moves ollama's host, as it does for the `ollama` CLI itself.
+
+### Best first (`--rank`)
+
+Matches print in file order, as grep prints them. With many, `--rank` prints the results best first, each under a
+numbered header. A result is a match with its `-A/-B/-C` lines; matches whose context touches are one result.
+
+```sh
+$ sys1grep -n -C1 --rank -e "the refund was refused" tickets/
+1. tickets/b.txt
+tickets/b.txt-2-Order #1234, placed 2026-08-01.
+tickets/b.txt:3:Your refund was declined: the order is older than 30 days.
+tickets/b.txt-4-Please contact support.
+
+2. tickets/a.txt
+tickets/a.txt-11-Order #88 arrived.
+tickets/a.txt:12:Refund request received.
+tickets/a.txt-13-We will check it.
+```
+
+- `--rank` is `--rank=jev`: after the search, Jev is asked of each result, lines and context together, whether it
+  is relevant to the meanings that are not negated. One more question per result, `--chunk` results to a request;
+  `--dry-run` / `-i` show an upper bound, since which results there are is known only after the search.
+- `--rank=match` sorts by each result's highest match probability, with no request. It is the answer to a yes/no
+  question on one line, so clear matches sit close together, and the context is not read.
+- `-p` puts the score on the header (`1. [0.96] tickets/b.txt`). `-l` lists the files by their best result.
+- `--format=markdown` writes a `## 1. tickets/b.txt` heading and a fenced block per result; `--format=html` one
+  self-contained HTML document from a template (below), light and dark, with a relevance bar per result. The lines
+  are as they print, escaped and uncolored.
+- `--summarize` gets the results in ranked order; with `--dedup` a representative is one result.
+- It needs a meaning (a regex, `!` or `-v` alone ranks nothing), and cannot be combined with `-c`, `-o` or `-q`.
+  `--no-rank` turns off an earlier one, from `SYS1GREP_OPTS` say.
+
+### HTML templates (`--template`)
+
+`--rank --format=html` and `--summarize --format=html` fill in a template. Four come with sys1grep: `default`, `print` (black on white, serif,
+for paper and PDF), `search` (a results page like a web search engine's) and `terminal` (dark monospace).
+`--template=NAME` picks one; `SYS1GREP_TEMPLATE`, in the
+environment or `~/.config/sys1grep/.env`, sets the default.
+
+```sh
+$ sys1grep -r -n -C1 --rank -p --format=html --template=print -e "the refund was refused" tickets/ > refunds.html
+```
+
+A template is one HTML file. The part between `<!--result-->` and `<!--/result-->` is repeated for each result;
+what is before and after it is written once. sys1grep replaces these, escaped, and leaves any other `{{…}}` as
+written:
+
+| placeholder | where | value |
+|---|---|---|
+| `{{title}}` | anywhere | `sys1grep: ` and the meanings |
+| `{{query}}` | anywhere | the meanings, as in `"refund" and not "policy"` |
+| `{{count}}` | anywhere | the number of results (with `--summarize`, of matching lines) |
+| `{{answer}}` | anywhere | `--summarize`'s answer; empty with `--rank` |
+| `{{rank}}` | per result | 1, 2, … |
+| `{{score}}` | per result | the score as `-p` prints it (`0.96`); empty without `-p` |
+| `{{score_pct}}` | per result | the score as a whole number 0 to 100, set even without `-p` (for a bar: `style="--s:{{score_pct}}"`) |
+| `{{file}}` | per result | the file name; empty when one file is searched |
+| `{{lines}}` | per result | the result's lines as they print |
+
+`--template=NAME` reads `~/.config/sys1grep/templates/NAME.html` if it is there, else the bundled one, so your
+copy of `default` replaces it. A value containing `/` or ending in `.html` is a file path. To make your own, copy
+the bundled ones and edit:
+
+```sh
+$ sys1grep --install-templates            # "copied PATH" for each file
+$ cp ~/.config/sys1grep/templates/default.html ~/.config/sys1grep/templates/team.html
+$ sys1grep -r --rank --format=html --template=team -e "..." src/ > out.html
+$ sys1grep --template=list                # one name a line, "(user)" after those under ~/.config
+```
+
+`{{score_pct}}` is set even without `-p`, so a bar can show the relevance without printing the number.
+
+With `--summarize --format=html` the TOOL is asked for plain text, and that answer, escaped, goes into `{{answer}}`
+once it has finished; the part between `<!--result-->` and `<!--/result-->` is not written. A template for
+`--summarize` therefore needs `{{answer}}` outside that part (the bundled ones have it; with `--rank` it is empty).
+
+```sh
+$ sys1grep -r --summarize --format=html --template=print -e "why the refunds were refused" tickets/ > why.html
+```
+
+A template is trusted local input: a path you give, or a symlink in the templates directory, is read as it is;
+only the results put into it are escaped.
+
+`--install-templates` never overwrites a file that is already there (it prints `kept` for it). A missing
+template, or one without exactly one `<!--result-->` before one `<!--/result-->`, is exit 2 naming the file.
+`--template` on the command line needs `--format=html` with `--rank` or `--summarize` (not `-l`); `SYS1GREP_TEMPLATE` is simply unused elsewhere.
+`--template=list` and `--install-templates` take no other arguments and are refused in `SYS1GREP_OPTS`.
+
 ## Use it from Claude Code
 
 There is a Claude Code skill that runs sys1grep for you: describe what you are looking for in plain words
@@ -651,7 +917,11 @@ usage: sys1grep [OPTION]... -e MEANING|-Q QUESTION [-a MEANING] [-v MEANING]...
   -T THRESH    negative threshold: "not X" when probability < THRESH (overrides --level)
                with -t 0.6 -T 0.3 a line at 0.3..0.6 matches neither X nor not-X
   -r           recurse into directories (current directory when FILE is omitted);
-               skips .git, node_modules, binary files, likely secrets and what git ignores
+               skips .git, node_modules, binary files, likely secrets, generated files and what git ignores
+  --cached     git sys1grep only: search the index instead of the working tree, as git grep --cached
+  --untracked  git sys1grep only: also search untracked files (.gitignore still applies), as git grep --untracked
+  <tree>...    git sys1grep only: a branch, tag, commit or @{u} before -- searches that revision's tree
+               instead, as git grep does; output is prefixed <tree>: with the name as typed
   --include=GLOB, --exclude=GLOB  with -r and git sys1grep, only files whose name matches GLOB, or not
                (with -r a file named on the command line is always searched; git sys1grep's pathspecs are filtered)
   --changed-within=WHEN  with -r and git sys1grep, only files modified within 30m / 2h / 7d / 2w, since a date
@@ -670,14 +940,38 @@ usage: sys1grep [OPTION]... -e MEANING|-Q QUESTION [-a MEANING] [-v MEANING]...
                on ambiguous lines, not just speed
   -j N         concurrent requests (default 8)
   -n           print line numbers
-  -z, --null-data  judge NUL-terminated records instead of lines, and print them NUL-terminated (see "Records that span several lines" above)
-  --sentence[=HOW] judge each sentence instead of each line; HOW is jev (default) or rules (see "One sentence at a time" above)
-  -o           with --sentence, print only the matching sentences
+  --unit=UNIT  the unit of judgement: line (default), zero, sentence-by-jev, sentence-by-rule or function
+  -z, --null-data  --unit=zero: judge NUL-terminated records instead of lines, and print them NUL-terminated (see "Records that span several lines" above);
+               with --unit=sentence-by-*, each record is split into sentences
+  --unit=sentence-by-jev|sentence-by-rule  judge each sentence instead of each line (see "One sentence at a time" above)
+  --unit=function  judge each function and print its lines (see "One function at a time" above)
+  --step-to EXPRESSION  print the paths from the functions the expression before it matches, along the
+               calls, to the functions the expression after it matches (see "From one function to another" above)
+  --edges=FILE walk the edges in FILE instead of the calls; --reverse walks them backwards
+  --hops=N|M..N|M..  the hops an end may be at (default 0..)
+  -o           with --unit=sentence-by-*, print only the matching sentences
   -p           print each meaning's probability at the end of the line
-  --dry-run    send nothing; print the endpoint, each file searched and each request with its questions
+  --dry-run    send nothing; print the settings the search would run with (and their source, when not the
+               command line), each file searched and each request with its questions
   --verbose    print the same to stderr while searching
   -i, --interactive  show what --dry-run would send, and search only after y on the terminal
-  --dedup      judge one line per template and reuse its answer for the rest (see "One line per template" above)
+  -M NUM, --max-columns=NUM  send at most the first NUM characters of a line (or record, with -z); still
+               searched and judged, just not past that cutoff (default 2000, 8000 with -z)
+  --max-filesize=SIZE  size every target first (K/M/G, default 10M); over it, skip it outright, like rg's own
+               --max-filesize (-y does not affect it). stdin is sized once read, and skipped the same way if
+               it is over; a --cached / <tree>: target is a blob, sized from its content; -g's commits stay
+               out of this (each is already bounded by -M when sent)
+  --max-cost=USD  price the input about to be sent; over it (default 1), ask to continue on the terminal; -y
+               answers yes without asking; no terminal and the limit exceeded is exit 2, unchanged by -q
+               (scripts pass -y); -i already asks unconditionally and earlier, so this does not ask again
+  --dedup[=auto|always|never]  judge one line per template and reuse its answer for the rest (default never,
+               for now; see "One line per template" above)
+  --rank[=jev|match]  print the results (a match with its context) best first, under numbered headers; jev
+               (bare --rank) asks Jev of each result, match sorts by its best match probability (see "Best first"); --no-rank: file order
+  --template=NAME  the document --rank or --summarize --format=html writes: ~/.config/sys1grep/templates/NAME.html, else the
+               bundled one (default, print, terminal), or a file; default SYS1GREP_TEMPLATE, else default.
+               --template=list prints the names (see "HTML templates")
+  --install-templates  copy the bundled templates to ~/.config/sys1grep/templates, keeping existing files
   --color[=WHEN] auto (default: color when stdout is a terminal) / always / never; bare --color means auto
                file and line number use grep's colors; with -p, probabilities are
                green at or above the positive threshold, red below the negative one,
@@ -766,14 +1060,14 @@ fmt -w 100000 essay.txt | sys1grep -n -e "the author admits they made a mistake"
 
 Each output line is then a whole paragraph, and `-n` counts the lines of `fmt`'s output, not of the file.
 `fmt` puts a space where it joins two Japanese lines; Jev reads through it. To judge sentences across
-wrapped lines you need neither: `--sentence` already joins them.
+wrapped lines you need neither: `--unit=sentence-by-jev` already joins them.
 
 ### How do I search a JSONL chat log one message at a time?
 
 Take the text out with `jq` and end each message with NUL, then judge records with `-z`:
 
 ```sh
-jq -j '.content + "\u0000"' chat.jsonl | sys1grep -z -n --sentence -e "the customer is asking for a refund" | tr '\0' '\n'
+jq -j '.content + "\u0000"' chat.jsonl | sys1grep -z -n --unit=sentence-by-jev -e "the customer is asking for a refund" | tr '\0' '\n'
 ```
 
 Adjust `.content` to where your log keeps the text. Each message becomes one record, so a sentence never

package/docs/sentence.svg

@@ -1,6 +1,6 @@
 <svg xmlns="http://www.w3.org/2000/svg" width="808" height="240" font-family="SFMono-Regular,Menlo,Consolas,monospace" font-size="13">
 <rect width="100%" height="100%" rx="6" fill="#1e1e2e"/>
-<text x="16" y="29" xml:space="preserve"><tspan fill="#cdd6f4">$ sys1grep -n --sentence -C 3 --color \</tspan></text>
+<text x="16" y="29" xml:space="preserve"><tspan fill="#cdd6f4">$ sys1grep -n --unit=sentence-by-jev -C 3 --color \</tspan></text>
 <text x="16" y="49" xml:space="preserve"><tspan fill="#cdd6f4">    -e "the author admits they made a mistake" -e "customer is asking for a refund" tests/prose.txt</tspan></text>
 <text x="16" y="69" xml:space="preserve"><tspan fill="#8ce99a">1</tspan><tspan fill="#66d9e8">:</tspan><tspan fill="#ffd43b" font-weight="700">I should have checked the input</tspan></text>
 <text x="16" y="89" xml:space="preserve"><tspan fill="#8ce99a">2</tspan><tspan fill="#66d9e8">:</tspan><tspan fill="#ffd43b" font-weight="700">before shipping, and that was my</tspan></text>

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@uehaj/sys1grep",
-  "version": "0.5.0-next.0",
+  "version": "0.5.0-next.2",
   "description": "grep by meaning, across languages. TypeSafe Jev scores every line against a meaning; combine meanings with AND/OR/NOT.",
   "keywords": [
     "grep",
@@ -29,7 +29,11 @@
     "README.ja.md",
     "docs/color.svg",
     "docs/demo.svg",
-    "docs/sentence.svg"
+    "docs/sentence.svg",
+    "templates/default.html",
+    "templates/print.html",
+    "templates/search.html",
+    "templates/terminal.html"
   ],
   "engines": {
     "node": ">=20.16"
@@ -39,6 +43,8 @@
     "test:offline": "sh tests/offline.sh",
     "judge": "node --no-warnings --env-file-if-exists=.env tests/judge.mts",
     "scope-eval": "node tests/scope-eval.mjs",
+    "dedup-eval": "node tests/dedup-eval.mjs",
+    "skill-eval": "bash tests/skill-eval/run.sh",
     "release": "sh scripts/release.sh"
   }
 }