@uehaj/sys1grep 0.5.0-next.1 → 0.5.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -4,6 +4,8 @@
4
4
 
5
5
  Background and design notes (Japanese): [Jevのキラーアプリ、「意味で探す grep」を作った](https://zenn.dev/uehaj/articles/jev-semgrep-grep-by-meaning) on Zenn.
6
6
 
7
+ > **Renamed from semgrep (0.5.0).** Two reasons. The name collided with the static-analysis tool [Semgrep](https://semgrep.dev/), a trademark. And the tool was never meant for Jev alone: it targets System 1 models in general, and I'd like it to work with any models TypeSafe AI may release after Jev under other names :). So from 0.5.0 this is **sys1grep** (System 1 + grep): npm package `@uehaj/sys1grep` (was `@uehaj/semgrep`), commands `sys1grep` / `git sys1grep`, environment `SYS1GREP_*`, config `~/.config/sys1grep/.env`. The old `SEMGREP_*` names and `~/.config/semgrep/.env` still work for one minor release, each use printing a deprecation line; 1.0.0 drops them. Details: [CHANGELOG](CHANGELOG.md#renamed). The Zenn article's URL keeps its old slug.
8
+
7
9
  A grep that finds lines by **what they mean**, not by regular expressions.
8
10
  Matching is done by **Jev**, the System One model from [TypeSafe AI](https://typesafe.ai/).
9
11
  Jev does not generate text. It answers typed questions with probabilities, so for every line
@@ -11,15 +13,15 @@ sys1grep asks "does this line match the meaning *network failure*?", gets a prob
11
13
  and applies a threshold.
12
14
 
13
15
  ```sh
14
- ./sys1grep -n -e "customer is angry or frustrated" tickets.txt
16
+ sys1grep -n -e "customer is angry or frustrated" tickets.txt
15
17
  ```
16
18
 
17
- [![sys1grep demo: a Japanese meaning finds refund requests in six languages; "asking for a refund" vs "about a refund"; -Q finds the answer](docs/demo.svg)](https://uehaj.github.io/jev-semgrep/)
19
+ [![sys1grep demo: a Japanese meaning finds refund requests in six languages; "asking for a refund" vs "about a refund"; -Q finds the answer](docs/demo.svg)](https://sys1grep.js.org/)
18
20
 
19
- <sub>▶ Click the demo, or open <a href="https://uehaj.github.io/sys1grep/">uehaj.github.io/sys1grep</a>, for the full demo on the landing page.</sub>
21
+ <sub>▶ Click the demo, or open <a href="https://sys1grep.js.org/">sys1grep.js.org</a>, for the full demo on the landing page.</sub>
20
22
 
21
- - Zero dependencies. One file, Node.js 20.16+ and `fetch`.
22
- - Fast. 30 lines go into one request, requests run 8 at a time. A 210-line file finishes in under a second.
23
+ - Zero dependencies. Node.js 20.16+ and `fetch`, nothing else.
24
+ - Fast. 30 lines go into one request, requests run 8 at a time (1 for a local URL). A 210-line file finishes in under a second.
23
25
  - Meanings combine with AND / OR / NOT.
24
26
  - **Language-agnostic.** The meaning and the text can each be in any language. A Japanese meaning finds French, Russian, Chinese and Korean lines alike. No translation step, same speed, same cost.
25
27
 
@@ -31,7 +33,7 @@ so one query finds matching lines in every language the file contains.
31
33
  An **English** meaning finds **Japanese** lines. None of the hits contain "angry" or "frustrated", and two of them are in Japanese:
32
34
 
33
35
  ```sh
34
- $ ./sys1grep -n -e "customer is angry or frustrated" tests/corpus.txt
36
+ $ sys1grep -n -e "customer is angry or frustrated" tests/corpus.txt
35
37
  14:ユーザー山田さんからの問い合わせ: 返金してほしい、商品が壊れていた
36
38
  16:ユーザー佐藤さんからの問い合わせ: 注文した覚えのない請求が来ています。至急確認してください
37
39
  18:I want my money back. The item arrived broken and customer service ignored me.
@@ -42,7 +44,7 @@ $ ./sys1grep -n -e "customer is angry or frustrated" tests/corpus.txt
42
44
  A **Japanese** meaning finds **English** lines, with the same confidence as the Japanese ones:
43
45
 
44
46
  ```sh
45
- $ ./sys1grep -n -p -e "返金の要求" tests/corpus.txt
47
+ $ sys1grep -n -p -e "返金の要求" tests/corpus.txt
46
48
  14:ユーザー山田さんからの問い合わせ: 返金してほしい、商品が壊れていた [0.97]
47
49
  18:I want my money back. The item arrived broken and customer service ignored me. [0.95]
48
50
  ```
@@ -52,7 +54,7 @@ Russian, German, Spanish, Chinese and Korean. One Japanese meaning finds all six
52
54
  meaning does the same:
53
55
 
54
56
  ```sh
55
- $ ./sys1grep -n -p -e "顧客が返金を求めている" tests/multi.txt
57
+ $ sys1grep -n -p -e "顧客が返金を求めている" tests/multi.txt
56
58
  1:Je veux être remboursé, le produit est arrivé cassé. [0.98]
57
59
  3:Я требую вернуть деньги, товар не работает. [0.97]
58
60
  5:Ich möchte mein Geld zurück, das Gerät ist defekt. [0.97]
@@ -60,7 +62,7 @@ $ ./sys1grep -n -p -e "顧客が返金を求めている" tests/multi.txt
60
62
  9:我要求退款,商品坏了。 [0.97]
61
63
  11:환불해 주세요. 제품이 고장났어요. [0.97]
62
64
 
63
- $ ./sys1grep -n -p -e "клиент требует возврат денег" tests/multi.txt
65
+ $ sys1grep -n -p -e "клиент требует возврат денег" tests/multi.txt
64
66
  1:Je veux être remboursé, le produit est arrivé cassé. [0.94]
65
67
  3:Я требую вернуть деньги, товар не работает. [0.97]
66
68
  5:Ich möchte mein Geld zurück, das Gerät ist defekt. [0.92]
@@ -82,12 +84,12 @@ Jev reads the line and the question together (a cross-encoder shape), so who did
82
84
  "asked for" versus "already done" all change the answer. An embedding of the line is fixed before it
83
85
  ever sees your query, so it can only measure topical closeness.
84
86
 
85
- All eight lines below are "about a refund" (the same contrast as `tests/contrast.txt`, translated to
87
+ The first six lines below are "about a refund", and the last two are the pair the AND example further down uses (the same contrast as `tests/contrast.txt`, translated to
86
88
  English as `tests/contrast.en.txt` — see [Search across languages](#search-across-languages) above for
87
89
  the case where the meaning and the text differ in language). Only two are a customer asking for one:
88
90
 
89
91
  ```sh
90
- $ ./sys1grep -n -p -t 0 -e "customer is asking for a refund" tests/contrast.en.txt
92
+ $ sys1grep -n -p -t 0 -e "customer is asking for a refund" tests/contrast.en.txt
91
93
  1:I want a refund. The item was broken. [0.99]
92
94
  2:The refund has been processed. Please check your account. [0.11]
93
95
  3:Our refund policy is within 30 days of purchase. [0.09]
@@ -103,14 +105,14 @@ operations**, not a trick with set differences or "negative queries":
103
105
 
104
106
  ```sh
105
107
  # about a refund, but NOT a customer asking for one → completed, policy, denied, timelines
106
- $ ./sys1grep -n -e "about a refund" -v "the customer is asking for a refund" tests/contrast.en.txt
108
+ $ sys1grep -n -e "about a refund" -v "the customer is asking for a refund" tests/contrast.en.txt
107
109
  2:The refund has been processed. Please check your account.
108
110
  3:Our refund policy is within 30 days of purchase.
109
111
  4:The manager denied the refund request yesterday
110
112
  6:Refunds are processed within 5 business days
111
113
 
112
114
  # angry AND it is the customer, not the staff
113
- $ ./sys1grep -n -e "someone is angry" -a "the customer, not the staff, is the one acting" tests/contrast.en.txt
115
+ $ sys1grep -n -e "someone is angry" -a "the customer, not the staff, is the one acting" tests/contrast.en.txt
114
116
  5:I demand a full refund immediately
115
117
  8:The customer got angry and hung up the phone.
116
118
  ```
@@ -129,18 +131,24 @@ repeated queries over a large, fixed corpus a vector index is cheaper and faster
129
131
 
130
132
  ## Regex terms
131
133
 
132
- `-e`/`-a`/`-v '/pattern/flags'` (first and last character `/`, JavaScript flags) is matched locally,
133
- as a plain regex, with no request at all. It prefilters its AND term: only the lines it holds for
134
- ever ask that term's meanings, so a cheap regex in front of a meaning cuts both the bill and the wait. A line is sent
135
- only if some term's regexes all hold for it (a term with no regex holds for every line), so with
136
- `-e '/re/' -a A -e B` a line without `re` is still sent, asked `B` only. A query of regex terms alone
137
- sends nothing, even with `--unit=sentence-by-jev`: the rules alone then decide where wrapped lines join. Anything that isn't shaped like `/…/flags` is still a meaning, so
138
- `-e '/etc 以下のファイルを変更している'` (no closing `/`) is unaffected; a meaning that really starts
139
- and ends with `/` can be written with a leading space to dodge the regex reading.
134
+ `-P '/pattern/flags'` (first and last character `/`, JavaScript flags) starts a regex term, OR'd like
135
+ `-e`, and is matched locally with no request at all. `-a`/`-v '/pattern/flags'` AND / AND NOT a regex
136
+ onto the preceding term, the same shape deciding regex from meaning there. It prefilters its AND term:
137
+ only the lines it holds for ever ask that term's meanings, so a cheap regex in front of a meaning cuts
138
+ both the bill and the wait. A line is sent only if some term's regexes all hold for it (a term with no
139
+ regex holds for every line), so with `-P '/re/' -a A -e B` a line without `re` is still sent, asked `B`
140
+ only. A query of regex terms alone sends nothing, even with `--unit=sentence-by-jev`: the rules alone
141
+ then decide where wrapped lines join. `-P '!/RE/'` is not allowed; negate a regex with `-v '/RE/'`
142
+ instead. `-e` never reads its value as a regex: a `/RE/flags`-shaped value there is refused, with a
143
+ hint naming `-P` (e.g. `-e '/catch/' is no longer a regex: use -P '/catch/'`). `-Q` is unaffected: it
144
+ was already always a question, never a regex, so `-e '/etc 以下のファイルを変更している'` (no closing
145
+ `/`) still goes through as a meaning, unrefused. A meaning for `-e`/`-a`/`-v` that really starts and
146
+ ends with `/` can still be written with a leading space to dodge the regex reading (`-e ' /cat/'`
147
+ sends the text `" /cat/"`, not a refusal).
140
148
 
141
149
  ```sh
142
- $ ./sys1grep -e '/ERROR|FATAL/' app.log # no requests at all
143
- $ ./sys1grep -e '/timeout/i' -a '顧客に影響が出ている' app.log # only lines with "timeout" go to Jev
150
+ $ sys1grep -P '/ERROR|FATAL/' app.log # no requests at all
151
+ $ sys1grep -P '/timeout/i' -a '顧客に影響が出ている' app.log # only lines with "timeout" go to Jev
144
152
  ```
145
153
 
146
154
  A regex's named and numbered groups pass to the other meanings of the *same* AND term as
@@ -150,7 +158,7 @@ an error rather than an empty string (so is a reference to a negated regex's gro
150
158
  no group stays literal, as in ECMAScript, so `$100 以上の請求` is unaffected.
151
159
 
152
160
  ```sh
153
- $ ./sys1grep -e '/(?<date>\d{4}-\d\d-\d\d) (?<time>\d\d:\d\d)/' \
161
+ $ sys1grep -P '/(?<date>\d{4}-\d\d-\d\d) (?<time>\d\d:\d\d)/' \
154
162
  -a '$<time> が深夜(0時〜5時)であり、$<date> が週末である' app.log
155
163
  # 2026-09-19 03:12 ... → asks "03:12 が深夜(0時〜5時)であり、2026-09-19 が週末である"
156
164
  ```
@@ -163,7 +171,7 @@ is never colored. `-o` prints each match on a line of its own, as `grep -o`; a l
163
171
  whole, since a meaning has no matching part.
164
172
 
165
173
  ```sh
166
- $ ./sys1grep -o -n -e '/[A-Z]+-\d+/' -a 'the ticket is still open' notes.txt # the ticket ids, one per line
174
+ $ sys1grep -o -n -P '/[A-Z]+-\d+/' -a 'the ticket is still open' notes.txt # the ticket ids, one per line
167
175
  ```
168
176
 
169
177
  ## Sending less
@@ -180,10 +188,18 @@ to the narrowest:
180
188
  its AND term's meanings. Blank lines are never sent. Only the first `-M`/`--max-columns` characters of a line
181
189
  (default 2000, 8000 with `-z`) are sent: it is still searched and judged, just not past that cutoff.
182
190
  - **How many times.** [`--dedup`](#one-line-per-template---dedup) judges one line per template: lines that
183
- differ only in ids, numbers, times or paths share one answer.
191
+ differ only in ids, numbers, times or paths share one answer. `auto`/`always`/`never`, off by default for
192
+ now: a hint says when `--dedup=auto` would pay.
193
+ - **Never twice.** `--cache` keeps Jev's answers in `~/.config/sys1grep/cache`, keyed by the request sent (model,
194
+ lines, questions) and the endpoint's host, so the same search again sends nothing and costs nothing. It is off
195
+ by default, since the cache holds the lines searched; `SYS1GREP_CACHE=on` (in the environment or
196
+ `~/.config/sys1grep/.env`, not settings.json) sets the default. `--cache=only` answers from the cache alone: a
197
+ request not there is exit 2 unless `-q` already found a match, named, and nothing is sent. `--dry-run` says how
198
+ many requests are cached and how many it would send. There is no expiry: the model is in the key, and Jev
199
+ drifts about ±0.05 between runs, so a cached answer is one of those runs. `--cache=clear` empties it.
184
200
  - **Check before paying.** `--dry-run` sends nothing and prints the settings the search would run with, the
185
201
  files, how many lines each would send and every request with its questions. Its last line estimates the input
186
- tokens and, for TypeSafe itself, the price (`~3178 input tokens, ~$0.000133`; within about 10%). `-i` shows the
202
+ tokens and the price (`~3178 input tokens, ~$0.000133`; within about 10%; see "Other endpoints" for whose price). `-i` shows the
187
203
  same totals on the terminal and sends only after `y`.
188
204
  Before the bulk of requests, every target is sized (`--max-filesize`, default 10M); one over it is skipped
189
205
  outright, like `rg`'s own `--max-filesize`, named on stderr (`-y` does not affect it). A `.gz` is sized
@@ -194,20 +210,24 @@ to the narrowest:
194
210
  is sent. The input about to be sent is priced too (`--max-cost`,
195
211
  default 1 USD); over it, one question asks to continue on the terminal, `-y` answers it yes, and without a
196
212
  terminal it is exit 2 (this applies with `-q` too — scripts pass `-y`).
213
+ With `-r` or `git sys1grep`, a term no regex narrows that is about to send more than 10,000 lines gets one
214
+ stderr line with the same totals, naming the term (`sys1grep: sending 214,913 of 231,502 lines from 1,247 files
215
+ (~9.1M input tokens, ~$0.38); the term "…" has no regex to narrow it. Add -a '/RE/' to it, …`). The search goes
216
+ on; `-q` silences the line, `-y` does not.
197
217
 
198
218
  ```sh
199
- $ sys1grep --dry-run -r --include='*.log' --changed-within=today -e '/ERROR|FATAL/' -a 'a customer is affected' logs/
219
+ $ sys1grep --dry-run -r --include='*.log' --changed-within=today -P '/ERROR|FATAL/' -a 'a customer is affected' logs/
200
220
  ```
201
221
 
202
222
  `--verbose` (or `--dry-run`) also shows *where* a setting that was not typed on the command line came from
203
- (`SYS1GREP_OPTS`, an environment variable, `~/.config/sys1grep/.env`, or a preset's default), so a result that
223
+ (`SYS1GREP_OPTS`, an environment variable, `~/.config/sys1grep/settings.json` or `.env`, or a preset's default), so a result that
204
224
  surprises you can be traced back to its source. The key's value never appears, only which option or variable
205
225
  supplied it:
206
226
 
207
227
  ```sh
208
228
  $ SYS1GREP_OPTS='--level strict' sys1grep --verbose -e "the API key is read from a file" .
209
229
  sys1grep: endpoint api.typesafe.ai/v1/systemone (default), model jev-latest (default)
210
- sys1grep: key: SYS1GREP_API_KEY (~/.config/sys1grep/.env)
230
+ sys1grep: key: key (~/.config/sys1grep/settings.json)
211
231
  sys1grep: SYS1GREP_OPTS: --level strict
212
232
  sys1grep: options: --level strict (SYS1GREP_OPTS) = -t 0.7 -T 0.3, --chunk 30, -j 8, scope on, -M 2000, --max-filesize 10M, --max-cost 1
213
233
  sys1grep: file ./a.py: 120 lines, 120 to send
@@ -217,7 +237,7 @@ sys1grep: file ./a.py: 120 lines, 120 to send
217
237
  ## Install
218
238
 
219
239
  Two ways to use it: as a command-line tool (this section), or as a Claude Code skill
220
- (see [Use it from Claude Code](#use-it-from-claude-code) below). The skill falls back to `npx @uehaj/sys1grep`,
240
+ (see [Use it from Claude Code](#use-it-from-claude-code) below). The skill runs sys1grep through `npx`,
221
241
  so if you only use it through Claude Code you can skip the install here entirely and just set the API key.
222
242
 
223
243
  Requires Node.js 20.16 or later. No other dependencies.
@@ -237,14 +257,34 @@ Then give it an API key from the [TypeSafe console](https://console.typesafe.ai/
237
257
 
238
258
  ```sh
239
259
  export SYS1GREP_API_KEY=your-key # environment variable
240
- echo 'SYS1GREP_API_KEY=your-key' > ~/.config/sys1grep/.env # per user (mkdir -p first)
260
+ # per user: ~/.config/sys1grep/settings.json (mkdir -p first; keep it at 0600)
261
+ (umask 077; echo '{"key": "your-key"}' > ~/.config/sys1grep/settings.json)
241
262
  ```
242
263
 
243
- Variables already in the environment win; otherwise `~/.config/sys1grep/.env` fills them in. A `.env` in the current
264
+ Variables already in the environment win; otherwise `~/.config/sys1grep/settings.json` fills them in, one field per
265
+ variable:
266
+
267
+ | field | variable |
268
+ |---|---|
269
+ | `url` | `SYS1GREP_URL` |
270
+ | `key` | `SYS1GREP_API_KEY` (and `TYPESAFE_API_KEY`) |
271
+ | `model` | `SYS1GREP_MODEL` |
272
+ | `opts` | `SYS1GREP_OPTS`, as an array: `["--level", "strict", "-n"]` |
273
+ | `summarizer` | `SYS1GREP_SUMMARIZER` |
274
+ | `summarizerModel` | `SYS1GREP_SUMMARIZER_MODEL` |
275
+ | `summarizerKey` | `SYS1GREP_SUMMARIZER_API_KEY` |
276
+ | `lang` | `SYS1GREP_LANG` (`en` or `ja`: `--help`'s language, and `--serve`'s page default) |
277
+
278
+ Every field is optional. A value of the wrong type is an error, as is a file that is not JSON. A field the file does
279
+ not know only warns (on stderr, naming the field) and is ignored, so an older sys1grep keeps working once a newer one
280
+ adds a field. A file that holds a key and that others can read gets a warning. `~/.config/sys1grep/.env` (`NAME=value`
281
+ lines) still works, below settings.json: a value in settings.json wins over the same variable there. A `.env` in the current
244
282
  directory is never read: it may belong to a repository you just cloned, and could send your key elsewhere through
245
283
  `SYS1GREP_URL`. For per-project settings, load a file yourself: `node --env-file=.env "$(command -v sys1grep)" ...`.
246
284
  `TYPESAFE_API_KEY` is accepted too when `SYS1GREP_API_KEY` is not set.
247
285
 
286
+ The `--serve` page's **Settings** dialog edits this settings.json too (see [A search page](#a-search-page---serve)).
287
+
248
288
  ### Default options
249
289
 
250
290
  `SYS1GREP_OPTS` holds options to apply on every call, read like the settings above. It is split on spaces and put in
@@ -256,15 +296,18 @@ sys1grep -e "payment failed" app.log # strict, 8 at once, line
256
296
  sys1grep --level loose --no-n -e "payment failed" app.log
257
297
  ```
258
298
 
299
+ The same defaults can live in settings.json as `"opts": ["--level", "strict", "-j", "8", "-n"]`, one argument per
300
+ element (so a value may hold spaces). `SYS1GREP_OPTS`, when set, replaces them rather than adding to them.
301
+
259
302
  A script calling sys1grep would pick these up too (grep dropped `GREP_OPTIONS` for that reason). Call it as
260
- `SYS1GREP_OPTS= sys1grep ...` in scripts.
303
+ `SYS1GREP_OPTS= sys1grep ...` in scripts: the empty variable drops settings.json's `opts` as well.
261
304
 
262
305
  ### Other endpoints
263
306
 
264
307
  The API is configured by exactly three settings: `SYS1GREP_API_KEY` (or `TYPESAFE_API_KEY`), `SYS1GREP_URL` and `SYS1GREP_MODEL`.
265
308
  Any endpoint that speaks TypeSafe's `POST /v1/systemone` works. The key is sent to `SYS1GREP_URL` as is, so set the
266
309
  two together. On the command line, `--sys1-model=ID`, `--sys1-url=URL` and `--sys1-api-key=KEY` override
267
- the three. A key given this way shows up in `ps` and shell history, so prefer `~/.config/sys1grep/.env` for it.
310
+ the three. A key given this way shows up in `ps` and shell history, so prefer `~/.config/sys1grep/settings.json` for it.
268
311
 
269
312
  ```sh
270
313
  # OpenRouter
@@ -275,8 +318,31 @@ SYS1GREP_URL=https://ai-gateway.vercel.sh/typesafe/v1/systemone SYS1GREP_MODEL=t
275
318
  SYS1GREP_URL=http://localhost:8000/v1/systemone sys1grep -e ...
276
319
  ```
277
320
 
278
- The summary line shows the cost the endpoint reports (`usage.cost`), or for TypeSafe itself an estimate at list
279
- price marked `~`.
321
+ Every price sys1grep shows (`--dry-run`, the large-send warning, the `--max-cost` question, the summary line) is the cost
322
+ the endpoint reports (`usage.cost`), or else an estimate marked `~`. The estimate's price per million input tokens: a local
323
+ URL is free (`localhost`, `127.0.0.1` or `[::1]`, judged by the host as typed with no DNS lookup, and only for a URL you set
324
+ with `SYS1GREP_URL` or `--sys1-url`); a model in this table has its own price, `jev-latest` 0.042, `clef` 0.24 and
325
+ `clef-flash` 0.09 (USD); any other model is priced as Jev, 0.042. With another URL the estimate says whose price it is:
326
+ `at TypeSafe's list price`, `at clef's list price` or `(local URL, free)`. `--max-cost` uses the same price, so it asks
327
+ about nothing on a local URL.
328
+
329
+ **Local decision models.** A model that serves the same `POST /v1/systemone` can be used instead of Jev, with no key and no
330
+ network, but both tried so far are less accurate than Jev: raise `-t`, tune with `npm run judge`, and expect more false
331
+ positives. Both also run one request at a time, so a local URL (`localhost`, `127.0.0.1`, `[::1]`) defaults to `-j 1`;
332
+ a `-j` you give (on the command line, in `SYS1GREP_OPTS` or settings.json) still wins.
333
+
334
+ - [Jeff](https://github.com/firelex/jeff) (`jeff-serve`, 0.8B, base v1.2, zero-shot, MLX): works with
335
+ `--sys1-url=http://127.0.0.1:8765/v1/systemone --sys1-model=jeff-latest`. It answers a second request at once with
336
+ `529 "The model is busy"`, and `-j 8` runs out of retries. On `npm run judge` (51 lines, 10 cases, `-j 1`)
337
+ precision 0.68, recall 0.61, F1 0.64 at the default thresholds and F1 0.77 at `-t 0.75 -T 0.85`, against F1 0.96 for Jev
338
+ (a run of another day): not a drop-in. v1.2 is a community preview; adapters do not carry across versions.
339
+ - [strands-decider](https://github.com/strands-labs/strands-decider) (`strands-decider serve`, 2B, an M4 with `mps`):
340
+ `SYS1GREP_URL=http://127.0.0.1:8000/v1/systemone` with no key works as is (30 lines in 2.7 s). Two requests at once crash
341
+ its server (exit 134, a Metal assertion). On the same `npm run judge` set: precision 0.64, recall 0.94, F1 0.76 at the
342
+ default thresholds and F1 0.81 at `-t 0.6 -T 0.45`.
343
+ - Not tried with sys1grep yet: Cloudflare's Clef.
344
+
345
+ A request carries at most 64 questions for every backend.
280
346
 
281
347
  From source: `git clone https://github.com/uehaj/sys1grep.git && cd sys1grep && npm install -g .`,
282
348
  or run it in place with `node sys1grep.mjs ...`.
@@ -289,10 +355,12 @@ otherwise show up in the output below, this section instead uses [`tests/corpus.
289
355
  the same 51 lines with the Japanese ones translated to English — the cross-language behaviour itself is
290
356
  shown once, in [Search across languages](#search-across-languages) above.
291
357
 
358
+ For examples organised by job (on-call, triage, release notes, ...) and run on real data, see [docs/use-cases.md](docs/use-cases.md).
359
+
292
360
  ### Find lines by a concept, in any language
293
361
 
294
362
  ```sh
295
- $ ./sys1grep -n -e "customer is angry or frustrated" tests/corpus.en.txt
363
+ $ sys1grep -n -e "customer is angry or frustrated" tests/corpus.en.txt
296
364
  14:Inquiry from user Yamada: I want a refund, the item was broken
297
365
  16:Inquiry from user Sato: I'm being charged for an order I never placed, it looks like fraud, please check urgently
298
366
  18:I want my money back. The item arrived broken and customer service ignored me.
@@ -309,8 +377,8 @@ None of these lines contain the words "angry" or "frustrated".
309
377
  so you can write the question as you would ask it:
310
378
 
311
379
  ```sh
312
- $ echo "The job failed." | ./sys1grep -e "Did the job succeed?"
313
- $ echo "The job failed." | ./sys1grep -Q "Did the job succeed?"
380
+ $ echo "The job failed." | sys1grep -e "Did the job succeed?"
381
+ $ echo "The job failed." | sys1grep -Q "Did the job succeed?"
314
382
  The job failed.
315
383
  ```
316
384
 
@@ -319,7 +387,7 @@ the line answers it: no, it failed (0.83). A line that *asks* the question is cl
319
387
  answers nothing, and for a yes/no question a line that *denies* it still answers it:
320
388
 
321
389
  ```sh
322
- $ ./sys1grep -n -Q "whether the server is down" tests/intent.txt
390
+ $ sys1grep -n -Q "whether the server is down" tests/intent.txt
323
391
  5:The server is down.
324
392
  6:The server is healthy and responding normally.
325
393
  2 of 17 lines matched; 17 sent to Jev in 1 request, 1087 input tokens, ~$0.000046
@@ -333,7 +401,7 @@ would match it instead. `-Q X` is shorthand for `-e "the line answers: X"`, so i
333
401
  ### OR: two meanings, and see the probabilities with `-p`
334
402
 
335
403
  ```sh
336
- $ ./sys1grep -n -p -e "customer is asking for a refund" -e "delivery address change request" tests/corpus.en.txt
404
+ $ sys1grep -n -p -e "customer is asking for a refund" -e "delivery address change request" tests/corpus.en.txt
337
405
  14:Inquiry from user Yamada: I want a refund, the item was broken [0.99 0.01]
338
406
  17:Inquiry from user Takahashi: I'd like to change the delivery address [0.01 0.99]
339
407
  18:I want my money back. The item arrived broken and customer service ignored me. [0.96 0.01]
@@ -351,7 +419,7 @@ green at or above `-t`, red below `-T`, yellow in between. Line numbers and file
351
419
  ### AND NOT: network errors, excluding retries
352
420
 
353
421
  ```sh
354
- $ ./sys1grep -n -e "network or remote connection failure" -v "a retry is happening or was attempted" tests/corpus.txt
422
+ $ sys1grep -n -e "network or remote connection failure" -v "a retry is happening or was attempted" tests/corpus.txt
355
423
  4:2026-09-19 08:02:30 ERROR connection reset by peer while calling payment-gateway
356
424
  6:2026-09-19 08:02:35 ERROR timeout after 5000ms waiting for payment-gateway
357
425
  9:2026-09-19 08:10:44 ERROR DNS lookup failed for api.example.com
@@ -359,7 +427,7 @@ $ ./sys1grep -n -e "network or remote connection failure" -v "a retry is happeni
359
427
  13:network unreachable: no route to host 10.0.0.5
360
428
  30:except ConnectionError as e:
361
429
  31: logger.error("upstream unreachable: %s", e)
362
- 7 of 51 lines matched; 51 sent to Jev in 2 requests, 5112 input tokens
430
+ 7 of 51 lines matched; 51 sent to Jev in 2 requests, 5112 input tokens, ~$0.000215
363
431
  ```
364
432
 
365
433
  Line 5, `retrying payment-gateway request (attempt 2/3)`, is a network failure but is dropped by `-v`.
@@ -367,7 +435,7 @@ Line 5, `retrying payment-gateway request (attempt 2/3)`, is a network failure b
367
435
  ### Mixed: (finance AND negative) OR weather
368
436
 
369
437
  ```sh
370
- $ ./sys1grep -n -e "about economy, finance or markets" -a "the news is negative or a decline" -e "about weather" tests/corpus.en.txt
438
+ $ sys1grep -n -e "about economy, finance or markets" -a "the news is negative or a decline" -e "about weather" tests/corpus.en.txt
371
439
  36:Today's weather is sunny, high of 28 degrees
372
440
  43:Stock prices fell 3% after the earnings report missed expectations.
373
441
  48:Tomorrow's forecast is rain so I'll bring an umbrella
@@ -379,11 +447,11 @@ $ ./sys1grep -n -e "about economy, finance or markets" -a "the news is negative
379
447
  ### Strictness presets
380
448
 
381
449
  ```sh
382
- $ ./sys1grep --level strict -n -e "a security risk or dangerous destructive operation" tests/corpus.en.txt
450
+ $ sys1grep --level strict -n -e "a security risk or dangerous destructive operation" tests/corpus.en.txt
383
451
  33:DROP TABLE sessions;
384
452
  49:API keys must never be committed to the repository.
385
453
 
386
- $ ./sys1grep --level loose -n -e "a security risk or dangerous destructive operation" tests/corpus.en.txt
454
+ $ sys1grep --level loose -n -e "a security risk or dangerous destructive operation" tests/corpus.en.txt
387
455
  11:2026-09-19 09:00:00 ERROR SSL handshake failed: certificate expired
388
456
  16:Inquiry from user Sato: I'm being charged for an order I never placed, it looks like fraud, please check urgently
389
457
  33:DROP TABLE sessions;
@@ -395,11 +463,11 @@ $ ./sys1grep --level loose -n -e "a security risk or dangerous destructive opera
395
463
  ### Recursive search and file names only
396
464
 
397
465
  ```sh
398
- $ ./sys1grep -r -n -e "customer is asking for a refund" tests/tickets/
466
+ $ sys1grep -r -n -e "customer is asking for a refund" tests/tickets/
399
467
  tests/tickets/a.txt:7:Ticket #16: I want a refund, the item was broken.
400
468
  tests/tickets/sub/b.txt:1:The customer wants a refund for the broken lamp.
401
469
 
402
- $ ./sys1grep -rl -e "customer is asking for a refund" tests/tickets/
470
+ $ sys1grep -rl -e "customer is asking for a refund" tests/tickets/
403
471
  tests/tickets/a.txt
404
472
  tests/tickets/sub/b.txt
405
473
  ```
@@ -409,8 +477,8 @@ tests/tickets/sub/b.txt
409
477
  that usually hold secrets (`.env*`, `.netrc`, `.npmrc`, `.pypirc`, `.pgpass`, `.git-credentials`, `*.pem`,
410
478
  `*.key`, `*.p12`, `*.pfx`, `*.jks`, `*.keystore`, `id_rsa*` and friends; names compared without case) and
411
479
  generated files (`*.map`, `*.min.js`, `*.min.css`, `package-lock.json`, `yarn.lock`, `pnpm-lock.yaml`,
412
- `Cargo.lock`, `poetry.lock`, `composer.lock`, `Gemfile.lock`, `go.sum`). **Every line that is searched is sent
413
- to the TypeSafe API**, so point `-r` at a directory
480
+ `Cargo.lock`, `poetry.lock`, `composer.lock`, `Gemfile.lock`, `go.sum`), symlinks and `~/.config/sys1grep/settings.json` itself. **Every line that is searched is sent
481
+ to the TypeSafe API** (or wherever `SYS1GREP_URL` points), so point `-r` at a directory
414
482
  you mean to scan. Inside a git repository, `-r` also skips what git ignores (`.gitignore`, `.git/info/exclude`, the
415
483
  global excludes file), so build output and local files stay home; a tracked file is searched even if it matches.
416
484
  A file named `*.gz` (a rotated log: `app.log.1.gz`) is read decompressed, as `zgrep` does, and printed by its
@@ -434,7 +502,7 @@ $ sys1grep -r -e 'Python でリトライ処理を書いている箇所' .
434
502
  sys1grep: scope: *.py *.pyi *.pyw (from "Python files: 0.94")
435
503
  sys1grep: scope: 12 of 340 files
436
504
  $ sys1grep -r -e '昨日変えた箇所で認証を扱っている' src/
437
- sys1grep: scope: modified since 2026-09-25 00:00 (from "what was changed yesterday: 0.91")
505
+ sys1grep: scope: changed since 2026-09-25 00:00 (git commits; the mtime for files git does not have committed) (from "what was changed yesterday: 0.91")
438
506
  sys1grep: scope: 3 of 120 files
439
507
  ```
440
508
 
@@ -503,12 +571,25 @@ tree, not `git ls-tree`'s own literal/directory-prefix match); `--include` / `--
503
571
  **An old `<tree>` can hold a secret an ordinary file once carried and was later removed from**: the skip list
504
572
  drops files by name only, not by what changed since.
505
573
 
574
+ ### Commits instead of files (`-g`)
575
+
576
+ `-g` (`--gitlog`) searches the commits of `git log` instead of files, one record each (hash, date, subject, body):
577
+
578
+ ```sh
579
+ $ sys1grep -g -e 'a performance fix to the .mjs files today'
580
+ ```
581
+
582
+ With one term, auto-scope becomes `git log`'s own arguments, printed on stderr: a time becomes `--since-as-filter`
583
+ (`--since` on git before 2.37, and `--until` when the span has an end: yesterday, last week, last month), a language
584
+ a pathspec, an author or "mine" `--author`, "this branch" or "not pushed" a range. FILE arguments are pathspecs.
585
+ Each commit is a record, as with `-z`, so `-M` defaults to 8000. Not with `-r` or `git sys1grep`.
586
+
506
587
  ### Everything that is *not* something
507
588
 
508
589
  ```sh
509
- ./sys1grep -v "a timestamped server log line" mixed.txt # like grep -v
510
- ./sys1grep -e "source code or SQL" -v "SQL" src.txt # code, but not SQL
511
- cat app.log | ./sys1grep -e "the deploy failed or was rolled back"
590
+ sys1grep -v "a timestamped server log line" mixed.txt # like grep -v
591
+ sys1grep -e "source code or SQL" -v "SQL" src.txt # code, but not SQL
592
+ cat app.log | sys1grep -e "the deploy failed or was rolled back"
512
593
  ```
513
594
 
514
595
  ### Records that span several lines (`-z`)
@@ -521,9 +602,9 @@ A proposition like "this commit changes user-visible behaviour" is true of a who
521
602
  line in it:
522
603
 
523
604
  ```sh
524
- $ git log -z --format='%h %s %b' | ./sys1grep -z -n -e "the change alters user-visible behaviour" -v "documentation only"
525
- 7:21120e9 Revert "feat: ship the /sys1grep Claude Code skill" ...
526
- 8:51ae333 feat: ship the /sys1grep Claude Code skill
605
+ $ git log -z --format='%h %s %b' | sys1grep -z -n -e "the change alters user-visible behaviour" -v "documentation only"
606
+ 7:21120e9 Revert "feat: ship the /semgrep Claude Code skill" ...
607
+ 8:51ae333 feat: ship the /semgrep Claude Code skill
527
608
  ```
528
609
 
529
610
  Matching records are printed NUL-terminated too, so pipe them through `tr '\0' '\n'` to read them.
@@ -538,7 +619,7 @@ Wrapped lines are joined before splitting, so a sentence that runs over several
538
619
  [`tests/prose.txt`](tests/prose.txt) wraps an English paragraph and a Japanese one:
539
620
 
540
621
  ```sh
541
- $ ./sys1grep -n --unit=sentence-by-jev -e "the author admits they made a mistake" tests/prose.txt
622
+ $ sys1grep -n --unit=sentence-by-jev -e "the author admits they made a mistake" tests/prose.txt
542
623
  1:I should have checked the input
543
624
  2:before shipping, and that was my
544
625
  3:mistake. Next time I will add a test
@@ -557,7 +638,7 @@ inside a line: lines 3 and 9 are colored only up to the end of the matching sent
557
638
  Thai, Lao, Khmer, Myanmar and Tibetan, which do not put spaces between words:
558
639
 
559
640
  ```sh
560
- $ ./sys1grep -n -o --unit=sentence-by-jev -e "the author admits they made a mistake" -e "customer is asking for a refund" tests/prose.txt
641
+ $ sys1grep -n -o --unit=sentence-by-jev -e "the author admits they made a mistake" -e "customer is asking for a refund" tests/prose.txt
561
642
  1:I should have checked the input before shipping, and that was my mistake.
562
643
  8:先週買った掃除機が初日から動かないので返金してほしいです。
563
644
  ```
@@ -594,7 +675,7 @@ In code the question is usually about a function ("retries on network failure"),
594
675
  says so. `--unit=function` judges each function and prints its lines, with `-n` giving file line numbers:
595
676
 
596
677
  ```sh
597
- $ ./sys1grep -n --unit=function -e "retries on network failure" src/net.js
678
+ $ sys1grep -H -n --unit=function -e "retries on network failure" src/net.js
598
679
  src/net.js:3:async function fetchWithRetry(url) {
599
680
  src/net.js:4: for (let i = 0; i < 5; i++) {
600
681
  src/net.js:5: try { return await fetch(url); }
@@ -620,6 +701,59 @@ $ git config diff.golang.xfuncname '^(func|type)[[:space:]]'
620
701
  `--chunk` counts functions, `-A`/`-B`/`-C` and `-c` count lines, as with the sentence units. `-z`, `-g` and
621
702
  `-o` are refused.
622
703
 
704
+ ### From one function to another (`--step-to`)
705
+
706
+ A question about code often comes in two steps: where does `--summarize` hand the lines to the tool, and then,
707
+ what happens when that fails. The "that" in the second step is what the first one found. Multi-step matching
708
+ answers both in one run: the expression before `--step-to` finds the start, sys1grep walks the calls from it,
709
+ and the expression after `--step-to` picks the functions it reaches that match. Jev judges the two
710
+ expressions; no score decides which call to follow.
711
+
712
+ ```sh
713
+ $ sys1grep -e "--summarize hands the lines to the tool" --step-to "what happens when the tool fails to start or answer" sys1grep.mjs
714
+ ```
715
+
716
+ A meaning may start with a dash, as the first one does, on any search and for `-e`, `-a`, `-v` and `-Q`: a value that starts with `-` is taken as the meaning unless it is an option (an option holds no space).
717
+
718
+ Each path prints as a tree: the hop (the number of calls from the start), `file:line` and the function's name,
719
+ `:` after an end and `-` after a function on the way, as grep marks a match and its context. A regex-only
720
+ start and end send nothing, which makes a free reachability search. In Python 3.6's `json`, the functions
721
+ that `load` reaches and that hold a `raise`:
722
+
723
+ ```
724
+ $ cd /usr/lib64/python3.6/json && sys1grep -P '/^\s*def load\b/' --step-to -P '/\braise\b/' *.py
725
+ sys1grep: walk: 1 at hop 0, 1 at hop 1, 3 at hop 2, 1 at hop 3, 1 at hop 4, 1 at hop 5; stopped: no new unit
726
+ 0 __init__.py-274-load
727
+ 1 __init__.py:302:loads
728
+ 2 decoder.py:334:decode
729
+ 3 decoder.py:345:raw_decode
730
+ 4 scanner.py-65-scan_once
731
+ 5 scanner.py:28:_scan_once
732
+ ```
733
+
734
+ - **The expressions.** The expression before `--step-to` is the start's, as in any search; every `-e` / `-P` /
735
+ `-a` / `-v` / `-Q` after it is the end's. A bare `--step-to MEANING` is `--step-to -e MEANING` and a bare
736
+ `--step-to '/RE/'` is auto-detected and becomes `--step-to -P '/RE/'`; the file names follow. Write the
737
+ explicit `-P` form for a regex end, as in the example above: it reads the same but names the option. A step
738
+ is a search, not a hop: one `--step-to` is two steps, whatever the hops between them.
739
+ - **The edges.** Without `--edges` the unit is a function (`--unit=function`) and an edge is a call: `name(` in
740
+ a function's body, its comments, docstrings and strings left out, links to every function of that name. A name
741
+ comes from the funcname line (the `def` under a decorator). `--edges=FILE` walks any other relation instead,
742
+ one edge per line, `FROM_FILE:LINE<TAB>FROM_NAME<TAB>TO_FILE:LINE<TAB>TO_NAME`; a line stands for the unit that
743
+ holds it. `--step-reverse-to` in place of `--step-to` walks the edges backwards, callee to caller: the same
744
+ arguments, with `--hops` and `--edges`, and not together with `--step-to`.
745
+ - **The walk.** Breadth first from every start at once. A function's hop is its shortest distance from any
746
+ start, and each function is reached once, so a cycle ends. `--hops=N`, `--hops=M..N` and `--hops=M..` pick the
747
+ hops an end may be at (default `0..`: a start that matches the end expression is an end at hop 0). A path that
748
+ reaches no end is left out. stderr says how many functions each hop reached and why the walk stopped: no new
749
+ unit, or `--hops`.
750
+ - **The cost.** Jev judges every function against the start expression in one batch, then the functions reached
751
+ within `--hops` against the end expression in another. `--max-cost` asks again before the second, counting the
752
+ first. `--dry-run` answers every question 0, so it walks from every function the start expression could hold
753
+ for and shows that bound.
754
+ - **Not with** `-z`, `-g`, `-o`, `-c`, `-l`, `-A`/`-B`/`-C`, `--rank`, `--summarize` or `--dedup`. `-p` adds the
755
+ end expression's scores to each function judged against it; `-q` prints nothing and exits 0 when there is an end.
756
+
623
757
  ### One line per template (`--dedup`)
624
758
 
625
759
  Cost is proportional to the text sent, and machine-generated logs are mostly one skeleton with a different
@@ -627,8 +761,21 @@ id or number in it. `--dedup` masks ids, hashes, numbers, dates and times, paths
627
761
  the result, sends one line per group and reuses its answer for the rest. What is sent is that line's
628
762
  original text, and every line is printed as itself:
629
763
 
764
+ `--dedup` takes `auto`, `always` or `never`; a bare `--dedup` is `--dedup=always`. **Default is `never`, for
765
+ now** (#143): `auto` is held back until real-run stats say it should be the default. Every run still
766
+ estimates locally, before anything is sent, what folding every kind (the best case) would save, and prints
767
+ one stderr hint when that is worth at least twice the cost of `--dedup`'s own question:
768
+
769
+ ```
770
+ $ sys1grep -e "a request failed" app.log
771
+ sys1grep: 2040 units fold to at most 31 templates; --dedup=auto would save ~68 requests (~44k tokens)
772
+ ```
773
+
774
+ `--dedup=auto` folds only when that estimate says it pays; a file that barely repeats sends every
775
+ line, asking nothing extra. `--verbose` / `--dry-run` print the decision and its numbers for every value.
776
+
630
777
  ```sh
631
- $ ./sys1grep --dedup -n -e "a request failed" app.log
778
+ $ sys1grep --dedup -n -e "a request failed" app.log
632
779
  1:worker request 3fa9c1e27b failed: connection reset
633
780
  2:worker request 88d0e41a5c failed: connection reset
634
781
  4:worker request 0b7f2a9e13 failed: connection reset
@@ -636,9 +783,21 @@ $ ./sys1grep --dedup -n -e "a request failed" app.log
636
783
  ```
637
784
 
638
785
  Whether a value may be folded depends on the meaning: a number decides "disk usage is above 90%", a time
639
- decides "happened at night". So Jev is first asked, one small request per meaning, which kinds of value
640
- could change a match, and those kinds are kept apart (one of the two requests above). With
641
- `-e "disk usage is above 90%"` the same file keeps `95%` and `12%` apart and prints only `5:disk usage 95%`.
786
+ decides "happened at night". So `--dedup=always` (and a bare `--dedup`) first asks Jev, one small request per
787
+ meaning, which kinds of value could change a match, and keeps those kinds apart (one of the two requests above).
788
+ With `-e "disk usage is above 90%"` the same file keeps `95%` and `12%` apart and prints only `5:disk usage 95%`.
789
+
790
+ `--dedup=auto` asks no such question (#152). It folds every kind and judges a sample of each template instead:
791
+ up to 5 members whose values differ, for each varying value the member holding its lowest, its highest and its
792
+ middle value. When every sample gets the same verdict, the members not sampled share the first sample's answers,
793
+ and the samples keep their own (`-p` prints each sample's own probability). When the verdicts differ, the rest are
794
+ sent in a second wave, and `--max-cost` is checked again before it. A split costs slightly more than `never`, at
795
+ worst one extra request per wave boundary, a few percent: one template of 600 members sent 21 requests where
796
+ `never` sent 20. A value only between the samples is missed: among `usage 1` to `usage 120`, the meaning
797
+ `usage 50` matches no sample, so `auto` prints nothing where `never` prints line 50. `--verbose` prints how many
798
+ templates agreed and how many units were sent again. `--dry-run` assumes every sample agrees, so its count is the
799
+ floor: 100 templates of 6 members show 10 requests and 300 of 600 lines, and a run where every template splits
800
+ sends all 600 lines in 20 requests.
642
801
 
643
802
  Measured on real logs, with nothing kept: a 43,071-line system log folds into 550 templates (1.2% of the
644
803
  bytes), `install.log` to 21.1%, a Claude Code transcript (jsonl) only to 56.8%. It is for machine-generated
@@ -675,9 +834,11 @@ $0.094 to $0.013, with the same answer (#69). It pays when the answer sits in a
675
834
  - `-n`, `-A/-B/-C`, `-p` and file names go in as they would print; colors never do. No match runs nothing (exit 1).
676
835
  - `SYS1GREP_SUMMARIZER` picks the TOOL of a bare `--summarize`, `SYS1GREP_SUMMARIZER_MODEL` its model.
677
836
  - `-q`, `-l` and `-c` print no lines, so they cannot be combined with it.
837
+ - In `SYS1GREP_OPTS` it summarizes every search, so every match goes to TOOL's provider too;
838
+ `--no-summarize` turns it off for one search.
678
839
  - The answer is plain text: an LLM that is not told writes Markdown, noise on a terminal, so plain is asked for.
679
- `--summarize-format=markdown` or `=html` asks for those instead (`… --summarize-format=html … > summary.html`).
680
- The answer prints as it comes, unchecked. In `SYS1GREP_OPTS` it is a standing preference, ignored without `--summarize`.
840
+ `--format=markdown` or `=html` asks for those instead (`… --format=html … > summary.html`).
841
+ The answer prints as it comes, unchecked. In `SYS1GREP_OPTS` it is a standing preference, ignored without `--summarize` or `--rank`.
681
842
  - With `--dedup`, the TOOL gets what Jev got: each template's representative once, marked `(×N like it)`,
682
843
  not every line its answer was reused for.
683
844
  - More than 200 KB (about 50k tokens) is not sent at all: exit 2, with the size, before the TOOL is paid.
@@ -690,9 +851,19 @@ $ sys1grep -r -n --summarize --summarize-prompt="3 lines or fewer, just which fi
690
851
  -e "the API key is read from a file" .
691
852
  ```
692
853
 
693
- It needs `--summarize`; empty TEXT is the same as leaving it out. It comes after the `--summarize-format` sentence,
854
+ It needs `--summarize`; empty TEXT is the same as leaving it out. It comes after the `--format` sentence,
694
855
  so TEXT can override the format.
695
856
 
857
+ To decide after seeing the lines, `--summarize=ask` prints them as usual and then asks on the terminal
858
+ `Summarize these matches? [y/N]`. On `y` the lines just printed go to the TOOL of a bare `--summarize`: no second
859
+ search, nothing more sent to Jev. Enter or anything else ends there.
860
+
861
+ - The question goes over `/dev/tty`, so a stdin target still works. With no terminal, or stdout not one (a pipe,
862
+ `$(...)`, a script), nothing is asked and nothing hangs.
863
+ - `-y` summarizes without asking, so `-y` in `SYS1GREP_OPTS` makes it summarize every time; `-q` and `--dry-run` never ask.
864
+ - With `--dedup`, every matching line prints; the TOOL gets each template's representative, as with `--summarize`.
865
+ - The exit status is the search's, whatever the summary does. Over 200 KB, a `y` says so instead of sending.
866
+
696
867
  Other TOOLs:
697
868
 
698
869
  ```sh
@@ -709,6 +880,218 @@ OpenAI-compatible server. All three need `SYS1GREP_SUMMARIZER_MODEL`: none has a
709
880
  `SYS1GREP_SUMMARIZER_API_KEY` goes as `Authorization: Bearer` to a URL TOOL only (never `SYS1GREP_API_KEY`,
710
881
  which is Jev's). `OLLAMA_HOST` moves ollama's host, as it does for the `ollama` CLI itself.
711
882
 
883
+ ### A search page (`--serve`)
884
+
885
+ `--rank --format=html` answers one question per run. `--serve` opens such a page (the `search` template, unless `--template` is given) with a search box, so you can rephrase
886
+ and narrow without rerunning the command:
887
+
888
+ ```
889
+ $ sys1grep --serve -r src/
890
+ http://127.0.0.1:51234/?k=3f9e...
891
+ $ sys1grep --serve=8080 -y --level=strict -r src/ # a fixed port; the options given here are the page's first values
892
+ ```
893
+
894
+ From a clone, `npm run serve -- -r src/` is the same (`--` keeps npm from reading the options; `npm run serve -- --serve=8080 -r src/` fixes the port). With no target, `--serve` searches its own bundled sample corpus (`examples/`, #237) instead of reading an empty stdin, so every example button has something to find; name a target to search that instead. A folder named without `-r` is refused at launch, with the command that adds it (`--serve: src is a directory: add -r to search it (sys1grep --serve -r src)`, #222).
895
+
896
+ The printed URL carries a token made fresh at launch (`?k=...`). Every route needs it, the page itself included: without
897
+ it (in the query, or in a header for the page's own requests) the server answers 404 to everything, so another local
898
+ account or process cannot read the page, its token, or a search's results merely by knowing the port.
899
+
900
+ `EN` / `日本語` in the title bar switches the whole page's language (headings, labels, buttons, hints, the settings
901
+ dialog, the page's own messages), remembered in the browser; its default is the saved choice, else `--lang` /
902
+ `SYS1GREP_LANG` / settings.json's `lang` (below), else the browser's own language, else English. What sys1grep itself
903
+ prints (the results, the `--dry-run` estimate, an error), the command line and option names are never translated.
904
+
905
+ The page is a set of cards under a blue header band: Target and Search conditions in one card with a large Search
906
+ button (Estimate cost and Stop under it) at its right, the examples and Options side by side below it (one column on
907
+ a phone), then the command line and Details. **Target** shows what every search reads: the files and folders given at launch
908
+ (`npm run serve -- src docs/a.md`), or with none what sys1grep falls back to: `.` with `-r`, every file git tracks
909
+ with `git sys1grep`, or, with neither, the bundled sample corpus (`demo corpus (bundled): examples/`, #237); while
910
+ `-g` (in Details) is on, it says git log's commits instead. It is shown only: the page cannot
911
+ change it, and a target in a request is ignored. **Search conditions**
912
+ holds the meaning fields and multi-step, **Options** `rank` and `summarize`,
913
+ and **Details** the other options; the results come below them. What they hold: the meaning fields (`+` adds one; each takes `-e`, `-P`, `-a`, `-v` or `-Q`), `rank`, `summarize`, a **multi-step**
914
+ switch (on, it shows the end fields, `--step-to` and one or more meanings, and the edges, `--hops` and a reverse switch that makes it `--step-reverse-to`; off, they
915
+ leave the command and the search and keep their values), example buttons (searches from this README and
916
+ [docs/use-cases.md](docs/use-cases.md), then multi-step walks; one fills the fields and the controls it needs, puts the rest back to their launch values, and waits for Search), a
917
+ **Details** fold with the other options (`--level`, `-t`/`-T`, `-C`, `-A`/`-B`, `--auto-scope`, `-n`, `-p`, `--dedup`,
918
+ `--unit`, `-o` (shown only with a sentence unit), `-j`, `--max-cost`, `--include`, `--exclude`, `--changed-within`,
919
+ `-g`), and the
920
+ **command line** for the controls as they are now, with a Copy button; pasted in a terminal it runs the
921
+ same search. An option at its launch value is left out of what the controls add. `rank` off shows the matches in file order
922
+ as text, on shows the ranked cards; `summarize` adds a right column: Search runs once, and once its results are on
923
+ screen the page summarizes those results (POSTs the text shown, rank's cards included, to the summarizer; no second
924
+ search, nothing extra sent to Jev, up to 200 KB as `--summarize` takes). Checking `summarize` while results are
925
+ already on screen asks, in the page, whether to summarize those results as they are (#227): OK sends their text back
926
+ the same way; Not now leaves the box checked for the next Search. If a field or option changed since
927
+ those results came, the question says so. No results on screen, or multi-step on: no question. Estimate cost is `--dry-run`. The first field is a question (`-Q`) by default. `--suggest=Q:TEXT` / `--suggest=e:TEXT` (or `--suggest TEXT`, repeatable; a bare TEXT is a question) puts suggested-question buttons first in the examples row; pressing one fills the page, then Search runs it. The options in Details each have a "?" hint saying what they do. While a search runs, Search and Estimate are disabled and **Stop** ends it (the child process is killed). The matches are colored as `--color=always` colors them (file names, line numbers, regex matches), the cards mark them with `<mark>`.
928
+
929
+ Each search runs sys1grep itself (the options given at launch, then the controls'), so everything the command line does
930
+ the page does, and only that. The server listens on `127.0.0.1` only, answers only to that host name, and the page never
931
+ sees the API key. The targets, `--chunk`, `-M`, `--max-filesize`, `-y`, `--edges`, `--template`, the
932
+ API settings and `--summarize`'s TOOL are fixed at launch: the page can narrow what was given but not widen it (`--chunk`,
933
+ `-M` and `--max-filesize` still reach the Settings dialog's default options). A search over `--max-cost` is refused at
934
+ once (`--serve` has no `/dev/tty` to ask `-y`'s question on, or launch with `-y` to let it through unasked); the error
935
+ carries the same estimate `--dry-run` would show, and a button next to it reruns that one search, once, with
936
+ `--max-cost` just above the estimate (not a change to the Details field).
937
+ The meanings come from the page, so `-e -P -a -v -Q --step-to --step-reverse-to` and `-l -c -q -o -z --unit=zero -H -i --format --color --dry-run --verbose` are refused at launch.
938
+
939
+ **Settings** edits `~/.config/sys1grep/settings.json` (see [Install](#install)): `url`, `model`, the key, `opts` (one argument
940
+ per line), the summarizer's three and `lang` (the page's own default language). Save writes the file, and the next search uses it, as does every later `sys1grep`
941
+ run. Next to each field the page says where the value a search uses comes from, and says so when an environment variable or a
942
+ launch option wins over the file, since then saving does not change the search. A key comes back only as saved / not
943
+ saved; type a new one to replace it, or tick remove. A `--sys1-api-key` inside `opts`, and a `url`'s userinfo, come back
944
+ masked as `***`, the same way `--verbose` masks them; `opts` refuses a target, a flag the page itself refuses at launch,
945
+ or `--sys1-url` / `--sys1-api-key` (their own fields), and `url` and a URL summarizer must be `http(s)`. The save is a
946
+ `POST` with the page's token and this request's own Host as its required `Origin`, and the file is written whole (0600,
947
+ its directory 0700 when created) and renamed into place, so a reader never sees half of it.
948
+
949
+ ### Best first (`--rank`)
950
+
951
+ Matches print in file order, as grep prints them. With many, `--rank` prints the results best first, each under a
952
+ numbered header. A result is a match with its `-A/-B/-C` lines; matches whose context touches are one result.
953
+
954
+ ```sh
955
+ $ sys1grep -n -C1 --rank -e "the refund was refused" tickets/
956
+ 1. tickets/b.txt
957
+ tickets/b.txt-2-Order #1234, placed 2026-08-01.
958
+ tickets/b.txt:3:Your refund was declined: the order is older than 30 days.
959
+ tickets/b.txt-4-Please contact support.
960
+
961
+ 2. tickets/a.txt
962
+ tickets/a.txt-11-Order #88 arrived.
963
+ tickets/a.txt:12:Refund request received.
964
+ tickets/a.txt-13-We will check it.
965
+ ```
966
+
967
+ - `--rank` is `--rank=jev`: after the search, Jev is asked of each result, lines and context together, whether it
968
+ is relevant to the meanings that are not negated. One more question per result, `--chunk` results (at most 64) to a request;
969
+ `--dry-run` / `-i` show an upper bound, since which results there are is known only after the search.
970
+ - `--rank=match` sorts by each result's highest match probability, with no request. It is the answer to a yes/no
971
+ question on one line, so clear matches sit close together, and the context is not read.
972
+ - `-p` puts the score on the header (`1. [0.96] tickets/b.txt`). `-l` lists the files by their best result.
973
+ - `--format=markdown` writes a `## 1. tickets/b.txt` heading and a fenced block per result; `--format=html` one
974
+ self-contained HTML document from a template (below), light and dark, with a relevance bar per result. The lines
975
+ are as they print, escaped; what matched is in `<mark>` (the regex matches, else the matching sentences, else the whole
976
+ matching line; context lines are not marked; `--color=never` turns it off).
977
+ - `--summarize` gets the results in ranked order; with `--dedup` a representative is one result.
978
+ - It needs a meaning (a regex, `!` or `-v` alone ranks nothing), and cannot be combined with `-c`, `-o` or `-q`.
979
+ `--no-rank` turns off an earlier one, from `SYS1GREP_OPTS` say.
980
+
981
+ ### Finding the cause of a failed command
982
+
983
+ A failed job's last error usually names a symptom (`214 accounts out of balance ... day-close aborted`), and the
984
+ other `ERROR` / `WARN` lines are retries that recovered. The cause is often one plain `INFO` or `DEBUG` line far
985
+ above. Asking for the symptom, or `-Q "why did the job fail"`, finds that noise: on the 30,000-line log of #156 it
986
+ returned only recovered `ERROR` lines. Ask by kind of cause instead, and rank the results:
987
+
988
+ ```sh
989
+ $ sys1grep -n --dedup --level strict --rank \
990
+ -e "a component, writer, job or feature was paused, disabled, skipped or put on hold" \
991
+ -e "a configuration, endpoint, region, credential or data source was changed or switched" \
992
+ -e "an input, file or table was empty, missing, or had fewer records or columns than expected" \
993
+ -e "a fallback or default value was used instead of the real one" \
994
+ job.log
995
+ ```
996
+
997
+ Measured on 24 generated logs of failed jobs, six per kind of cause, 10,008 lines each, with these four meanings
998
+ (`npm run cause-eval`, rows in `tests/cause-results.tsv`; an earlier version of the script wrote its 24 rows, and
999
+ the `top 2 jev` column is filled only in the two rerun rows). Each search returned 366 to 464 results. The
1000
+ `--rank=match` column is not a separate run: it is recomputed from the `-p` columns of the `--rank` run, since
1001
+ `--rank=match` scores a result by the highest of them. That recomputation was checked against a real `--rank=match`
1002
+ run only on config-5, where the cause was not a result. A cause "not matched" may have been judged below `strict` or
1003
+ folded by `--dedup` into a like line; the rows do not tell the two apart.
1004
+
1005
+ | cause | cause is result 1, `--rank` | cause is result 1, `--rank=match` | otherwise |
1006
+ |---|---|---|---|
1007
+ | a configuration, endpoint or credential switched | 5 of 6 | 5 of 6 | 1 not matched |
1008
+ | an input empty or short | 2 of 6 | 2 of 6 | 2 at results 62 (`match`: 343-371) and 81 (`match`: 191-219); 2 not matched |
1009
+ | a default used instead of the real value | 6 of 6 | 5 of 6 | `match`: 2-9 (one result above it, seven tied with it at 0.90) |
1010
+ | a line that never came (a start with no finish) | 0 of 6 | 0 of 6 | the start line never a result |
1011
+
1012
+ A log cost about $0.088 (2.1M input tokens, 643 to 646 requests) and 16 to 17 s. `--rank` (`jev`) adds one question
1013
+ per result, about 3% more input tokens than `--rank=match` (one log run both ways; not recorded in the TSV). It put
1014
+ the cause higher than `--rank=match` on three logs and lower on none.
1015
+
1016
+ When the top results look unrelated (rotations, refreshes, the final error itself), the cause is likely one of these:
1017
+
1018
+ - **A line that states only a value**: `stock snapshot downloaded: 1204 bytes`, `input directory ... holds 2 files`,
1019
+ `resolved db-primary.internal to 10.0.4.22 (replica-2)`. Nothing in the line says it is wrong, so no hypothesis
1020
+ holds, nor did a meaning written from the final error
1021
+ (`-e "the job connected to a read-only replica instead of the primary database"`). Search for the thing the error
1022
+ names instead (`-P '/snapshot|replica/'`) and read the numbers.
1023
+ - **A line that never came**: a worker that claimed a shard and never finished it, a step or a migration that started
1024
+ and never ended. On all six such logs the start line was not among the results. The top result was a routine line,
1025
+ the final error, or once the previous run's `alias products switched to products-20260927`, whose 09-28
1026
+ counterpart never came. A meaning for it
1027
+ (`-e "a worker, step or upload started but never reported that it finished"`) put the final error first, not the
1028
+ start, and `--dedup` can fold one start into a like one, since only the missing finish tells them apart. Compare
1029
+ starts with finishes by eye or with grep.
1030
+ - **The score will not tell you**: where the cause was result 1 it scored 0.93 to 0.97 under `--rank`; on two logs
1031
+ where it never matched, the top result scored 0.88 and 0.91.
1032
+
1033
+ The follow-ups behind the first two points (key lines judged alone in a 31-line window, the meanings written from
1034
+ the error or for a missing finish) are in the description of PR #192; `tests` holds no way to rerun them.
1035
+
1036
+ Every searched line goes to Jev (TypeSafe, or whatever `SYS1GREP_URL` names), and with `--summarize` the matching
1037
+ lines go to its TOOL's provider as well. Build logs and job logs often hold tokens, passwords and customer data.
1038
+
1039
+ ### HTML templates (`--template`)
1040
+
1041
+ `--rank --format=html` and `--summarize --format=html` fill in a template. Four come with sys1grep: `default`, `print` (black on white, serif,
1042
+ for paper and PDF), `search` (a results page like a web search engine's) and `terminal` (dark monospace).
1043
+ `--template=NAME` picks one; `SYS1GREP_TEMPLATE`, in the
1044
+ environment or `~/.config/sys1grep/.env`, sets the default.
1045
+
1046
+ ```sh
1047
+ $ sys1grep -r -n -C1 --rank -p --format=html --template=print -e "the refund was refused" tickets/ > refunds.html
1048
+ ```
1049
+
1050
+ A template is one HTML file. The part between `<!--result-->` and `<!--/result-->` is repeated for each result;
1051
+ what is before and after it is written once. sys1grep replaces these, escaped, and leaves any other `{{…}}` as
1052
+ written:
1053
+
1054
+ | placeholder | where | value |
1055
+ |---|---|---|
1056
+ | `{{title}}` | anywhere | `sys1grep: ` and the meanings |
1057
+ | `{{query}}` | anywhere | the meanings, as in `"refund" and not "policy"` |
1058
+ | `{{count}}` | anywhere | the number of results (with `--summarize`, of matching lines) |
1059
+ | `{{answer}}` | anywhere | `--summarize`'s answer; empty with `--rank` |
1060
+ | `{{rank}}` | per result | 1, 2, … |
1061
+ | `{{score}}` | per result | the score as `-p` prints it (`0.96`); empty without `-p` |
1062
+ | `{{score_pct}}` | per result | the score as a whole number 0 to 100, set even without `-p` (for a bar: `style="--s:{{score_pct}}"`) |
1063
+ | `{{file}}` | per result | the file name; empty when one file is searched |
1064
+ | `{{lines}}` | per result | the result's lines as they print |
1065
+
1066
+ `--template=NAME` reads `~/.config/sys1grep/templates/NAME.html` if it is there, else the bundled one, so your
1067
+ copy of `default` replaces it. A value containing `/` or ending in `.html` is a file path. To make your own, copy
1068
+ the bundled ones and edit:
1069
+
1070
+ ```sh
1071
+ $ sys1grep --install-templates # "copied PATH" for each file
1072
+ $ cp ~/.config/sys1grep/templates/default.html ~/.config/sys1grep/templates/team.html
1073
+ $ sys1grep -r --rank --format=html --template=team -e "..." src/ > out.html
1074
+ $ sys1grep --template=list # one name a line, "(user)" after those under ~/.config
1075
+ ```
1076
+
1077
+ `{{score_pct}}` is set even without `-p`, so a bar can show the relevance without printing the number.
1078
+
1079
+ With `--summarize --format=html` the TOOL is asked for plain text, and that answer, escaped, goes into `{{answer}}`
1080
+ once it has finished; the part between `<!--result-->` and `<!--/result-->` is not written. A template for
1081
+ `--summarize` therefore needs `{{answer}}` outside that part (the bundled ones have it; with `--rank` it is empty).
1082
+
1083
+ ```sh
1084
+ $ sys1grep -r --summarize --format=html --template=print -e "why the refunds were refused" tickets/ > why.html
1085
+ ```
1086
+
1087
+ A template is trusted local input: a path you give, or a symlink in the templates directory, is read as it is;
1088
+ only the results put into it are escaped.
1089
+
1090
+ `--install-templates` never overwrites a file that is already there (it prints `kept` for it). A missing
1091
+ template, or one without exactly one `<!--result-->` before one `<!--/result-->`, is exit 2 naming the file.
1092
+ `--template` on the command line needs `--format=html` with `--rank` or `--summarize` (not `-l`); `SYS1GREP_TEMPLATE` is simply unused elsewhere.
1093
+ `--template=list` and `--install-templates` take no other arguments and are refused in `SYS1GREP_OPTS`.
1094
+
712
1095
  ## Use it from Claude Code
713
1096
 
714
1097
  There is a Claude Code skill that runs sys1grep for you: describe what you are looking for in plain words
@@ -720,8 +1103,8 @@ claude plugin marketplace add uehaj/uehaj-marketplace
720
1103
  claude plugin install uehaj@uehaj-marketplace
721
1104
  ```
722
1105
 
723
- No separate install of the command-line tool is needed: the skill uses `sys1grep` from your PATH if present,
724
- otherwise `npx @uehaj/sys1grep`. Only the API key has to be set (see [Install](#install)).
1106
+ No separate install of the command-line tool is needed: the skill runs a version it pins,
1107
+ through `npx -y @uehaj/sys1grep@VERSION`, not the `sys1grep` on your PATH. Only the API key has to be set (see [Install](#install)).
725
1108
 
726
1109
  Then, inside Claude Code:
727
1110
 
@@ -737,22 +1120,27 @@ Claude Code installs plugins, not single skills. If you want just this one skill
737
1120
  npx skills add uehaj/uehaj-marketplace --skill sys1grep -a claude-code -g
738
1121
  ```
739
1122
 
740
- The skill writes the meaning in English, picks `-e` / `-a` / `-v` for AND / OR / NOT, adds `-n`, narrows large
1123
+ The skill writes the meaning in English, picks `-e` / `-a` / `-v` for OR / AND / NOT, adds `-n`, narrows large
741
1124
  directories to files worth paying for, and re-runs with `--level loose` or `strict` when the first result looks off.
742
- The API key and endpoint are read the same way as on the command line (`SYS1GREP_API_KEY`, `~/.config/sys1grep/.env`).
1125
+ The API key and endpoint are read the same way as on the command line (`SYS1GREP_API_KEY`, `~/.config/sys1grep/settings.json`, `~/.config/sys1grep/.env`).
743
1126
 
744
1127
  ## Usage
745
1128
 
746
1129
  ```
747
- usage: sys1grep [OPTION]... -e MEANING|-Q QUESTION [-a MEANING] [-v MEANING]... [FILE...]
748
-
749
- -e MEANING lines matching this meaning (several -e are OR'd)
1130
+ usage: sys1grep [OPTION]... -e MEANING|-Q QUESTION|-P /RE/FLAGS [-a MEANING] [-v MEANING]... [FILE...]
1131
+ sys1grep [OPTION]... -e START1 [-e START2]... --step-to END1 [-e END2]... [FILE...]
1132
+
1133
+ -e MEANING lines matching this meaning (several -e are OR'd); MEANING is always sent to Jev, never
1134
+ read as a regex — a /RE/flags-shaped value is refused, with a hint naming -P. A meaning
1135
+ that really starts and ends with / gets through with a leading space (-e ' /cat/')
1136
+ -P '/RE/FLAGS' a regex term, OR'd like -e; matched locally, no request; -a/-v after it AND/AND NOT
1137
+ onto it, same as after -e; -P '!/RE/' is not allowed, negate with -v '/RE/'
750
1138
  -Q, --question QUESTION lines that answer QUESTION, not lines asking it; the same as
751
1139
  -e "the line answers: QUESTION" (see "Lines that answer a question" above)
752
- -a MEANING AND onto the preceding -e/-Q term. -e A -a B -e C = (A and B) or C
753
- -v MEANING AND NOT onto the preceding -e/-Q term. -e A -v B = A and not B
1140
+ -a MEANING AND onto the preceding -e/-Q/-P term. -e A -a B -e C = (A and B) or C
1141
+ -v MEANING AND NOT onto the preceding -e/-Q/-P term. -e A -v B = A and not B
754
1142
  At the front it is a bare negation. -v B = not B (like grep -v)
755
- !MEANING a leading ! negates just that meaning, in -e / -Q / -a / -v alike
1143
+ !MEANING a leading ! negates just that meaning, in -e / -Q / -a / -v alike (not -P)
756
1144
  -e A -e '!B' = A or not B. -a '!C' is the same as -v C
757
1145
  --level=LEVEL strictness preset, sets both thresholds (default normal)
758
1146
  loose : -t 0.3 -T 0.7 catch more, accept some noise
@@ -773,6 +1161,8 @@ usage: sys1grep [OPTION]... -e MEANING|-Q QUESTION [-a MEANING] [-v MEANING]...
773
1161
  or date-time, today, this-week or this-month
774
1162
  --no-auto-scope do not narrow those files by what a meaning says about them (see Scope from the meaning);
775
1163
  --auto-scope turns it back on
1164
+ -g, --gitlog search git log's commits, one record each, instead of files; FILE arguments are pathspecs
1165
+ (see "Scope from the meaning"). Not with -r or git sys1grep
776
1166
  -l print only the names of files with a match, not the lines
777
1167
  -H, --with-filename prefix file names even for a single file; --no-filename never prefixes them
778
1168
  -A NUM print NUM lines of trailing context after each match (context lines use - as separator)
@@ -780,24 +1170,31 @@ usage: sys1grep [OPTION]... -e MEANING|-Q QUESTION [-a MEANING] [-v MEANING]...
780
1170
  -C NUM print NUM lines of context before and after (-A NUM -B NUM)
781
1171
  -c print only a count of matching lines per file (like grep -c)
782
1172
  -q, --quiet print nothing, stop at the first match; exit 0 on a match, even after an error (like grep -q)
783
- --chunk=LINES lines per request (default 30)
1173
+ --chunk=LINES lines per request (default 30; a request also carries at most 64 questions, so fewer lines
1174
+ with 3 or more meanings)
784
1175
  Lines in one request are each other's context, so a small chunk changes verdicts
785
1176
  on ambiguous lines, not just speed
786
- -j N concurrent requests (default 8)
1177
+ -j N concurrent requests (default 8; 1 for a local URL: localhost, 127.0.0.1, [::1])
787
1178
  -n print line numbers
788
1179
  --unit=UNIT the unit of judgement: line (default), zero, sentence-by-jev, sentence-by-rule or function
789
1180
  -z, --null-data --unit=zero: judge NUL-terminated records instead of lines, and print them NUL-terminated (see "Records that span several lines" above);
790
1181
  with --unit=sentence-by-*, each record is split into sentences
791
1182
  --unit=sentence-by-jev|sentence-by-rule judge each sentence instead of each line (see "One sentence at a time" above)
792
1183
  --unit=function judge each function and print its lines (see "One function at a time" above)
793
- -o with --unit=sentence-by-*, print only the matching sentences
1184
+ --step-to EXPRESSION print the paths from the functions the expression before it matches, along the
1185
+ calls, to the functions the expression after it matches (see "From one function to another" above)
1186
+ --step-reverse-to EXPRESSION --step-to walking the edges backwards, callee to caller
1187
+ --edges=FILE walk the edges in FILE instead of the calls
1188
+ --hops=N|M..N|M.. the hops an end may be at (default 0..)
1189
+ -o print only each regex match, one per line, as grep -o (a line only meanings matched prints
1190
+ whole); with --unit=sentence-by-*, only the matching sentences
794
1191
  -p print each meaning's probability at the end of the line
795
1192
  --dry-run send nothing; print the settings the search would run with (and their source, when not the
796
1193
  command line), each file searched and each request with its questions
797
1194
  --verbose print the same to stderr while searching
798
1195
  -i, --interactive show what --dry-run would send, and search only after y on the terminal
799
1196
  -M NUM, --max-columns=NUM send at most the first NUM characters of a line (or record, with -z); still
800
- searched and judged, just not past that cutoff (default 2000, 8000 with -z)
1197
+ searched and judged, just not past that cutoff (default 2000, 8000 with -z or --unit=function)
801
1198
  --max-filesize=SIZE size every target first (K/M/G, default 10M); over it, skip it outright, like rg's own
802
1199
  --max-filesize (-y does not affect it). stdin is sized once read, and skipped the same way if
803
1200
  it is over; a --cached / <tree>: target is a blob, sized from its content; -g's commits stay
@@ -805,24 +1202,51 @@ usage: sys1grep [OPTION]... -e MEANING|-Q QUESTION [-a MEANING] [-v MEANING]...
805
1202
  --max-cost=USD price the input about to be sent; over it (default 1), ask to continue on the terminal; -y
806
1203
  answers yes without asking; no terminal and the limit exceeded is exit 2, unchanged by -q
807
1204
  (scripts pass -y); -i already asks unconditionally and earlier, so this does not ask again
808
- --dedup judge one line per template and reuse its answer for the rest (see "One line per template" above)
1205
+ -y, --yes answer that question yes without asking
1206
+ --dedup[=auto|always|never] judge one line per template and reuse its answer for the rest (default never,
1207
+ for now; see "One line per template" above)
1208
+ --cache[=on|off|only] answer a request sent before from ~/.config/sys1grep/cache, free; only: never send,
1209
+ a request not cached is exit 2 unless -q already found a match (default off,
1210
+ SYS1GREP_CACHE; see "Sending less");
1211
+ --cache=clear (alone) empties it
1212
+ --rank[=jev|match] print the results (a match with its context) best first, under numbered headers; jev
1213
+ (bare --rank) asks Jev of each result, match sorts by its best match probability (see "Best first"); --no-rank: file order
1214
+ --summarize[=TOOL] pipe what would print to TOOL (claude by default, or SYS1GREP_SUMMARIZER) and print its
1215
+ summary instead; --summarize=ask prints the lines, then asks on the terminal; --no-summarize
1216
+ turns it off (see "A summary instead of the lines")
1217
+ --summarize-prompt=TEXT your own instruction, added after the fixed one; needs --summarize
1218
+ --format=FORMAT plain (default) / markdown / html: --rank's output, or asked of --summarize's TOOL
1219
+ --serve[=PORT] serve a search page on 127.0.0.1 instead of searching; --suggest=Q:TEXT / e:TEXT adds
1220
+ suggested-question buttons (see "A search page (`--serve`)")
1221
+ --template=NAME the document --rank or --summarize --format=html writes: ~/.config/sys1grep/templates/NAME.html, else the
1222
+ bundled one (default, print, search, terminal), or a file; default SYS1GREP_TEMPLATE, else default.
1223
+ --template=list prints the names (see "HTML templates")
1224
+ --install-templates copy the bundled templates to ~/.config/sys1grep/templates, keeping existing files
809
1225
  --color[=WHEN] auto (default: color when stdout is a terminal) / always / never; bare --color means auto
810
1226
  file and line number use grep's colors; with -p, probabilities are
811
1227
  green at or above the positive threshold, red below the negative one,
812
1228
  yellow in between. NO_COLOR is honored
813
1229
  --sys1-model=ID, --sys1-url=URL, --sys1-api-key=KEY
814
1230
  the API settings, overriding SYS1GREP_MODEL, SYS1GREP_URL, SYS1GREP_API_KEY
815
- -h, --help this help (Japanese when LANG / LC_ALL / LC_MESSAGES starts with ja)
1231
+ --lang=en|ja this help's language, and --serve's page default; overrides SYS1GREP_LANG
1232
+ -h, --help this help (--lang / SYS1GREP_LANG when set, else Japanese when the first set of LC_ALL /
1233
+ LC_MESSAGES / LANG starts with ja)
816
1234
  -V, --version print the version and exit
817
1235
  ```
818
1236
 
819
1237
  Without FILE, stdin is read. With several files, output is prefixed with `file:`.
820
1238
  Exit codes follow grep: 0 matched, 1 no match, 2 error (bad arguments, unreadable file, API failure).
1239
+ Exit 1 after lines were sent prints one line on stderr: the highest probability, its line, the option that
1240
+ loosens the threshold (unless even that would not match), and `-p -t 0` (`-T 1` too with a negation) to see every
1241
+ probability (`sys1grep: no line reached 0.5 for "…"; the highest was 0.42 (app.log:118). ...`).
1242
+ Not with `-q`, `--step-to` or `--step-reverse-to`, nor when only a negated meaning kept lines out.
821
1243
 
822
1244
  ### Expression grammar
823
1245
 
824
- `-e` starts an OR term. `-a` and `-v` extend the previous term with AND and AND NOT.
825
- A leading `!` on a meaning negates just that meaning (quote it, `!` is history expansion in most shells).
1246
+ `-e` and `-P` each start an OR term, the first always a meaning, the second always a regex. `-a` and
1247
+ `-v` extend the previous term with AND and AND NOT.
1248
+ A leading `!` on a meaning negates just that meaning (quote it, `!` is history expansion in most shells);
1249
+ a leading `!` on a `-P` value is refused instead — negate a regex term with `-v '/RE/'`.
826
1250
 
827
1251
  | command | means |
828
1252
  |---|---|
@@ -835,10 +1259,10 @@ A leading `!` on a meaning negates just that meaning (quote it, `!` is history e
835
1259
 
836
1260
  ## How it works
837
1261
 
838
- 1. Non-blank lines are cut into chunks of 30 lines (with a character cap).
1262
+ 1. Non-blank lines are cut into chunks of 30 lines (with a character cap, and at most 64 questions a request).
839
1263
  2. Each chunk goes into `state` as an object `{"L000": "line 1", "L001": "line 2", ...}`,
840
1264
  and one `noul` (yes/no probability) question per line × meaning goes into the same request.
841
- 3. Up to 8 requests run concurrently. Output is printed in file order.
1265
+ 3. Up to 8 requests run concurrently (`-j`; 1 for a local URL). Output is printed in file order.
842
1266
  4. Per line, each meaning's probability is thresholded to a boolean and the AND / OR / NOT expression is evaluated.
843
1267
 
844
1268
  Lines sent together are each other's context: Jev judges a line against what the rest of the chunk shows
@@ -863,19 +1287,24 @@ the runner evaluates the boolean expression on those verdicts and compares with
863
1287
  reporting precision and recall. It also sweeps `-t` × `-T` in 0.05 steps and reports the best pair.
864
1288
 
865
1289
  ```sh
866
- node --no-warnings tests/judge.mts [--model sonnet] [--rejudge]
1290
+ npm run judge [-- --model sonnet] [--rejudge] # runs tests/judge.mts; .mts needs Node 23.6+
867
1291
  ```
868
1292
 
869
1293
  Judge verdicts are cached in `tests/verdicts.json`; later runs do not call the judge.
870
1294
  The result is written to `tests/report.md`. Latest: precision 0.94, recall 0.98.
871
1295
 
1296
+ The judge is one of several test entry points. `npm run test:offline` checks sys1grep itself against a fake Jev
1297
+ (`tests/fake-jev.mjs`): no key, no network, the same result every run (`OFFLINE_ONLY=PATTERN` runs only the matching
1298
+ sections). `npm test` adds `tests/check.sh` against real Jev. [CLAUDE.md](CLAUDE.md) lists every entry point with what it
1299
+ checks and what it needs.
1300
+
872
1301
  ## Limits
873
1302
 
874
- - Every searched line is sent to api.typesafe.ai. Do not run it over files you would not upload there.
1303
+ - Every searched line is sent to api.typesafe.ai (or wherever `SYS1GREP_URL` points). Do not run it over files you would not upload there.
875
1304
  - Blank lines are not sent; they count as probability 0 for every meaning, so `-v X` prints them and `-e X` never does.
876
- - Lines are truncated to 2,000 characters before sending.
877
- - The maximum number of questions per request is undocumented; 420 worked.
878
- - 429 / 529 are retried up to 6 times with exponential backoff.
1305
+ - Lines are truncated to 2,000 characters before sending (8,000 with `-z` or `--unit=function`; `-M` sets it).
1306
+ - The maximum number of questions per request is undocumented; 420 worked. sys1grep itself sends at most 64 a request (Clef's cap); only a single unit asked more than 64 meanings goes out whole.
1307
+ - 429, 529 and other 5xx answers, connection errors and timeouts are retried up to 6 times with exponential backoff.
879
1308
  - Accuracy is best in English. Japanese works but is noisier.
880
1309
 
881
1310
  ## FAQ