@uehaj/sys1grep 0.5.0-next.1 → 0.5.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.ja.md +521 -95
- package/README.md +522 -93
- package/examples/app.js +49 -0
- package/examples/app.py +54 -0
- package/examples/logs.txt +19 -0
- package/examples/prose.txt +10 -0
- package/examples/tickets.txt +9 -0
- package/lib/sentences.mjs +49 -0
- package/options.mjs +101 -0
- package/package.json +29 -5
- package/serve.mjs +1121 -0
- package/settings.mjs +58 -0
- package/sys1grep.mjs +1260 -442
- package/templates/default.html +102 -0
- package/templates/print.html +51 -0
- package/templates/search.html +114 -0
- package/templates/terminal.html +57 -0
package/README.md
CHANGED
|
@@ -4,6 +4,8 @@
|
|
|
4
4
|
|
|
5
5
|
Background and design notes (Japanese): [Jevのキラーアプリ、「意味で探す grep」を作った](https://zenn.dev/uehaj/articles/jev-semgrep-grep-by-meaning) on Zenn.
|
|
6
6
|
|
|
7
|
+
> **Renamed from semgrep (0.5.0).** Two reasons. The name collided with the static-analysis tool [Semgrep](https://semgrep.dev/), a trademark. And the tool was never meant for Jev alone: it targets System 1 models in general, and I'd like it to work with any models TypeSafe AI may release after Jev under other names :). So from 0.5.0 this is **sys1grep** (System 1 + grep): npm package `@uehaj/sys1grep` (was `@uehaj/semgrep`), commands `sys1grep` / `git sys1grep`, environment `SYS1GREP_*`, config `~/.config/sys1grep/.env`. The old `SEMGREP_*` names and `~/.config/semgrep/.env` still work for one minor release, each use printing a deprecation line; 1.0.0 drops them. Details: [CHANGELOG](CHANGELOG.md#renamed). The Zenn article's URL keeps its old slug.
|
|
8
|
+
|
|
7
9
|
A grep that finds lines by **what they mean**, not by regular expressions.
|
|
8
10
|
Matching is done by **Jev**, the System One model from [TypeSafe AI](https://typesafe.ai/).
|
|
9
11
|
Jev does not generate text. It answers typed questions with probabilities, so for every line
|
|
@@ -11,15 +13,15 @@ sys1grep asks "does this line match the meaning *network failure*?", gets a prob
|
|
|
11
13
|
and applies a threshold.
|
|
12
14
|
|
|
13
15
|
```sh
|
|
14
|
-
|
|
16
|
+
sys1grep -n -e "customer is angry or frustrated" tickets.txt
|
|
15
17
|
```
|
|
16
18
|
|
|
17
|
-
[](https://
|
|
19
|
+
[](https://sys1grep.js.org/)
|
|
18
20
|
|
|
19
|
-
<sub>▶ Click the demo, or open <a href="https://
|
|
21
|
+
<sub>▶ Click the demo, or open <a href="https://sys1grep.js.org/">sys1grep.js.org</a>, for the full demo on the landing page.</sub>
|
|
20
22
|
|
|
21
|
-
- Zero dependencies.
|
|
22
|
-
- Fast. 30 lines go into one request, requests run 8 at a time. A 210-line file finishes in under a second.
|
|
23
|
+
- Zero dependencies. Node.js 20.16+ and `fetch`, nothing else.
|
|
24
|
+
- Fast. 30 lines go into one request, requests run 8 at a time (1 for a local URL). A 210-line file finishes in under a second.
|
|
23
25
|
- Meanings combine with AND / OR / NOT.
|
|
24
26
|
- **Language-agnostic.** The meaning and the text can each be in any language. A Japanese meaning finds French, Russian, Chinese and Korean lines alike. No translation step, same speed, same cost.
|
|
25
27
|
|
|
@@ -31,7 +33,7 @@ so one query finds matching lines in every language the file contains.
|
|
|
31
33
|
An **English** meaning finds **Japanese** lines. None of the hits contain "angry" or "frustrated", and two of them are in Japanese:
|
|
32
34
|
|
|
33
35
|
```sh
|
|
34
|
-
$
|
|
36
|
+
$ sys1grep -n -e "customer is angry or frustrated" tests/corpus.txt
|
|
35
37
|
14:ユーザー山田さんからの問い合わせ: 返金してほしい、商品が壊れていた
|
|
36
38
|
16:ユーザー佐藤さんからの問い合わせ: 注文した覚えのない請求が来ています。至急確認してください
|
|
37
39
|
18:I want my money back. The item arrived broken and customer service ignored me.
|
|
@@ -42,7 +44,7 @@ $ ./sys1grep -n -e "customer is angry or frustrated" tests/corpus.txt
|
|
|
42
44
|
A **Japanese** meaning finds **English** lines, with the same confidence as the Japanese ones:
|
|
43
45
|
|
|
44
46
|
```sh
|
|
45
|
-
$
|
|
47
|
+
$ sys1grep -n -p -e "返金の要求" tests/corpus.txt
|
|
46
48
|
14:ユーザー山田さんからの問い合わせ: 返金してほしい、商品が壊れていた [0.97]
|
|
47
49
|
18:I want my money back. The item arrived broken and customer service ignored me. [0.95]
|
|
48
50
|
```
|
|
@@ -52,7 +54,7 @@ Russian, German, Spanish, Chinese and Korean. One Japanese meaning finds all six
|
|
|
52
54
|
meaning does the same:
|
|
53
55
|
|
|
54
56
|
```sh
|
|
55
|
-
$
|
|
57
|
+
$ sys1grep -n -p -e "顧客が返金を求めている" tests/multi.txt
|
|
56
58
|
1:Je veux être remboursé, le produit est arrivé cassé. [0.98]
|
|
57
59
|
3:Я требую вернуть деньги, товар не работает. [0.97]
|
|
58
60
|
5:Ich möchte mein Geld zurück, das Gerät ist defekt. [0.97]
|
|
@@ -60,7 +62,7 @@ $ ./sys1grep -n -p -e "顧客が返金を求めている" tests/multi.txt
|
|
|
60
62
|
9:我要求退款,商品坏了。 [0.97]
|
|
61
63
|
11:환불해 주세요. 제품이 고장났어요. [0.97]
|
|
62
64
|
|
|
63
|
-
$
|
|
65
|
+
$ sys1grep -n -p -e "клиент требует возврат денег" tests/multi.txt
|
|
64
66
|
1:Je veux être remboursé, le produit est arrivé cassé. [0.94]
|
|
65
67
|
3:Я требую вернуть деньги, товар не работает. [0.97]
|
|
66
68
|
5:Ich möchte mein Geld zurück, das Gerät ist defekt. [0.92]
|
|
@@ -82,12 +84,12 @@ Jev reads the line and the question together (a cross-encoder shape), so who did
|
|
|
82
84
|
"asked for" versus "already done" all change the answer. An embedding of the line is fixed before it
|
|
83
85
|
ever sees your query, so it can only measure topical closeness.
|
|
84
86
|
|
|
85
|
-
|
|
87
|
+
The first six lines below are "about a refund", and the last two are the pair the AND example further down uses (the same contrast as `tests/contrast.txt`, translated to
|
|
86
88
|
English as `tests/contrast.en.txt` — see [Search across languages](#search-across-languages) above for
|
|
87
89
|
the case where the meaning and the text differ in language). Only two are a customer asking for one:
|
|
88
90
|
|
|
89
91
|
```sh
|
|
90
|
-
$
|
|
92
|
+
$ sys1grep -n -p -t 0 -e "customer is asking for a refund" tests/contrast.en.txt
|
|
91
93
|
1:I want a refund. The item was broken. [0.99]
|
|
92
94
|
2:The refund has been processed. Please check your account. [0.11]
|
|
93
95
|
3:Our refund policy is within 30 days of purchase. [0.09]
|
|
@@ -103,14 +105,14 @@ operations**, not a trick with set differences or "negative queries":
|
|
|
103
105
|
|
|
104
106
|
```sh
|
|
105
107
|
# about a refund, but NOT a customer asking for one → completed, policy, denied, timelines
|
|
106
|
-
$
|
|
108
|
+
$ sys1grep -n -e "about a refund" -v "the customer is asking for a refund" tests/contrast.en.txt
|
|
107
109
|
2:The refund has been processed. Please check your account.
|
|
108
110
|
3:Our refund policy is within 30 days of purchase.
|
|
109
111
|
4:The manager denied the refund request yesterday
|
|
110
112
|
6:Refunds are processed within 5 business days
|
|
111
113
|
|
|
112
114
|
# angry AND it is the customer, not the staff
|
|
113
|
-
$
|
|
115
|
+
$ sys1grep -n -e "someone is angry" -a "the customer, not the staff, is the one acting" tests/contrast.en.txt
|
|
114
116
|
5:I demand a full refund immediately
|
|
115
117
|
8:The customer got angry and hung up the phone.
|
|
116
118
|
```
|
|
@@ -129,18 +131,24 @@ repeated queries over a large, fixed corpus a vector index is cheaper and faster
|
|
|
129
131
|
|
|
130
132
|
## Regex terms
|
|
131
133
|
|
|
132
|
-
`-
|
|
133
|
-
|
|
134
|
-
|
|
135
|
-
only
|
|
136
|
-
|
|
137
|
-
|
|
138
|
-
|
|
139
|
-
|
|
134
|
+
`-P '/pattern/flags'` (first and last character `/`, JavaScript flags) starts a regex term, OR'd like
|
|
135
|
+
`-e`, and is matched locally with no request at all. `-a`/`-v '/pattern/flags'` AND / AND NOT a regex
|
|
136
|
+
onto the preceding term, the same shape deciding regex from meaning there. It prefilters its AND term:
|
|
137
|
+
only the lines it holds for ever ask that term's meanings, so a cheap regex in front of a meaning cuts
|
|
138
|
+
both the bill and the wait. A line is sent only if some term's regexes all hold for it (a term with no
|
|
139
|
+
regex holds for every line), so with `-P '/re/' -a A -e B` a line without `re` is still sent, asked `B`
|
|
140
|
+
only. A query of regex terms alone sends nothing, even with `--unit=sentence-by-jev`: the rules alone
|
|
141
|
+
then decide where wrapped lines join. `-P '!/RE/'` is not allowed; negate a regex with `-v '/RE/'`
|
|
142
|
+
instead. `-e` never reads its value as a regex: a `/RE/flags`-shaped value there is refused, with a
|
|
143
|
+
hint naming `-P` (e.g. `-e '/catch/' is no longer a regex: use -P '/catch/'`). `-Q` is unaffected: it
|
|
144
|
+
was already always a question, never a regex, so `-e '/etc 以下のファイルを変更している'` (no closing
|
|
145
|
+
`/`) still goes through as a meaning, unrefused. A meaning for `-e`/`-a`/`-v` that really starts and
|
|
146
|
+
ends with `/` can still be written with a leading space to dodge the regex reading (`-e ' /cat/'`
|
|
147
|
+
sends the text `" /cat/"`, not a refusal).
|
|
140
148
|
|
|
141
149
|
```sh
|
|
142
|
-
$
|
|
143
|
-
$
|
|
150
|
+
$ sys1grep -P '/ERROR|FATAL/' app.log # no requests at all
|
|
151
|
+
$ sys1grep -P '/timeout/i' -a '顧客に影響が出ている' app.log # only lines with "timeout" go to Jev
|
|
144
152
|
```
|
|
145
153
|
|
|
146
154
|
A regex's named and numbered groups pass to the other meanings of the *same* AND term as
|
|
@@ -150,7 +158,7 @@ an error rather than an empty string (so is a reference to a negated regex's gro
|
|
|
150
158
|
no group stays literal, as in ECMAScript, so `$100 以上の請求` is unaffected.
|
|
151
159
|
|
|
152
160
|
```sh
|
|
153
|
-
$
|
|
161
|
+
$ sys1grep -P '/(?<date>\d{4}-\d\d-\d\d) (?<time>\d\d:\d\d)/' \
|
|
154
162
|
-a '$<time> が深夜(0時〜5時)であり、$<date> が週末である' app.log
|
|
155
163
|
# 2026-09-19 03:12 ... → asks "03:12 が深夜(0時〜5時)であり、2026-09-19 が週末である"
|
|
156
164
|
```
|
|
@@ -163,7 +171,7 @@ is never colored. `-o` prints each match on a line of its own, as `grep -o`; a l
|
|
|
163
171
|
whole, since a meaning has no matching part.
|
|
164
172
|
|
|
165
173
|
```sh
|
|
166
|
-
$
|
|
174
|
+
$ sys1grep -o -n -P '/[A-Z]+-\d+/' -a 'the ticket is still open' notes.txt # the ticket ids, one per line
|
|
167
175
|
```
|
|
168
176
|
|
|
169
177
|
## Sending less
|
|
@@ -180,10 +188,18 @@ to the narrowest:
|
|
|
180
188
|
its AND term's meanings. Blank lines are never sent. Only the first `-M`/`--max-columns` characters of a line
|
|
181
189
|
(default 2000, 8000 with `-z`) are sent: it is still searched and judged, just not past that cutoff.
|
|
182
190
|
- **How many times.** [`--dedup`](#one-line-per-template---dedup) judges one line per template: lines that
|
|
183
|
-
differ only in ids, numbers, times or paths share one answer.
|
|
191
|
+
differ only in ids, numbers, times or paths share one answer. `auto`/`always`/`never`, off by default for
|
|
192
|
+
now: a hint says when `--dedup=auto` would pay.
|
|
193
|
+
- **Never twice.** `--cache` keeps Jev's answers in `~/.config/sys1grep/cache`, keyed by the request sent (model,
|
|
194
|
+
lines, questions) and the endpoint's host, so the same search again sends nothing and costs nothing. It is off
|
|
195
|
+
by default, since the cache holds the lines searched; `SYS1GREP_CACHE=on` (in the environment or
|
|
196
|
+
`~/.config/sys1grep/.env`, not settings.json) sets the default. `--cache=only` answers from the cache alone: a
|
|
197
|
+
request not there is exit 2 unless `-q` already found a match, named, and nothing is sent. `--dry-run` says how
|
|
198
|
+
many requests are cached and how many it would send. There is no expiry: the model is in the key, and Jev
|
|
199
|
+
drifts about ±0.05 between runs, so a cached answer is one of those runs. `--cache=clear` empties it.
|
|
184
200
|
- **Check before paying.** `--dry-run` sends nothing and prints the settings the search would run with, the
|
|
185
201
|
files, how many lines each would send and every request with its questions. Its last line estimates the input
|
|
186
|
-
tokens and
|
|
202
|
+
tokens and the price (`~3178 input tokens, ~$0.000133`; within about 10%; see "Other endpoints" for whose price). `-i` shows the
|
|
187
203
|
same totals on the terminal and sends only after `y`.
|
|
188
204
|
Before the bulk of requests, every target is sized (`--max-filesize`, default 10M); one over it is skipped
|
|
189
205
|
outright, like `rg`'s own `--max-filesize`, named on stderr (`-y` does not affect it). A `.gz` is sized
|
|
@@ -194,20 +210,24 @@ to the narrowest:
|
|
|
194
210
|
is sent. The input about to be sent is priced too (`--max-cost`,
|
|
195
211
|
default 1 USD); over it, one question asks to continue on the terminal, `-y` answers it yes, and without a
|
|
196
212
|
terminal it is exit 2 (this applies with `-q` too — scripts pass `-y`).
|
|
213
|
+
With `-r` or `git sys1grep`, a term no regex narrows that is about to send more than 10,000 lines gets one
|
|
214
|
+
stderr line with the same totals, naming the term (`sys1grep: sending 214,913 of 231,502 lines from 1,247 files
|
|
215
|
+
(~9.1M input tokens, ~$0.38); the term "…" has no regex to narrow it. Add -a '/RE/' to it, …`). The search goes
|
|
216
|
+
on; `-q` silences the line, `-y` does not.
|
|
197
217
|
|
|
198
218
|
```sh
|
|
199
|
-
$ sys1grep --dry-run -r --include='*.log' --changed-within=today -
|
|
219
|
+
$ sys1grep --dry-run -r --include='*.log' --changed-within=today -P '/ERROR|FATAL/' -a 'a customer is affected' logs/
|
|
200
220
|
```
|
|
201
221
|
|
|
202
222
|
`--verbose` (or `--dry-run`) also shows *where* a setting that was not typed on the command line came from
|
|
203
|
-
(`SYS1GREP_OPTS`, an environment variable, `~/.config/sys1grep
|
|
223
|
+
(`SYS1GREP_OPTS`, an environment variable, `~/.config/sys1grep/settings.json` or `.env`, or a preset's default), so a result that
|
|
204
224
|
surprises you can be traced back to its source. The key's value never appears, only which option or variable
|
|
205
225
|
supplied it:
|
|
206
226
|
|
|
207
227
|
```sh
|
|
208
228
|
$ SYS1GREP_OPTS='--level strict' sys1grep --verbose -e "the API key is read from a file" .
|
|
209
229
|
sys1grep: endpoint api.typesafe.ai/v1/systemone (default), model jev-latest (default)
|
|
210
|
-
sys1grep: key:
|
|
230
|
+
sys1grep: key: key (~/.config/sys1grep/settings.json)
|
|
211
231
|
sys1grep: SYS1GREP_OPTS: --level strict
|
|
212
232
|
sys1grep: options: --level strict (SYS1GREP_OPTS) = -t 0.7 -T 0.3, --chunk 30, -j 8, scope on, -M 2000, --max-filesize 10M, --max-cost 1
|
|
213
233
|
sys1grep: file ./a.py: 120 lines, 120 to send
|
|
@@ -217,7 +237,7 @@ sys1grep: file ./a.py: 120 lines, 120 to send
|
|
|
217
237
|
## Install
|
|
218
238
|
|
|
219
239
|
Two ways to use it: as a command-line tool (this section), or as a Claude Code skill
|
|
220
|
-
(see [Use it from Claude Code](#use-it-from-claude-code) below). The skill
|
|
240
|
+
(see [Use it from Claude Code](#use-it-from-claude-code) below). The skill runs sys1grep through `npx`,
|
|
221
241
|
so if you only use it through Claude Code you can skip the install here entirely and just set the API key.
|
|
222
242
|
|
|
223
243
|
Requires Node.js 20.16 or later. No other dependencies.
|
|
@@ -237,14 +257,34 @@ Then give it an API key from the [TypeSafe console](https://console.typesafe.ai/
|
|
|
237
257
|
|
|
238
258
|
```sh
|
|
239
259
|
export SYS1GREP_API_KEY=your-key # environment variable
|
|
240
|
-
|
|
260
|
+
# per user: ~/.config/sys1grep/settings.json (mkdir -p first; keep it at 0600)
|
|
261
|
+
(umask 077; echo '{"key": "your-key"}' > ~/.config/sys1grep/settings.json)
|
|
241
262
|
```
|
|
242
263
|
|
|
243
|
-
Variables already in the environment win; otherwise `~/.config/sys1grep
|
|
264
|
+
Variables already in the environment win; otherwise `~/.config/sys1grep/settings.json` fills them in, one field per
|
|
265
|
+
variable:
|
|
266
|
+
|
|
267
|
+
| field | variable |
|
|
268
|
+
|---|---|
|
|
269
|
+
| `url` | `SYS1GREP_URL` |
|
|
270
|
+
| `key` | `SYS1GREP_API_KEY` (and `TYPESAFE_API_KEY`) |
|
|
271
|
+
| `model` | `SYS1GREP_MODEL` |
|
|
272
|
+
| `opts` | `SYS1GREP_OPTS`, as an array: `["--level", "strict", "-n"]` |
|
|
273
|
+
| `summarizer` | `SYS1GREP_SUMMARIZER` |
|
|
274
|
+
| `summarizerModel` | `SYS1GREP_SUMMARIZER_MODEL` |
|
|
275
|
+
| `summarizerKey` | `SYS1GREP_SUMMARIZER_API_KEY` |
|
|
276
|
+
| `lang` | `SYS1GREP_LANG` (`en` or `ja`: `--help`'s language, and `--serve`'s page default) |
|
|
277
|
+
|
|
278
|
+
Every field is optional. A value of the wrong type is an error, as is a file that is not JSON. A field the file does
|
|
279
|
+
not know only warns (on stderr, naming the field) and is ignored, so an older sys1grep keeps working once a newer one
|
|
280
|
+
adds a field. A file that holds a key and that others can read gets a warning. `~/.config/sys1grep/.env` (`NAME=value`
|
|
281
|
+
lines) still works, below settings.json: a value in settings.json wins over the same variable there. A `.env` in the current
|
|
244
282
|
directory is never read: it may belong to a repository you just cloned, and could send your key elsewhere through
|
|
245
283
|
`SYS1GREP_URL`. For per-project settings, load a file yourself: `node --env-file=.env "$(command -v sys1grep)" ...`.
|
|
246
284
|
`TYPESAFE_API_KEY` is accepted too when `SYS1GREP_API_KEY` is not set.
|
|
247
285
|
|
|
286
|
+
The `--serve` page's **Settings** dialog edits this settings.json too (see [A search page](#a-search-page---serve)).
|
|
287
|
+
|
|
248
288
|
### Default options
|
|
249
289
|
|
|
250
290
|
`SYS1GREP_OPTS` holds options to apply on every call, read like the settings above. It is split on spaces and put in
|
|
@@ -256,15 +296,18 @@ sys1grep -e "payment failed" app.log # strict, 8 at once, line
|
|
|
256
296
|
sys1grep --level loose --no-n -e "payment failed" app.log
|
|
257
297
|
```
|
|
258
298
|
|
|
299
|
+
The same defaults can live in settings.json as `"opts": ["--level", "strict", "-j", "8", "-n"]`, one argument per
|
|
300
|
+
element (so a value may hold spaces). `SYS1GREP_OPTS`, when set, replaces them rather than adding to them.
|
|
301
|
+
|
|
259
302
|
A script calling sys1grep would pick these up too (grep dropped `GREP_OPTIONS` for that reason). Call it as
|
|
260
|
-
`SYS1GREP_OPTS= sys1grep ...` in scripts.
|
|
303
|
+
`SYS1GREP_OPTS= sys1grep ...` in scripts: the empty variable drops settings.json's `opts` as well.
|
|
261
304
|
|
|
262
305
|
### Other endpoints
|
|
263
306
|
|
|
264
307
|
The API is configured by exactly three settings: `SYS1GREP_API_KEY` (or `TYPESAFE_API_KEY`), `SYS1GREP_URL` and `SYS1GREP_MODEL`.
|
|
265
308
|
Any endpoint that speaks TypeSafe's `POST /v1/systemone` works. The key is sent to `SYS1GREP_URL` as is, so set the
|
|
266
309
|
two together. On the command line, `--sys1-model=ID`, `--sys1-url=URL` and `--sys1-api-key=KEY` override
|
|
267
|
-
the three. A key given this way shows up in `ps` and shell history, so prefer `~/.config/sys1grep
|
|
310
|
+
the three. A key given this way shows up in `ps` and shell history, so prefer `~/.config/sys1grep/settings.json` for it.
|
|
268
311
|
|
|
269
312
|
```sh
|
|
270
313
|
# OpenRouter
|
|
@@ -275,8 +318,31 @@ SYS1GREP_URL=https://ai-gateway.vercel.sh/typesafe/v1/systemone SYS1GREP_MODEL=t
|
|
|
275
318
|
SYS1GREP_URL=http://localhost:8000/v1/systemone sys1grep -e ...
|
|
276
319
|
```
|
|
277
320
|
|
|
278
|
-
|
|
279
|
-
|
|
321
|
+
Every price sys1grep shows (`--dry-run`, the large-send warning, the `--max-cost` question, the summary line) is the cost
|
|
322
|
+
the endpoint reports (`usage.cost`), or else an estimate marked `~`. The estimate's price per million input tokens: a local
|
|
323
|
+
URL is free (`localhost`, `127.0.0.1` or `[::1]`, judged by the host as typed with no DNS lookup, and only for a URL you set
|
|
324
|
+
with `SYS1GREP_URL` or `--sys1-url`); a model in this table has its own price, `jev-latest` 0.042, `clef` 0.24 and
|
|
325
|
+
`clef-flash` 0.09 (USD); any other model is priced as Jev, 0.042. With another URL the estimate says whose price it is:
|
|
326
|
+
`at TypeSafe's list price`, `at clef's list price` or `(local URL, free)`. `--max-cost` uses the same price, so it asks
|
|
327
|
+
about nothing on a local URL.
|
|
328
|
+
|
|
329
|
+
**Local decision models.** A model that serves the same `POST /v1/systemone` can be used instead of Jev, with no key and no
|
|
330
|
+
network, but both tried so far are less accurate than Jev: raise `-t`, tune with `npm run judge`, and expect more false
|
|
331
|
+
positives. Both also run one request at a time, so a local URL (`localhost`, `127.0.0.1`, `[::1]`) defaults to `-j 1`;
|
|
332
|
+
a `-j` you give (on the command line, in `SYS1GREP_OPTS` or settings.json) still wins.
|
|
333
|
+
|
|
334
|
+
- [Jeff](https://github.com/firelex/jeff) (`jeff-serve`, 0.8B, base v1.2, zero-shot, MLX): works with
|
|
335
|
+
`--sys1-url=http://127.0.0.1:8765/v1/systemone --sys1-model=jeff-latest`. It answers a second request at once with
|
|
336
|
+
`529 "The model is busy"`, and `-j 8` runs out of retries. On `npm run judge` (51 lines, 10 cases, `-j 1`)
|
|
337
|
+
precision 0.68, recall 0.61, F1 0.64 at the default thresholds and F1 0.77 at `-t 0.75 -T 0.85`, against F1 0.96 for Jev
|
|
338
|
+
(a run of another day): not a drop-in. v1.2 is a community preview; adapters do not carry across versions.
|
|
339
|
+
- [strands-decider](https://github.com/strands-labs/strands-decider) (`strands-decider serve`, 2B, an M4 with `mps`):
|
|
340
|
+
`SYS1GREP_URL=http://127.0.0.1:8000/v1/systemone` with no key works as is (30 lines in 2.7 s). Two requests at once crash
|
|
341
|
+
its server (exit 134, a Metal assertion). On the same `npm run judge` set: precision 0.64, recall 0.94, F1 0.76 at the
|
|
342
|
+
default thresholds and F1 0.81 at `-t 0.6 -T 0.45`.
|
|
343
|
+
- Not tried with sys1grep yet: Cloudflare's Clef.
|
|
344
|
+
|
|
345
|
+
A request carries at most 64 questions for every backend.
|
|
280
346
|
|
|
281
347
|
From source: `git clone https://github.com/uehaj/sys1grep.git && cd sys1grep && npm install -g .`,
|
|
282
348
|
or run it in place with `node sys1grep.mjs ...`.
|
|
@@ -289,10 +355,12 @@ otherwise show up in the output below, this section instead uses [`tests/corpus.
|
|
|
289
355
|
the same 51 lines with the Japanese ones translated to English — the cross-language behaviour itself is
|
|
290
356
|
shown once, in [Search across languages](#search-across-languages) above.
|
|
291
357
|
|
|
358
|
+
For examples organised by job (on-call, triage, release notes, ...) and run on real data, see [docs/use-cases.md](docs/use-cases.md).
|
|
359
|
+
|
|
292
360
|
### Find lines by a concept, in any language
|
|
293
361
|
|
|
294
362
|
```sh
|
|
295
|
-
$
|
|
363
|
+
$ sys1grep -n -e "customer is angry or frustrated" tests/corpus.en.txt
|
|
296
364
|
14:Inquiry from user Yamada: I want a refund, the item was broken
|
|
297
365
|
16:Inquiry from user Sato: I'm being charged for an order I never placed, it looks like fraud, please check urgently
|
|
298
366
|
18:I want my money back. The item arrived broken and customer service ignored me.
|
|
@@ -309,8 +377,8 @@ None of these lines contain the words "angry" or "frustrated".
|
|
|
309
377
|
so you can write the question as you would ask it:
|
|
310
378
|
|
|
311
379
|
```sh
|
|
312
|
-
$ echo "The job failed." |
|
|
313
|
-
$ echo "The job failed." |
|
|
380
|
+
$ echo "The job failed." | sys1grep -e "Did the job succeed?"
|
|
381
|
+
$ echo "The job failed." | sys1grep -Q "Did the job succeed?"
|
|
314
382
|
The job failed.
|
|
315
383
|
```
|
|
316
384
|
|
|
@@ -319,7 +387,7 @@ the line answers it: no, it failed (0.83). A line that *asks* the question is cl
|
|
|
319
387
|
answers nothing, and for a yes/no question a line that *denies* it still answers it:
|
|
320
388
|
|
|
321
389
|
```sh
|
|
322
|
-
$
|
|
390
|
+
$ sys1grep -n -Q "whether the server is down" tests/intent.txt
|
|
323
391
|
5:The server is down.
|
|
324
392
|
6:The server is healthy and responding normally.
|
|
325
393
|
2 of 17 lines matched; 17 sent to Jev in 1 request, 1087 input tokens, ~$0.000046
|
|
@@ -333,7 +401,7 @@ would match it instead. `-Q X` is shorthand for `-e "the line answers: X"`, so i
|
|
|
333
401
|
### OR: two meanings, and see the probabilities with `-p`
|
|
334
402
|
|
|
335
403
|
```sh
|
|
336
|
-
$
|
|
404
|
+
$ sys1grep -n -p -e "customer is asking for a refund" -e "delivery address change request" tests/corpus.en.txt
|
|
337
405
|
14:Inquiry from user Yamada: I want a refund, the item was broken [0.99 0.01]
|
|
338
406
|
17:Inquiry from user Takahashi: I'd like to change the delivery address [0.01 0.99]
|
|
339
407
|
18:I want my money back. The item arrived broken and customer service ignored me. [0.96 0.01]
|
|
@@ -351,7 +419,7 @@ green at or above `-t`, red below `-T`, yellow in between. Line numbers and file
|
|
|
351
419
|
### AND NOT: network errors, excluding retries
|
|
352
420
|
|
|
353
421
|
```sh
|
|
354
|
-
$
|
|
422
|
+
$ sys1grep -n -e "network or remote connection failure" -v "a retry is happening or was attempted" tests/corpus.txt
|
|
355
423
|
4:2026-09-19 08:02:30 ERROR connection reset by peer while calling payment-gateway
|
|
356
424
|
6:2026-09-19 08:02:35 ERROR timeout after 5000ms waiting for payment-gateway
|
|
357
425
|
9:2026-09-19 08:10:44 ERROR DNS lookup failed for api.example.com
|
|
@@ -359,7 +427,7 @@ $ ./sys1grep -n -e "network or remote connection failure" -v "a retry is happeni
|
|
|
359
427
|
13:network unreachable: no route to host 10.0.0.5
|
|
360
428
|
30:except ConnectionError as e:
|
|
361
429
|
31: logger.error("upstream unreachable: %s", e)
|
|
362
|
-
7 of 51 lines matched; 51 sent to Jev in 2 requests, 5112 input tokens
|
|
430
|
+
7 of 51 lines matched; 51 sent to Jev in 2 requests, 5112 input tokens, ~$0.000215
|
|
363
431
|
```
|
|
364
432
|
|
|
365
433
|
Line 5, `retrying payment-gateway request (attempt 2/3)`, is a network failure but is dropped by `-v`.
|
|
@@ -367,7 +435,7 @@ Line 5, `retrying payment-gateway request (attempt 2/3)`, is a network failure b
|
|
|
367
435
|
### Mixed: (finance AND negative) OR weather
|
|
368
436
|
|
|
369
437
|
```sh
|
|
370
|
-
$
|
|
438
|
+
$ sys1grep -n -e "about economy, finance or markets" -a "the news is negative or a decline" -e "about weather" tests/corpus.en.txt
|
|
371
439
|
36:Today's weather is sunny, high of 28 degrees
|
|
372
440
|
43:Stock prices fell 3% after the earnings report missed expectations.
|
|
373
441
|
48:Tomorrow's forecast is rain so I'll bring an umbrella
|
|
@@ -379,11 +447,11 @@ $ ./sys1grep -n -e "about economy, finance or markets" -a "the news is negative
|
|
|
379
447
|
### Strictness presets
|
|
380
448
|
|
|
381
449
|
```sh
|
|
382
|
-
$
|
|
450
|
+
$ sys1grep --level strict -n -e "a security risk or dangerous destructive operation" tests/corpus.en.txt
|
|
383
451
|
33:DROP TABLE sessions;
|
|
384
452
|
49:API keys must never be committed to the repository.
|
|
385
453
|
|
|
386
|
-
$
|
|
454
|
+
$ sys1grep --level loose -n -e "a security risk or dangerous destructive operation" tests/corpus.en.txt
|
|
387
455
|
11:2026-09-19 09:00:00 ERROR SSL handshake failed: certificate expired
|
|
388
456
|
16:Inquiry from user Sato: I'm being charged for an order I never placed, it looks like fraud, please check urgently
|
|
389
457
|
33:DROP TABLE sessions;
|
|
@@ -395,11 +463,11 @@ $ ./sys1grep --level loose -n -e "a security risk or dangerous destructive opera
|
|
|
395
463
|
### Recursive search and file names only
|
|
396
464
|
|
|
397
465
|
```sh
|
|
398
|
-
$
|
|
466
|
+
$ sys1grep -r -n -e "customer is asking for a refund" tests/tickets/
|
|
399
467
|
tests/tickets/a.txt:7:Ticket #16: I want a refund, the item was broken.
|
|
400
468
|
tests/tickets/sub/b.txt:1:The customer wants a refund for the broken lamp.
|
|
401
469
|
|
|
402
|
-
$
|
|
470
|
+
$ sys1grep -rl -e "customer is asking for a refund" tests/tickets/
|
|
403
471
|
tests/tickets/a.txt
|
|
404
472
|
tests/tickets/sub/b.txt
|
|
405
473
|
```
|
|
@@ -409,8 +477,8 @@ tests/tickets/sub/b.txt
|
|
|
409
477
|
that usually hold secrets (`.env*`, `.netrc`, `.npmrc`, `.pypirc`, `.pgpass`, `.git-credentials`, `*.pem`,
|
|
410
478
|
`*.key`, `*.p12`, `*.pfx`, `*.jks`, `*.keystore`, `id_rsa*` and friends; names compared without case) and
|
|
411
479
|
generated files (`*.map`, `*.min.js`, `*.min.css`, `package-lock.json`, `yarn.lock`, `pnpm-lock.yaml`,
|
|
412
|
-
`Cargo.lock`, `poetry.lock`, `composer.lock`, `Gemfile.lock`, `go.sum`). **Every line that is searched is sent
|
|
413
|
-
to the TypeSafe API
|
|
480
|
+
`Cargo.lock`, `poetry.lock`, `composer.lock`, `Gemfile.lock`, `go.sum`), symlinks and `~/.config/sys1grep/settings.json` itself. **Every line that is searched is sent
|
|
481
|
+
to the TypeSafe API** (or wherever `SYS1GREP_URL` points), so point `-r` at a directory
|
|
414
482
|
you mean to scan. Inside a git repository, `-r` also skips what git ignores (`.gitignore`, `.git/info/exclude`, the
|
|
415
483
|
global excludes file), so build output and local files stay home; a tracked file is searched even if it matches.
|
|
416
484
|
A file named `*.gz` (a rotated log: `app.log.1.gz`) is read decompressed, as `zgrep` does, and printed by its
|
|
@@ -434,7 +502,7 @@ $ sys1grep -r -e 'Python でリトライ処理を書いている箇所' .
|
|
|
434
502
|
sys1grep: scope: *.py *.pyi *.pyw (from "Python files: 0.94")
|
|
435
503
|
sys1grep: scope: 12 of 340 files
|
|
436
504
|
$ sys1grep -r -e '昨日変えた箇所で認証を扱っている' src/
|
|
437
|
-
sys1grep: scope:
|
|
505
|
+
sys1grep: scope: changed since 2026-09-25 00:00 (git commits; the mtime for files git does not have committed) (from "what was changed yesterday: 0.91")
|
|
438
506
|
sys1grep: scope: 3 of 120 files
|
|
439
507
|
```
|
|
440
508
|
|
|
@@ -503,12 +571,25 @@ tree, not `git ls-tree`'s own literal/directory-prefix match); `--include` / `--
|
|
|
503
571
|
**An old `<tree>` can hold a secret an ordinary file once carried and was later removed from**: the skip list
|
|
504
572
|
drops files by name only, not by what changed since.
|
|
505
573
|
|
|
574
|
+
### Commits instead of files (`-g`)
|
|
575
|
+
|
|
576
|
+
`-g` (`--gitlog`) searches the commits of `git log` instead of files, one record each (hash, date, subject, body):
|
|
577
|
+
|
|
578
|
+
```sh
|
|
579
|
+
$ sys1grep -g -e 'a performance fix to the .mjs files today'
|
|
580
|
+
```
|
|
581
|
+
|
|
582
|
+
With one term, auto-scope becomes `git log`'s own arguments, printed on stderr: a time becomes `--since-as-filter`
|
|
583
|
+
(`--since` on git before 2.37, and `--until` when the span has an end: yesterday, last week, last month), a language
|
|
584
|
+
a pathspec, an author or "mine" `--author`, "this branch" or "not pushed" a range. FILE arguments are pathspecs.
|
|
585
|
+
Each commit is a record, as with `-z`, so `-M` defaults to 8000. Not with `-r` or `git sys1grep`.
|
|
586
|
+
|
|
506
587
|
### Everything that is *not* something
|
|
507
588
|
|
|
508
589
|
```sh
|
|
509
|
-
|
|
510
|
-
|
|
511
|
-
cat app.log |
|
|
590
|
+
sys1grep -v "a timestamped server log line" mixed.txt # like grep -v
|
|
591
|
+
sys1grep -e "source code or SQL" -v "SQL" src.txt # code, but not SQL
|
|
592
|
+
cat app.log | sys1grep -e "the deploy failed or was rolled back"
|
|
512
593
|
```
|
|
513
594
|
|
|
514
595
|
### Records that span several lines (`-z`)
|
|
@@ -521,9 +602,9 @@ A proposition like "this commit changes user-visible behaviour" is true of a who
|
|
|
521
602
|
line in it:
|
|
522
603
|
|
|
523
604
|
```sh
|
|
524
|
-
$ git log -z --format='%h %s %b' |
|
|
525
|
-
7:21120e9 Revert "feat: ship the /
|
|
526
|
-
8:51ae333 feat: ship the /
|
|
605
|
+
$ git log -z --format='%h %s %b' | sys1grep -z -n -e "the change alters user-visible behaviour" -v "documentation only"
|
|
606
|
+
7:21120e9 Revert "feat: ship the /semgrep Claude Code skill" ...
|
|
607
|
+
8:51ae333 feat: ship the /semgrep Claude Code skill
|
|
527
608
|
```
|
|
528
609
|
|
|
529
610
|
Matching records are printed NUL-terminated too, so pipe them through `tr '\0' '\n'` to read them.
|
|
@@ -538,7 +619,7 @@ Wrapped lines are joined before splitting, so a sentence that runs over several
|
|
|
538
619
|
[`tests/prose.txt`](tests/prose.txt) wraps an English paragraph and a Japanese one:
|
|
539
620
|
|
|
540
621
|
```sh
|
|
541
|
-
$
|
|
622
|
+
$ sys1grep -n --unit=sentence-by-jev -e "the author admits they made a mistake" tests/prose.txt
|
|
542
623
|
1:I should have checked the input
|
|
543
624
|
2:before shipping, and that was my
|
|
544
625
|
3:mistake. Next time I will add a test
|
|
@@ -557,7 +638,7 @@ inside a line: lines 3 and 9 are colored only up to the end of the matching sent
|
|
|
557
638
|
Thai, Lao, Khmer, Myanmar and Tibetan, which do not put spaces between words:
|
|
558
639
|
|
|
559
640
|
```sh
|
|
560
|
-
$
|
|
641
|
+
$ sys1grep -n -o --unit=sentence-by-jev -e "the author admits they made a mistake" -e "customer is asking for a refund" tests/prose.txt
|
|
561
642
|
1:I should have checked the input before shipping, and that was my mistake.
|
|
562
643
|
8:先週買った掃除機が初日から動かないので返金してほしいです。
|
|
563
644
|
```
|
|
@@ -594,7 +675,7 @@ In code the question is usually about a function ("retries on network failure"),
|
|
|
594
675
|
says so. `--unit=function` judges each function and prints its lines, with `-n` giving file line numbers:
|
|
595
676
|
|
|
596
677
|
```sh
|
|
597
|
-
$
|
|
678
|
+
$ sys1grep -H -n --unit=function -e "retries on network failure" src/net.js
|
|
598
679
|
src/net.js:3:async function fetchWithRetry(url) {
|
|
599
680
|
src/net.js:4: for (let i = 0; i < 5; i++) {
|
|
600
681
|
src/net.js:5: try { return await fetch(url); }
|
|
@@ -620,6 +701,59 @@ $ git config diff.golang.xfuncname '^(func|type)[[:space:]]'
|
|
|
620
701
|
`--chunk` counts functions, `-A`/`-B`/`-C` and `-c` count lines, as with the sentence units. `-z`, `-g` and
|
|
621
702
|
`-o` are refused.
|
|
622
703
|
|
|
704
|
+
### From one function to another (`--step-to`)
|
|
705
|
+
|
|
706
|
+
A question about code often comes in two steps: where does `--summarize` hand the lines to the tool, and then,
|
|
707
|
+
what happens when that fails. The "that" in the second step is what the first one found. Multi-step matching
|
|
708
|
+
answers both in one run: the expression before `--step-to` finds the start, sys1grep walks the calls from it,
|
|
709
|
+
and the expression after `--step-to` picks the functions it reaches that match. Jev judges the two
|
|
710
|
+
expressions; no score decides which call to follow.
|
|
711
|
+
|
|
712
|
+
```sh
|
|
713
|
+
$ sys1grep -e "--summarize hands the lines to the tool" --step-to "what happens when the tool fails to start or answer" sys1grep.mjs
|
|
714
|
+
```
|
|
715
|
+
|
|
716
|
+
A meaning may start with a dash, as the first one does, on any search and for `-e`, `-a`, `-v` and `-Q`: a value that starts with `-` is taken as the meaning unless it is an option (an option holds no space).
|
|
717
|
+
|
|
718
|
+
Each path prints as a tree: the hop (the number of calls from the start), `file:line` and the function's name,
|
|
719
|
+
`:` after an end and `-` after a function on the way, as grep marks a match and its context. A regex-only
|
|
720
|
+
start and end send nothing, which makes a free reachability search. In Python 3.6's `json`, the functions
|
|
721
|
+
that `load` reaches and that hold a `raise`:
|
|
722
|
+
|
|
723
|
+
```
|
|
724
|
+
$ cd /usr/lib64/python3.6/json && sys1grep -P '/^\s*def load\b/' --step-to -P '/\braise\b/' *.py
|
|
725
|
+
sys1grep: walk: 1 at hop 0, 1 at hop 1, 3 at hop 2, 1 at hop 3, 1 at hop 4, 1 at hop 5; stopped: no new unit
|
|
726
|
+
0 __init__.py-274-load
|
|
727
|
+
1 __init__.py:302:loads
|
|
728
|
+
2 decoder.py:334:decode
|
|
729
|
+
3 decoder.py:345:raw_decode
|
|
730
|
+
4 scanner.py-65-scan_once
|
|
731
|
+
5 scanner.py:28:_scan_once
|
|
732
|
+
```
|
|
733
|
+
|
|
734
|
+
- **The expressions.** The expression before `--step-to` is the start's, as in any search; every `-e` / `-P` /
|
|
735
|
+
`-a` / `-v` / `-Q` after it is the end's. A bare `--step-to MEANING` is `--step-to -e MEANING` and a bare
|
|
736
|
+
`--step-to '/RE/'` is auto-detected and becomes `--step-to -P '/RE/'`; the file names follow. Write the
|
|
737
|
+
explicit `-P` form for a regex end, as in the example above: it reads the same but names the option. A step
|
|
738
|
+
is a search, not a hop: one `--step-to` is two steps, whatever the hops between them.
|
|
739
|
+
- **The edges.** Without `--edges` the unit is a function (`--unit=function`) and an edge is a call: `name(` in
|
|
740
|
+
a function's body, its comments, docstrings and strings left out, links to every function of that name. A name
|
|
741
|
+
comes from the funcname line (the `def` under a decorator). `--edges=FILE` walks any other relation instead,
|
|
742
|
+
one edge per line, `FROM_FILE:LINE<TAB>FROM_NAME<TAB>TO_FILE:LINE<TAB>TO_NAME`; a line stands for the unit that
|
|
743
|
+
holds it. `--step-reverse-to` in place of `--step-to` walks the edges backwards, callee to caller: the same
|
|
744
|
+
arguments, with `--hops` and `--edges`, and not together with `--step-to`.
|
|
745
|
+
- **The walk.** Breadth first from every start at once. A function's hop is its shortest distance from any
|
|
746
|
+
start, and each function is reached once, so a cycle ends. `--hops=N`, `--hops=M..N` and `--hops=M..` pick the
|
|
747
|
+
hops an end may be at (default `0..`: a start that matches the end expression is an end at hop 0). A path that
|
|
748
|
+
reaches no end is left out. stderr says how many functions each hop reached and why the walk stopped: no new
|
|
749
|
+
unit, or `--hops`.
|
|
750
|
+
- **The cost.** Jev judges every function against the start expression in one batch, then the functions reached
|
|
751
|
+
within `--hops` against the end expression in another. `--max-cost` asks again before the second, counting the
|
|
752
|
+
first. `--dry-run` answers every question 0, so it walks from every function the start expression could hold
|
|
753
|
+
for and shows that bound.
|
|
754
|
+
- **Not with** `-z`, `-g`, `-o`, `-c`, `-l`, `-A`/`-B`/`-C`, `--rank`, `--summarize` or `--dedup`. `-p` adds the
|
|
755
|
+
end expression's scores to each function judged against it; `-q` prints nothing and exits 0 when there is an end.
|
|
756
|
+
|
|
623
757
|
### One line per template (`--dedup`)
|
|
624
758
|
|
|
625
759
|
Cost is proportional to the text sent, and machine-generated logs are mostly one skeleton with a different
|
|
@@ -627,8 +761,21 @@ id or number in it. `--dedup` masks ids, hashes, numbers, dates and times, paths
|
|
|
627
761
|
the result, sends one line per group and reuses its answer for the rest. What is sent is that line's
|
|
628
762
|
original text, and every line is printed as itself:
|
|
629
763
|
|
|
764
|
+
`--dedup` takes `auto`, `always` or `never`; a bare `--dedup` is `--dedup=always`. **Default is `never`, for
|
|
765
|
+
now** (#143): `auto` is held back until real-run stats say it should be the default. Every run still
|
|
766
|
+
estimates locally, before anything is sent, what folding every kind (the best case) would save, and prints
|
|
767
|
+
one stderr hint when that is worth at least twice the cost of `--dedup`'s own question:
|
|
768
|
+
|
|
769
|
+
```
|
|
770
|
+
$ sys1grep -e "a request failed" app.log
|
|
771
|
+
sys1grep: 2040 units fold to at most 31 templates; --dedup=auto would save ~68 requests (~44k tokens)
|
|
772
|
+
```
|
|
773
|
+
|
|
774
|
+
`--dedup=auto` folds only when that estimate says it pays; a file that barely repeats sends every
|
|
775
|
+
line, asking nothing extra. `--verbose` / `--dry-run` print the decision and its numbers for every value.
|
|
776
|
+
|
|
630
777
|
```sh
|
|
631
|
-
$
|
|
778
|
+
$ sys1grep --dedup -n -e "a request failed" app.log
|
|
632
779
|
1:worker request 3fa9c1e27b failed: connection reset
|
|
633
780
|
2:worker request 88d0e41a5c failed: connection reset
|
|
634
781
|
4:worker request 0b7f2a9e13 failed: connection reset
|
|
@@ -636,9 +783,21 @@ $ ./sys1grep --dedup -n -e "a request failed" app.log
|
|
|
636
783
|
```
|
|
637
784
|
|
|
638
785
|
Whether a value may be folded depends on the meaning: a number decides "disk usage is above 90%", a time
|
|
639
|
-
decides "happened at night". So
|
|
640
|
-
could change a match, and those kinds
|
|
641
|
-
`-e "disk usage is above 90%"` the same file keeps `95%` and `12%` apart and prints only `5:disk usage 95%`.
|
|
786
|
+
decides "happened at night". So `--dedup=always` (and a bare `--dedup`) first asks Jev, one small request per
|
|
787
|
+
meaning, which kinds of value could change a match, and keeps those kinds apart (one of the two requests above).
|
|
788
|
+
With `-e "disk usage is above 90%"` the same file keeps `95%` and `12%` apart and prints only `5:disk usage 95%`.
|
|
789
|
+
|
|
790
|
+
`--dedup=auto` asks no such question (#152). It folds every kind and judges a sample of each template instead:
|
|
791
|
+
up to 5 members whose values differ, for each varying value the member holding its lowest, its highest and its
|
|
792
|
+
middle value. When every sample gets the same verdict, the members not sampled share the first sample's answers,
|
|
793
|
+
and the samples keep their own (`-p` prints each sample's own probability). When the verdicts differ, the rest are
|
|
794
|
+
sent in a second wave, and `--max-cost` is checked again before it. A split costs slightly more than `never`, at
|
|
795
|
+
worst one extra request per wave boundary, a few percent: one template of 600 members sent 21 requests where
|
|
796
|
+
`never` sent 20. A value only between the samples is missed: among `usage 1` to `usage 120`, the meaning
|
|
797
|
+
`usage 50` matches no sample, so `auto` prints nothing where `never` prints line 50. `--verbose` prints how many
|
|
798
|
+
templates agreed and how many units were sent again. `--dry-run` assumes every sample agrees, so its count is the
|
|
799
|
+
floor: 100 templates of 6 members show 10 requests and 300 of 600 lines, and a run where every template splits
|
|
800
|
+
sends all 600 lines in 20 requests.
|
|
642
801
|
|
|
643
802
|
Measured on real logs, with nothing kept: a 43,071-line system log folds into 550 templates (1.2% of the
|
|
644
803
|
bytes), `install.log` to 21.1%, a Claude Code transcript (jsonl) only to 56.8%. It is for machine-generated
|
|
@@ -675,9 +834,11 @@ $0.094 to $0.013, with the same answer (#69). It pays when the answer sits in a
|
|
|
675
834
|
- `-n`, `-A/-B/-C`, `-p` and file names go in as they would print; colors never do. No match runs nothing (exit 1).
|
|
676
835
|
- `SYS1GREP_SUMMARIZER` picks the TOOL of a bare `--summarize`, `SYS1GREP_SUMMARIZER_MODEL` its model.
|
|
677
836
|
- `-q`, `-l` and `-c` print no lines, so they cannot be combined with it.
|
|
837
|
+
- In `SYS1GREP_OPTS` it summarizes every search, so every match goes to TOOL's provider too;
|
|
838
|
+
`--no-summarize` turns it off for one search.
|
|
678
839
|
- The answer is plain text: an LLM that is not told writes Markdown, noise on a terminal, so plain is asked for.
|
|
679
|
-
`--
|
|
680
|
-
The answer prints as it comes, unchecked. In `SYS1GREP_OPTS` it is a standing preference, ignored without `--summarize`.
|
|
840
|
+
`--format=markdown` or `=html` asks for those instead (`… --format=html … > summary.html`).
|
|
841
|
+
The answer prints as it comes, unchecked. In `SYS1GREP_OPTS` it is a standing preference, ignored without `--summarize` or `--rank`.
|
|
681
842
|
- With `--dedup`, the TOOL gets what Jev got: each template's representative once, marked `(×N like it)`,
|
|
682
843
|
not every line its answer was reused for.
|
|
683
844
|
- More than 200 KB (about 50k tokens) is not sent at all: exit 2, with the size, before the TOOL is paid.
|
|
@@ -690,9 +851,19 @@ $ sys1grep -r -n --summarize --summarize-prompt="3 lines or fewer, just which fi
|
|
|
690
851
|
-e "the API key is read from a file" .
|
|
691
852
|
```
|
|
692
853
|
|
|
693
|
-
It needs `--summarize`; empty TEXT is the same as leaving it out. It comes after the `--
|
|
854
|
+
It needs `--summarize`; empty TEXT is the same as leaving it out. It comes after the `--format` sentence,
|
|
694
855
|
so TEXT can override the format.
|
|
695
856
|
|
|
857
|
+
To decide after seeing the lines, `--summarize=ask` prints them as usual and then asks on the terminal
|
|
858
|
+
`Summarize these matches? [y/N]`. On `y` the lines just printed go to the TOOL of a bare `--summarize`: no second
|
|
859
|
+
search, nothing more sent to Jev. Enter or anything else ends there.
|
|
860
|
+
|
|
861
|
+
- The question goes over `/dev/tty`, so a stdin target still works. With no terminal, or stdout not one (a pipe,
|
|
862
|
+
`$(...)`, a script), nothing is asked and nothing hangs.
|
|
863
|
+
- `-y` summarizes without asking, so `-y` in `SYS1GREP_OPTS` makes it summarize every time; `-q` and `--dry-run` never ask.
|
|
864
|
+
- With `--dedup`, every matching line prints; the TOOL gets each template's representative, as with `--summarize`.
|
|
865
|
+
- The exit status is the search's, whatever the summary does. Over 200 KB, a `y` says so instead of sending.
|
|
866
|
+
|
|
696
867
|
Other TOOLs:
|
|
697
868
|
|
|
698
869
|
```sh
|
|
@@ -709,6 +880,218 @@ OpenAI-compatible server. All three need `SYS1GREP_SUMMARIZER_MODEL`: none has a
|
|
|
709
880
|
`SYS1GREP_SUMMARIZER_API_KEY` goes as `Authorization: Bearer` to a URL TOOL only (never `SYS1GREP_API_KEY`,
|
|
710
881
|
which is Jev's). `OLLAMA_HOST` moves ollama's host, as it does for the `ollama` CLI itself.
|
|
711
882
|
|
|
883
|
+
### A search page (`--serve`)
|
|
884
|
+
|
|
885
|
+
`--rank --format=html` answers one question per run. `--serve` opens such a page (the `search` template, unless `--template` is given) with a search box, so you can rephrase
|
|
886
|
+
and narrow without rerunning the command:
|
|
887
|
+
|
|
888
|
+
```
|
|
889
|
+
$ sys1grep --serve -r src/
|
|
890
|
+
http://127.0.0.1:51234/?k=3f9e...
|
|
891
|
+
$ sys1grep --serve=8080 -y --level=strict -r src/ # a fixed port; the options given here are the page's first values
|
|
892
|
+
```
|
|
893
|
+
|
|
894
|
+
From a clone, `npm run serve -- -r src/` is the same (`--` keeps npm from reading the options; `npm run serve -- --serve=8080 -r src/` fixes the port). With no target, `--serve` searches its own bundled sample corpus (`examples/`, #237) instead of reading an empty stdin, so every example button has something to find; name a target to search that instead. A folder named without `-r` is refused at launch, with the command that adds it (`--serve: src is a directory: add -r to search it (sys1grep --serve -r src)`, #222).
|
|
895
|
+
|
|
896
|
+
The printed URL carries a token made fresh at launch (`?k=...`). Every route needs it, the page itself included: without
|
|
897
|
+
it (in the query, or in a header for the page's own requests) the server answers 404 to everything, so another local
|
|
898
|
+
account or process cannot read the page, its token, or a search's results merely by knowing the port.
|
|
899
|
+
|
|
900
|
+
`EN` / `日本語` in the title bar switches the whole page's language (headings, labels, buttons, hints, the settings
|
|
901
|
+
dialog, the page's own messages), remembered in the browser; its default is the saved choice, else `--lang` /
|
|
902
|
+
`SYS1GREP_LANG` / settings.json's `lang` (below), else the browser's own language, else English. What sys1grep itself
|
|
903
|
+
prints (the results, the `--dry-run` estimate, an error), the command line and option names are never translated.
|
|
904
|
+
|
|
905
|
+
The page is a set of cards under a blue header band: Target and Search conditions in one card with a large Search
|
|
906
|
+
button (Estimate cost and Stop under it) at its right, the examples and Options side by side below it (one column on
|
|
907
|
+
a phone), then the command line and Details. **Target** shows what every search reads: the files and folders given at launch
|
|
908
|
+
(`npm run serve -- src docs/a.md`), or with none what sys1grep falls back to: `.` with `-r`, every file git tracks
|
|
909
|
+
with `git sys1grep`, or, with neither, the bundled sample corpus (`demo corpus (bundled): examples/`, #237); while
|
|
910
|
+
`-g` (in Details) is on, it says git log's commits instead. It is shown only: the page cannot
|
|
911
|
+
change it, and a target in a request is ignored. **Search conditions**
|
|
912
|
+
holds the meaning fields and multi-step, **Options** `rank` and `summarize`,
|
|
913
|
+
and **Details** the other options; the results come below them. What they hold: the meaning fields (`+` adds one; each takes `-e`, `-P`, `-a`, `-v` or `-Q`), `rank`, `summarize`, a **multi-step**
|
|
914
|
+
switch (on, it shows the end fields, `--step-to` and one or more meanings, and the edges, `--hops` and a reverse switch that makes it `--step-reverse-to`; off, they
|
|
915
|
+
leave the command and the search and keep their values), example buttons (searches from this README and
|
|
916
|
+
[docs/use-cases.md](docs/use-cases.md), then multi-step walks; one fills the fields and the controls it needs, puts the rest back to their launch values, and waits for Search), a
|
|
917
|
+
**Details** fold with the other options (`--level`, `-t`/`-T`, `-C`, `-A`/`-B`, `--auto-scope`, `-n`, `-p`, `--dedup`,
|
|
918
|
+
`--unit`, `-o` (shown only with a sentence unit), `-j`, `--max-cost`, `--include`, `--exclude`, `--changed-within`,
|
|
919
|
+
`-g`), and the
|
|
920
|
+
**command line** for the controls as they are now, with a Copy button; pasted in a terminal it runs the
|
|
921
|
+
same search. An option at its launch value is left out of what the controls add. `rank` off shows the matches in file order
|
|
922
|
+
as text, on shows the ranked cards; `summarize` adds a right column: Search runs once, and once its results are on
|
|
923
|
+
screen the page summarizes those results (POSTs the text shown, rank's cards included, to the summarizer; no second
|
|
924
|
+
search, nothing extra sent to Jev, up to 200 KB as `--summarize` takes). Checking `summarize` while results are
|
|
925
|
+
already on screen asks, in the page, whether to summarize those results as they are (#227): OK sends their text back
|
|
926
|
+
the same way; Not now leaves the box checked for the next Search. If a field or option changed since
|
|
927
|
+
those results came, the question says so. No results on screen, or multi-step on: no question. Estimate cost is `--dry-run`. The first field is a question (`-Q`) by default. `--suggest=Q:TEXT` / `--suggest=e:TEXT` (or `--suggest TEXT`, repeatable; a bare TEXT is a question) puts suggested-question buttons first in the examples row; pressing one fills the page, then Search runs it. The options in Details each have a "?" hint saying what they do. While a search runs, Search and Estimate are disabled and **Stop** ends it (the child process is killed). The matches are colored as `--color=always` colors them (file names, line numbers, regex matches), the cards mark them with `<mark>`.
|
|
928
|
+
|
|
929
|
+
Each search runs sys1grep itself (the options given at launch, then the controls'), so everything the command line does
|
|
930
|
+
the page does, and only that. The server listens on `127.0.0.1` only, answers only to that host name, and the page never
|
|
931
|
+
sees the API key. The targets, `--chunk`, `-M`, `--max-filesize`, `-y`, `--edges`, `--template`, the
|
|
932
|
+
API settings and `--summarize`'s TOOL are fixed at launch: the page can narrow what was given but not widen it (`--chunk`,
|
|
933
|
+
`-M` and `--max-filesize` still reach the Settings dialog's default options). A search over `--max-cost` is refused at
|
|
934
|
+
once (`--serve` has no `/dev/tty` to ask `-y`'s question on, or launch with `-y` to let it through unasked); the error
|
|
935
|
+
carries the same estimate `--dry-run` would show, and a button next to it reruns that one search, once, with
|
|
936
|
+
`--max-cost` just above the estimate (not a change to the Details field).
|
|
937
|
+
The meanings come from the page, so `-e -P -a -v -Q --step-to --step-reverse-to` and `-l -c -q -o -z --unit=zero -H -i --format --color --dry-run --verbose` are refused at launch.
|
|
938
|
+
|
|
939
|
+
**Settings** edits `~/.config/sys1grep/settings.json` (see [Install](#install)): `url`, `model`, the key, `opts` (one argument
|
|
940
|
+
per line), the summarizer's three and `lang` (the page's own default language). Save writes the file, and the next search uses it, as does every later `sys1grep`
|
|
941
|
+
run. Next to each field the page says where the value a search uses comes from, and says so when an environment variable or a
|
|
942
|
+
launch option wins over the file, since then saving does not change the search. A key comes back only as saved / not
|
|
943
|
+
saved; type a new one to replace it, or tick remove. A `--sys1-api-key` inside `opts`, and a `url`'s userinfo, come back
|
|
944
|
+
masked as `***`, the same way `--verbose` masks them; `opts` refuses a target, a flag the page itself refuses at launch,
|
|
945
|
+
or `--sys1-url` / `--sys1-api-key` (their own fields), and `url` and a URL summarizer must be `http(s)`. The save is a
|
|
946
|
+
`POST` with the page's token and this request's own Host as its required `Origin`, and the file is written whole (0600,
|
|
947
|
+
its directory 0700 when created) and renamed into place, so a reader never sees half of it.
|
|
948
|
+
|
|
949
|
+
### Best first (`--rank`)
|
|
950
|
+
|
|
951
|
+
Matches print in file order, as grep prints them. With many, `--rank` prints the results best first, each under a
|
|
952
|
+
numbered header. A result is a match with its `-A/-B/-C` lines; matches whose context touches are one result.
|
|
953
|
+
|
|
954
|
+
```sh
|
|
955
|
+
$ sys1grep -n -C1 --rank -e "the refund was refused" tickets/
|
|
956
|
+
1. tickets/b.txt
|
|
957
|
+
tickets/b.txt-2-Order #1234, placed 2026-08-01.
|
|
958
|
+
tickets/b.txt:3:Your refund was declined: the order is older than 30 days.
|
|
959
|
+
tickets/b.txt-4-Please contact support.
|
|
960
|
+
|
|
961
|
+
2. tickets/a.txt
|
|
962
|
+
tickets/a.txt-11-Order #88 arrived.
|
|
963
|
+
tickets/a.txt:12:Refund request received.
|
|
964
|
+
tickets/a.txt-13-We will check it.
|
|
965
|
+
```
|
|
966
|
+
|
|
967
|
+
- `--rank` is `--rank=jev`: after the search, Jev is asked of each result, lines and context together, whether it
|
|
968
|
+
is relevant to the meanings that are not negated. One more question per result, `--chunk` results (at most 64) to a request;
|
|
969
|
+
`--dry-run` / `-i` show an upper bound, since which results there are is known only after the search.
|
|
970
|
+
- `--rank=match` sorts by each result's highest match probability, with no request. It is the answer to a yes/no
|
|
971
|
+
question on one line, so clear matches sit close together, and the context is not read.
|
|
972
|
+
- `-p` puts the score on the header (`1. [0.96] tickets/b.txt`). `-l` lists the files by their best result.
|
|
973
|
+
- `--format=markdown` writes a `## 1. tickets/b.txt` heading and a fenced block per result; `--format=html` one
|
|
974
|
+
self-contained HTML document from a template (below), light and dark, with a relevance bar per result. The lines
|
|
975
|
+
are as they print, escaped; what matched is in `<mark>` (the regex matches, else the matching sentences, else the whole
|
|
976
|
+
matching line; context lines are not marked; `--color=never` turns it off).
|
|
977
|
+
- `--summarize` gets the results in ranked order; with `--dedup` a representative is one result.
|
|
978
|
+
- It needs a meaning (a regex, `!` or `-v` alone ranks nothing), and cannot be combined with `-c`, `-o` or `-q`.
|
|
979
|
+
`--no-rank` turns off an earlier one, from `SYS1GREP_OPTS` say.
|
|
980
|
+
|
|
981
|
+
### Finding the cause of a failed command
|
|
982
|
+
|
|
983
|
+
A failed job's last error usually names a symptom (`214 accounts out of balance ... day-close aborted`), and the
|
|
984
|
+
other `ERROR` / `WARN` lines are retries that recovered. The cause is often one plain `INFO` or `DEBUG` line far
|
|
985
|
+
above. Asking for the symptom, or `-Q "why did the job fail"`, finds that noise: on the 30,000-line log of #156 it
|
|
986
|
+
returned only recovered `ERROR` lines. Ask by kind of cause instead, and rank the results:
|
|
987
|
+
|
|
988
|
+
```sh
|
|
989
|
+
$ sys1grep -n --dedup --level strict --rank \
|
|
990
|
+
-e "a component, writer, job or feature was paused, disabled, skipped or put on hold" \
|
|
991
|
+
-e "a configuration, endpoint, region, credential or data source was changed or switched" \
|
|
992
|
+
-e "an input, file or table was empty, missing, or had fewer records or columns than expected" \
|
|
993
|
+
-e "a fallback or default value was used instead of the real one" \
|
|
994
|
+
job.log
|
|
995
|
+
```
|
|
996
|
+
|
|
997
|
+
Measured on 24 generated logs of failed jobs, six per kind of cause, 10,008 lines each, with these four meanings
|
|
998
|
+
(`npm run cause-eval`, rows in `tests/cause-results.tsv`; an earlier version of the script wrote its 24 rows, and
|
|
999
|
+
the `top 2 jev` column is filled only in the two rerun rows). Each search returned 366 to 464 results. The
|
|
1000
|
+
`--rank=match` column is not a separate run: it is recomputed from the `-p` columns of the `--rank` run, since
|
|
1001
|
+
`--rank=match` scores a result by the highest of them. That recomputation was checked against a real `--rank=match`
|
|
1002
|
+
run only on config-5, where the cause was not a result. A cause "not matched" may have been judged below `strict` or
|
|
1003
|
+
folded by `--dedup` into a like line; the rows do not tell the two apart.
|
|
1004
|
+
|
|
1005
|
+
| cause | cause is result 1, `--rank` | cause is result 1, `--rank=match` | otherwise |
|
|
1006
|
+
|---|---|---|---|
|
|
1007
|
+
| a configuration, endpoint or credential switched | 5 of 6 | 5 of 6 | 1 not matched |
|
|
1008
|
+
| an input empty or short | 2 of 6 | 2 of 6 | 2 at results 62 (`match`: 343-371) and 81 (`match`: 191-219); 2 not matched |
|
|
1009
|
+
| a default used instead of the real value | 6 of 6 | 5 of 6 | `match`: 2-9 (one result above it, seven tied with it at 0.90) |
|
|
1010
|
+
| a line that never came (a start with no finish) | 0 of 6 | 0 of 6 | the start line never a result |
|
|
1011
|
+
|
|
1012
|
+
A log cost about $0.088 (2.1M input tokens, 643 to 646 requests) and 16 to 17 s. `--rank` (`jev`) adds one question
|
|
1013
|
+
per result, about 3% more input tokens than `--rank=match` (one log run both ways; not recorded in the TSV). It put
|
|
1014
|
+
the cause higher than `--rank=match` on three logs and lower on none.
|
|
1015
|
+
|
|
1016
|
+
When the top results look unrelated (rotations, refreshes, the final error itself), the cause is likely one of these:
|
|
1017
|
+
|
|
1018
|
+
- **A line that states only a value**: `stock snapshot downloaded: 1204 bytes`, `input directory ... holds 2 files`,
|
|
1019
|
+
`resolved db-primary.internal to 10.0.4.22 (replica-2)`. Nothing in the line says it is wrong, so no hypothesis
|
|
1020
|
+
holds, nor did a meaning written from the final error
|
|
1021
|
+
(`-e "the job connected to a read-only replica instead of the primary database"`). Search for the thing the error
|
|
1022
|
+
names instead (`-P '/snapshot|replica/'`) and read the numbers.
|
|
1023
|
+
- **A line that never came**: a worker that claimed a shard and never finished it, a step or a migration that started
|
|
1024
|
+
and never ended. On all six such logs the start line was not among the results. The top result was a routine line,
|
|
1025
|
+
the final error, or once the previous run's `alias products switched to products-20260927`, whose 09-28
|
|
1026
|
+
counterpart never came. A meaning for it
|
|
1027
|
+
(`-e "a worker, step or upload started but never reported that it finished"`) put the final error first, not the
|
|
1028
|
+
start, and `--dedup` can fold one start into a like one, since only the missing finish tells them apart. Compare
|
|
1029
|
+
starts with finishes by eye or with grep.
|
|
1030
|
+
- **The score will not tell you**: where the cause was result 1 it scored 0.93 to 0.97 under `--rank`; on two logs
|
|
1031
|
+
where it never matched, the top result scored 0.88 and 0.91.
|
|
1032
|
+
|
|
1033
|
+
The follow-ups behind the first two points (key lines judged alone in a 31-line window, the meanings written from
|
|
1034
|
+
the error or for a missing finish) are in the description of PR #192; `tests` holds no way to rerun them.
|
|
1035
|
+
|
|
1036
|
+
Every searched line goes to Jev (TypeSafe, or whatever `SYS1GREP_URL` names), and with `--summarize` the matching
|
|
1037
|
+
lines go to its TOOL's provider as well. Build logs and job logs often hold tokens, passwords and customer data.
|
|
1038
|
+
|
|
1039
|
+
### HTML templates (`--template`)
|
|
1040
|
+
|
|
1041
|
+
`--rank --format=html` and `--summarize --format=html` fill in a template. Four come with sys1grep: `default`, `print` (black on white, serif,
|
|
1042
|
+
for paper and PDF), `search` (a results page like a web search engine's) and `terminal` (dark monospace).
|
|
1043
|
+
`--template=NAME` picks one; `SYS1GREP_TEMPLATE`, in the
|
|
1044
|
+
environment or `~/.config/sys1grep/.env`, sets the default.
|
|
1045
|
+
|
|
1046
|
+
```sh
|
|
1047
|
+
$ sys1grep -r -n -C1 --rank -p --format=html --template=print -e "the refund was refused" tickets/ > refunds.html
|
|
1048
|
+
```
|
|
1049
|
+
|
|
1050
|
+
A template is one HTML file. The part between `<!--result-->` and `<!--/result-->` is repeated for each result;
|
|
1051
|
+
what is before and after it is written once. sys1grep replaces these, escaped, and leaves any other `{{…}}` as
|
|
1052
|
+
written:
|
|
1053
|
+
|
|
1054
|
+
| placeholder | where | value |
|
|
1055
|
+
|---|---|---|
|
|
1056
|
+
| `{{title}}` | anywhere | `sys1grep: ` and the meanings |
|
|
1057
|
+
| `{{query}}` | anywhere | the meanings, as in `"refund" and not "policy"` |
|
|
1058
|
+
| `{{count}}` | anywhere | the number of results (with `--summarize`, of matching lines) |
|
|
1059
|
+
| `{{answer}}` | anywhere | `--summarize`'s answer; empty with `--rank` |
|
|
1060
|
+
| `{{rank}}` | per result | 1, 2, … |
|
|
1061
|
+
| `{{score}}` | per result | the score as `-p` prints it (`0.96`); empty without `-p` |
|
|
1062
|
+
| `{{score_pct}}` | per result | the score as a whole number 0 to 100, set even without `-p` (for a bar: `style="--s:{{score_pct}}"`) |
|
|
1063
|
+
| `{{file}}` | per result | the file name; empty when one file is searched |
|
|
1064
|
+
| `{{lines}}` | per result | the result's lines as they print |
|
|
1065
|
+
|
|
1066
|
+
`--template=NAME` reads `~/.config/sys1grep/templates/NAME.html` if it is there, else the bundled one, so your
|
|
1067
|
+
copy of `default` replaces it. A value containing `/` or ending in `.html` is a file path. To make your own, copy
|
|
1068
|
+
the bundled ones and edit:
|
|
1069
|
+
|
|
1070
|
+
```sh
|
|
1071
|
+
$ sys1grep --install-templates # "copied PATH" for each file
|
|
1072
|
+
$ cp ~/.config/sys1grep/templates/default.html ~/.config/sys1grep/templates/team.html
|
|
1073
|
+
$ sys1grep -r --rank --format=html --template=team -e "..." src/ > out.html
|
|
1074
|
+
$ sys1grep --template=list # one name a line, "(user)" after those under ~/.config
|
|
1075
|
+
```
|
|
1076
|
+
|
|
1077
|
+
`{{score_pct}}` is set even without `-p`, so a bar can show the relevance without printing the number.
|
|
1078
|
+
|
|
1079
|
+
With `--summarize --format=html` the TOOL is asked for plain text, and that answer, escaped, goes into `{{answer}}`
|
|
1080
|
+
once it has finished; the part between `<!--result-->` and `<!--/result-->` is not written. A template for
|
|
1081
|
+
`--summarize` therefore needs `{{answer}}` outside that part (the bundled ones have it; with `--rank` it is empty).
|
|
1082
|
+
|
|
1083
|
+
```sh
|
|
1084
|
+
$ sys1grep -r --summarize --format=html --template=print -e "why the refunds were refused" tickets/ > why.html
|
|
1085
|
+
```
|
|
1086
|
+
|
|
1087
|
+
A template is trusted local input: a path you give, or a symlink in the templates directory, is read as it is;
|
|
1088
|
+
only the results put into it are escaped.
|
|
1089
|
+
|
|
1090
|
+
`--install-templates` never overwrites a file that is already there (it prints `kept` for it). A missing
|
|
1091
|
+
template, or one without exactly one `<!--result-->` before one `<!--/result-->`, is exit 2 naming the file.
|
|
1092
|
+
`--template` on the command line needs `--format=html` with `--rank` or `--summarize` (not `-l`); `SYS1GREP_TEMPLATE` is simply unused elsewhere.
|
|
1093
|
+
`--template=list` and `--install-templates` take no other arguments and are refused in `SYS1GREP_OPTS`.
|
|
1094
|
+
|
|
712
1095
|
## Use it from Claude Code
|
|
713
1096
|
|
|
714
1097
|
There is a Claude Code skill that runs sys1grep for you: describe what you are looking for in plain words
|
|
@@ -720,8 +1103,8 @@ claude plugin marketplace add uehaj/uehaj-marketplace
|
|
|
720
1103
|
claude plugin install uehaj@uehaj-marketplace
|
|
721
1104
|
```
|
|
722
1105
|
|
|
723
|
-
No separate install of the command-line tool is needed: the skill
|
|
724
|
-
|
|
1106
|
+
No separate install of the command-line tool is needed: the skill runs a version it pins,
|
|
1107
|
+
through `npx -y @uehaj/sys1grep@VERSION`, not the `sys1grep` on your PATH. Only the API key has to be set (see [Install](#install)).
|
|
725
1108
|
|
|
726
1109
|
Then, inside Claude Code:
|
|
727
1110
|
|
|
@@ -737,22 +1120,27 @@ Claude Code installs plugins, not single skills. If you want just this one skill
|
|
|
737
1120
|
npx skills add uehaj/uehaj-marketplace --skill sys1grep -a claude-code -g
|
|
738
1121
|
```
|
|
739
1122
|
|
|
740
|
-
The skill writes the meaning in English, picks `-e` / `-a` / `-v` for
|
|
1123
|
+
The skill writes the meaning in English, picks `-e` / `-a` / `-v` for OR / AND / NOT, adds `-n`, narrows large
|
|
741
1124
|
directories to files worth paying for, and re-runs with `--level loose` or `strict` when the first result looks off.
|
|
742
|
-
The API key and endpoint are read the same way as on the command line (`SYS1GREP_API_KEY`, `~/.config/sys1grep/.env`).
|
|
1125
|
+
The API key and endpoint are read the same way as on the command line (`SYS1GREP_API_KEY`, `~/.config/sys1grep/settings.json`, `~/.config/sys1grep/.env`).
|
|
743
1126
|
|
|
744
1127
|
## Usage
|
|
745
1128
|
|
|
746
1129
|
```
|
|
747
|
-
usage: sys1grep [OPTION]... -e MEANING|-Q QUESTION [-a MEANING] [-v MEANING]... [FILE...]
|
|
748
|
-
|
|
749
|
-
|
|
1130
|
+
usage: sys1grep [OPTION]... -e MEANING|-Q QUESTION|-P /RE/FLAGS [-a MEANING] [-v MEANING]... [FILE...]
|
|
1131
|
+
sys1grep [OPTION]... -e START1 [-e START2]... --step-to END1 [-e END2]... [FILE...]
|
|
1132
|
+
|
|
1133
|
+
-e MEANING lines matching this meaning (several -e are OR'd); MEANING is always sent to Jev, never
|
|
1134
|
+
read as a regex — a /RE/flags-shaped value is refused, with a hint naming -P. A meaning
|
|
1135
|
+
that really starts and ends with / gets through with a leading space (-e ' /cat/')
|
|
1136
|
+
-P '/RE/FLAGS' a regex term, OR'd like -e; matched locally, no request; -a/-v after it AND/AND NOT
|
|
1137
|
+
onto it, same as after -e; -P '!/RE/' is not allowed, negate with -v '/RE/'
|
|
750
1138
|
-Q, --question QUESTION lines that answer QUESTION, not lines asking it; the same as
|
|
751
1139
|
-e "the line answers: QUESTION" (see "Lines that answer a question" above)
|
|
752
|
-
-a MEANING AND onto the preceding -e/-Q term. -e A -a B -e C = (A and B) or C
|
|
753
|
-
-v MEANING AND NOT onto the preceding -e/-Q term. -e A -v B = A and not B
|
|
1140
|
+
-a MEANING AND onto the preceding -e/-Q/-P term. -e A -a B -e C = (A and B) or C
|
|
1141
|
+
-v MEANING AND NOT onto the preceding -e/-Q/-P term. -e A -v B = A and not B
|
|
754
1142
|
At the front it is a bare negation. -v B = not B (like grep -v)
|
|
755
|
-
!MEANING a leading ! negates just that meaning, in -e / -Q / -a / -v alike
|
|
1143
|
+
!MEANING a leading ! negates just that meaning, in -e / -Q / -a / -v alike (not -P)
|
|
756
1144
|
-e A -e '!B' = A or not B. -a '!C' is the same as -v C
|
|
757
1145
|
--level=LEVEL strictness preset, sets both thresholds (default normal)
|
|
758
1146
|
loose : -t 0.3 -T 0.7 catch more, accept some noise
|
|
@@ -773,6 +1161,8 @@ usage: sys1grep [OPTION]... -e MEANING|-Q QUESTION [-a MEANING] [-v MEANING]...
|
|
|
773
1161
|
or date-time, today, this-week or this-month
|
|
774
1162
|
--no-auto-scope do not narrow those files by what a meaning says about them (see Scope from the meaning);
|
|
775
1163
|
--auto-scope turns it back on
|
|
1164
|
+
-g, --gitlog search git log's commits, one record each, instead of files; FILE arguments are pathspecs
|
|
1165
|
+
(see "Scope from the meaning"). Not with -r or git sys1grep
|
|
776
1166
|
-l print only the names of files with a match, not the lines
|
|
777
1167
|
-H, --with-filename prefix file names even for a single file; --no-filename never prefixes them
|
|
778
1168
|
-A NUM print NUM lines of trailing context after each match (context lines use - as separator)
|
|
@@ -780,24 +1170,31 @@ usage: sys1grep [OPTION]... -e MEANING|-Q QUESTION [-a MEANING] [-v MEANING]...
|
|
|
780
1170
|
-C NUM print NUM lines of context before and after (-A NUM -B NUM)
|
|
781
1171
|
-c print only a count of matching lines per file (like grep -c)
|
|
782
1172
|
-q, --quiet print nothing, stop at the first match; exit 0 on a match, even after an error (like grep -q)
|
|
783
|
-
--chunk=LINES lines per request (default 30
|
|
1173
|
+
--chunk=LINES lines per request (default 30; a request also carries at most 64 questions, so fewer lines
|
|
1174
|
+
with 3 or more meanings)
|
|
784
1175
|
Lines in one request are each other's context, so a small chunk changes verdicts
|
|
785
1176
|
on ambiguous lines, not just speed
|
|
786
|
-
-j N concurrent requests (default 8)
|
|
1177
|
+
-j N concurrent requests (default 8; 1 for a local URL: localhost, 127.0.0.1, [::1])
|
|
787
1178
|
-n print line numbers
|
|
788
1179
|
--unit=UNIT the unit of judgement: line (default), zero, sentence-by-jev, sentence-by-rule or function
|
|
789
1180
|
-z, --null-data --unit=zero: judge NUL-terminated records instead of lines, and print them NUL-terminated (see "Records that span several lines" above);
|
|
790
1181
|
with --unit=sentence-by-*, each record is split into sentences
|
|
791
1182
|
--unit=sentence-by-jev|sentence-by-rule judge each sentence instead of each line (see "One sentence at a time" above)
|
|
792
1183
|
--unit=function judge each function and print its lines (see "One function at a time" above)
|
|
793
|
-
|
|
1184
|
+
--step-to EXPRESSION print the paths from the functions the expression before it matches, along the
|
|
1185
|
+
calls, to the functions the expression after it matches (see "From one function to another" above)
|
|
1186
|
+
--step-reverse-to EXPRESSION --step-to walking the edges backwards, callee to caller
|
|
1187
|
+
--edges=FILE walk the edges in FILE instead of the calls
|
|
1188
|
+
--hops=N|M..N|M.. the hops an end may be at (default 0..)
|
|
1189
|
+
-o print only each regex match, one per line, as grep -o (a line only meanings matched prints
|
|
1190
|
+
whole); with --unit=sentence-by-*, only the matching sentences
|
|
794
1191
|
-p print each meaning's probability at the end of the line
|
|
795
1192
|
--dry-run send nothing; print the settings the search would run with (and their source, when not the
|
|
796
1193
|
command line), each file searched and each request with its questions
|
|
797
1194
|
--verbose print the same to stderr while searching
|
|
798
1195
|
-i, --interactive show what --dry-run would send, and search only after y on the terminal
|
|
799
1196
|
-M NUM, --max-columns=NUM send at most the first NUM characters of a line (or record, with -z); still
|
|
800
|
-
searched and judged, just not past that cutoff (default 2000, 8000 with -z)
|
|
1197
|
+
searched and judged, just not past that cutoff (default 2000, 8000 with -z or --unit=function)
|
|
801
1198
|
--max-filesize=SIZE size every target first (K/M/G, default 10M); over it, skip it outright, like rg's own
|
|
802
1199
|
--max-filesize (-y does not affect it). stdin is sized once read, and skipped the same way if
|
|
803
1200
|
it is over; a --cached / <tree>: target is a blob, sized from its content; -g's commits stay
|
|
@@ -805,24 +1202,51 @@ usage: sys1grep [OPTION]... -e MEANING|-Q QUESTION [-a MEANING] [-v MEANING]...
|
|
|
805
1202
|
--max-cost=USD price the input about to be sent; over it (default 1), ask to continue on the terminal; -y
|
|
806
1203
|
answers yes without asking; no terminal and the limit exceeded is exit 2, unchanged by -q
|
|
807
1204
|
(scripts pass -y); -i already asks unconditionally and earlier, so this does not ask again
|
|
808
|
-
--
|
|
1205
|
+
-y, --yes answer that question yes without asking
|
|
1206
|
+
--dedup[=auto|always|never] judge one line per template and reuse its answer for the rest (default never,
|
|
1207
|
+
for now; see "One line per template" above)
|
|
1208
|
+
--cache[=on|off|only] answer a request sent before from ~/.config/sys1grep/cache, free; only: never send,
|
|
1209
|
+
a request not cached is exit 2 unless -q already found a match (default off,
|
|
1210
|
+
SYS1GREP_CACHE; see "Sending less");
|
|
1211
|
+
--cache=clear (alone) empties it
|
|
1212
|
+
--rank[=jev|match] print the results (a match with its context) best first, under numbered headers; jev
|
|
1213
|
+
(bare --rank) asks Jev of each result, match sorts by its best match probability (see "Best first"); --no-rank: file order
|
|
1214
|
+
--summarize[=TOOL] pipe what would print to TOOL (claude by default, or SYS1GREP_SUMMARIZER) and print its
|
|
1215
|
+
summary instead; --summarize=ask prints the lines, then asks on the terminal; --no-summarize
|
|
1216
|
+
turns it off (see "A summary instead of the lines")
|
|
1217
|
+
--summarize-prompt=TEXT your own instruction, added after the fixed one; needs --summarize
|
|
1218
|
+
--format=FORMAT plain (default) / markdown / html: --rank's output, or asked of --summarize's TOOL
|
|
1219
|
+
--serve[=PORT] serve a search page on 127.0.0.1 instead of searching; --suggest=Q:TEXT / e:TEXT adds
|
|
1220
|
+
suggested-question buttons (see "A search page (`--serve`)")
|
|
1221
|
+
--template=NAME the document --rank or --summarize --format=html writes: ~/.config/sys1grep/templates/NAME.html, else the
|
|
1222
|
+
bundled one (default, print, search, terminal), or a file; default SYS1GREP_TEMPLATE, else default.
|
|
1223
|
+
--template=list prints the names (see "HTML templates")
|
|
1224
|
+
--install-templates copy the bundled templates to ~/.config/sys1grep/templates, keeping existing files
|
|
809
1225
|
--color[=WHEN] auto (default: color when stdout is a terminal) / always / never; bare --color means auto
|
|
810
1226
|
file and line number use grep's colors; with -p, probabilities are
|
|
811
1227
|
green at or above the positive threshold, red below the negative one,
|
|
812
1228
|
yellow in between. NO_COLOR is honored
|
|
813
1229
|
--sys1-model=ID, --sys1-url=URL, --sys1-api-key=KEY
|
|
814
1230
|
the API settings, overriding SYS1GREP_MODEL, SYS1GREP_URL, SYS1GREP_API_KEY
|
|
815
|
-
|
|
1231
|
+
--lang=en|ja this help's language, and --serve's page default; overrides SYS1GREP_LANG
|
|
1232
|
+
-h, --help this help (--lang / SYS1GREP_LANG when set, else Japanese when the first set of LC_ALL /
|
|
1233
|
+
LC_MESSAGES / LANG starts with ja)
|
|
816
1234
|
-V, --version print the version and exit
|
|
817
1235
|
```
|
|
818
1236
|
|
|
819
1237
|
Without FILE, stdin is read. With several files, output is prefixed with `file:`.
|
|
820
1238
|
Exit codes follow grep: 0 matched, 1 no match, 2 error (bad arguments, unreadable file, API failure).
|
|
1239
|
+
Exit 1 after lines were sent prints one line on stderr: the highest probability, its line, the option that
|
|
1240
|
+
loosens the threshold (unless even that would not match), and `-p -t 0` (`-T 1` too with a negation) to see every
|
|
1241
|
+
probability (`sys1grep: no line reached 0.5 for "…"; the highest was 0.42 (app.log:118). ...`).
|
|
1242
|
+
Not with `-q`, `--step-to` or `--step-reverse-to`, nor when only a negated meaning kept lines out.
|
|
821
1243
|
|
|
822
1244
|
### Expression grammar
|
|
823
1245
|
|
|
824
|
-
`-e`
|
|
825
|
-
|
|
1246
|
+
`-e` and `-P` each start an OR term, the first always a meaning, the second always a regex. `-a` and
|
|
1247
|
+
`-v` extend the previous term with AND and AND NOT.
|
|
1248
|
+
A leading `!` on a meaning negates just that meaning (quote it, `!` is history expansion in most shells);
|
|
1249
|
+
a leading `!` on a `-P` value is refused instead — negate a regex term with `-v '/RE/'`.
|
|
826
1250
|
|
|
827
1251
|
| command | means |
|
|
828
1252
|
|---|---|
|
|
@@ -835,10 +1259,10 @@ A leading `!` on a meaning negates just that meaning (quote it, `!` is history e
|
|
|
835
1259
|
|
|
836
1260
|
## How it works
|
|
837
1261
|
|
|
838
|
-
1. Non-blank lines are cut into chunks of 30 lines (with a character cap).
|
|
1262
|
+
1. Non-blank lines are cut into chunks of 30 lines (with a character cap, and at most 64 questions a request).
|
|
839
1263
|
2. Each chunk goes into `state` as an object `{"L000": "line 1", "L001": "line 2", ...}`,
|
|
840
1264
|
and one `noul` (yes/no probability) question per line × meaning goes into the same request.
|
|
841
|
-
3. Up to 8 requests run concurrently. Output is printed in file order.
|
|
1265
|
+
3. Up to 8 requests run concurrently (`-j`; 1 for a local URL). Output is printed in file order.
|
|
842
1266
|
4. Per line, each meaning's probability is thresholded to a boolean and the AND / OR / NOT expression is evaluated.
|
|
843
1267
|
|
|
844
1268
|
Lines sent together are each other's context: Jev judges a line against what the rest of the chunk shows
|
|
@@ -863,19 +1287,24 @@ the runner evaluates the boolean expression on those verdicts and compares with
|
|
|
863
1287
|
reporting precision and recall. It also sweeps `-t` × `-T` in 0.05 steps and reports the best pair.
|
|
864
1288
|
|
|
865
1289
|
```sh
|
|
866
|
-
|
|
1290
|
+
npm run judge [-- --model sonnet] [--rejudge] # runs tests/judge.mts; .mts needs Node 23.6+
|
|
867
1291
|
```
|
|
868
1292
|
|
|
869
1293
|
Judge verdicts are cached in `tests/verdicts.json`; later runs do not call the judge.
|
|
870
1294
|
The result is written to `tests/report.md`. Latest: precision 0.94, recall 0.98.
|
|
871
1295
|
|
|
1296
|
+
The judge is one of several test entry points. `npm run test:offline` checks sys1grep itself against a fake Jev
|
|
1297
|
+
(`tests/fake-jev.mjs`): no key, no network, the same result every run (`OFFLINE_ONLY=PATTERN` runs only the matching
|
|
1298
|
+
sections). `npm test` adds `tests/check.sh` against real Jev. [CLAUDE.md](CLAUDE.md) lists every entry point with what it
|
|
1299
|
+
checks and what it needs.
|
|
1300
|
+
|
|
872
1301
|
## Limits
|
|
873
1302
|
|
|
874
|
-
- Every searched line is sent to api.typesafe.ai. Do not run it over files you would not upload there.
|
|
1303
|
+
- Every searched line is sent to api.typesafe.ai (or wherever `SYS1GREP_URL` points). Do not run it over files you would not upload there.
|
|
875
1304
|
- Blank lines are not sent; they count as probability 0 for every meaning, so `-v X` prints them and `-e X` never does.
|
|
876
|
-
- Lines are truncated to 2,000 characters before sending.
|
|
877
|
-
- The maximum number of questions per request is undocumented; 420 worked.
|
|
878
|
-
- 429
|
|
1305
|
+
- Lines are truncated to 2,000 characters before sending (8,000 with `-z` or `--unit=function`; `-M` sets it).
|
|
1306
|
+
- The maximum number of questions per request is undocumented; 420 worked. sys1grep itself sends at most 64 a request (Clef's cap); only a single unit asked more than 64 meanings goes out whole.
|
|
1307
|
+
- 429, 529 and other 5xx answers, connection errors and timeouts are retried up to 6 times with exponential backoff.
|
|
879
1308
|
- Accuracy is best in English. Japanese works but is noisier.
|
|
880
1309
|
|
|
881
1310
|
## FAQ
|