jev-grep 0.1.0__tar.gz

jev_grep-0.1.0/.github/workflows/publish.yml

@@ -0,0 +1,22 @@
+name: publish
+# Push a tag like v0.1.0 to release. PyPI trusts this workflow directly (Trusted Publishing),
+# so there is no API token to store, leak or rotate.
+on:
+  push:
+    tags: ["v*"]
+jobs:
+  pypi:
+    runs-on: ubuntu-latest
+    environment: pypi
+    permissions:
+      id-token: write
+      contents: read
+    steps:
+      - uses: actions/checkout@v4
+      - uses: astral-sh/setup-uv@v5
+      - name: The tag must match the version in pyproject.toml
+        run: test "v$(grep -m1 '^version' pyproject.toml | cut -d'"' -f2)" = "$GITHUB_REF_NAME"
+      - run: uv sync
+      - run: uv run pytest -q
+      - run: uv build
+      - run: uv publish

jev_grep-0.1.0/.github/workflows/test.yml

@@ -0,0 +1,13 @@
+name: test
+on: [push, pull_request]
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        python: ["3.10", "3.13"]
+    steps:
+      - uses: actions/checkout@v4
+      - uses: astral-sh/setup-uv@v5
+      - run: uv sync --python ${{ matrix.python }}
+      - run: uv run pytest -q

jev_grep-0.1.0/.gitignore

@@ -0,0 +1,7 @@
+.venv/
+__pycache__/
+.pytest_cache/
+dist/
+bench/out/
+.env
+*.key

jev_grep-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Khaled Eltokhy
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

jev_grep-0.1.0/PKG-INFO

@@ -0,0 +1,185 @@
+Metadata-Version: 2.5
+Name: jev-grep
+Version: 0.1.0
+Summary: grep by meaning: filter lines with a plain-English description, judged by TypeSafe's Jev model
+Project-URL: Repository, https://github.com/keltokhy/jgrep
+Author: Khaled Eltokhy
+License-Expression: MIT
+License-File: LICENSE
+Keywords: cli,grep,jev,openrouter,semantic search,typesafe
+Classifier: Environment :: Console
+Classifier: Intended Audience :: Developers
+Classifier: Topic :: Text Processing :: Filters
+Classifier: Topic :: Utilities
+Requires-Python: >=3.10
+Requires-Dist: httpx>=0.27
+Description-Content-Type: text/markdown
+
+# jgrep
+
+grep, but the pattern is a description.
+
+```console
+$ tail -f app.log | jgrep "a user is getting frustrated"
+user 12: this is the third time checkout has failed, I am done with this app
+user 77: WHY does it log me out every five minutes??
+
+$ jgrep -o "announces or releases a new AI model" titles.txt | sort -rn | head -3
+0.980	PrismML Launches Bonsai 2 27B, Its Most Capable Model Yet
+0.970	Alibaba Releases Qwen3.8-Omni-Flash
+0.940	Google announces new experimental "CC" AI agent for families
+```
+
+Each line becomes one yes/no question to [Jev](https://docs.typesafe.ai), TypeSafe's decision
+model. Jev does not generate text. It returns a probability in about 200 ms for about a
+thousandth of a cent, which is fast and cheap enough to sit in a pipe. jgrep reads lines as
+they arrive, judges them concurrently and prints matches in input order, so it works on
+`tail -f` as well as on files.
+
+Measured on 994 Hacker News titles: 4.6 seconds and $0.012 for one description, and the same
+time for three descriptions at once.
+
+## Install
+
+```bash
+uv tool install git+https://github.com/keltokhy/jgrep
+```
+
+jgrep needs a key for one of two APIs. With keys for both, it uses TypeSafe's.
+
+| API | Key | Get one |
+|---|---|---|
+| TypeSafe | `TYPESAFE_API_KEY` | [console.typesafe.ai](https://console.typesafe.ai/settings/keys) |
+| OpenRouter | `OPENROUTER_API_KEY` | [openrouter.ai/keys](https://openrouter.ai/keys) |
+
+Set the environment variable, or put the key in `~/.config/jev/typesafe.key` or
+`~/.config/jev/openrouter.key`. Force a choice with `--api` or `JEV_API`.
+
+## Use
+
+```bash
+jgrep "a complaint about noise" complaints.txt          # lines that fit
+jgrep -v "spam" inbox.txt                               # lines that do not
+jgrep -c "asks a question" *.txt                        # counts per file
+jgrep -p 0.9 "mentions a specific dollar amount" f.txt  # only confident matches
+jgrep -o -p 0 "the writer is losing sleep" f.txt | sort -rn   # rank every line
+jgrep -e "about economics" -e "about New York" f.txt    # either; add --all for both
+jgrep --para "describes an identification strategy" paper.txt
+jgrep --whole "uses a bunching estimator" abstracts/*.txt     # prints matching file names
+jgrep -q "a stack trace" build.log && notify "build broke"
+```
+
+| Option | Meaning |
+|---|---|
+| `-p P` | Match when the probability is at least P. Default 0.5. |
+| `-o` | Put the probability in a first, tab-separated column. |
+| `-v`, `-c`, `-n`, `-H`, `-m NUM`, `-q` | As in grep. |
+| `-e DESC` | Another description. All of them go in one call per line. A line matches if any fits, or all with `--all`. |
+| `--para`, `--whole` | Judge paragraphs or whole files in place of lines. |
+| `--json` | One JSON object per match, with the probability. |
+| `--unordered` | Print matches as answers arrive. |
+| `-j N` | Calls in flight. Default 32. |
+| `--budget DOLLARS` | Stop once this much is spent. Default 1.00, or `$JGREP_BUDGET`; 0 for no limit. |
+| `--timeout SECONDS` | Give up on a line after this long, retries included. Default 15. |
+| `--no-cache`, `--api`, `--model`, `--stats` | See `jgrep --help`. |
+
+Exit status follows grep: 0 if anything matched, 1 if nothing did, 2 on error.
+
+## Cost
+
+A call bills roughly 270 tokens of fixed overhead plus the line and the description, so a
+typical line costs about 300 tokens, or $0.0000126 at $0.042 per million. A million lines is
+about $13. Blank lines, repeated lines and anything answered before are free: answers are
+cached in `~/.cache/jev/answers.sqlite`, keyed on the exact model, line and description.
+Extra `-e` descriptions add about 27 tokens each and no time.
+
+jgrep stops at `--budget`, one dollar by default, so a stray `jgrep pattern huge.log` cannot
+run up a bill. A dollar is about 80,000 lines. A stopped run loses nothing: rerun with a higher
+budget and everything already judged comes from the cache. For a long-lived `tail -f` monitor,
+set your own default once with `export JGREP_BUDGET=20`, or `0` for no limit. With `--stats`, or whenever stderr is a terminal, it prints what the run cost:
+
+```
+jgrep: 994 records, 33 matched; 994 calls, 0 cached; 292,839 tokens; $0.0123; 4.6s
+```
+
+## How well does it work
+
+Three benchmarks on public labeled text, run on 2026-09-18 with Jev 1.13 through OpenRouter.
+Each one runs the installed `jgrep` command itself, uncached, at its default threshold of 0.5.
+Reproduce them with `bench/accuracy.py`.
+
+**Against a keyword grep.** The UCI SMS Spam Collection: 5,574 text messages, 747 of them spam.
+
+| Filter | Precision | Recall | F1 | Time | Cost |
+|---|---:|---:|---:|---:|---:|
+| `jgrep "an unsolicited spam, scam or marketing text message"` | 0.87 | 0.95 | **0.91** | 27 s | $0.07 |
+| the same with `-p 0.9` | 0.98 | 0.84 | 0.90 | | |
+| `grep -iE "free\|win\|prize\|claim\|urgent\|cash\|txt\|call now\|..."` (17 terms) | 0.64 | 0.81 | 0.72 | 0.03 s | free |
+
+The regular expression was written before looking at any results and is in the script.
+
+**Against asking a chat model.** The do-it-yourself alternative is a loop that asks an LLM the
+same yes/no question about each line. On 300 of those messages, 32 requests in flight, all
+through OpenRouter:
+
+| Judge | F1 | Wall time | Cost | Median latency |
+|---|---:|---:|---:|---:|
+| **jgrep (Jev 1.13)** | 0.90 | **2.7 s** | $0.0039 | about 210 ms |
+| GPT Luna | 0.88 | 9.3 s | $0.0060 | 802 ms |
+| GPT Terra | 0.92 | 10.8 s | $0.0571 | 988 ms |
+| Qwen 3.7 Flash, thinking off | 0.78 | 8.4 s | $0.0007 | 789 ms |
+
+jgrep finished three to four times sooner than any of them. Its accuracy sits between the two
+GPT tiers; with 45 spam messages in the sample, those three F1 scores are within noise of each
+other. It is not the cheapest per line: a small open model costs a sixth as much and is
+clearly less accurate. Against the model that matched its accuracy, jgrep cost a fifteenth
+as much.
+
+**Several descriptions at once.** AG News test set, 7,600 articles, four descriptions
+(`-e "news about sports" -e "news about business, markets or the economy" ...`) judged in one
+call per article: 37 seconds and $0.13 for all four. Taking the most probable description as the
+label gives 86.6% accuracy with no training. One-vs-rest F1 at 0.5 was 0.97 for sports, 0.82 for
+science and technology, 0.82 for world affairs and 0.72 for business, which over-triggers
+(precision 0.58) because so much technology news is also business news.
+
+**Does the wording of a description matter?** `bench/phrasing.py` scores 30 hand-labeled lines
+against five descriptions of different grammatical shapes, including a negation and a question.
+Jev got all 150 right under each of four ways of wording the question; that set is easy on
+purpose. Asking five descriptions in one call changed no decision and moved probabilities by
+0.001 on average. Latency was flat at about 210 ms from 1 to 64 questions per call.
+
+On borderline lines the probabilities land in between, which is what `-p` is for:
+
+```
+0.65  [a complaint about noise]  The music from the church on Sunday mornings is lovely but it does start early.
+0.46  [does not mention a landlord]  The owner of the building never answers the phone.
+```
+
+Things to know:
+
+- These are a model's judgments. Check a sample before you rely on a filter.
+- Jev answers the description you wrote, not the one you meant. TypeSafe
+  [documents](https://docs.typesafe.ai/model-jaggedness/jev-1.13) weak spots: counting,
+  comparing numbers or dates, double negatives, and long inputs full of irrelevant detail.
+- Each line is judged alone. jgrep does not show Jev the lines around it.
+- Jev is close to deterministic, not exactly so. Asking 150 questions three times without the
+  cache gave identical probabilities for 128; the rest moved by up to 0.03 and no decision
+  flipped. The cache makes reruns exact.
+- The default model ID is an alias for the latest Jev. For results that must reproduce, pin
+  one with `--model` (for example `typesafe/jev-1.13` on OpenRouter).
+- Text in the input can try to steer the answer. Do not use jgrep as a security boundary.
+
+## Development
+
+```bash
+uv sync && uv run pytest        # 23 tests against a fake API; no key, no network
+uv run python bench/phrasing.py # live; costs about a cent
+uv run python bench/accuracy.py prepare && uv run python bench/accuracy.py spam   # also: news, llm
+```
+
+`src/jgrep/core.py` is the client: two backends, retries inside a time budget, the cache,
+in-flight deduplication and the cost meter. It is shared verbatim with
+[jlink](https://github.com/keltokhy/jlink), which links records across datasets with the same
+model.
+
+MIT license.

jev_grep-0.1.0/README.md

@@ -0,0 +1,168 @@
+# jgrep
+
+grep, but the pattern is a description.
+
+```console
+$ tail -f app.log | jgrep "a user is getting frustrated"
+user 12: this is the third time checkout has failed, I am done with this app
+user 77: WHY does it log me out every five minutes??
+
+$ jgrep -o "announces or releases a new AI model" titles.txt | sort -rn | head -3
+0.980	PrismML Launches Bonsai 2 27B, Its Most Capable Model Yet
+0.970	Alibaba Releases Qwen3.8-Omni-Flash
+0.940	Google announces new experimental "CC" AI agent for families
+```
+
+Each line becomes one yes/no question to [Jev](https://docs.typesafe.ai), TypeSafe's decision
+model. Jev does not generate text. It returns a probability in about 200 ms for about a
+thousandth of a cent, which is fast and cheap enough to sit in a pipe. jgrep reads lines as
+they arrive, judges them concurrently and prints matches in input order, so it works on
+`tail -f` as well as on files.
+
+Measured on 994 Hacker News titles: 4.6 seconds and $0.012 for one description, and the same
+time for three descriptions at once.
+
+## Install
+
+```bash
+uv tool install git+https://github.com/keltokhy/jgrep
+```
+
+jgrep needs a key for one of two APIs. With keys for both, it uses TypeSafe's.
+
+| API | Key | Get one |
+|---|---|---|
+| TypeSafe | `TYPESAFE_API_KEY` | [console.typesafe.ai](https://console.typesafe.ai/settings/keys) |
+| OpenRouter | `OPENROUTER_API_KEY` | [openrouter.ai/keys](https://openrouter.ai/keys) |
+
+Set the environment variable, or put the key in `~/.config/jev/typesafe.key` or
+`~/.config/jev/openrouter.key`. Force a choice with `--api` or `JEV_API`.
+
+## Use
+
+```bash
+jgrep "a complaint about noise" complaints.txt          # lines that fit
+jgrep -v "spam" inbox.txt                               # lines that do not
+jgrep -c "asks a question" *.txt                        # counts per file
+jgrep -p 0.9 "mentions a specific dollar amount" f.txt  # only confident matches
+jgrep -o -p 0 "the writer is losing sleep" f.txt | sort -rn   # rank every line
+jgrep -e "about economics" -e "about New York" f.txt    # either; add --all for both
+jgrep --para "describes an identification strategy" paper.txt
+jgrep --whole "uses a bunching estimator" abstracts/*.txt     # prints matching file names
+jgrep -q "a stack trace" build.log && notify "build broke"
+```
+
+| Option | Meaning |
+|---|---|
+| `-p P` | Match when the probability is at least P. Default 0.5. |
+| `-o` | Put the probability in a first, tab-separated column. |
+| `-v`, `-c`, `-n`, `-H`, `-m NUM`, `-q` | As in grep. |
+| `-e DESC` | Another description. All of them go in one call per line. A line matches if any fits, or all with `--all`. |
+| `--para`, `--whole` | Judge paragraphs or whole files in place of lines. |
+| `--json` | One JSON object per match, with the probability. |
+| `--unordered` | Print matches as answers arrive. |
+| `-j N` | Calls in flight. Default 32. |
+| `--budget DOLLARS` | Stop once this much is spent. Default 1.00, or `$JGREP_BUDGET`; 0 for no limit. |
+| `--timeout SECONDS` | Give up on a line after this long, retries included. Default 15. |
+| `--no-cache`, `--api`, `--model`, `--stats` | See `jgrep --help`. |
+
+Exit status follows grep: 0 if anything matched, 1 if nothing did, 2 on error.
+
+## Cost
+
+A call bills roughly 270 tokens of fixed overhead plus the line and the description, so a
+typical line costs about 300 tokens, or $0.0000126 at $0.042 per million. A million lines is
+about $13. Blank lines, repeated lines and anything answered before are free: answers are
+cached in `~/.cache/jev/answers.sqlite`, keyed on the exact model, line and description.
+Extra `-e` descriptions add about 27 tokens each and no time.
+
+jgrep stops at `--budget`, one dollar by default, so a stray `jgrep pattern huge.log` cannot
+run up a bill. A dollar is about 80,000 lines. A stopped run loses nothing: rerun with a higher
+budget and everything already judged comes from the cache. For a long-lived `tail -f` monitor,
+set your own default once with `export JGREP_BUDGET=20`, or `0` for no limit. With `--stats`, or whenever stderr is a terminal, it prints what the run cost:
+
+```
+jgrep: 994 records, 33 matched; 994 calls, 0 cached; 292,839 tokens; $0.0123; 4.6s
+```
+
+## How well does it work
+
+Three benchmarks on public labeled text, run on 2026-09-18 with Jev 1.13 through OpenRouter.
+Each one runs the installed `jgrep` command itself, uncached, at its default threshold of 0.5.
+Reproduce them with `bench/accuracy.py`.
+
+**Against a keyword grep.** The UCI SMS Spam Collection: 5,574 text messages, 747 of them spam.
+
+| Filter | Precision | Recall | F1 | Time | Cost |
+|---|---:|---:|---:|---:|---:|
+| `jgrep "an unsolicited spam, scam or marketing text message"` | 0.87 | 0.95 | **0.91** | 27 s | $0.07 |
+| the same with `-p 0.9` | 0.98 | 0.84 | 0.90 | | |
+| `grep -iE "free\|win\|prize\|claim\|urgent\|cash\|txt\|call now\|..."` (17 terms) | 0.64 | 0.81 | 0.72 | 0.03 s | free |
+
+The regular expression was written before looking at any results and is in the script.
+
+**Against asking a chat model.** The do-it-yourself alternative is a loop that asks an LLM the
+same yes/no question about each line. On 300 of those messages, 32 requests in flight, all
+through OpenRouter:
+
+| Judge | F1 | Wall time | Cost | Median latency |
+|---|---:|---:|---:|---:|
+| **jgrep (Jev 1.13)** | 0.90 | **2.7 s** | $0.0039 | about 210 ms |
+| GPT Luna | 0.88 | 9.3 s | $0.0060 | 802 ms |
+| GPT Terra | 0.92 | 10.8 s | $0.0571 | 988 ms |
+| Qwen 3.7 Flash, thinking off | 0.78 | 8.4 s | $0.0007 | 789 ms |
+
+jgrep finished three to four times sooner than any of them. Its accuracy sits between the two
+GPT tiers; with 45 spam messages in the sample, those three F1 scores are within noise of each
+other. It is not the cheapest per line: a small open model costs a sixth as much and is
+clearly less accurate. Against the model that matched its accuracy, jgrep cost a fifteenth
+as much.
+
+**Several descriptions at once.** AG News test set, 7,600 articles, four descriptions
+(`-e "news about sports" -e "news about business, markets or the economy" ...`) judged in one
+call per article: 37 seconds and $0.13 for all four. Taking the most probable description as the
+label gives 86.6% accuracy with no training. One-vs-rest F1 at 0.5 was 0.97 for sports, 0.82 for
+science and technology, 0.82 for world affairs and 0.72 for business, which over-triggers
+(precision 0.58) because so much technology news is also business news.
+
+**Does the wording of a description matter?** `bench/phrasing.py` scores 30 hand-labeled lines
+against five descriptions of different grammatical shapes, including a negation and a question.
+Jev got all 150 right under each of four ways of wording the question; that set is easy on
+purpose. Asking five descriptions in one call changed no decision and moved probabilities by
+0.001 on average. Latency was flat at about 210 ms from 1 to 64 questions per call.
+
+On borderline lines the probabilities land in between, which is what `-p` is for:
+
+```
+0.65  [a complaint about noise]  The music from the church on Sunday mornings is lovely but it does start early.
+0.46  [does not mention a landlord]  The owner of the building never answers the phone.
+```
+
+Things to know:
+
+- These are a model's judgments. Check a sample before you rely on a filter.
+- Jev answers the description you wrote, not the one you meant. TypeSafe
+  [documents](https://docs.typesafe.ai/model-jaggedness/jev-1.13) weak spots: counting,
+  comparing numbers or dates, double negatives, and long inputs full of irrelevant detail.
+- Each line is judged alone. jgrep does not show Jev the lines around it.
+- Jev is close to deterministic, not exactly so. Asking 150 questions three times without the
+  cache gave identical probabilities for 128; the rest moved by up to 0.03 and no decision
+  flipped. The cache makes reruns exact.
+- The default model ID is an alias for the latest Jev. For results that must reproduce, pin
+  one with `--model` (for example `typesafe/jev-1.13` on OpenRouter).
+- Text in the input can try to steer the answer. Do not use jgrep as a security boundary.
+
+## Development
+
+```bash
+uv sync && uv run pytest        # 23 tests against a fake API; no key, no network
+uv run python bench/phrasing.py # live; costs about a cent
+uv run python bench/accuracy.py prepare && uv run python bench/accuracy.py spam   # also: news, llm
+```
+
+`src/jgrep/core.py` is the client: two backends, retries inside a time budget, the cache,
+in-flight deduplication and the cost meter. It is shared verbatim with
+[jlink](https://github.com/keltokhy/jlink), which links records across datasets with the same
+model.
+
+MIT license.

jev_grep-0.1.0/bench/accuracy.py

@@ -0,0 +1,193 @@
+"""How accurate, fast and cheap is jgrep on public labeled text, next to the obvious alternatives?
+
+    uv run python bench/accuracy.py prepare          # download the two datasets into bench/out/
+    uv run python bench/accuracy.py spam             # jgrep vs a keyword grep on SMS spam
+    uv run python bench/accuracy.py news             # four descriptions in one call on AG News
+    uv run python bench/accuracy.py llm [--n 300]    # jgrep vs chat LLMs asked the same yes/no question
+
+Everything runs the installed `jgrep` command itself, uncached, so the numbers are what a user gets.
+Live; the whole file costs well under a dollar. Results land in bench/out/accuracy-*.json.
+"""
+
+from __future__ import annotations
+
+import argparse
+import asyncio
+import csv
+import io
+import json
+import re
+import subprocess
+import sys
+import time
+import urllib.request
+import zipfile
+from pathlib import Path
+
+import httpx
+
+from jgrep.core import BACKENDS
+
+OUT = Path(__file__).parent / "out"
+SPAM_URL = "https://archive.ics.uci.edu/static/public/228/sms+spam+collection.zip"  # UCI, CC BY 4.0
+NEWS_URL = "https://raw.githubusercontent.com/mhjabreel/CharCnn_Keras/master/data/ag_news_csv/test.csv"
+SPAM_DESCRIPTION = "an unsolicited spam, scam or marketing text message"
+# A keyword filter of the kind people actually write. Fixed before looking at any results.
+SPAM_REGEX = r"free|win|won|prize|claim|urgent|cash|txt|text .* to|call now|reply|offer|guaranteed|£|\$|www\.|http"
+NEWS = {1: "news about world affairs, politics or conflict", 2: "news about sports",
+        3: "news about business, markets or the economy", 4: "news about science or technology"}
+
+
+def one_line(text: str) -> str:
+    return " ".join(text.split())
+
+
+def prepare() -> None:
+    OUT.mkdir(exist_ok=True)
+    raw = urllib.request.urlopen(SPAM_URL, timeout=60).read()
+    rows = zipfile.ZipFile(io.BytesIO(raw)).read("SMSSpamCollection").decode("utf-8").splitlines()
+    pairs = [r.split("\t", 1) for r in rows if "\t" in r]
+    (OUT / "spam.txt").write_text("".join(one_line(t) + "\n" for _, t in pairs))
+    (OUT / "spam.labels").write_text("".join(("1" if y == "spam" else "0") + "\n" for y, _ in pairs))
+    news = list(csv.reader(io.StringIO(urllib.request.urlopen(NEWS_URL, timeout=60).read().decode("utf-8"))))
+    (OUT / "news.txt").write_text("".join(one_line(f"{title}. {body}").replace("\\", " ") + "\n" for _, title, body in news))
+    (OUT / "news.labels").write_text("".join(y + "\n" for y, _, _ in news))
+    print(f"spam: {len(pairs):,} messages, {sum(y == 'spam' for y, _ in pairs):,} spam;  news: {len(news):,} articles")
+
+
+def load(name: str) -> tuple[list[str], list[int]]:
+    lines = (OUT / f"{name}.txt").read_text().splitlines()
+    labels = [int(x) for x in (OUT / f"{name}.labels").read_text().split()]
+    assert len(lines) == len(labels), "a text spans lines; rerun prepare"
+    return lines, labels
+
+
+def jgrep(descriptions: list[str], path: Path, *, jobs: int = 64) -> tuple[list[dict], dict]:
+    """Every line's probabilities from the real command, uncached, plus its own stats line."""
+    cmd = ["jgrep", "--json", "-p", "0", "-j", str(jobs), "--no-cache", "--stats", "--budget", "2"]
+    for d in descriptions:
+        cmd += ["-e", d]
+    t0 = time.perf_counter()
+    run = subprocess.run(cmd + [str(path)], capture_output=True, text=True)
+    seconds = time.perf_counter() - t0
+    rows = [json.loads(line) for line in run.stdout.splitlines()]
+    stats = re.search(r"([\d,]+) calls.*?([\d,]+) tokens; \$([\d.]+)", run.stderr)
+    if run.returncode == 2 or not stats:
+        sys.exit(f"jgrep failed: {run.stderr[-400:]}")
+    return rows, {"seconds": round(seconds, 1), "calls": int(stats[1].replace(",", "")),
+                  "tokens": int(stats[2].replace(",", "")), "dollars": float(stats[3])}
+
+
+def prf(predicted: list[bool], truth: list[bool]) -> dict:
+    tp = sum(p and t for p, t in zip(predicted, truth))
+    fp = sum(p and not t for p, t in zip(predicted, truth))
+    fn = sum(t and not p for p, t in zip(predicted, truth))
+    precision, recall = tp / max(tp + fp, 1), tp / max(tp + fn, 1)
+    return {"precision": round(precision, 4), "recall": round(recall, 4),
+            "f1": round(2 * precision * recall / max(precision + recall, 1e-9), 4),
+            "accuracy": round(sum(p == t for p, t in zip(predicted, truth)) / len(truth), 4)}
+
+
+def spam() -> None:
+    lines, labels = load("spam")
+    truth = [y == 1 for y in labels]
+    rows, stats = jgrep([SPAM_DESCRIPTION], OUT / "spam.txt")
+    p = {r["line"]: r["p"] for r in rows}
+    ps = [p.get(i + 1, 0.0) for i in range(len(lines))]
+    t0 = time.perf_counter()
+    keyword = [bool(re.search(SPAM_REGEX, line, re.I)) for line in lines]
+    result = {
+        "dataset": "UCI SMS Spam Collection", "lines": len(lines), "positives": sum(truth),
+        "description": SPAM_DESCRIPTION, "jgrep_at_0.5": prf([x >= 0.5 for x in ps], truth),
+        "jgrep_at_0.9": prf([x >= 0.9 for x in ps], truth), "jgrep_run": stats,
+        "keyword_grep": prf(keyword, truth) | {"regex": SPAM_REGEX, "seconds": round(time.perf_counter() - t0, 3)},
+    }
+    (OUT / "accuracy-spam.json").write_text(json.dumps(result, indent=2))
+    print(json.dumps(result, indent=2))
+
+
+def news() -> None:
+    lines, labels = load("news")
+    rows, stats = jgrep(list(NEWS.values()), OUT / "news.txt")
+    by_line = {r["line"]: r.get("ps", [r["p"]]) for r in rows}
+    ps = [by_line.get(i + 1, [0.0] * 4) for i in range(len(lines))]
+    per_class = {NEWS[c]: prf([x[c - 1] >= 0.5 for x in ps], [y == c for y in labels]) for c in NEWS}
+    top1 = [max(range(4), key=lambda j: x[j]) + 1 for x in ps]
+    result = {
+        "dataset": "AG News test set", "lines": len(lines), "descriptions": list(NEWS.values()),
+        "note": "four descriptions packed into one call per line; each scored one-vs-rest at 0.5",
+        "per_description": per_class, "macro_f1": round(sum(v["f1"] for v in per_class.values()) / 4, 4),
+        "top1_accuracy": round(sum(a == b for a, b in zip(top1, labels)) / len(labels), 4), "jgrep_run": stats,
+    }
+    (OUT / "accuracy-news.json").write_text(json.dumps(result, indent=2))
+    print(json.dumps(result, indent=2))
+
+
+async def chat(model: str, lines: list[str], key: str, jobs: int) -> tuple[list[bool | None], dict]:
+    """The do-it-yourself alternative: ask a chat model the same question, one line per request."""
+    sem, latencies, usage = asyncio.Semaphore(jobs), [], {"cost": 0.0, "prompt": 0, "completion": 0}
+    prompt = f'Does the text fit this description: "{SPAM_DESCRIPTION}"? Answer with one word, yes or no.'
+
+    async def one(client: httpx.AsyncClient, text: str) -> bool | None:
+        # Hybrid models that think by default (Qwen) spend the whole allowance thinking and never answer,
+        # so their thinking is switched off; the GPT models take the lowest effort they offer.
+        reasoning = {"enabled": False} if model.startswith("qwen/") else {"effort": "low", "exclude": True}
+        body = {"model": model, "temperature": 0, "max_tokens": 200, "usage": {"include": True},
+                "reasoning": reasoning,
+                "messages": [{"role": "system", "content": prompt}, {"role": "user", "content": text}]}
+        async with sem:
+            for attempt in range(4):
+                t0 = time.perf_counter()
+                r = await client.post("https://openrouter.ai/api/v1/chat/completions", json=body)
+                if r.status_code == 200 and "choices" in r.json():
+                    break
+                await asyncio.sleep(0.5 * 2 ** attempt)
+            else:
+                return None
+        latencies.append(time.perf_counter() - t0)
+        data = r.json()
+        u = data.get("usage") or {}
+        usage["cost"] += u.get("cost") or 0.0
+        usage["prompt"] += u.get("prompt_tokens") or 0
+        usage["completion"] += u.get("completion_tokens") or 0
+        answer = (data["choices"][0]["message"].get("content") or "").strip().lower()
+        return True if answer.startswith("yes") else False if answer.startswith("no") else None
+
+    t0 = time.perf_counter()
+    async with httpx.AsyncClient(headers={"Authorization": f"Bearer {key}"}, timeout=90) as client:
+        answers = await asyncio.gather(*(one(client, t) for t in lines))
+    latencies.sort()
+    return answers, {"seconds": round(time.perf_counter() - t0, 1), "dollars": round(usage["cost"], 5),
+                     "median_latency_ms": round(latencies[len(latencies) // 2] * 1000) if latencies else None,
+                     "prompt_tokens": usage["prompt"], "completion_tokens": usage["completion"],
+                     "unparseable": sum(a is None for a in answers)}
+
+
+def llm(n: int, models: list[str], jobs: int) -> None:
+    lines, labels = load("spam")
+    step = len(lines) // n
+    idx = list(range(0, step * n, step))  # an even spread through the file, fixed
+    sub_lines, truth = [lines[i] for i in idx], [labels[i] == 1 for i in idx]
+    sub = OUT / "spam-subset.txt"
+    sub.write_text("".join(t + "\n" for t in sub_lines))
+    rows, stats = jgrep([SPAM_DESCRIPTION], sub, jobs=jobs)
+    p = {r["line"]: r["p"] for r in rows}
+    result = {"dataset": f"UCI SMS Spam Collection, every {step}th message", "lines": n, "positives": sum(truth),
+              "concurrency": jobs, "jgrep": prf([p.get(i + 1, 0.0) >= 0.5 for i in range(n)], truth) | stats, "chat_models": {}}
+    key = BACKENDS["openrouter"].key()
+    for model in models:
+        answers, run = asyncio.run(chat(model, sub_lines, key, jobs))
+        result["chat_models"][model] = prf([a is True for a in answers], truth) | run
+        print(model, result["chat_models"][model], flush=True)
+    (OUT / "accuracy-llm.json").write_text(json.dumps(result, indent=2))
+    print(json.dumps(result, indent=2))
+
+
+if __name__ == "__main__":
+    ap = argparse.ArgumentParser()
+    ap.add_argument("what", choices=["prepare", "spam", "news", "llm"])
+    ap.add_argument("--n", type=int, default=300)
+    ap.add_argument("--jobs", type=int, default=32)
+    ap.add_argument("--models", nargs="+", default=["~openai/gpt-luna-latest", "qwen/qwen3.7-flash", "~openai/gpt-terra-latest"])
+    a = ap.parse_args()
+    {"prepare": prepare, "spam": spam, "news": news, "llm": lambda: llm(a.n, a.models, a.jobs)}[a.what]()

jev_grep-0.1.0/bench/corpus.py

@@ -0,0 +1,47 @@
+"""Thirty hand-labeled complaint lines and five descriptions of different grammatical shapes.
+
+Labels are in the order of DESCRIPTIONS. The lines were written to be clear-cut, with a few
+deliberate near-misses (brown tap water is not a hot-water complaint; "how much is the fine"
+names no amount).
+"""
+
+DESCRIPTIONS = [
+    "a complaint about noise",              # noun phrase
+    "the writer asks a question",           # statement
+    "mentions a specific dollar amount",    # verb phrase, number-adjacent
+    "does not mention a landlord",          # negation, a documented weak spot
+    "is this about heat or hot water?",     # question form
+]
+
+LINES = [
+    ("The bar downstairs blasts music until 4am every single night.", 1, 0, 0, 1, 0),
+    ("No heat in the apartment for three days and it is 20 degrees outside.", 0, 0, 0, 1, 1),
+    ("My landlord raised the rent by $400 with no notice.", 0, 0, 1, 0, 0),
+    ("Can someone tell me when the hot water will be back?", 0, 1, 0, 1, 1),
+    ("Construction next door starts jackhammering at 6am, is that even legal?", 1, 1, 0, 1, 0),
+    ("The landlord still has not fixed the boiler and we have no hot water.", 0, 0, 0, 0, 1),
+    ("I was charged a $75 late fee even though I paid on time.", 0, 0, 1, 1, 0),
+    ("There is a pothole on Atlantic Avenue near 4th that has been there for months.", 0, 0, 0, 1, 0),
+    ("Why does my landlord get to ignore the broken radiator all winter?", 0, 1, 0, 0, 1),
+    ("The upstairs neighbor's dog barks all day while they are at work.", 1, 0, 0, 1, 0),
+    ("Thank you to the sanitation crew for the quick pickup on our block.", 0, 0, 0, 1, 0),
+    ("Street light out at the corner of Dekalb and Adelphi.", 0, 0, 0, 1, 0),
+    ("The super said the repair would cost $1,200 and the landlord refuses to pay.", 0, 0, 1, 0, 0),
+    ("Car alarms going off for hours on my street, I cannot sleep.", 1, 0, 0, 1, 0),
+    ("Is there a number I can call about rats in the building?", 0, 1, 0, 1, 0),
+    ("Our radiators are stone cold and the kids are sleeping in coats.", 0, 0, 0, 1, 1),
+    ("The parking ticket was $115 for a sign that was covered by scaffolding.", 0, 0, 1, 1, 0),
+    ("Helicopters fly over the park every ten minutes and the noise is unbearable.", 1, 0, 0, 1, 0),
+    ("How much is the fine for a blocked fire hydrant?", 0, 1, 0, 1, 0),
+    ("The landlord's contractor drills through the wall at night and it is deafening.", 1, 0, 0, 0, 0),
+    ("Bus stop shelter glass is shattered on Flatbush Ave.", 0, 0, 0, 1, 0),
+    ("Water comes out brown from the kitchen tap.", 0, 0, 0, 1, 0),
+    ("They want $3,000 for a security deposit, is that allowed?", 0, 1, 1, 1, 0),
+    ("Ice cream truck jingle plays outside my window for an hour every evening.", 1, 0, 0, 1, 0),
+    ("The elevator has been out of service since Monday.", 0, 0, 0, 1, 0),
+    ("When will the landlord turn the heat on? It is November.", 0, 1, 0, 0, 1),
+    ("Garbage has not been collected on our street in over a week.", 0, 0, 0, 1, 0),
+    ("A tree branch fell on my car during the storm.", 0, 0, 0, 1, 0),
+    ("The shower only runs cold, no hot water since the weekend.", 0, 0, 0, 1, 1),
+    ("Rent is going up to $2,850 next month according to the letter from my landlord.", 0, 0, 1, 0, 0),
+]