@niroai/niro 0.16.0 → 0.18.0

package/docs/evaluate-what-niro-did.md

@@ -0,0 +1,271 @@
+# See and judge what Niro did for you
+
+**You can check Niro's help yourself, in four ways:** read the line it prints at the end of each task, look at the record of which tools your agent asked, add up the last 7 and 30 days with `niro impact`, and run the same task with and without Niro. This page says what each number means, how it is worked out, and which numbers are estimates.
+
+**An everyday comparison:** a taxi meter tells you the fare and the distance, and it is right about both. It does not know what the bus would have cost you. The end-of-task line is the same: its counts are measured, and its "instead of" figures are an estimate of the other route.
+
+## 1. The end-of-task line
+
+At the end of a task your agent calls a tool, `mark_task_complete`, and copies the line it gets back into its reply. It looks like this:
+
+```
+[NIRO-CLI] Listed the 4 module-detection paths that reach the changed premake reader · 2 calls instead of ~3 searches · ~1k tokens instead of ~5k · this session: ~39k tokens instead of ~96k
+```
+
+**Example from a real session on Niro's own code. Your numbers will depend on your code and your tasks.** It is copied exactly as the engine printed it. Of the 38 end-of-task lines with a token saving recorded on one developer's computer over two days, it ranks 29th by the size of that saving: higher than most, not the largest.
+<!-- source: the result of a real mark_task_complete call in a Claude Code session working on niro-local, 2026-11-10 03:53 UTC. Chosen from every [NIRO-CLI] line returned by mark_task_complete in this machine's Claude Code transcripts, 2026-11-10 00:19 to 2026-11-11 12:47 UTC (python3 -I receipts_mtc.py 4, scratchpad niro-cli-docs-methods-only/receipts-mtc.jsonl and ranked.txt): 85 lines, 38 of them with a task token pair. Ranked by the ratio of the two task token figures (here 5k / 1k = 5.0), it is 29th of 38 (76th percentile); counting the 47 lines with no token pair as no saving, it is 76th of 85 (89th percentile). Format: niro-local src/niro_local/mcp/receipt.py:9-16 (the three clauses, "[NIRO-CLI] <what it did> · <N calls instead of ~M searches> · <~X tokens instead of ~Yk>"), :204-208 ("this session: ...") -->
+
+| Part | What it is | Measured or estimated |
+|---|---|---|
+| "Listed the 4 module-detection paths ..." | Your agent's own sentence about what Niro found, cut to 80 characters | Written by the agent |
+| "2 calls" | How many Niro tool calls answered in this task | **Measured**, counted by the engine |
+| "instead of ~3 searches" | How many extra searches and file reads the agent thinks it would have needed | **Estimated by the agent**: it types this number itself |
+| "~1k tokens" | The size of every answer Niro returned in this task | **Measured characters**, turned into tokens at 2.09 characters a token |
+| "instead of ~5k" | What finding the same things by hand would have cost | **Estimated** by the engine, as below |
+| "this session: ..." | The same two token figures, added up over the session so far | Same as above; shown from the second priced task |
+<!-- source: niro-local src/niro_local/mcp/receipt.py:12-15 (measured vs estimated, "There is no path through this module that prints an estimate without a tilde"), :33 (CHARS_PER_TOKEN = 2.09), :35 (MAX_CLAIM_CHARS = 80); src/niro_local/mcp/server.py:370-373 (Turn: "Two numbers are MEASURED here ... how many calls actually answered, and how many characters those answers contained"), :2209-2210 (niro_did, without_niro_extra_tool_calls typed by the agent), :533 (session total only after more than one priced turn) -->
+
+**Where the 2.09 comes from.** It was measured against the model's own token counts, in Keycloak A/B transcripts, on results of more than 2,000 characters each. Over 69 Niro answers the median was 2.09 characters a token (middle half 1.83 to 2.25). Over 367 shell-tool results it was 2.10. So one rate fits both sides of the line.
+<!-- source: niro-local src/niro_local/mcp/receipt.py:27-33; scratchpad niro-local-receipt-tokens/HOW-MEASURED-AND-VERDICT.md (69 answers median 2.09, quartiles 1.83 to 2.25; 367 native results median 2.10) -->
+
+### How "instead of ~Y" is estimated
+
+The engine prices the by-hand route the way agents with only shell tools were measured to work on Keycloak: they search for the name they were asked about, then read about 40 lines around each place that matters, and almost never open a whole file. So for each task it counts:
+
+- one text search per name the question asked about, at one output line per place the name appears (capped at 250 lines);
+- one 40-line read for every place (`file:line`) an answer showed you; places in one file within 40 lines of each other count once;
+- one more search for things a name search cannot reach, such as the classes that run when code calls an interface.
+<!-- source: niro-local src/niro_local/mcp/by_hand.py:1-21 (the agreed method, Niraj 2026-11-09), :29-38 (READ_LINES = 40, median of 375 reads, quartiles 26/40/80; MAX_GREP_LINES = 250); scratchpad niro-local-receipt-recheck/NATIVE-CHOICES.md (sets c4 and c3, 99 runs) -->
+
+**Only what you were shown is priced.** A page of an answer that was cut off adds nothing, and a short `answer=` form is priced for what it showed.
+<!-- source: niro-local src/niro_local/mcp/by_hand.py:15-17; docs/usecases/answer-shapes.md rule 12 -->
+
+**Honest limits of the estimate:**
+- It is a model of a typical agent, not a replay of yours. An agent that reads whole files would have spent more; one that guessed the right file first would have spent less.
+- An earlier version priced every named file in full. Checked against what shell-only agents really read on Keycloak, it was far too high, so it was replaced by the method above.
+<!-- source: scratchpad niro-local-receipt-tokens/HOW-MEASURED-AND-VERDICT.md (set v17, whole-file pricing against the shell-only arm's real tokens) -->
+- How it is checked: the line prices only the work Niro answered, so it is compared, in A/B runs, with what the shell-only session of the same task read, less what the Niro session still read with shell tools. In our A/B runs so far, added up over the tasks that printed a token figure, the two came close; for one task they can differ by several times either way. Read a single line's "instead of" as a rough order of size. More in [tokens-speed-and-accuracy.md](tokens-speed-and-accuracy.md).
+<!-- source: scratchpad niro-cli-task-playbooks/AB-RESULTS.md "Receipt figures against measured tokens" (fix round 1); ab/receipt-adjusted.json (python3 -I ab/replay_receipts.py p1 p2) -->
+- The "~3 searches" figure is the agent's own guess, and nothing checks it. It can be compared with an A/B run's call counts: the shell-only session's calls, against the Niro session's Niro calls and the shell calls it still made.
+<!-- source: niro-local src/niro_local/mcp/server.py:2209-2210 (without_niro_extra_tool_calls typed by the agent); scratchpad niro-local-receipt-tokens/HOW-MEASURED-AND-VERDICT.md (calls replaced, worked from the two arms' call counts) -->
+
+### How the two token figures are made
+
+**What the answers cost.** The engine adds up the characters of every Niro answer in the task, divides by the characters-a-token rate above, and rounds the result for the line.
+
+**The by-hand estimate.** The engine looks only at the answers that showed you places (`file:line`). It prices one read of the lines around each place, counting nearby places in one file once, from each file's real size and line count. It adds one search for each name the question asked about, and one for what a name search cannot reach. It turns those characters into tokens at the same rate. An answer that showed no places, such as a count or a list of names, adds nothing.
+
+**The calls and the agent's guess.** The line shows how many calls answered. It shows the agent's guess of searches only when the guess is more than the calls made.
+<!-- source: niro-local src/niro_local/mcp/receipt.py:33 (CHARS_PER_TOKEN), :110-114 tokens_from_chars, :117-121 compact, :177-208 (calls clause only when searches_avoided > answered_calls); src/niro_local/mcp/by_hand.py:174-198 (reads, line_chars, read_chars) -->
+
+**How the searches figure is kept.** The agent's number is stored with the task in the engine's record of the session (`niro journal`). It is not added up: the session total in the line covers tokens only.
+<!-- source: niro-local src/niro_local/journal/store.py:114 (without_niro_extra_tool_calls column), :367-380 (Totals: answered_calls, tokens_returned, tokens_to_read_covered, priced_turns; no searches field); src/niro_local/mcp/server.py:464-498 (price adds tokens only) -->
+
+**How the session total is kept.** Each task adds its two token figures to the session's running totals, in memory. A place or a search already priced earlier in the session is not priced again, because whoever made that read by hand would still have it. The total is shown from the second priced task on.
+<!-- source: niro-local src/niro_local/mcp/server.py:374-376 (running totals in memory), :392-396 (priced places and searches), :464-498 (Turn.price: "leaves out reads and searches already priced this session"), :533 (session_priced_turns > 1) -->
+
+### When the line is shown, and when it is not
+
+| What you see | Why |
+|---|---|
+| The line | Niro answered at least one call in this task |
+| "N calls" with no "instead of" | The agent's guess was not more than the calls made, so no saving is claimed |
+| No token pair | The by-hand estimate was not more than what Niro returned, or the two round to the same figure |
+| No session total | It is the first priced task of the session, or the session grew past 20,000 files or 100,000 places, after which a total would be incomplete |
+| "Not consulted: this turn used file reads instead" | The graph was there, and the agent searched the files itself |
+| "Nothing to report: ..." | No Niro call answered in this task, so the agent should print nothing |
+| Nothing at all | The line is switched off (`niro mcp receipt --off`), or the agent did not call `mark_task_complete` |
+<!-- source: niro-local src/niro_local/mcp/receipt.py:177-208 (calls clause only when searches_avoided > answered_calls; token pair only when `comparison` shows a saving; `comparison` returns None when the compact figures are equal); src/niro_local/mcp/server.py:352-353 (MAX_SESSION_FILES 20,000, MAX_SESSION_PLACES 100,000), :523-524 (not_consulted), :2304-2313 ("Nothing to report"); docs/faq-feedback-and-receipts.md (niro mcp receipt --off) -->
+
+**In Claude Code, a forgotten line is asked for once.** A Stop hook (a small command Claude Code runs when the agent is about to stop) checks the reply. If Niro answered and the line is missing, it asks the agent, once, to finish. It never prints anything itself, and it lets the agent stop if anything goes wrong.
+<!-- source: src/mcp/lineCheck.js:1-26 -->
+
+**The line is not saved.** It is shown once, in your agent's reply. See [faq-feedback-and-receipts.md](faq-feedback-and-receipts.md).
+<!-- source: src/lib/receiptStore.js:1-15 -->
+
+## 2. The record of what your agent asked
+
+`niro journal` lists the sessions the engine recorded and not yet sent, with each tool call, whether it worked, and how long it took:
+
+```
+niro journal --limit 3
+
+9 session(s) waiting, 31 tool call(s) in total
+  adb7d35bb51e  5 call(s)  started 2026-11-10T19:45:35+00:00  ended 2026-11-10T19:45:53+00:00
+      get_impact_analysis          ok        122ms  {"http_path": "/api/shipments/{id}", ...
+      resolve_name                 ok         19ms  {"name": "ShippingMain.shipment"}
+      ...
+      find_api_endpoints           ok         17ms  {"limit": 20, "offset": 0, "path_pattern": "shipments"}
+      ...
+```
+<!-- source: `niro-local journal --limit 3` on the scratch home used for the A/B of 2026-11-10 (a snapshot taken during the run, trimmed to 3 of the session's 5 calls; sessions of the synthetic java-mono corpus); `niro journal` passes to the engine (src/commands/engineRoutes.js:36, "What the MCP server was asked, waiting to be sent") -->
+
+This is the place to see which tools your agent really uses, which fail, and which are slow. The record is sent to Niro (by the background watcher, or `niro journal-send`), and a sent session leaves this list. What it holds, and what it does not, is in [what-leaves-your-machine.md](what-leaves-your-machine.md).
+<!-- source: src/commands/journalSend.js:1-12 ("in normal use the watcher calls the same library"); src/lib/planUsage.js:118 (JOURNAL_SENTENCE, "to improve Niro and to show you its value") -->
+
+**The lines themselves are not kept, but their figures are.** Each task's counts, tokens and the agent's sentence stay on this computer after the session is sent, and `niro impact` adds them up (next section).
+<!-- source: niro-local src/niro_local/journal/store.py turns table (niro_did, niro_helpfulness, task_outcome, searches_avoided on the task's row, kept after the upload; at most 20,000 rows), Journal.mark_sent (deletes only sessions and calls) -->
+
+## 3. What Niro did for you over time: `niro impact`
+
+`niro impact` adds up the tasks on this computer for the last 7 and the last 30 days. It reads only the engine's record on this computer: nothing is sent, and no model is called.
+
+Each window prints one figure per line, then the projects, then the good moments:
+
+| Line | What it is | Measured or estimated |
+|---|---|---|
+| The record on this computer starts on ... | Shown only when the record starts after the window's first day, so "last 30 days" never quietly means fewer days. It sits directly under the window's heading | **Measured** |
+| Tasks that used Niro: "N of M" | Tasks with at least one Niro answer, of all the tasks in the window | **Measured** |
+| Number of calls Niro answered | The Niro answers in those tasks | **Measured** |
+| Tokens used: "~X tokens instead of ~Y (estimated)" | X is what the answers cost; Y is the by-hand estimate of section 1, added up over the tasks that had both | X **measured**, Y **estimated** |
+| Searches and file reads avoided: "~N" | The count the agents typed in `mark_task_complete`, added up. The kinds of call they named are kept in `--json` | **Estimated by the agents**; nothing checks them |
+| Look-ups avoided: "~N" | The searches and file reads the graph answers stood for, less the Niro calls: the size of the work. See below | **Counted** |
+| Round trips avoided: "at least ~M" | The searches your agent would have had to wait for one after another, plus reading the files, less the Niro calls: a floor on the waiting. See below | **Counted** |
+| Answers by kind | The answers, grouped by the kind of question: impact check, caller lookup, trace of what a method calls, route and handler lookup, event trace, database lookup, type or name lookup, exception trace, text search, source read | **Measured** |
+| Median answer time | The middle of the answer times recorded in the window | **Measured** |
+| By project | The tasks, answers and tokens per project, by the project's name | As above |
+| Good moments | The tasks with the best measured saving, tokens first and then calls, in the agents' own words, largest saving first: 3 for a window of up to 7 days, 10 for a longer one. A task rated not helpful or not used is never one | Chosen by numbers, never by the words |
+| Some figures cover only tasks recorded since ... | Shown at the end of a window when a row covers fewer of its tasks, because some were recorded by an earlier version. The row itself shows its number as it is | **Measured** |
+| Graph kept current | Per project, the files the graph holds and when it was last updated. Per window, the graph updates (full and incremental), the files they rebuilt, and the median update time | **Measured** |
+<!-- source: niro-local src/niro_local/impact.py (module note; moments_for; TOOL_KINDS, ANSWER_KINDS; avoided_by_kind; _render_window, _tokens_line, _fewer_tasks_since; graph_updates, graph_current, _render_updates), src/niro_local/journal/store.py Journal.impact_groups, impact_moments (ORDER BY token_saving DESC, calls_saving DESC; niro_helpfulness NOT IN not_helpful, not_used), answer_tools, median_answer_ms; src/niro_local/build/state.py build_history -->
+
+**An example.** This is `niro impact` exactly as it printed after five real tasks on two projects on one computer (four on Keycloak, one on a small Java project), unedited:
+
+```
+What Niro did for you, from the record on this computer. Nothing here is sent anywhere.
+
+Last 7 days (since 2026-11-04):
+  The record on this computer starts on 2026-11-11.
+  Tasks that used Niro: 5 of 5
+  Number of calls Niro answered: 7
+  Tokens used: ~12k tokens instead of ~60k (estimated)
+  Searches and file reads avoided: ~29
+  Look-ups avoided: ~238
+  Round trips avoided: at least ~5
+  Answers by kind: 3 caller lookups, 2 route and handler lookups, 1 impact check before a change, 1 text search.
+  Median answer time: 70 ms
+
+  By project:
+    keycloak: 4 task(s) with Niro, 5 answer(s), ~8k tokens instead of ~56k (estimated)
+    java-mono: 1 task(s) with Niro, 2 answer(s)
+
+  Good moments, in the agents' own words, largest saving first:
+    2026-11-11  keycloak: Counted the callers of the error helper across 45 files (~5k tokens instead of ~47k, 2 call(s) instead of ~10 searches)
+    2026-11-11  keycloak: Listed the 9 direct callers of the lookup, 2 of them in tests (~1k tokens instead of ~7k, 1 call(s) instead of ~6 searches)
+    2026-11-11  keycloak: Found where the token lifespan constant is read (1 call(s) instead of ~3 searches)
+
+Last 30 days (since 2026-10-12):
+  The record on this computer starts on 2026-11-11.
+  Tasks that used Niro: 5 of 5
+  Number of calls Niro answered: 7
+  Tokens used: ~12k tokens instead of ~60k (estimated)
+  Searches and file reads avoided: ~29
+  Look-ups avoided: ~238
+  Round trips avoided: at least ~5
+  Answers by kind: 3 caller lookups, 2 route and handler lookups, 1 impact check before a change, 1 text search.
+  Median answer time: 70 ms
+
+  By project:
+    keycloak: 4 task(s) with Niro, 5 answer(s), ~8k tokens instead of ~56k (estimated)
+    java-mono: 1 task(s) with Niro, 2 answer(s)
+
+  Good moments, in the agents' own words, largest saving first:
+    2026-11-11  keycloak: Counted the callers of the error helper across 45 files (~5k tokens instead of ~47k, 2 call(s) instead of ~10 searches)
+    2026-11-11  keycloak: Listed the 9 direct callers of the lookup, 2 of them in tests (~1k tokens instead of ~7k, 1 call(s) instead of ~6 searches)
+    2026-11-11  keycloak: Found where the token lifespan constant is read (1 call(s) instead of ~3 searches)
+    2026-11-11  java-mono: Listed the routes each service serves and which service calls which (2 call(s) instead of ~8 searches)
+
+Graph kept current, per project:
+  java-mono: 155 files indexed in 1 folder(s), last updated 2026-11-11 10:43 UTC.
+  keycloak: 9,099 files indexed in 1 folder(s), last updated 2026-11-10 19:27 UTC.
+  typescript-mono: 366 files indexed in 1 folder(s), last updated 2026-11-10 19:28 UTC.
+  Graph updates:
+    Last 7 days: 1 graph update (1 full, 0 incremental), 155 files rebuilt, median update time 3.6 s.
+    Last 30 days: 1 graph update (1 full, 0 incremental), 155 files rebuilt, median update time 3.6 s.
+    Updates are recorded from 2026-11-11.
+```
+<!-- source: a real run of niro-local a38ac9c6 (feat/impact-layout) over the keycloak and java-mono tasks, 2026-11-11; scratchpad niro-coverage-run/impact-work/real-impact-layout.txt -->
+
+**How "look-ups avoided" is counted: the size of the work.** For each Niro answer that walks the graph (a call chain, the callers of a method, a blast radius or impact check, a route, handler, event or database lookup), the engine counts one search for each method the answer expanded or found, and one file read for each different file it covered. Each method and each file counts once in a task: when a second answer covers a file the first already covered, the file is not counted again, and asking the same question twice counts once. It then takes away the Niro calls themselves, because each of those is a call too, and never counts below 0 for one task. A text search or a source read adds nothing, because by hand it would be the same single call. The count is the work the answers stand for, not a claim that your agent would have made every one of those calls: an agent working by hand often stops early.
+<!-- source: niro-local src/niro_local/mcp/round_trips.py (TOOLS; PerTask.add joins the receipt's covered files and counts each method and file once per task); src/niro_local/journal/store.py trips_avoided (MAX(0, methods_expanded + files_covered - graph_answers) per task, then summed); tests/test_round_trips.py -->
+
+**A page counts only what it shows,** in both rows. When an answer is cut by `limit`, is a later page asked with `offset` or `from_line`, or is too long for one reply and shown a window of lines at a time, the engine counts only the files the page shows and the methods whose names it prints, the same way the end-of-task line prices only what was shown. For example, the callers of a method with `limit: 5` count the files of those 5 callers, not the files of every caller. An answer shortened with `answer=count` is not a page: it reports on everything the full answer covers, so it counts all of it.
+<!-- source: niro-local src/niro_local/mcp/round_trips.py (cut, shown_part); src/niro_local/mcp/server.py answer (trips.paged or an offset: shown_part over the places _by_hand_route read off the page), _more_line and _fit (round_trips.cut); src/niro_local/mcp/shapes.py render (round_trips.cut when a listing shows fewer than it has); tests/test_round_trips.py (test_a_page_cut_by_limit_counts_only_the_callers_it_shows, test_a_window_of_lines_counts_only_what_the_window_shows, test_must_not_change_a_limit_above_the_list_or_a_count_counts_every_caller) -->
+
+**How "round trips avoided" is counted: a floor on the waiting.** A round trip is one call your agent makes and then waits for. Working by hand, an agent can put the names of one level into one search (all the callers of a method, say), but it cannot search the next level before that search comes back, because it does not know the names yet. So a walk three levels deep takes at least three searches one after another, then one more round to read the files. For each Niro answer that added at least one method or file new to the task, the engine counts the deepest level the answer walked, plus 1 for reading the files, minus 1 for the Niro call itself. An answer that adds nothing new to the task counts -1: the Niro call was a round trip of its own, and it stood in for none. A list of a method's direct callers counts 1; a blast radius three levels deep counts 3; asking the same question again in the task counts -1; a later page of a list already paged in the task counts -1 too, or only the levels deeper than any page before it, since by hand the names of one level go into one search; a text search or a source read counts 0. Never below 0 for one task. It says "at least" because an agent working by hand often needs more than one search for a level, so the real number of waits is this or more. In the example above, the 5 is 1 (a list of direct callers) + 2 (a blast radius three levels deep, 3, then the callers of the same method asked again in the task, -1) + 1 (a list of callers that found none) + 1 (a list of routes); the text search counts 0.
+<!-- source: niro-local src/niro_local/mcp/round_trips.py (levels_avoided: depth + READING_THE_FILES - THE_NIRO_CALL; Counted.depth; PerTask.add gives -THE_NIRO_CALL when nothing is new); src/niro_local/mcp/chains.py walk (level=spent + 1); src/niro_local/journal/store.py levels_avoided (MAX(0, levels_walked) per task, then summed); tests/test_round_trips.py (test_a_three_level_walk_counts_three_round_trips) -->
+
+**The agents' words are checked again before they are shown.** They were masked when they were stored; `niro impact` masks them again, and replaces anything that looks like a file path, a piece of code or a network address with `[path]`, `[code]` or `[address]`. Only the project's name is shown.
+<!-- source: niro-local src/niro_local/impact.py shown_words (mask_text, then redact/shapes.py hide_shapes: code spans, calls, statements, paths, IP addresses and host:port, dotted names, then render_claim); tests/test_impact.py test_a_secret_and_code_in_the_agents_words_are_not_printed -->
+
+**Options:** `--days N` shows one window of N days (1 to 365) instead of both; `--project <name>` shows one project; `--json` prints the same report as JSON.
+<!-- source: niro-local src/niro_local/cli.py _cmd_impact (1 to 365, upstream's ProjectImpactService MIN_DAYS and MAX_DAYS) -->
+
+**How far back it goes.** The record keeps the newest 20,000 tasks, the newest 100,000 answers and the newest 1,000 graph updates. When the record starts after a window's first day, the report says on which day it starts, directly under the window's heading. Tasks recorded by an earlier version count in the numbers, with no words. When the look-ups, round trips or answers-by-kind rows cover fewer of the window's tasks because of them, each row shows its number as it is, and one line at the end of the window says "Some figures cover only tasks recorded since" the day they start. A graph update is recorded only when a build changed the graph: a build that failed, was refused, or found nothing changed is not one.
+<!-- source: niro-local src/niro_local/journal/store.py MAX_KEPT_ROWS = 20_000, MAX_KEPT_ANSWERS = 100_000, _trim; src/niro_local/build/state.py MAX_BUILD_HISTORY = 1000; src/niro_local/build/builder.py Builder._record_history; impact.py _render_window ("The record on this computer starts on", only when its day is after the window's first day), _fewer_tasks_since ("Some figures cover only tasks recorded since"), _render_answers, _render_updates; src/niro_local/journal/store.py Journal.counts_recorded_from -->
+
+**On a team server,** this release shows only what was recorded on your own computer. Answers from a teammate's shared server are recorded on that server.
+<!-- source: niro-local feat/serve-http (not merged): mcp/connections.py, one journal session per connection in the server's journal.db -->
+
+**`niro ab`, an account-wide on and off test, is hosted Niro only.** See [hosted-only.md](hosted-only.md).
+<!-- source: src/index.js:478-480 ("`ab` switches the hosted MCP's A/B window"; left out when cloud is off); docs/reference-cli.md:86 -->
+
+## 4. Judge it yourself: a small A/B
+
+The most honest test is to run the same task twice, once with Niro and once without, and compare.
+
+### How
+
+1. **Pick 3 to 5 real tasks** from your own work, with an answer you can check: "list every caller of X", "which tests reach Y", "who writes table Z".
+2. **Write down the right answer first,** by hand, so neither run grades itself.
+3. **Run each task twice in a fresh session:** once as you normally work, and once with Niro switched off for that session. In Claude Code, start the second run with `claude --strict-mcp-config --mcp-config '{"mcpServers":{}}'`, which starts it with no MCP servers.
+4. **Keep everything else the same:** the same model, the same prompt, the same agent version.
+<!-- source: the same switches the measured A/B uses for its "without" arm (scratchpad niro-cli-task-playbooks/ab/bench.py session_cmd: --strict-mcp-config with an empty server list) -->
+
+### Which numbers to compare
+
+| Number | Where to read it | What it tells you |
+|---|---|---|
+| Correctness: items found, items invented | Your answer key | Whether the answer was complete and true. This matters most |
+| Cost, and tokens: cache writes, cache reads, output | Claude Code's `/cost`, or the `result` event of `claude -p --output-format stream-json` | What the run really cost |
+| Tool calls | The transcript | How much searching and reading it did |
+| Wall time | A clock, or the result event's `duration_ms` | How long you waited |
+<!-- source: the fields bench.py reads from the result event (total_cost_usd, modelUsage cacheCreationInputTokens / cacheReadInputTokens / outputTokens, duration_ms): scratchpad niro-cli-task-playbooks/ab/bench.py parse_stream -->
+
+Report all of them together. Cost alone hides why it moved: in long sessions, cache reads (the conversation re-read on every turn) are most of the tokens.
+<!-- source: docs/tokens-speed-and-accuracy.md "How accuracy pays off over a long session" (cache reads were most of the tokens in both arms) -->
+
+How we run ours, and what to keep in mind when reading the results, is in [tokens-speed-and-accuracy.md](tokens-speed-and-accuracy.md).
+
+### Check that an answer was complete
+
+A list of callers can be checked against a text search. On Keycloak, `ErrorResponse.error` has two overloads (two methods with the same name and different parameters), and a grep for the name finds the calls to both. So ask for both:
+
+```
+niro ask get_reverse_call_chain "method_fqn=org.keycloak.services.ErrorResponse.error(String, Response.Status)" answer=locations limit=300
+# 100 callers, in 45 files
+
+niro ask get_reverse_call_chain "method_fqn=org.keycloak.services.ErrorResponse.error(String, Object[], Response.Status)" answer=locations limit=300
+# 3 callers, in 3 files
+
+grep -rln --include=*.java "ErrorResponse.error(" .
+# 47 files
+```
+
+Keep the file part of each caller line, put the files of both answers in one sorted list, and compare it with grep's sorted list (`comm -3 a.txt b.txt` prints lines in only one of them). Here both lists had the same 47 files. A file only grep finds is a caller Niro missed, or a mention that is not a call; open it and see which.
+<!-- source: scratchpad niro-cli-task-playbooks/runs/eval-02-loc-all.txt (overload 1: 100 callers, 45 files) and runs/eval-04-ov2.txt (overload 2: 3 callers in ErrorResponse.java, ClientResource.java, ClientsResource.java); union 47 files against `grep -rln --include=*.java "ErrorResponse.error(" .` (47 files); `comm` both ways empty (Keycloak 968726fe66, 2026-11-10) -->
+
+**It does not always agree, and that is the point of checking.** For `JpaUserProvider.getUserByEmail`, Niro answered "0 callers" while a grep for `getUserByEmail(` found calls in many files. The callers use a child interface's name; asking `UserProvider.getUserByEmail` gives 17. That is a known problem, listed in [faq-wrong-or-missing-answers.md](faq-wrong-or-missing-answers.md).
+<!-- source: runs/chg-03-impl-full.txt (0 callers), runs/chg-07-iface-loc.txt (17); `grep -rn --include=*.java "getUserByEmail(" .` = 46 lines in 35 files -->
+
+**For a class or interface,** compare `get_class_structure`'s subtypes with `grep -rln "implements <Name>"`. A grep cannot see a class that gets the interface through its parent class, so Niro listing more is expected; Niro listing fewer is worth reporting.
+<!-- source: docs/playbook-add-a-feature.md (15 subtypes against 13 grep files, the 2 extra extend a parent class) -->
+
+## 5. Report a wrong or missing answer
+
+Send the question, the answer, and what you expected (with the grep that showed it) to help@niro.ai. The known gaps are in [faq-wrong-or-missing-answers.md](faq-wrong-or-missing-answers.md), and the feedback FAQ is [faq-feedback-and-receipts.md](faq-feedback-and-receipts.md).
+<!-- source: docs/faq-feedback-and-receipts.md ("Email help@niro.ai"); docs/faq-wrong-or-missing-answers.md ("How do I report a wrong answer?") -->
+
+## Related
+
+[tokens-speed-and-accuracy.md](tokens-speed-and-accuracy.md), [answer-shapes.md](answer-shapes.md).

package/docs/faq-assistants-and-mcp.md

@@ -0,0 +1,122 @@
+# FAQ: coding agents and MCP
+
+### Which coding agents does Niro support?
+Fourteen:
+
+- Claude Code
+- Codex
+- Claude Desktop
+- Cursor
+- Windsurf
+- VS Code (GitHub Copilot)
+- Gemini CLI
+- GitHub Copilot CLI
+- Factory (droid)
+- Goose
+- OpenCode
+- Cline
+- Kiro
+- Antigravity
+
+See [connect-your-ai-assistant.md](connect-your-ai-assistant.md).
+<!-- source: src/mcp/setup.js:279-414 -->
+
+### My agent is not in the list.
+Run `niro mcp install` and choose "Other (print the config)". Paste the block it prints into your agent's MCP settings.
+<!-- source: src/mcp/setup.js:658-667 -->
+
+### "Unknown client: X"
+The name after `--client` is not one Niro knows. `niro mcp install --list` shows the names.
+<!-- source: src/mcp/setup.js:683, :3020 -->
+
+### "Failed to register MCP server. Is the Codex CLI installed?" (or the same for Gemini, Copilot, droid or Claude Code)
+For these agents, Niro registers itself through the agent's own command. Install the agent's command first, then run `niro mcp install` again. The message gives the command to run by hand.
+<!-- source: src/mcp/setup.js:775-777, :2123, :2620, :2645, :2671 -->
+
+### "Failed to write config ... Manually add to <path>" or "No config path known for <client>."
+Niro could not write the agent's settings file. Add the block it prints by hand, at the path it names.
+<!-- source: src/mcp/setup.js:2799, :2867 -->
+
+### I installed, but my agent shows no Niro tools.
+Restart the agent. It reads its settings only when it starts.
+<!-- source: src/mcp/setup.js:3071-3085 -->
+
+### My agent still searches files instead of asking Niro.
+Only Claude Code gets hooks that remind it at every step. Other agents get instruction text, or nothing beyond the connection. Ask your agent to use the Niro tools, or start your request with "Use Niro to ...".
+<!-- source: docs/usecases/impact-gate.md:155; src/mcp/setup.js:782-800, :2079-2100, :2436-2592 -->
+
+### A `.cursor/rules/niro.mdc` file appeared in my repository.
+For Cursor, Niro writes its instructions as a rule in the folder where you ran `niro mcp install` or `niro upgrade`. Delete it, or add it to `.gitignore`, if you do not want it there.
+<!-- source: src/mcp/setup.js:2079-2100; src/commands/upgrade.js:349-353 -->
+
+### What did install change in my agent's settings?
+Each agent's page lists the files. Three agents also let Niro's tools run without asking each time:
+
+- **Cursor:** an allowlist entry.
+- **Kiro:** auto-approve for all of Niro's tools.
+- **Codex:** approval mode "writes".
+<!-- source: src/mcp/setup.js:445-503, :382-395, :2305-2318 -->
+
+### What does "Answered from the local graph on this machine (niro-local)." mean?
+The answer came from the engine on your computer. Every answer starts with this line.
+<!-- source: src/mcp/localBackend.js:189 -->
+
+### The Niro tool list is empty.
+The engine did not start. Run `niro engine status` and `niro status` in a terminal to see why. Then restart the agent.
+<!-- source: src/mcp/server.js:1266-1268 -->
+
+### "The local engine could not answer: <message>"
+The engine stopped or failed on this question. The message says why. Run the same question with `niro ask` to see the full error.
+<!-- source: src/mcp/server.js:1279 -->
+
+### "the local engine did not answer <method> within Nms" or "Niro MCP request timed out"
+The question took too long: 30 seconds for one engine call, 2 minutes for one tool call. Ask a narrower question. Raise the tool limit with `NIRO_TOOL_TIMEOUT_MS`.
+<!-- source: src/mcp/localBackend.js:183, :684; src/lib/pendingRequests.js:24, :64-68 -->
+
+### The first answer of every session takes about 20 seconds.
+This happens when Niro's licence service cannot be reached. The engine waits for its licence. Known problem.
+<!-- source: niro-local docs/NIRO-LOCAL-ISSUES.md:3617 (open) -->
+
+### My agent says "THIS FOLDER IS NOT INDEXED" and offers to index it.
+The folder your agent is working in has no graph. Indexing stays on your computer. Say yes if you want Niro's help in that folder. It counts towards your line limit.
+<!-- source: niro-local src/niro_local/offer.py:297-303, :409-414 -->
+
+### Will my agent run `niro start`, `niro login` or `niro uninstall` by itself?
+No. The skill Niro installs tells it never to. It also tells the agent to ask you before it stops indexing a folder.
+<!-- source: src/mcp/localSkill.js:8-90 -->
+
+### I work in two folders. Which one do the tools answer for?
+The folder your agent's session started in. For the other folder, run `niro status` there to see whether it is indexed. Run `niro build <folder>` after editing it.
+<!-- source: src/mcp/localMode.js:39 (RELEASE_WORKTREE_SENTENCE) -->
+
+### Claude Code shows "Niro impact check, before you edit this file". How do I turn it off?
+It is advisory only. Turn it off with `touch ~/.niro/impact-gate-off`. Delete that file to turn it back on.
+<!-- source: src/mcp/setup.js:1376-1405, :1512-1514; src/lib/impactLog.js:64 -->
+
+### Claude Code checks for a Niro line at the end of each task.
+This is the end-of-task line. Turn it off with `niro mcp receipt --off`. See [faq-feedback-and-receipts.md](faq-feedback-and-receipts.md).
+<!-- source: bin/niro.js:3-10; src/mcp/lineCheck.js:329-330; src/commands/mcp.js:79-90 -->
+
+### Can I use Niro with my agent on Windows?
+Windows is supported, and the details are coming.
+<!-- source: Niraj, 2026-11-10 -->
+
+### Cline does not show Niro after a restart.
+Niro writes Cline's settings to the path Cline's users report. Cline's own documentation does not confirm it. Open Cline's MCP settings, and check where it keeps them.
+<!-- source: src/mcp/setup.js:362-366 -->
+
+### Goose printed a block for me to paste.
+Your Goose settings file is laid out in a way Niro does not change by itself. Paste the block into it.
+<!-- source: src/mcp/setup.js:2693-2717 -->
+
+### "Niro is not set up in any AI assistant on this machine, so there is nothing to remove."
+`niro mcp uninstall` found no agent with Niro in it.
+<!-- source: src/mcp/teardown.js:663 -->
+
+### "Invalid --url" or "Refusing to use MCP server URL ..."
+`--url` must start with `http://` or `https://`, with no spaces. It is for hosted Niro only.
+<!-- source: src/mcp/setup.js:2897, :3036 -->
+
+### "MCP <label> wizard exited with code N"
+The install or uninstall step failed. The lines above say why.
+<!-- source: src/commands/mcp.js:43 -->

package/docs/faq-feedback-and-receipts.md

@@ -0,0 +1,33 @@
+# FAQ: feedback and receipts
+
+### What is the "[NIRO-CLI]" line at the end of a task?
+At the end of a task, your agent calls a tool, `mark_task_complete`. Niro answers with one line saying what its answers did in that task: how many calls answered (counted), how many searches the agent thinks they replaced (the agent's own estimate), and the tokens the answers cost against an estimate of the same work by hand. What each number means, and how each is worked out, is in [evaluate-what-niro-did.md](evaluate-what-niro-did.md). The agent copies the line into its reply. The line is not saved on your computer. What your agent wrote in that call (a summary of the task, and what Niro did) is part of the tool-usage record sent to Niro. See [what-leaves-your-machine.md](what-leaves-your-machine.md).
+<!-- source: niro-local src/niro_local/mcp/server.py:2107; src/niro_local/journal/store.py:86-90, :121; src/lib/receiptStore.js:1-15 ("Nothing is stored locally to display it") -->
+
+### Why does the line not appear?
+The agent must call `mark_task_complete` and copy the line. Claude Code checks for it at the end of each task. Other agents may skip it.
+<!-- source: niro-local docs/HOW-TO.md:694-790; src/mcp/lineCheck.js -->
+
+### How do I turn the line off?
+Run `niro mcp receipt --off`. Turn it on again with `--on`. To remove the tool altogether, set `NIRO_LOCAL_TASK_COMPLETE=0`.
+<!-- source: src/commands/mcp.js:79-90; src/lib/receiptStore.js:33-35; niro-local src/niro_local/mcp/server.py:302 -->
+
+### "Nothing to report: no niro-local call answered this turn."
+Niro was not asked anything during the task, so there is no line to show.
+<!-- source: niro-local src/niro_local/mcp/server.py:2207-2215 -->
+
+### "The end-of-task Niro line is switched off on this machine"
+You turned it off with `niro mcp receipt --off`.
+<!-- source: src/mcp/localBackend.js:206-207 -->
+
+### Can I see past receipts?
+Not the lines: each is shown once, in your agent's reply, and is not saved. Their figures are kept, and `niro impact` adds them up for the last 7 and 30 days, with a line per project. See [evaluate-what-niro-did.md](evaluate-what-niro-did.md). A `~/.niro/receipts` folder left by an earlier version is removed when you remove Niro from Claude Code (`niro mcp uninstall`), or by `niro uninstall`.
+<!-- source: niro-local src/niro_local/impact.py and journal/store.py turns table (kept after upload); src/commands/engineRoutes.js (impact); src/lib/receiptStore.js:4-11, :64-75 (purge); src/mcp/teardown.js:396 (purge runs when Niro is removed from Claude Code); src/commands/uninstall.js:66, :157-159 (all of ~/.niro is deleted) -->
+
+### Two tool calls were counted as two sessions.
+Known problem when two calls arrive at the same moment.
+<!-- source: niro-local docs/NIRO-LOCAL-ISSUES.md:3663 (open) -->
+
+### How do I send feedback or ask for help?
+Email help@niro.ai. The screen at the end of `niro start` also links the help site, GitHub and Discord.
+<!-- source: help@niro.ai: Niraj, 2026-11-10; src/commands/start.js:131-136 -->

package/docs/faq-indexing.md

@@ -0,0 +1,127 @@
+# FAQ: indexing and builds
+
+Indexing means reading a folder's code and building its graph.
+
+### "This folder is not added yet. Run niro init to add it and build it."
+`niro build` only rebuilds folders Niro already knows. Add this one with `niro init`.
+<!-- source: src/commands/initLocal.js:53 -->
+
+### "Two folders were given (<a> and --path <b>). Give one."
+You named a folder twice, once directly and once with `--path`. Give it once.
+<!-- source: src/commands/initLocal.js:46 -->
+
+### "This folder is not indexed on this machine. Run niro start here to set it up."
+Niro has no graph for this folder. On a computer that is already set up, `niro init` adds a folder. On a new computer, run `niro start`.
+<!-- source: src/commands/status.js:97; src/index.js:490-502 -->
+
+### A file I know exists is missing from every answer.
+It may be larger than 2 MB, which is the largest file Niro reads. `get_index_status` lists files left out for size. Raise the limit with `NIRO_LOCAL_MAX_FILE_BYTES`, then run `niro build`. Some answers say only "not found", with no word about the size, so check `get_index_status`.
+<!-- source: niro-local src/niro_local/walk.py:79; src/niro_local/mcp/server.py (_render_status); niro-local docs/NIRO-LOCAL-ISSUES.md:3530 (open) -->
+
+### A file is listed as one "whose parse passed the 20 s time limit".
+A file over 256 KB that takes longer than 20 seconds to read is left out. Raise the limit with `NIRO_LOCAL_PARSE_TIMEOUT_SECONDS`, then run `niro build`.
+<!-- source: niro-local src/niro_local/build/parse_time_limit.py:44-57; src/niro_local/too_large.py:174 -->
+
+### "SKIPPED: N source file(s) larger than 2 MB ... were not indexed"
+An answer may be missing something because these files were left out. The line names the first 5 of them, and says how to raise the limits.
+<!-- source: niro-local src/niro_local/too_large.py:41, :164-182 -->
+
+### Which files and folders does Niro skip?
+Niro skips:
+
+- files your `.gitignore` covers (it honours `.gitignore` the way git does);
+- build output folders;
+- editor scratch files;
+- links to other places on disk (symlinks);
+- files in languages it does not read.
+<!-- source: niro-local src/niro_local/mcp/source_tools.py:825-850; src/niro_local/gitindex.py; src/niro_local/walk.py -->
+
+### Minified or bundled JavaScript shows up as hundreds of methods.
+A minified file is code squeezed onto a few lines for shipping. Niro filters many of them out, but one committed to the repository can still be read as code. Add it to `.gitignore` if it is generated. Mapping it back to its real source is not supported.
+<!-- source: niro-local docs/NIRO-LOCAL-ISSUES.md:2849 (open); niro-local TODO.md:168 -->
+
+### Are HTML, Vue and Svelte files read?
+Only their script blocks. The page markup itself is not read.
+<!-- source: niro-local src/niro_local/languages.py, the comment above the javascript entry -->
+
+### My F#, Visual Basic or PowerShell code is not in the graph.
+Niro has no reader for these languages. It counts F# and Visual Basic files in a build's summary, but nothing in them reaches the graph.
+<!-- source: niro-local src/niro_local/profiles/local/unparsed-languages.json; coverage catalog powershell.json -->
+
+### Why are my tests in the graph?
+Niro reads test code too, so you can see which tests reach a change. Hosted Niro sometimes leaves tests out, so counts can differ between the two.
+<!-- source: niro-local TODO.md:3679 -->
+
+### "another niro-local was writing the graph, and this gave up (it waits at most 120 s ...)"
+Two builds ran at once, and the second waited 2 minutes for the first. Run your build again when the first has finished. Raise the wait with `NIRO_LOCAL_BUILD_LOCK_WAIT`.
+<!-- source: niro-local src/niro_local/cli.py:105-143; src/niro_local/build/lock.py:59 -->
+
+### A build of a very large repository crashed.
+This is a known problem on some very large repositories (a "RecursionError"). Send the error to help@niro.ai.
+<!-- source: niro-local docs/NIRO-LOCAL-ISSUES.md:528 (open) -->
+
+### The watcher's log says "Could not rebuild <folder>: ... Edits are not reaching the graph."
+The background rebuild failed. Run `niro build` in that folder to see the full error. After you fix it, save any file and the watcher tries again.
+<!-- source: src/lib/localBuild.js:172, :208-210 -->
+
+### An answer says "NOTE FROM niro-local: ACTION REQUIRED. The following file(s) were modified after the last index build".
+Those files changed after the last build, so the graph may be out of date for them. The answer lists up to 20 of them. Read them directly. The watcher rebuilds them shortly.
+<!-- source: niro-local src/niro_local/mcp/freshness.py:583-590, :346 -->
+
+### An answer says "niro-local is rewriting this folder's code graph right now".
+A full build is running. For this one request, search the files (or use `code_grep` and `get_source_code`, which keep working), and ask again when the build ends. It does not mean "nothing found".
+<!-- source: niro-local src/niro_local/mcp/freshness.py:309-334 -->
+
+### How long does a build take?
+A full build of a repository of about 9,000 files was measured at about 103 seconds. Later rebuilds handle only what changed, and are much quicker.
+<!-- source: niro-local TODO.md:189 -->
+
+### Does Niro run git, or read my history?
+No. It reads three small git files: `.git/HEAD` (your branch), `.git/index` (which files git tracks) and `.git/config` (to tell copies of a repository apart). It never runs git, and never reads your history.
+<!-- source: niro-local CLAUDE.md amendments 2026-11-03, 2026-11-04, 2026-11-07 -->
+
+### When does the graph update after I save a file?
+About 3 seconds after the folder stops changing, if the background watcher is running. Check it with `niro status`.
+<!-- source: src/lib/localBuild.js:36; src/commands/status.js:35-40 -->
+
+### A service has the wrong name, or a library shows up as a service.
+Niro guesses which folders are separate services from build files and layouts, and sometimes guesses wrong. Two examples:
+
+- some .NET projects are never treated as services;
+- a Go library can be labelled a service.
+<!-- source: niro-local docs/NIRO-LOCAL-ISSUES.md:612 (an ASP.NET Core project is never a service), :698 (a Go library is labelled a service) -->
+
+`niro modules` shows what it decided. Fixing a wrong guess by hand is not supported yet.
+<!-- source: niro-local docs/NIRO-LOCAL-ISSUES.md:586, :607, :672, :1606, :3459 (open); src/commands/engineRoutes.js:34 -->
+
+### How does Niro find the services in a monorepo?
+A monorepo is one repository holding several services. Niro uses each service's build file, and deployment files such as a Docker Compose file, to tell them apart. Some library modules in Cargo, Go, Gradle and .NET are not yet separated out. See [configuring-a-folder.md](configuring-a-folder.md).
+<!-- source: niro-local docs/usecases/compose-addresses.md; niro-local docs/NIRO-LOCAL-ISSUES.md:2184, :2195 (open) -->
+
+### "the graph at <path> is damaged ... It is rebuilt from what was indexed."
+The graph file was damaged, and the engine rebuilds it from its saved build state. Run the command it names.
+<!-- source: niro-local src/niro_local/graph/sqlite_writer.py:520 -->
+
+### "the graph was written by an older major version of niro-local; run niro build"
+After an upgrade, the graph needs one rebuild in the new format. Run `niro build`.
+<!-- source: niro-local src/niro_local/graph/sqlite_writer.py:311-313 -->
+
+### "the graph ... was written by a newer major version of niro-local"
+A newer Niro wrote this graph, and this older one will not change it. Run `niro upgrade`.
+<!-- source: niro-local src/niro_local/graph/sqlite_writer.py:618-620 -->
+
+### "The niro-local home ... was last used by a newer niro-local ... Upgrade with: niro upgrade"
+Same cause: a newer version used this folder. Run `niro upgrade`.
+<!-- source: niro-local src/niro_local/registry.py:368-373 -->
+
+### When should I use `niro build --full`?
+When you suspect the graph is wrong in a way a normal rebuild does not fix. It reads every file again. A normal rebuild gives the same graph, so you rarely need it.
+<!-- source: src/index.js:517-527; niro-local CLAUDE.md (incremental must equal full) -->
+
+### My folder is in two projects. Which one does `niro build` build?
+Both. Name one with `niro build --project <name>`.
+<!-- source: src/index.js:517-527; niro-local src/niro_local/cli.py:300-306 -->
+
+### I deleted a folder. Is its graph still on disk?
+Yes, for 14 days after Niro first saw it missing. Then it is removed. To remove it now, run `niro clean`. If you move the folder back, it is used again.
+<!-- source: niro-local src/niro_local/cleanup.py:37-58; src/commands/engineRoutes.js:35 -->