localagentbrowser 1.52__tar.gz → 1.57__tar.gz

{localagentbrowser-1.52 → localagentbrowser-1.57}/CHANGELOG.md

@@ -1,9 +1,98 @@
 # Changelog
 
-Every version of `agent_browser.py`, newest first. Each version is also a tagged commit (`v1.11` … `v1.28`).
+Every version of `agent_browser.py`, newest first. Each released version is also a tagged commit (`v1.11` … `v1.57`); 1.54 to 1.56 were internal steps
+measured on the regression suite and released as part of 1.57 (see
+[docs/regression-history.md](docs/regression-history.md)).
 Timings are from live runs on the development machine (Intel i5-9500, 6 cores, no GPU, Qwen2.5-7B-Instruct
 Q4_K_M in LM Studio), where reading the prompt takes about 90% of every model call.
 
+## [1.57] - 2026-10-19
+Released together with the internal steps 1.54, 1.55 and 1.56, each of which was measured on the regression suite
+but not released on its own. [docs/regression-history.md](docs/regression-history.md) has every step's results,
+task by task, and what each one showed.
+### Added
+- Coverage check before finishing. The 1.53 suite left two problems open: an explanation finished from one overview
+  paragraph that the model said covered every item (no oxygen or water), and a list named WPS Office as
+  "free, open-source". Before a FINISH goes through, each checklist item is now checked against the full findings
+  in a short call of its own, with the item last so the instructions are read once (about 1-3 minutes for a
+  checklist). The code checks the verdict rather than taking it on trust: an item nothing was copied for counts as
+  unanswered; an item asking for a number of things gets "only 1 of 3 found" when fewer are named; for such items,
+  each thing named must be shown to meet the item's conditions in words copied from a finding about that thing; and
+  a claim that something is missing is dropped when every word of it is already in the findings. With gaps, the
+  FINISH is sent back once, naming the next step for a condition ("SEARCH for 'OnlyOffice free and open-source' to
+  check it, or replace it"), and the next turn starts from a fresh prompt. Finishing with nothing recorded is sent
+  back too.
+- What no finding answers is reported by the code. Checked once more just before the final result, checklist items
+  nothing answers and things not shown to meet a condition are left out of what the final result is asked to write
+  and added after it ("Not found in the pages read", "Left out, not shown to meet the conditions in the pages read").
+  The final result had filled a gap from the model's own knowledge (a care routine for repotted plants).
+- The first turn asks for one short search for the whole objective, keeping its conditions: with the checklist, the
+  first search had gone after the first item only, and the documentation task had dropped "official". Two system
+  prompt rules say the same for later searches.
+### Changed
+- A DISCOVER refused as not on the page now quotes the page sentence closest to it (chosen by the refused text and
+  the objective), to record instead, or names what the parts shown never mention ("Japan, Navajo"); it is no longer
+  wrapped in "Do NOT choose DISCOVER again ... choose a different action", which outweighed the quoted sentence.
+  Replayed on the saved runs: the page's real aftercare line was recorded 6 times out of 6 (0 of 3 before), a refusal
+  where the real steps came on the next part still led to READ_MORE 6 of 6, and an owl case read on 5 of 6.
+- The final result gives as many items as asked for without ever repeating one, and says how many were not found:
+  a list run had listed LibreOffice three times.
+- Opinion items are shown in the checklist as "written by you at FINISH, not found on a page": shown as having no
+  discoveries yet, an owl task kept collecting cultures after it had the three asked for.
+- Fundraising banners and other site notices are hidden like cookie banners: Wikipedia's filled the whole first part
+  of an article.
+- Regression suite: the core how-to is now "make cold brew coffee at home" (`howto-coldbrew`), since the repotting
+  task went to the same guide in three runs and that guide has no section on care in the first two weeks; repotting
+  stays as an extra task under its old id. The summary flags a task that fails again on the same site as before. The
+  documentation task has 45 minutes, like the others.
+### Fixed
+- The not-on-page check refused the real Kilimanjaro answer as unseen because the banner hid the article (above).
+- Numbered headings ("#### 1. Setup") were flagged as numbers missing from the findings (1.53).
+
+Regression runs, finished out of 7 (model minutes): 1.53 5 (200), 1.54 4 (239), 1.54 with fixes 5 (225), 1.55 5
+(233), 1.56 4 (272), 1.57 4 (270). Across these runs the result quality rose where the checks fired: the
+explanation went from thin to complete and cited, the comparison stopped inventing features, the list stopped naming
+WPS Office as open source, and every documentation finding since 1.55 is from python.org. What is not better: about
+half of the coverage check's send-backs were for gaps that were not real, and each costs 8-10 minutes on this
+hardware (a fresh prompt reads 3,000-4,000 tokens), so finishes were lost: on 1.57, a list of three correct
+open-source suites sent back because no words showed them to be "free", and a comparison sent back for "official
+documentation" it was not asked for. The next step is to send the agent back only for gaps the code verifies.
+Also still open: on the documentation task the Windows line repeats the create command instead of the activate
+one (1.56 and 1.57), and on 1.57 the opinion task ran out of time on a page that covered two of the three
+cultures asked for.
+
+## [1.53] - 2026-10-17
+### Added
+- Regression suite (`tools/regression.py`): seven objectives, one of each kind of task tested so far, run one
+  after the other with a time limit each and scored the same way (finished or not, turns, model time, checklist
+  progress, checks fired, and word checks for missing answers, missed conditions, uncited paragraphs, numbers
+  missing from the findings and a missing opinion), with a comparison to an earlier run. Single runs had been
+  flattering: on 1.52 the suite finished only 2 of 7 tasks within their limits, including tasks earlier single
+  runs had finished. Refused actions in the journal now name the check that refused them, so the suite can count
+  each check, and the live test runner times every model call and saves the model's full replies.
+- Objective checklist. The 1.52 suite showed the main failure: every check asked the model to split the
+  objective into parts again, from memory, and the split drifted. On the list task it blocked its own FINISH
+  over "migration strategies", which the objective never asked for (an idea from a page it had read), and never
+  looked for a third suite; on the comparison it added "data types" and never covered "many simultaneous users".
+  Now one short call at the start splits the objective into a fixed checklist; asked on its own, the model split
+  all seven test objectives cleanly and kept their conditions. Each DISCOVER names the items it serves
+  (`checklist_items` in the reply), the checklist with its discoveries is in fresh prompts and in the note after
+  each discovery, the finish, progress and objective checks go through its items ("do not add other parts"),
+  the read-on check names the items still open, the final result covers every item or says it was not found,
+  and the report lists the checklist. The split call uses the same system prompt as turn 1, so LM Studio's cache
+  of it still applies there (without that, turn 1 took 128 s instead of about 40).
+### Fixed
+- The report flagged the numbers of numbered headings ("#### 1. Setup") as numbers missing from the findings.
+
+Regression run against 1.52 (model minutes in brackets): 5 of 7 tasks finished instead of 2, and the total model
+time fell from 254 to 200 minutes. The list (30.1, was a 45-minute time limit), comparison (39.0, every part
+covered), explanation (12.9, was 60 without finishing) and opinion (38.1) tasks now finish; the factual question
+still does (13.6, was 11.9). What is not better: the list names WPS Office, which is not open source; the
+explanation finished from a single overview paragraph that the model claimed covered all four items, without
+oxygen or water; the how-to found the materials and steps but not the aftercare before its time limit; and the
+documentation task now went to python.org, as asked, but worked through the items one search at a time and hit
+its 30-minute limit before the last one (it had finished in 15 minutes on 1.52, from a tutorial site).
+
 ## [1.52] - 2026-10-16
 ### Changed
 - The note after each DISCOVER also says to work on parts of the objective that are not covered yet before

{localagentbrowser-1.52 → localagentbrowser-1.57}/MANIFEST.in

@@ -1,3 +1,4 @@
-# Files the source package needs beyond the defaults, so its tests can run
+# Files the source package needs beyond the defaults, so its tests (and the tools they test) can run
 include CHANGELOG.md
 recursive-include tests *.py
+recursive-include tools *.py

{localagentbrowser-1.52/localagentbrowser.egg-info → localagentbrowser-1.57}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: localagentbrowser
-Version: 1.52
+Version: 1.57
 Summary: An autonomous web agent driven by a local LLM (LM Studio) and Playwright. Runs entirely on your own machine.
 Author: SnowFX
 License-Expression: MIT
@@ -56,8 +56,8 @@ and tuned to work well with a **small 7B model on a CPU, no GPU required**.
 - ⚡ **Built for slow hardware.** It reuses LM Studio's prompt cache, so most turns process only what changed.
 - 📝 **A readable report every time.** `research_report.md` holds the final result and every finding with its
   source, written however the run ends; `research_log.txt` keeps every action and reason.
-- ✅ **Tested.** 112 tests on Linux and Windows, Python 3.10–3.13, run in about a second with no model or browser,
-  plus live runs on seven different kinds of task (see [results](#results-on-different-kinds-of-tasks)).
+- ✅ **Tested.** 165 tests on Linux and Windows, Python 3.10–3.13, run in a few seconds with no model or browser,
+  plus a [regression suite](#regression-suite) of live runs on seven different kinds of task.
 
 📖 **The story behind it:** [Teaching a 7B model on a CPU to finish what it starts](https://dev.to/snowfxxx/teaching-a-7b-model-on-a-cpu-to-finish-what-it-starts-1amc),
 six lessons from 30 versions of keeping a small local model on track.
@@ -193,7 +193,25 @@ does not generalise; the [changelog](https://github.com/SnowFXXX/LocalAgentBrows
 | List of items | Three free, open-source alternatives to Microsoft Office and what each is best at | 1.46, 1.51 | LibreOffice and OnlyOffice with their strengths, cited, in 18 turns (49 min); the third, FreeOffice, is free but not open source, and LibreOffice was recorded four times in different words; on 1.51: LibreOffice, OnlyOffice and Apache OpenOffice, all open source, in 7 turns (16 min) |
 | Step-by-step how-to | Make a sourdough starter, with quantities and what to do each day | 1.36–1.37 | A clear plan in 7 turns (19 min), but it invented the first water amount; 1.37 stops guessing and flags unsupported numbers, and the agent then went looking for the missing details |
 | Step-by-step how-to | Repot a houseplant: what you need, the steps, and care for the first two weeks | 1.43–1.44, 1.51 | 1.43 recorded six steps it wrote itself before reading the part that had them; 1.44 recorded the guide's real steps but took 68 min and its result left one out. 1.51: every step from the page, in 7 turns (29 min), and it says that no care instructions beyond the first watering were found |
-| Explanation of a topic | How photosynthesis works: inputs and outputs, both stages and where each takes place | 1.41–1.42 | A complete, cited explanation in 18 turns (45 min); 1.41 read four parts of a relevant page without recording them and had not finished after 49 min |
+| Explanation of a topic | How photosynthesis works: inputs and outputs, both stages and where each takes place | 1.41–1.42, 1.52 | A complete, cited explanation in 18 turns (45 min); 1.41 read four parts of a relevant page without recording them and had not finished after 49 min. 1.52 recorded both stages and where each takes place, but did not finish within 65 min: it read the history sections of a long encyclopedia article twice and never recorded the inputs and outputs |
+
+### Regression suite
+
+Since 1.53, every change is measured on the same seven objectives, one of each kind above, run one after the other
+with a time limit each (`tools/regression.py`, see [Testing](#testing)). Single runs vary with the pages a search
+turns up, so the suite compares versions on all seven at once. 1.54 to 1.56 were internal steps measured this way
+and released together as 1.57; [docs/regression-history.md](https://github.com/SnowFXXX/LocalAgentBrowser/blob/main/docs/regression-history.md)
+has every step's results and what each one showed.
+
+| Task | 1.52 | 1.53 (objective checklist) | 1.57 (coverage check) |
+|---|---|---|---|
+| Factual question | Finished, 12 min | Finished, 14 min | Finished, 13 min |
+| List of three office suites | Time limit | Finished, 30 min; but WPS Office is not open source | Time limit: three correct open-source suites by turn 7, sent back because no words showed them to be "free" |
+| Comparison and recommendation | Time limit | Finished, 39 min | Time limit, after a send-back for "official documentation", which the objective does not ask for |
+| How-to (cold brew from 1.55; repotting before) | Time limit | Time limit | Finished, 36 min; but "how long it keeps" answered with the steeping time |
+| Explanation (photosynthesis) | Time limit | Finished, 13 min, but thin: no oxygen or water | Finished, 59 min: inputs, outputs and both stages, cited |
+| Opinion (owls) | Time limit | Finished, 38 min | Time limit: its first page covered two of the three cultures asked for |
+| Documentation (venv, official docs) | Finished, 15 min, from a tutorial site | Time limit (30 min) | Finished, 37 min, all from python.org; but the Windows line repeats the create command instead of the activate one |
 
 ## How it works
 
@@ -232,7 +250,22 @@ flowchart LR
 - **Findings must match the page.** A finding that shares too few word pairs with the page text shown so far
   (or has a sentence that does) is sent back once, asking for only what the page says in its own words, and
   one far from anything on the page is refused outright. A small model will otherwise "complete" a list from
-  its own knowledge when a page says "follow these 6 steps" and the steps come later.
+  its own knowledge when a page says "follow these 6 steps" and the steps come later. The refusal points the way:
+  it quotes the one page sentence closest to what the model wrote, to record instead, or names what the parts
+  shown never mention ("Japan, Navajo"), so the model reads on.
+- **An objective checklist.** At the start, one short model call splits the objective into a fixed checklist of
+  what the result must contain, keeping conditions like "free, open-source" or "three". Each finding names the
+  items it serves, the checklist with its findings is in every prompt, and every check goes through these items
+  instead of asking the model to split the objective again from memory, which drifted: it once blocked its own
+  finish over "migration strategies", which the objective never asked for. Opinion items are shown as written
+  at FINISH, not as gaps to fill, and the first search covers the whole objective in one short query, keeping
+  its conditions ("official python documentation venv").
+- **A coverage check before finishing.** Each checklist item is checked against the findings in a short call of
+  its own, and the code checks the verdict instead of taking it on trust: an item nothing was copied for, fewer
+  things than asked for ("only 1 of 3 found"), and things not shown, in words copied from a finding about them,
+  to meet a condition such as "open-source". A claim that something is missing is dropped when every word of it is
+  already in the findings. With gaps, the FINISH is sent back once, with the next step named ("SEARCH for
+  'OnlyOffice free and open-source' to check it, or replace it"), and the next turn starts from a fresh prompt.
 - **Numbered findings.** Findings are numbered in the prompt and listed with every check, and the model names
   the finding that covers each part of the objective ("covered by [2]") instead of just saying "covered".
 - **Knowing when it's done.** After every new finding the model is asked whether the objective is now fully
@@ -244,12 +277,16 @@ flowchart LR
   A part that cannot be found after several tries can be reported as missing instead of blocking for good.
 - **Writing is a separate step.** When the agent finishes, one more model call writes the final result from
   only the objective and the recorded findings, under a short list of rules: no guessing or outside knowledge,
-  cite findings by number, give exactly as many items as asked for, and add a "My view:" paragraph only when
-  the objective asks for an opinion. A small model writes far better when that is its only job, and every
-  claim stays traceable to a recorded finding. The report flags any number in the final result that appears
-  in none of the findings, the most harmful kind of slip in a how-to.
+  cite findings by number, give as many items as asked for (never repeating one, and saying how many were not
+  found), and add a "My view:" paragraph only when the objective asks for an opinion. A small model writes far
+  better when that is its only job, and every claim stays traceable to a recorded finding. What no finding
+  answers is not left to the model: those checklist items, and things not shown to meet a condition, are left out
+  of what it is asked to write, and the code adds them after the result as "Not found in the pages read" and
+  "Left out, not shown to meet the conditions". The report flags any number in the final result that appears in
+  none of the findings, the most harmful kind of slip in a how-to.
 - **Error pages are recognised.** A page answering 404, 403 and so on counts as a failed visit, not content.
-- **Readable pages.** Navigation menus, footers and cookie banners are hidden; code blocks and tables are
+- **Readable pages.** Navigation menus, footers, cookie banners and site notices such as fundraising banners are
+  hidden; code blocks and tables are
   never split between parts; links already shown are not repeated. Long pages come with an outline of the
   headings in each part, marking the parts already read, so the agent can jump to the part it needs instead of
   reading in order.
@@ -282,11 +319,13 @@ Measured on an Intel i5-9500 (6 cores, no GPU) with Qwen2.5-7B-Instruct Q4_K_M a
 | Follow-up on the same page (finding recorded, cheap retry) | 35–60 s |
 | Next part of a page, or a new page appended | 100–280 s |
 | First turn of a run | 45–110 s |
-| Fresh prompt after the context budget is reached | 160–380 s (longest with long pages) |
+| Fresh prompt (context budget reached, or after a coverage send-back) | 160–540 s (longest with long pages) |
+| Coverage check before finishing (all checklist items) | about 1–3 min |
 | Writing the final result (once, at the end) | 50–190 s, depending on how much was found |
 
-Whole runs from the results table: a factual question in about 12 minutes of model time, a list of three items
-in 24, a comparison in 33. A GPU or a smaller model will be much faster.
+Whole runs vary with the pages a search turns up. In the regression runs since 1.53, a factual question took 13 to
+19 minutes of model time; the list, comparison, how-to, opinion and documentation tasks 21 to 44 when they finished;
+and the explanation 13 to 59. A GPU or a smaller model will be much faster.
 
 ## FAQ
 
@@ -321,9 +360,19 @@ turn that brings in a new page takes a few minutes. See [Performance](#performan
   research, an opinion from several sources, a factual question, a comparison, a how-to, a list and an
   explanation of a topic. Each one
   exposed something that did not generalise and was fixed; other kinds of tasks may expose more.
-- **Reading without recording.** The model used to read long pages part by part without recording the
-  relevant parts. Since 1.42 it must say what a part contains before reading on, which fixed this on an
-  explanation task; it is still the area most likely to need more work on other pages.
+- **Judging whether a finding covers an item is the weakest area.** The model's verdict is unreliable both ways:
+  on 1.53 the explanation finished from one overview paragraph without oxygen or water, and in the regression runs
+  since 1.54 about half of the coverage check's send-backs were for gaps that were not real ("missing official
+  documentation" on a comparison that did not ask for it, "H2O, O2" already recorded). The code now verifies what
+  it can (an item nothing answers, a count, word-for-word proof of a condition), but each wrong send-back still
+  costs a finish on this hardware, since a send-back costs 8–10 minutes.
+- **Conditions in the objective.** A small model tends to drop conditions such as "open source" or "using the
+  official documentation": a list of open-source office suites once included FreeOffice (free but not open
+  source), and a task that asked for the official Python documentation used a tutorial site. The objective
+  checklist keeps the conditions in view (the documentation task uses python.org since 1.55), and a thing not
+  shown in the findings to meet a condition is left out of the result and listed after it. The check is strict:
+  on 1.57 it sent back three correct open-source suites because no words showed them to be "free". Check the
+  result against any conditions you set.
 - **A small model's judgement is the ceiling.** It sometimes drifts to loosely related pages, prefers
   secondary sources or stays with one source too long, and results vary with the first pages a search turns
   up; every finding is in `research_report.md` regardless.
@@ -332,8 +381,9 @@ turn that brings in a new page takes a few minutes. See [Performance](#performan
   minutes, but with thin findings and without naming what it could not find, so check the result against the
   objective.
 - **Check the citations.** The final result is told to use only recorded findings, but a 7B model sometimes
-  puts the wrong number on a citation, adds a line no finding supports or leaves out a step, so verify a
-  claim against the listed findings before relying on it.
+  puts the wrong number on a citation, adds a line no finding supports, leaves out a step or misreads a part of
+  the question (on 1.57 it answered "how long it keeps in the fridge" with the steeping time), so verify a claim
+  against the listed findings before relying on it.
 - **Some sites refuse automated browsers** (HTTP 403). The agent respects that and does not try to disguise
   itself; it moves on to other sources.
 - **Reading only.** The agent searches, opens pages and reads them. It does not click, fill in forms or sign in.
@@ -348,6 +398,15 @@ turn that brings in a new page takes a few minutes. See [Performance](#performan
 - [x] Finish as soon as the objective is covered (1.35)
 - [x] Get the model to weigh each part of a long page before reading on (1.42)
 - [ ] Let very strict objectives end with the gaps named instead of searching indefinitely (they finish since 1.50; naming the gaps is still open)
+- [x] Measure every change on a regression suite of seven kinds of task (1.53)
+- [x] Keep a fixed checklist of the objective's parts instead of re-splitting it at every check (1.53)
+- [ ] Check that a finding really covers its checklist item, including conditions like "open source" (partly, since
+  1.57: a coverage check before finishing, with what the code can verify)
+- [ ] Send the agent back only for gaps the code verifies, and let the model's other verdicts only shape the final
+  result (the next step after 1.57)
+- [ ] Rewind dead ends: save the conversation before an expensive step such as reading a long page, and if it
+  leads nowhere, return to that point with a short note instead of carrying the unused page text in every later
+  prompt (LM Studio reuses its cache up to the saved point)
 - [ ] Try more kinds of objectives (seven so far) and adjust the rules that don't generalise
 - [ ] Make fresh prompts (the slowest kind of turn) cheaper still (31% cheaper since 1.34)
 - [ ] Interacting with pages (clicking, forms) for tasks beyond reading
@@ -356,7 +415,7 @@ Ideas and requests are welcome in [issues](https://github.com/SnowFXXX/LocalAgen
 
 ## Testing
 
-The test suite runs in about a second and needs neither LM Studio nor a browser: the model and the website
+The test suite runs in a few seconds and needs neither LM Studio nor a browser: the model and the website
 are replaced by scripted fakes, including an end-to-end run of the main loop.
 
 ```bash
@@ -372,6 +431,23 @@ call:
 python tools/live_test.py "Your objective here" --minutes 25
 ```
 
+To measure a change on all seven kinds of task (about four hours on a CPU), compared with an earlier run:
+
+```bash
+python tools/regression.py --baseline regression_runs/<earlier run>/summary.json
+```
+
+Each run gets a folder in `regression_runs/` with every task's report, journal, console output and prompt
+trace, and a `summary.md` that is updated after each task: whether it finished, turns, model time, how many
+checklist items got findings, which checks fired, and problems found by simple word checks (missing answers,
+missed conditions such as a non-open-source item, uncited paragraphs, numbers missing from the findings, a
+missing opinion). It also flags a task that fails again on the same site as in the earlier run: one site that
+cannot answer part of an objective can decide a task whatever the agent does (the repotting how-to went to the same
+guide in three runs, which is why the core how-to is now the cold brew one). `--suite all` adds six more tasks,
+including the earlier how-to, `--only` runs some of them, and `--rescore` scores an earlier run again without the
+model. Every step since 1.53 and its results are in
+[docs/regression-history.md](https://github.com/SnowFXXX/LocalAgentBrowser/blob/main/docs/regression-history.md).
+
 ## Contributing
 
 Contributions are welcome! See [CONTRIBUTING.md](https://github.com/SnowFXXX/LocalAgentBrowser/blob/main/CONTRIBUTING.md) for setup and guidelines, and please follow
@@ -379,8 +455,10 @@ the [Code of Conduct](https://github.com/SnowFXXX/LocalAgentBrowser/blob/main/CO
 
 ## Project history
 
-Every version is a tagged commit, and [CHANGELOG.md](https://github.com/SnowFXXX/LocalAgentBrowser/blob/main/CHANGELOG.md) describes what changed in each one and
-why, with the measurements behind the decisions.
+Every released version is a tagged commit, and [CHANGELOG.md](https://github.com/SnowFXXX/LocalAgentBrowser/blob/main/CHANGELOG.md) describes what changed in each one and
+why, with the measurements behind the decisions. Versions 1.54 to 1.56 were internal steps, each measured on the
+regression suite and released together as 1.57; their results are in
+[docs/regression-history.md](https://github.com/SnowFXXX/LocalAgentBrowser/blob/main/docs/regression-history.md).
 
 ## Acknowledgements
 

{localagentbrowser-1.52 → localagentbrowser-1.57}/README.md

@@ -24,8 +24,8 @@ and tuned to work well with a **small 7B model on a CPU, no GPU required**.
 - ⚡ **Built for slow hardware.** It reuses LM Studio's prompt cache, so most turns process only what changed.
 - 📝 **A readable report every time.** `research_report.md` holds the final result and every finding with its
   source, written however the run ends; `research_log.txt` keeps every action and reason.
-- ✅ **Tested.** 112 tests on Linux and Windows, Python 3.10–3.13, run in about a second with no model or browser,
-  plus live runs on seven different kinds of task (see [results](#results-on-different-kinds-of-tasks)).
+- ✅ **Tested.** 165 tests on Linux and Windows, Python 3.10–3.13, run in a few seconds with no model or browser,
+  plus a [regression suite](#regression-suite) of live runs on seven different kinds of task.
 
 📖 **The story behind it:** [Teaching a 7B model on a CPU to finish what it starts](https://dev.to/snowfxxx/teaching-a-7b-model-on-a-cpu-to-finish-what-it-starts-1amc),
 six lessons from 30 versions of keeping a small local model on track.
@@ -161,7 +161,25 @@ does not generalise; the [changelog](https://github.com/SnowFXXX/LocalAgentBrows
 | List of items | Three free, open-source alternatives to Microsoft Office and what each is best at | 1.46, 1.51 | LibreOffice and OnlyOffice with their strengths, cited, in 18 turns (49 min); the third, FreeOffice, is free but not open source, and LibreOffice was recorded four times in different words; on 1.51: LibreOffice, OnlyOffice and Apache OpenOffice, all open source, in 7 turns (16 min) |
 | Step-by-step how-to | Make a sourdough starter, with quantities and what to do each day | 1.36–1.37 | A clear plan in 7 turns (19 min), but it invented the first water amount; 1.37 stops guessing and flags unsupported numbers, and the agent then went looking for the missing details |
 | Step-by-step how-to | Repot a houseplant: what you need, the steps, and care for the first two weeks | 1.43–1.44, 1.51 | 1.43 recorded six steps it wrote itself before reading the part that had them; 1.44 recorded the guide's real steps but took 68 min and its result left one out. 1.51: every step from the page, in 7 turns (29 min), and it says that no care instructions beyond the first watering were found |
-| Explanation of a topic | How photosynthesis works: inputs and outputs, both stages and where each takes place | 1.41–1.42 | A complete, cited explanation in 18 turns (45 min); 1.41 read four parts of a relevant page without recording them and had not finished after 49 min |
+| Explanation of a topic | How photosynthesis works: inputs and outputs, both stages and where each takes place | 1.41–1.42, 1.52 | A complete, cited explanation in 18 turns (45 min); 1.41 read four parts of a relevant page without recording them and had not finished after 49 min. 1.52 recorded both stages and where each takes place, but did not finish within 65 min: it read the history sections of a long encyclopedia article twice and never recorded the inputs and outputs |
+
+### Regression suite
+
+Since 1.53, every change is measured on the same seven objectives, one of each kind above, run one after the other
+with a time limit each (`tools/regression.py`, see [Testing](#testing)). Single runs vary with the pages a search
+turns up, so the suite compares versions on all seven at once. 1.54 to 1.56 were internal steps measured this way
+and released together as 1.57; [docs/regression-history.md](https://github.com/SnowFXXX/LocalAgentBrowser/blob/main/docs/regression-history.md)
+has every step's results and what each one showed.
+
+| Task | 1.52 | 1.53 (objective checklist) | 1.57 (coverage check) |
+|---|---|---|---|
+| Factual question | Finished, 12 min | Finished, 14 min | Finished, 13 min |
+| List of three office suites | Time limit | Finished, 30 min; but WPS Office is not open source | Time limit: three correct open-source suites by turn 7, sent back because no words showed them to be "free" |
+| Comparison and recommendation | Time limit | Finished, 39 min | Time limit, after a send-back for "official documentation", which the objective does not ask for |
+| How-to (cold brew from 1.55; repotting before) | Time limit | Time limit | Finished, 36 min; but "how long it keeps" answered with the steeping time |
+| Explanation (photosynthesis) | Time limit | Finished, 13 min, but thin: no oxygen or water | Finished, 59 min: inputs, outputs and both stages, cited |
+| Opinion (owls) | Time limit | Finished, 38 min | Time limit: its first page covered two of the three cultures asked for |
+| Documentation (venv, official docs) | Finished, 15 min, from a tutorial site | Time limit (30 min) | Finished, 37 min, all from python.org; but the Windows line repeats the create command instead of the activate one |
 
 ## How it works
 
@@ -200,7 +218,22 @@ flowchart LR
 - **Findings must match the page.** A finding that shares too few word pairs with the page text shown so far
   (or has a sentence that does) is sent back once, asking for only what the page says in its own words, and
   one far from anything on the page is refused outright. A small model will otherwise "complete" a list from
-  its own knowledge when a page says "follow these 6 steps" and the steps come later.
+  its own knowledge when a page says "follow these 6 steps" and the steps come later. The refusal points the way:
+  it quotes the one page sentence closest to what the model wrote, to record instead, or names what the parts
+  shown never mention ("Japan, Navajo"), so the model reads on.
+- **An objective checklist.** At the start, one short model call splits the objective into a fixed checklist of
+  what the result must contain, keeping conditions like "free, open-source" or "three". Each finding names the
+  items it serves, the checklist with its findings is in every prompt, and every check goes through these items
+  instead of asking the model to split the objective again from memory, which drifted: it once blocked its own
+  finish over "migration strategies", which the objective never asked for. Opinion items are shown as written
+  at FINISH, not as gaps to fill, and the first search covers the whole objective in one short query, keeping
+  its conditions ("official python documentation venv").
+- **A coverage check before finishing.** Each checklist item is checked against the findings in a short call of
+  its own, and the code checks the verdict instead of taking it on trust: an item nothing was copied for, fewer
+  things than asked for ("only 1 of 3 found"), and things not shown, in words copied from a finding about them,
+  to meet a condition such as "open-source". A claim that something is missing is dropped when every word of it is
+  already in the findings. With gaps, the FINISH is sent back once, with the next step named ("SEARCH for
+  'OnlyOffice free and open-source' to check it, or replace it"), and the next turn starts from a fresh prompt.
 - **Numbered findings.** Findings are numbered in the prompt and listed with every check, and the model names
   the finding that covers each part of the objective ("covered by [2]") instead of just saying "covered".
 - **Knowing when it's done.** After every new finding the model is asked whether the objective is now fully
@@ -212,12 +245,16 @@ flowchart LR
   A part that cannot be found after several tries can be reported as missing instead of blocking for good.
 - **Writing is a separate step.** When the agent finishes, one more model call writes the final result from
   only the objective and the recorded findings, under a short list of rules: no guessing or outside knowledge,
-  cite findings by number, give exactly as many items as asked for, and add a "My view:" paragraph only when
-  the objective asks for an opinion. A small model writes far better when that is its only job, and every
-  claim stays traceable to a recorded finding. The report flags any number in the final result that appears
-  in none of the findings, the most harmful kind of slip in a how-to.
+  cite findings by number, give as many items as asked for (never repeating one, and saying how many were not
+  found), and add a "My view:" paragraph only when the objective asks for an opinion. A small model writes far
+  better when that is its only job, and every claim stays traceable to a recorded finding. What no finding
+  answers is not left to the model: those checklist items, and things not shown to meet a condition, are left out
+  of what it is asked to write, and the code adds them after the result as "Not found in the pages read" and
+  "Left out, not shown to meet the conditions". The report flags any number in the final result that appears in
+  none of the findings, the most harmful kind of slip in a how-to.
 - **Error pages are recognised.** A page answering 404, 403 and so on counts as a failed visit, not content.
-- **Readable pages.** Navigation menus, footers and cookie banners are hidden; code blocks and tables are
+- **Readable pages.** Navigation menus, footers, cookie banners and site notices such as fundraising banners are
+  hidden; code blocks and tables are
   never split between parts; links already shown are not repeated. Long pages come with an outline of the
   headings in each part, marking the parts already read, so the agent can jump to the part it needs instead of
   reading in order.
@@ -250,11 +287,13 @@ Measured on an Intel i5-9500 (6 cores, no GPU) with Qwen2.5-7B-Instruct Q4_K_M a
 | Follow-up on the same page (finding recorded, cheap retry) | 35–60 s |
 | Next part of a page, or a new page appended | 100–280 s |
 | First turn of a run | 45–110 s |
-| Fresh prompt after the context budget is reached | 160–380 s (longest with long pages) |
+| Fresh prompt (context budget reached, or after a coverage send-back) | 160–540 s (longest with long pages) |
+| Coverage check before finishing (all checklist items) | about 1–3 min |
 | Writing the final result (once, at the end) | 50–190 s, depending on how much was found |
 
-Whole runs from the results table: a factual question in about 12 minutes of model time, a list of three items
-in 24, a comparison in 33. A GPU or a smaller model will be much faster.
+Whole runs vary with the pages a search turns up. In the regression runs since 1.53, a factual question took 13 to
+19 minutes of model time; the list, comparison, how-to, opinion and documentation tasks 21 to 44 when they finished;
+and the explanation 13 to 59. A GPU or a smaller model will be much faster.
 
 ## FAQ
 
@@ -289,9 +328,19 @@ turn that brings in a new page takes a few minutes. See [Performance](#performan
   research, an opinion from several sources, a factual question, a comparison, a how-to, a list and an
   explanation of a topic. Each one
   exposed something that did not generalise and was fixed; other kinds of tasks may expose more.
-- **Reading without recording.** The model used to read long pages part by part without recording the
-  relevant parts. Since 1.42 it must say what a part contains before reading on, which fixed this on an
-  explanation task; it is still the area most likely to need more work on other pages.
+- **Judging whether a finding covers an item is the weakest area.** The model's verdict is unreliable both ways:
+  on 1.53 the explanation finished from one overview paragraph without oxygen or water, and in the regression runs
+  since 1.54 about half of the coverage check's send-backs were for gaps that were not real ("missing official
+  documentation" on a comparison that did not ask for it, "H2O, O2" already recorded). The code now verifies what
+  it can (an item nothing answers, a count, word-for-word proof of a condition), but each wrong send-back still
+  costs a finish on this hardware, since a send-back costs 8–10 minutes.
+- **Conditions in the objective.** A small model tends to drop conditions such as "open source" or "using the
+  official documentation": a list of open-source office suites once included FreeOffice (free but not open
+  source), and a task that asked for the official Python documentation used a tutorial site. The objective
+  checklist keeps the conditions in view (the documentation task uses python.org since 1.55), and a thing not
+  shown in the findings to meet a condition is left out of the result and listed after it. The check is strict:
+  on 1.57 it sent back three correct open-source suites because no words showed them to be "free". Check the
+  result against any conditions you set.
 - **A small model's judgement is the ceiling.** It sometimes drifts to loosely related pages, prefers
   secondary sources or stays with one source too long, and results vary with the first pages a search turns
   up; every finding is in `research_report.md` regardless.
@@ -300,8 +349,9 @@ turn that brings in a new page takes a few minutes. See [Performance](#performan
   minutes, but with thin findings and without naming what it could not find, so check the result against the
   objective.
 - **Check the citations.** The final result is told to use only recorded findings, but a 7B model sometimes
-  puts the wrong number on a citation, adds a line no finding supports or leaves out a step, so verify a
-  claim against the listed findings before relying on it.
+  puts the wrong number on a citation, adds a line no finding supports, leaves out a step or misreads a part of
+  the question (on 1.57 it answered "how long it keeps in the fridge" with the steeping time), so verify a claim
+  against the listed findings before relying on it.
 - **Some sites refuse automated browsers** (HTTP 403). The agent respects that and does not try to disguise
   itself; it moves on to other sources.
 - **Reading only.** The agent searches, opens pages and reads them. It does not click, fill in forms or sign in.
@@ -316,6 +366,15 @@ turn that brings in a new page takes a few minutes. See [Performance](#performan
 - [x] Finish as soon as the objective is covered (1.35)
 - [x] Get the model to weigh each part of a long page before reading on (1.42)
 - [ ] Let very strict objectives end with the gaps named instead of searching indefinitely (they finish since 1.50; naming the gaps is still open)
+- [x] Measure every change on a regression suite of seven kinds of task (1.53)
+- [x] Keep a fixed checklist of the objective's parts instead of re-splitting it at every check (1.53)
+- [ ] Check that a finding really covers its checklist item, including conditions like "open source" (partly, since
+  1.57: a coverage check before finishing, with what the code can verify)
+- [ ] Send the agent back only for gaps the code verifies, and let the model's other verdicts only shape the final
+  result (the next step after 1.57)
+- [ ] Rewind dead ends: save the conversation before an expensive step such as reading a long page, and if it
+  leads nowhere, return to that point with a short note instead of carrying the unused page text in every later
+  prompt (LM Studio reuses its cache up to the saved point)
 - [ ] Try more kinds of objectives (seven so far) and adjust the rules that don't generalise
 - [ ] Make fresh prompts (the slowest kind of turn) cheaper still (31% cheaper since 1.34)
 - [ ] Interacting with pages (clicking, forms) for tasks beyond reading
@@ -324,7 +383,7 @@ Ideas and requests are welcome in [issues](https://github.com/SnowFXXX/LocalAgen
 
 ## Testing
 
-The test suite runs in about a second and needs neither LM Studio nor a browser: the model and the website
+The test suite runs in a few seconds and needs neither LM Studio nor a browser: the model and the website
 are replaced by scripted fakes, including an end-to-end run of the main loop.
 
 ```bash
@@ -340,6 +399,23 @@ call:
 python tools/live_test.py "Your objective here" --minutes 25
 ```
 
+To measure a change on all seven kinds of task (about four hours on a CPU), compared with an earlier run:
+
+```bash
+python tools/regression.py --baseline regression_runs/<earlier run>/summary.json
+```
+
+Each run gets a folder in `regression_runs/` with every task's report, journal, console output and prompt
+trace, and a `summary.md` that is updated after each task: whether it finished, turns, model time, how many
+checklist items got findings, which checks fired, and problems found by simple word checks (missing answers,
+missed conditions such as a non-open-source item, uncited paragraphs, numbers missing from the findings, a
+missing opinion). It also flags a task that fails again on the same site as in the earlier run: one site that
+cannot answer part of an objective can decide a task whatever the agent does (the repotting how-to went to the same
+guide in three runs, which is why the core how-to is now the cold brew one). `--suite all` adds six more tasks,
+including the earlier how-to, `--only` runs some of them, and `--rescore` scores an earlier run again without the
+model. Every step since 1.53 and its results are in
+[docs/regression-history.md](https://github.com/SnowFXXX/LocalAgentBrowser/blob/main/docs/regression-history.md).
+
 ## Contributing
 
 Contributions are welcome! See [CONTRIBUTING.md](https://github.com/SnowFXXX/LocalAgentBrowser/blob/main/CONTRIBUTING.md) for setup and guidelines, and please follow
@@ -347,8 +423,10 @@ the [Code of Conduct](https://github.com/SnowFXXX/LocalAgentBrowser/blob/main/CO
 
 ## Project history
 
-Every version is a tagged commit, and [CHANGELOG.md](https://github.com/SnowFXXX/LocalAgentBrowser/blob/main/CHANGELOG.md) describes what changed in each one and
-why, with the measurements behind the decisions.
+Every released version is a tagged commit, and [CHANGELOG.md](https://github.com/SnowFXXX/LocalAgentBrowser/blob/main/CHANGELOG.md) describes what changed in each one and
+why, with the measurements behind the decisions. Versions 1.54 to 1.56 were internal steps, each measured on the
+regression suite and released together as 1.57; their results are in
+[docs/regression-history.md](https://github.com/SnowFXXX/LocalAgentBrowser/blob/main/docs/regression-history.md).
 
 ## Acknowledgements