localagentbrowser 1.47__tar.gz → 1.53__tar.gz

{localagentbrowser-1.47 → localagentbrowser-1.53}/CHANGELOG.md

@@ -1,9 +1,132 @@
 # Changelog
 
-Every version of `agent_browser.py`, newest first. Each version is also a tagged commit (`v1.11` … `v1.28`).
+Every version of `agent_browser.py`, newest first. Each version is also a tagged commit (`v1.11` … `v1.53`).
 Timings are from live runs on the development machine (Intel i5-9500, 6 cores, no GPU, Qwen2.5-7B-Instruct
 Q4_K_M in LM Studio), where reading the prompt takes about 90% of every model call.
 
+## [1.53] - 2026-10-17
+### Added
+- Regression suite (`tools/regression.py`): seven objectives, one of each kind of task tested so far, run one
+  after the other with a time limit each and scored the same way (finished or not, turns, model time, checklist
+  progress, checks fired, and word checks for missing answers, missed conditions, uncited paragraphs, numbers
+  missing from the findings and a missing opinion), with a comparison to an earlier run. Single runs had been
+  flattering: on 1.52 the suite finished only 2 of 7 tasks within their limits, including tasks earlier single
+  runs had finished. Refused actions in the journal now name the check that refused them, so the suite can count
+  each check, and the live test runner times every model call and saves the model's full replies.
+- Objective checklist. The 1.52 suite showed the main failure: every check asked the model to split the
+  objective into parts again, from memory, and the split drifted. On the list task it blocked its own FINISH
+  over "migration strategies", which the objective never asked for (an idea from a page it had read), and never
+  looked for a third suite; on the comparison it added "data types" and never covered "many simultaneous users".
+  Now one short call at the start splits the objective into a fixed checklist; asked on its own, the model split
+  all seven test objectives cleanly and kept their conditions. Each DISCOVER names the items it serves
+  (`checklist_items` in the reply), the checklist with its discoveries is in fresh prompts and in the note after
+  each discovery, the finish, progress and objective checks go through its items ("do not add other parts"),
+  the read-on check names the items still open, the final result covers every item or says it was not found,
+  and the report lists the checklist. The split call uses the same system prompt as turn 1, so LM Studio's cache
+  of it still applies there (without that, turn 1 took 128 s instead of about 40).
+### Fixed
+- The report flagged the numbers of numbered headings ("#### 1. Setup") as numbers missing from the findings.
+
+Regression run against 1.52 (model minutes in brackets): 5 of 7 tasks finished instead of 2, and the total model
+time fell from 254 to 200 minutes. The list (30.1, was a 45-minute time limit), comparison (39.0, every part
+covered), explanation (12.9, was 60 without finishing) and opinion (38.1) tasks now finish; the factual question
+still does (13.6, was 11.9). What is not better: the list names WPS Office, which is not open source; the
+explanation finished from a single overview paragraph that the model claimed covered all four items, without
+oxygen or water; the how-to found the materials and steps but not the aftercare before its time limit; and the
+documentation task now went to python.org, as asked, but worked through the items one search at a time and hit
+its 30-minute limit before the last one (it had finished in 15 minutes on 1.52, from a tutorial site).
+
+## [1.52] - 2026-10-16
+### Changed
+- The note after each DISCOVER also says to work on parts of the objective that are not covered yet before
+  adding detail to parts that are. On the explanation task (how photosynthesis works), 1.51 recorded the
+  Calvin cycle enzyme by enzyme while the light reactions and the inputs and outputs were still missing.
+
+Rerun: after two Calvin cycle discoveries the agent left for a page on the light reactions and recorded where
+each stage happens and what the light reactions do. It still did not finish within 65 minutes: it lost about
+20 minutes reading the history and evolution sections of a long encyclopedia article twice, answering "none"
+to each read-on check, and never recorded the inputs and outputs that article's first part gives.
+
+Regression runs on 1.51 for the other kinds of task: the list task (three free, open-source alternatives to
+Microsoft Office) finished in 7 turns and 15.5 minutes of model time (49.3 on 1.46), naming LibreOffice,
+OnlyOffice and Apache OpenOffice, all open source; its final result wrongly says FreeOffice was "not
+mentioned" in the discoveries while leaving it out.
+
+## [1.51] - 2026-10-16
+### Fixed
+- On the repotting how-to, 1.50 sent back the model's invented steps once (part 2 still only said "follow
+  these 6 easy steps"), the model sent nearly the same steps again, and they went through as a confirmed
+  repeat. A DISCOVER far below anything real is now refused outright, and repeating it does not help: below
+  40% of its word pairs on the page text shown, or with a sentence below 25%. Real discoveries in the
+  earlier runs scored at least 50% overall and 40% per sentence; these invented steps scored 32%, one of
+  them 9%. Between those levels and 70% / 35%, the one-time confirmation stays.
+
+Rerun: the invented steps were refused, the agent read on and recorded the guide's real materials and steps,
+and FINISH came on turn 7 after 28.5 minutes of model time (67.5 on 1.44). The final result has every step,
+including putting the plant in the new pot, and says that no care instructions beyond the first watering
+were found. Regression on the comparison task with 1.50: 8 turns and 24.9 minutes of model time (38.2 on
+1.42), all parts covered and cited, with a recommendation.
+
+## [1.50] - 2026-10-16
+Three runs of the strict documentation objective (Playwright test agents, "do not FINISH until ... fully
+validated", including "parameter schemas" the documentation does not have).
+### Changed
+- `READ_MORE` with the number of the part already shown reads the next part. The model did this in most
+  long-page runs, meaning the next part, and refusing it cost a retry each time. On the last part it is still
+  refused.
+- The read-on check also says to leave a page that is not about the objective: shown an unrelated "agent"
+  project's command reference, the model read on part after part.
+### Fixed
+- After being told twice that a search was exhausted and that it could finish with the gap named, the model
+  tried FINISH, the finish check sent it to work on the uncovered part, and it repeated the search again (five
+  times in one run). Once a search is exhausted, FINISH goes through without the finish and gap checks.
+
+Third run: FINISH on turn 10 (25.2 minutes of model time), the first time this objective has finished; earlier
+versions searched for over an hour. The result has the official `npx playwright init-agents --loop=vscode`
+command and the file layout from the official docs, but it is thin: the finish check was answered with
+"covered by [1] covered by [2] covered by [3]" without naming the parts, one discovery was off topic, and the
+result does not say that no parameter schemas were found.
+
+## [1.49] - 2026-10-16
+A documentation task on 1.48 (create a Python virtual environment with venv, activate it on Windows and on
+macOS or Linux, and install packages) garbled commands in two places.
+### Fixed
+- Recording the Windows command `python -m venv C:\path\to\new\virtual\environment`, the model wrote single
+  backslashes in its JSON reply: `\t`, `\n` and `\r` became a tab, a newline and a carriage return, and since
+  the response format does not allow `\v`, the model turned `\virtual\environment` into `\real\name`. The
+  system prompt now asks for every backslash to be written twice, and a tab, carriage return, backspace or
+  form feed right before a letter is turned back into its backslash sequence when the text also has a
+  backslash or a drive letter (page tables keep their real tabs).
+- The final result turned the recorded `.\venv\Scripts\activate` and `source venv/bin/activate` into
+  `.venv\Scripts\activate` and `source .venv/bin/activate`, which do not match the `python -m venv venv` it
+  gave first. Rule 1 of the final result now says to copy commands, code, file paths and names exactly.
+  Replaying that step three times on 1.48 changed the commands twice; with the rule, all three were exact.
+
+Rerun with the backslash changes: FINISH on turn 6 (15.1 minutes of model time) with the create, activate
+and install commands, the Windows path intact. It used a tutorial site rather than the official
+documentation the objective named; checking conditions like that is still a weakness.
+
+## [1.48] - 2026-10-16
+An opinion task on 1.47 (how at least three cultures or religions have viewed owls, then an opinion on why
+they attract such different meanings) showed three problems.
+### Fixed
+- A discovery copied two sentences from the page and added a third the page never said ("...depicted
+  alongside other wise figures like the prophet Samuel"); copied text lifted the whole discovery to 75%, past
+  the page-match check. Each sentence of 8 or more word pairs is now also scored, and one below 35% gets the
+  same cheap retry. Measured on the earlier runs' real discoveries, sentences scored 40-100%; invented ones
+  0-41%.
+- With three cultures recorded where three were asked for, the read-on check pushed the agent to keep
+  collecting. It now also says to choose FINISH if the discoveries already cover every part of the objective.
+- The final result listed many traditions but gave no opinion, although the objective asked for one. When the
+  objective asks for an opinion, view or recommendation and the result has no "My view:" paragraph, a short
+  second call writes only that paragraph from the result.
+- The report's number check treated citation ranges like "[1-4]" as numbers.
+
+Rerun with the sentence and finish changes: FINISH on turn 11 (37.9 minutes of model time) with the views of
+Greek, English and many Native American peoples from a detailed source, all discoveries matching the page. Its
+three page-match checks were condensed summaries of real sentences; the model repeated each and it went
+through, at 37-61 s per check. Replaying the final-result step on that run added a "My view:" paragraph.
+
 ## [1.47] - 2026-10-16
 A list task on 1.46 (three free, open-source alternatives to Microsoft Office and what each is best at)
 finished on its own in 18 turns and 49 minutes of model time. LibreOffice and OnlyOffice were right, but the

{localagentbrowser-1.47 → localagentbrowser-1.53}/MANIFEST.in

@@ -1,3 +1,4 @@
-# Files the source package needs beyond the defaults, so its tests can run
+# Files the source package needs beyond the defaults, so its tests (and the tools they test) can run
 include CHANGELOG.md
 recursive-include tests *.py
+recursive-include tools *.py

{localagentbrowser-1.47/localagentbrowser.egg-info → localagentbrowser-1.53}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: localagentbrowser
-Version: 1.47
+Version: 1.53
 Summary: An autonomous web agent driven by a local LLM (LM Studio) and Playwright. Runs entirely on your own machine.
 Author: SnowFX
 License-Expression: MIT
@@ -56,8 +56,8 @@ and tuned to work well with a **small 7B model on a CPU, no GPU required**.
 - ⚡ **Built for slow hardware.** It reuses LM Studio's prompt cache, so most turns process only what changed.
 - 📝 **A readable report every time.** `research_report.md` holds the final result and every finding with its
   source, written however the run ends; `research_log.txt` keeps every action and reason.
-- ✅ **Tested.** 107 tests on Linux and Windows, Python 3.10–3.13, run in about a second with no model or browser,
-  plus live runs on seven different kinds of task (see [results](#results-on-different-kinds-of-tasks)).
+- ✅ **Tested.** 136 tests on Linux and Windows, Python 3.10–3.13, run in about a second with no model or browser,
+  plus a [regression suite](#regression-suite) of live runs on seven different kinds of task.
 
 📖 **The story behind it:** [Teaching a 7B model on a CPU to finish what it starts](https://dev.to/snowfxxx/teaching-a-7b-model-on-a-cpu-to-finish-what-it-starts-1amc),
 six lessons from 30 versions of keeping a small local model on track.
@@ -184,15 +184,32 @@ does not generalise; the [changelog](https://github.com/SnowFXXX/LocalAgentBrows
 | Kind of task | Objective (abridged) | Version | Outcome |
 |---|---|---|---|
 | Documentation research | Setup commands and file layout for Playwright's test agents | 1.27 | Finished on its own in 13 turns (~36 min) with commands, roles and file layout |
-| Documentation research, strict | The same, but "do not FINISH until fully validated", including "parameter schemas" | 1.38–1.40 | Its most detailed findings (per-client setup, seed test, agent definition files), but it keeps searching for schemas the docs do not contain and has not finished within an hour |
+| Documentation research, strict | The same, but "do not FINISH until fully validated", including "parameter schemas" | 1.38–1.40, 1.50 | 1.38–1.40: its most detailed findings (per-client setup, seed test, agent definition files), but it kept searching for schemas the docs do not contain and did not finish within an hour. 1.50: finished in 10 turns (25 min) with the official init command and file layout, but with thin findings and without saying the schemas were not found |
 | Opinion from several sources | How spiritual traditions view cats, then form an opinion | 1.30 | Six traditions from two sources and a stated opinion, in 18 turns (37 min of model time) |
+| Opinion from several sources | How at least three cultures or religions have viewed owls, then an opinion on why | 1.47–1.48 | Greek, English and many Native American views from one detailed source in 11 turns (38 min); 1.48 adds the opinion the result had left out and catches an invented sentence in a discovery |
 | Factual question | Tallest mountain in Africa, height in meters and feet | 1.35, 1.40, 1.46 | Correct, cited answer in 4 turns (about 12 min of model time) in all three versions; before 1.35 it kept reading after finding it |
-| Comparison and recommendation | Python vs Rust for command-line tools; recommend one for a beginner. Later: SQLite vs PostgreSQL for a small web app, then a recommendation for a hobby project | 1.36, 1.42 | Python vs Rust: cited comparison recommending Python in 16 turns (33 min), Rust's libraries not covered. SQLite vs PostgreSQL: all three parts covered and cited, with a recommendation, in 13 turns (38 min) |
+| Comparison and recommendation | Python vs Rust for command-line tools; recommend one for a beginner. Later: SQLite vs PostgreSQL for a small web app, then a recommendation for a hobby project | 1.36, 1.42, 1.50 | Python vs Rust: cited comparison recommending Python in 16 turns (33 min), Rust's libraries not covered. SQLite vs PostgreSQL: all three parts covered and cited, with a recommendation, in 13 turns (38 min); on 1.50 in 8 turns (25 min) |
 | List of items | Three free, open-source Photoshop alternatives and the strengths of each | 1.37–1.38 | GIMP, Krita and Darktable with their strengths, all cited, in 9 turns (24 min); 1.37 listed a fourth and added an unrequested opinion, which 1.38 fixes |
-| List of items | Three free, open-source alternatives to Microsoft Office and what each is best at | 1.46 | LibreOffice and OnlyOffice with their strengths, cited, in 18 turns (49 min); the third, FreeOffice, is free but not open source, and LibreOffice was recorded four times in different words |
+| List of items | Three free, open-source alternatives to Microsoft Office and what each is best at | 1.46, 1.51 | LibreOffice and OnlyOffice with their strengths, cited, in 18 turns (49 min); the third, FreeOffice, is free but not open source, and LibreOffice was recorded four times in different words; on 1.51: LibreOffice, OnlyOffice and Apache OpenOffice, all open source, in 7 turns (16 min) |
 | Step-by-step how-to | Make a sourdough starter, with quantities and what to do each day | 1.36–1.37 | A clear plan in 7 turns (19 min), but it invented the first water amount; 1.37 stops guessing and flags unsupported numbers, and the agent then went looking for the missing details |
-| Step-by-step how-to | Repot a houseplant: what you need, the steps, and care for the first two weeks | 1.43–1.44 | 1.43 recorded six steps it wrote itself before reading the part that had them; 1.44 checks findings against the page and recorded the guide's real materials, steps and aftercare, finishing in 24 turns (68 min), though the final result left out one step |
-| Explanation of a topic | How photosynthesis works: inputs and outputs, both stages and where each takes place | 1.41–1.42 | A complete, cited explanation in 18 turns (45 min); 1.41 read four parts of a relevant page without recording them and had not finished after 49 min |
+| Step-by-step how-to | Repot a houseplant: what you need, the steps, and care for the first two weeks | 1.43–1.44, 1.51 | 1.43 recorded six steps it wrote itself before reading the part that had them; 1.44 recorded the guide's real steps but took 68 min and its result left one out. 1.51: every step from the page, in 7 turns (29 min), and it says that no care instructions beyond the first watering were found |
+| Explanation of a topic | How photosynthesis works: inputs and outputs, both stages and where each takes place | 1.41–1.42, 1.52 | A complete, cited explanation in 18 turns (45 min); 1.41 read four parts of a relevant page without recording them and had not finished after 49 min. 1.52 recorded both stages and where each takes place, but did not finish within 65 min: it read the history sections of a long encyclopedia article twice and never recorded the inputs and outputs |
+
+### Regression suite
+
+Since 1.53, every change is measured on the same seven objectives, one of each kind above, run one after the other
+with a time limit each (`tools/regression.py`, see [Testing](#testing)). Single runs vary with the pages a search
+turns up, so the suite compares versions on all seven at once:
+
+| Task | 1.52 | 1.53 (objective checklist) |
+|---|---|---|
+| Factual question | Finished, 12 min | Finished, 14 min |
+| List of three office suites | Time limit (45 min) | Finished, 30 min; but one of the three (WPS Office) is not open source |
+| Comparison and recommendation | Time limit (45 min) | Finished, 39 min, every part covered |
+| How-to (repotting) | Time limit (45 min) | Time limit (45 min); materials and steps found, aftercare not |
+| Explanation (photosynthesis) | Time limit (60 min) | Finished, 13 min, but thin: one finding, no oxygen or water |
+| Opinion (owls) | Time limit (45 min) | Finished, 38 min, four cultures and an opinion |
+| Documentation (venv, official docs) | Finished, 15 min, from a tutorial site | Time limit (30 min); create and activate found, from python.org |
 
 ## How it works
 
@@ -229,8 +246,14 @@ flowchart LR
   check: record something, or repeat the action to leave. Moving on to the next part of a page works the same
   way, and the model must first list what the part it is on says about the objective (or "none").
 - **Findings must match the page.** A finding that shares too few word pairs with the page text shown so far
-  is sent back once, asking for only what the page says in its own words: a small model will otherwise
-  "complete" a list from its own knowledge when a page says "follow these 6 steps" and the steps come later.
+  (or has a sentence that does) is sent back once, asking for only what the page says in its own words, and
+  one far from anything on the page is refused outright. A small model will otherwise "complete" a list from
+  its own knowledge when a page says "follow these 6 steps" and the steps come later.
+- **An objective checklist.** At the start, one short model call splits the objective into a fixed checklist of
+  what the result must contain, keeping conditions like "free, open-source" or "three". Each finding names the
+  items it serves, the checklist with its findings is in every prompt, and every check goes through these items
+  instead of asking the model to split the objective again from memory, which drifted: it once blocked its own
+  finish over "migration strategies", which the objective never asked for.
 - **Numbered findings.** Findings are numbered in the prompt and listed with every check, and the model names
   the finding that covers each part of the objective ("covered by [2]") instead of just saying "covered".
 - **Knowing when it's done.** After every new finding the model is asked whether the objective is now fully
@@ -283,8 +306,9 @@ Measured on an Intel i5-9500 (6 cores, no GPU) with Qwen2.5-7B-Instruct Q4_K_M a
 | Fresh prompt after the context budget is reached | 160–380 s (longest with long pages) |
 | Writing the final result (once, at the end) | 50–190 s, depending on how much was found |
 
-Whole runs from the results table: a factual question in about 12 minutes of model time, a list of three items
-in 24, a comparison in 33. A GPU or a smaller model will be much faster.
+Whole runs vary with the pages a search turns up. In the results table, a factual question takes about 12
+minutes of model time, a list of three items 16 to 49, a comparison 25 to 38 and an explanation 45 or more. A
+GPU or a smaller model will be much faster.
 
 ## FAQ
 
@@ -319,16 +343,25 @@ turn that brings in a new page takes a few minutes. See [Performance](#performan
   research, an opinion from several sources, a factual question, a comparison, a how-to, a list and an
   explanation of a topic. Each one
   exposed something that did not generalise and was fixed; other kinds of tasks may expose more.
-- **Reading without recording.** The model used to read long pages part by part without recording the
-  relevant parts. Since 1.42 it must say what a part contains before reading on, which fixed this on an
-  explanation task; it is still the area most likely to need more work on other pages.
+- **Reading long pages and knowing when it has enough.** The model used to read long pages part by part
+  without recording the relevant parts; since 1.42 it must say what a part contains before reading on. On 1.52
+  the explanation task still lost about 20 minutes reading unrelated sections of a long article and did not
+  finish. On 1.53, with the objective checklist, it finished in 13 minutes, but from a single overview paragraph
+  that left out oxygen and water: the model can now claim one finding covers every item. Judging whether a
+  finding really covers an item is the weakest area.
+- **Conditions in the objective.** A small model tends to drop conditions such as "open source" or "using the
+  official documentation": a list of open-source office suites once included FreeOffice (free but not open
+  source), and a task that asked for the official Python documentation used a tutorial site. The objective
+  checklist keeps the conditions in view (the documentation task went to python.org on 1.53), but on the same
+  run the list task still named WPS Office, which is not open source. Check the result against any conditions
+  you set.
 - **A small model's judgement is the ceiling.** It sometimes drifts to loosely related pages, prefers
   secondary sources or stays with one source too long, and results vary with the first pages a search turns
   up; every finding is in `research_report.md` regardless.
-- **Very strict objectives can keep it searching.** An objective that forbids finishing until everything is
-  "fully validated" can keep the agent looking for something the web may not contain. It is told to finish
-  and name what it could not find after looking in several places, but a 7B model does not always judge
-  when that point has come; the turn limit (100) and Ctrl+C still apply, and the report is always current.
+- **Very strict objectives.** An objective that forbids finishing until everything is "fully validated" used
+  to keep the agent looking for something the web may not contain. On 1.50 it finished such an objective in 25
+  minutes, but with thin findings and without naming what it could not find, so check the result against the
+  objective.
 - **Check the citations.** The final result is told to use only recorded findings, but a 7B model sometimes
   puts the wrong number on a citation, adds a line no finding supports or leaves out a step, so verify a
   claim against the listed findings before relying on it.
@@ -345,7 +378,10 @@ turn that brings in a new page takes a few minutes. See [Performance](#performan
 - [x] Write the final result in a separate, rule-bound step and flag unsupported numbers (1.32–1.38)
 - [x] Finish as soon as the objective is covered (1.35)
 - [x] Get the model to weigh each part of a long page before reading on (1.42)
-- [ ] Let very strict objectives end with the gaps named instead of searching indefinitely
+- [ ] Let very strict objectives end with the gaps named instead of searching indefinitely (they finish since 1.50; naming the gaps is still open)
+- [x] Measure every change on a regression suite of seven kinds of task (1.53)
+- [x] Keep a fixed checklist of the objective's parts instead of re-splitting it at every check (1.53)
+- [ ] Check that a finding really covers its checklist item, including conditions like "open source"
 - [ ] Try more kinds of objectives (seven so far) and adjust the rules that don't generalise
 - [ ] Make fresh prompts (the slowest kind of turn) cheaper still (31% cheaper since 1.34)
 - [ ] Interacting with pages (clicking, forms) for tasks beyond reading
@@ -370,6 +406,19 @@ call:
 python tools/live_test.py "Your objective here" --minutes 25
 ```
 
+To measure a change on all seven kinds of task (about four hours on a CPU), compared with an earlier run:
+
+```bash
+python tools/regression.py --baseline regression_runs/<earlier run>/summary.json
+```
+
+Each run gets a folder in `regression_runs/` with every task's report, journal, console output and prompt
+trace, and a `summary.md` that is updated after each task: whether it finished, turns, model time, how many
+checklist items got findings, which checks fired, and problems found by simple word checks (missing answers,
+missed conditions such as a non-open-source item, uncited paragraphs, numbers missing from the findings, a
+missing opinion). `--suite all` adds five more tasks, `--only` runs some of them, and `--rescore` scores an
+earlier run again without the model.
+
 ## Contributing
 
 Contributions are welcome! See [CONTRIBUTING.md](https://github.com/SnowFXXX/LocalAgentBrowser/blob/main/CONTRIBUTING.md) for setup and guidelines, and please follow

{localagentbrowser-1.47 → localagentbrowser-1.53}/README.md

@@ -24,8 +24,8 @@ and tuned to work well with a **small 7B model on a CPU, no GPU required**.
 - ⚡ **Built for slow hardware.** It reuses LM Studio's prompt cache, so most turns process only what changed.
 - 📝 **A readable report every time.** `research_report.md` holds the final result and every finding with its
   source, written however the run ends; `research_log.txt` keeps every action and reason.
-- ✅ **Tested.** 107 tests on Linux and Windows, Python 3.10–3.13, run in about a second with no model or browser,
-  plus live runs on seven different kinds of task (see [results](#results-on-different-kinds-of-tasks)).
+- ✅ **Tested.** 136 tests on Linux and Windows, Python 3.10–3.13, run in about a second with no model or browser,
+  plus a [regression suite](#regression-suite) of live runs on seven different kinds of task.
 
 📖 **The story behind it:** [Teaching a 7B model on a CPU to finish what it starts](https://dev.to/snowfxxx/teaching-a-7b-model-on-a-cpu-to-finish-what-it-starts-1amc),
 six lessons from 30 versions of keeping a small local model on track.
@@ -152,15 +152,32 @@ does not generalise; the [changelog](https://github.com/SnowFXXX/LocalAgentBrows
 | Kind of task | Objective (abridged) | Version | Outcome |
 |---|---|---|---|
 | Documentation research | Setup commands and file layout for Playwright's test agents | 1.27 | Finished on its own in 13 turns (~36 min) with commands, roles and file layout |
-| Documentation research, strict | The same, but "do not FINISH until fully validated", including "parameter schemas" | 1.38–1.40 | Its most detailed findings (per-client setup, seed test, agent definition files), but it keeps searching for schemas the docs do not contain and has not finished within an hour |
+| Documentation research, strict | The same, but "do not FINISH until fully validated", including "parameter schemas" | 1.38–1.40, 1.50 | 1.38–1.40: its most detailed findings (per-client setup, seed test, agent definition files), but it kept searching for schemas the docs do not contain and did not finish within an hour. 1.50: finished in 10 turns (25 min) with the official init command and file layout, but with thin findings and without saying the schemas were not found |
 | Opinion from several sources | How spiritual traditions view cats, then form an opinion | 1.30 | Six traditions from two sources and a stated opinion, in 18 turns (37 min of model time) |
+| Opinion from several sources | How at least three cultures or religions have viewed owls, then an opinion on why | 1.47–1.48 | Greek, English and many Native American views from one detailed source in 11 turns (38 min); 1.48 adds the opinion the result had left out and catches an invented sentence in a discovery |
 | Factual question | Tallest mountain in Africa, height in meters and feet | 1.35, 1.40, 1.46 | Correct, cited answer in 4 turns (about 12 min of model time) in all three versions; before 1.35 it kept reading after finding it |
-| Comparison and recommendation | Python vs Rust for command-line tools; recommend one for a beginner. Later: SQLite vs PostgreSQL for a small web app, then a recommendation for a hobby project | 1.36, 1.42 | Python vs Rust: cited comparison recommending Python in 16 turns (33 min), Rust's libraries not covered. SQLite vs PostgreSQL: all three parts covered and cited, with a recommendation, in 13 turns (38 min) |
+| Comparison and recommendation | Python vs Rust for command-line tools; recommend one for a beginner. Later: SQLite vs PostgreSQL for a small web app, then a recommendation for a hobby project | 1.36, 1.42, 1.50 | Python vs Rust: cited comparison recommending Python in 16 turns (33 min), Rust's libraries not covered. SQLite vs PostgreSQL: all three parts covered and cited, with a recommendation, in 13 turns (38 min); on 1.50 in 8 turns (25 min) |
 | List of items | Three free, open-source Photoshop alternatives and the strengths of each | 1.37–1.38 | GIMP, Krita and Darktable with their strengths, all cited, in 9 turns (24 min); 1.37 listed a fourth and added an unrequested opinion, which 1.38 fixes |
-| List of items | Three free, open-source alternatives to Microsoft Office and what each is best at | 1.46 | LibreOffice and OnlyOffice with their strengths, cited, in 18 turns (49 min); the third, FreeOffice, is free but not open source, and LibreOffice was recorded four times in different words |
+| List of items | Three free, open-source alternatives to Microsoft Office and what each is best at | 1.46, 1.51 | LibreOffice and OnlyOffice with their strengths, cited, in 18 turns (49 min); the third, FreeOffice, is free but not open source, and LibreOffice was recorded four times in different words; on 1.51: LibreOffice, OnlyOffice and Apache OpenOffice, all open source, in 7 turns (16 min) |
 | Step-by-step how-to | Make a sourdough starter, with quantities and what to do each day | 1.36–1.37 | A clear plan in 7 turns (19 min), but it invented the first water amount; 1.37 stops guessing and flags unsupported numbers, and the agent then went looking for the missing details |
-| Step-by-step how-to | Repot a houseplant: what you need, the steps, and care for the first two weeks | 1.43–1.44 | 1.43 recorded six steps it wrote itself before reading the part that had them; 1.44 checks findings against the page and recorded the guide's real materials, steps and aftercare, finishing in 24 turns (68 min), though the final result left out one step |
-| Explanation of a topic | How photosynthesis works: inputs and outputs, both stages and where each takes place | 1.41–1.42 | A complete, cited explanation in 18 turns (45 min); 1.41 read four parts of a relevant page without recording them and had not finished after 49 min |
+| Step-by-step how-to | Repot a houseplant: what you need, the steps, and care for the first two weeks | 1.43–1.44, 1.51 | 1.43 recorded six steps it wrote itself before reading the part that had them; 1.44 recorded the guide's real steps but took 68 min and its result left one out. 1.51: every step from the page, in 7 turns (29 min), and it says that no care instructions beyond the first watering were found |
+| Explanation of a topic | How photosynthesis works: inputs and outputs, both stages and where each takes place | 1.41–1.42, 1.52 | A complete, cited explanation in 18 turns (45 min); 1.41 read four parts of a relevant page without recording them and had not finished after 49 min. 1.52 recorded both stages and where each takes place, but did not finish within 65 min: it read the history sections of a long encyclopedia article twice and never recorded the inputs and outputs |
+
+### Regression suite
+
+Since 1.53, every change is measured on the same seven objectives, one of each kind above, run one after the other
+with a time limit each (`tools/regression.py`, see [Testing](#testing)). Single runs vary with the pages a search
+turns up, so the suite compares versions on all seven at once:
+
+| Task | 1.52 | 1.53 (objective checklist) |
+|---|---|---|
+| Factual question | Finished, 12 min | Finished, 14 min |
+| List of three office suites | Time limit (45 min) | Finished, 30 min; but one of the three (WPS Office) is not open source |
+| Comparison and recommendation | Time limit (45 min) | Finished, 39 min, every part covered |
+| How-to (repotting) | Time limit (45 min) | Time limit (45 min); materials and steps found, aftercare not |
+| Explanation (photosynthesis) | Time limit (60 min) | Finished, 13 min, but thin: one finding, no oxygen or water |
+| Opinion (owls) | Time limit (45 min) | Finished, 38 min, four cultures and an opinion |
+| Documentation (venv, official docs) | Finished, 15 min, from a tutorial site | Time limit (30 min); create and activate found, from python.org |
 
 ## How it works
 
@@ -197,8 +214,14 @@ flowchart LR
   check: record something, or repeat the action to leave. Moving on to the next part of a page works the same
   way, and the model must first list what the part it is on says about the objective (or "none").
 - **Findings must match the page.** A finding that shares too few word pairs with the page text shown so far
-  is sent back once, asking for only what the page says in its own words: a small model will otherwise
-  "complete" a list from its own knowledge when a page says "follow these 6 steps" and the steps come later.
+  (or has a sentence that does) is sent back once, asking for only what the page says in its own words, and
+  one far from anything on the page is refused outright. A small model will otherwise "complete" a list from
+  its own knowledge when a page says "follow these 6 steps" and the steps come later.
+- **An objective checklist.** At the start, one short model call splits the objective into a fixed checklist of
+  what the result must contain, keeping conditions like "free, open-source" or "three". Each finding names the
+  items it serves, the checklist with its findings is in every prompt, and every check goes through these items
+  instead of asking the model to split the objective again from memory, which drifted: it once blocked its own
+  finish over "migration strategies", which the objective never asked for.
 - **Numbered findings.** Findings are numbered in the prompt and listed with every check, and the model names
   the finding that covers each part of the objective ("covered by [2]") instead of just saying "covered".
 - **Knowing when it's done.** After every new finding the model is asked whether the objective is now fully
@@ -251,8 +274,9 @@ Measured on an Intel i5-9500 (6 cores, no GPU) with Qwen2.5-7B-Instruct Q4_K_M a
 | Fresh prompt after the context budget is reached | 160–380 s (longest with long pages) |
 | Writing the final result (once, at the end) | 50–190 s, depending on how much was found |
 
-Whole runs from the results table: a factual question in about 12 minutes of model time, a list of three items
-in 24, a comparison in 33. A GPU or a smaller model will be much faster.
+Whole runs vary with the pages a search turns up. In the results table, a factual question takes about 12
+minutes of model time, a list of three items 16 to 49, a comparison 25 to 38 and an explanation 45 or more. A
+GPU or a smaller model will be much faster.
 
 ## FAQ
 
@@ -287,16 +311,25 @@ turn that brings in a new page takes a few minutes. See [Performance](#performan
   research, an opinion from several sources, a factual question, a comparison, a how-to, a list and an
   explanation of a topic. Each one
   exposed something that did not generalise and was fixed; other kinds of tasks may expose more.
-- **Reading without recording.** The model used to read long pages part by part without recording the
-  relevant parts. Since 1.42 it must say what a part contains before reading on, which fixed this on an
-  explanation task; it is still the area most likely to need more work on other pages.
+- **Reading long pages and knowing when it has enough.** The model used to read long pages part by part
+  without recording the relevant parts; since 1.42 it must say what a part contains before reading on. On 1.52
+  the explanation task still lost about 20 minutes reading unrelated sections of a long article and did not
+  finish. On 1.53, with the objective checklist, it finished in 13 minutes, but from a single overview paragraph
+  that left out oxygen and water: the model can now claim one finding covers every item. Judging whether a
+  finding really covers an item is the weakest area.
+- **Conditions in the objective.** A small model tends to drop conditions such as "open source" or "using the
+  official documentation": a list of open-source office suites once included FreeOffice (free but not open
+  source), and a task that asked for the official Python documentation used a tutorial site. The objective
+  checklist keeps the conditions in view (the documentation task went to python.org on 1.53), but on the same
+  run the list task still named WPS Office, which is not open source. Check the result against any conditions
+  you set.
 - **A small model's judgement is the ceiling.** It sometimes drifts to loosely related pages, prefers
   secondary sources or stays with one source too long, and results vary with the first pages a search turns
   up; every finding is in `research_report.md` regardless.
-- **Very strict objectives can keep it searching.** An objective that forbids finishing until everything is
-  "fully validated" can keep the agent looking for something the web may not contain. It is told to finish
-  and name what it could not find after looking in several places, but a 7B model does not always judge
-  when that point has come; the turn limit (100) and Ctrl+C still apply, and the report is always current.
+- **Very strict objectives.** An objective that forbids finishing until everything is "fully validated" used
+  to keep the agent looking for something the web may not contain. On 1.50 it finished such an objective in 25
+  minutes, but with thin findings and without naming what it could not find, so check the result against the
+  objective.
 - **Check the citations.** The final result is told to use only recorded findings, but a 7B model sometimes
   puts the wrong number on a citation, adds a line no finding supports or leaves out a step, so verify a
   claim against the listed findings before relying on it.
@@ -313,7 +346,10 @@ turn that brings in a new page takes a few minutes. See [Performance](#performan
 - [x] Write the final result in a separate, rule-bound step and flag unsupported numbers (1.32–1.38)
 - [x] Finish as soon as the objective is covered (1.35)
 - [x] Get the model to weigh each part of a long page before reading on (1.42)
-- [ ] Let very strict objectives end with the gaps named instead of searching indefinitely
+- [ ] Let very strict objectives end with the gaps named instead of searching indefinitely (they finish since 1.50; naming the gaps is still open)
+- [x] Measure every change on a regression suite of seven kinds of task (1.53)
+- [x] Keep a fixed checklist of the objective's parts instead of re-splitting it at every check (1.53)
+- [ ] Check that a finding really covers its checklist item, including conditions like "open source"
 - [ ] Try more kinds of objectives (seven so far) and adjust the rules that don't generalise
 - [ ] Make fresh prompts (the slowest kind of turn) cheaper still (31% cheaper since 1.34)
 - [ ] Interacting with pages (clicking, forms) for tasks beyond reading
@@ -338,6 +374,19 @@ call:
 python tools/live_test.py "Your objective here" --minutes 25
 ```
 
+To measure a change on all seven kinds of task (about four hours on a CPU), compared with an earlier run:
+
+```bash
+python tools/regression.py --baseline regression_runs/<earlier run>/summary.json
+```
+
+Each run gets a folder in `regression_runs/` with every task's report, journal, console output and prompt
+trace, and a `summary.md` that is updated after each task: whether it finished, turns, model time, how many
+checklist items got findings, which checks fired, and problems found by simple word checks (missing answers,
+missed conditions such as a non-open-source item, uncited paragraphs, numbers missing from the findings, a
+missing opinion). `--suite all` adds five more tasks, `--only` runs some of them, and `--rescore` scores an
+earlier run again without the model.
+
 ## Contributing
 
 Contributions are welcome! See [CONTRIBUTING.md](https://github.com/SnowFXXX/LocalAgentBrowser/blob/main/CONTRIBUTING.md) for setup and guidelines, and please follow