localagentbrowser 1.52__tar.gz → 1.53__tar.gz

{localagentbrowser-1.52 → localagentbrowser-1.53}/CHANGELOG.md

@@ -1,9 +1,41 @@
 # Changelog
 
-Every version of `agent_browser.py`, newest first. Each version is also a tagged commit (`v1.11` … `v1.28`).
+Every version of `agent_browser.py`, newest first. Each version is also a tagged commit (`v1.11` … `v1.53`).
 Timings are from live runs on the development machine (Intel i5-9500, 6 cores, no GPU, Qwen2.5-7B-Instruct
 Q4_K_M in LM Studio), where reading the prompt takes about 90% of every model call.
 
+## [1.53] - 2026-10-17
+### Added
+- Regression suite (`tools/regression.py`): seven objectives, one of each kind of task tested so far, run one
+  after the other with a time limit each and scored the same way (finished or not, turns, model time, checklist
+  progress, checks fired, and word checks for missing answers, missed conditions, uncited paragraphs, numbers
+  missing from the findings and a missing opinion), with a comparison to an earlier run. Single runs had been
+  flattering: on 1.52 the suite finished only 2 of 7 tasks within their limits, including tasks earlier single
+  runs had finished. Refused actions in the journal now name the check that refused them, so the suite can count
+  each check, and the live test runner times every model call and saves the model's full replies.
+- Objective checklist. The 1.52 suite showed the main failure: every check asked the model to split the
+  objective into parts again, from memory, and the split drifted. On the list task it blocked its own FINISH
+  over "migration strategies", which the objective never asked for (an idea from a page it had read), and never
+  looked for a third suite; on the comparison it added "data types" and never covered "many simultaneous users".
+  Now one short call at the start splits the objective into a fixed checklist; asked on its own, the model split
+  all seven test objectives cleanly and kept their conditions. Each DISCOVER names the items it serves
+  (`checklist_items` in the reply), the checklist with its discoveries is in fresh prompts and in the note after
+  each discovery, the finish, progress and objective checks go through its items ("do not add other parts"),
+  the read-on check names the items still open, the final result covers every item or says it was not found,
+  and the report lists the checklist. The split call uses the same system prompt as turn 1, so LM Studio's cache
+  of it still applies there (without that, turn 1 took 128 s instead of about 40).
+### Fixed
+- The report flagged the numbers of numbered headings ("#### 1. Setup") as numbers missing from the findings.
+
+Regression run against 1.52 (model minutes in brackets): 5 of 7 tasks finished instead of 2, and the total model
+time fell from 254 to 200 minutes. The list (30.1, was a 45-minute time limit), comparison (39.0, every part
+covered), explanation (12.9, was 60 without finishing) and opinion (38.1) tasks now finish; the factual question
+still does (13.6, was 11.9). What is not better: the list names WPS Office, which is not open source; the
+explanation finished from a single overview paragraph that the model claimed covered all four items, without
+oxygen or water; the how-to found the materials and steps but not the aftercare before its time limit; and the
+documentation task now went to python.org, as asked, but worked through the items one search at a time and hit
+its 30-minute limit before the last one (it had finished in 15 minutes on 1.52, from a tutorial site).
+
 ## [1.52] - 2026-10-16
 ### Changed
 - The note after each DISCOVER also says to work on parts of the objective that are not covered yet before

{localagentbrowser-1.52 → localagentbrowser-1.53}/MANIFEST.in

@@ -1,3 +1,4 @@
-# Files the source package needs beyond the defaults, so its tests can run
+# Files the source package needs beyond the defaults, so its tests (and the tools they test) can run
 include CHANGELOG.md
 recursive-include tests *.py
+recursive-include tools *.py

{localagentbrowser-1.52/localagentbrowser.egg-info → localagentbrowser-1.53}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: localagentbrowser
-Version: 1.52
+Version: 1.53
 Summary: An autonomous web agent driven by a local LLM (LM Studio) and Playwright. Runs entirely on your own machine.
 Author: SnowFX
 License-Expression: MIT
@@ -56,8 +56,8 @@ and tuned to work well with a **small 7B model on a CPU, no GPU required**.
 - ⚡ **Built for slow hardware.** It reuses LM Studio's prompt cache, so most turns process only what changed.
 - 📝 **A readable report every time.** `research_report.md` holds the final result and every finding with its
   source, written however the run ends; `research_log.txt` keeps every action and reason.
-- ✅ **Tested.** 112 tests on Linux and Windows, Python 3.10–3.13, run in about a second with no model or browser,
-  plus live runs on seven different kinds of task (see [results](#results-on-different-kinds-of-tasks)).
+- ✅ **Tested.** 136 tests on Linux and Windows, Python 3.10–3.13, run in about a second with no model or browser,
+  plus a [regression suite](#regression-suite) of live runs on seven different kinds of task.
 
 📖 **The story behind it:** [Teaching a 7B model on a CPU to finish what it starts](https://dev.to/snowfxxx/teaching-a-7b-model-on-a-cpu-to-finish-what-it-starts-1amc),
 six lessons from 30 versions of keeping a small local model on track.
@@ -193,7 +193,23 @@ does not generalise; the [changelog](https://github.com/SnowFXXX/LocalAgentBrows
 | List of items | Three free, open-source alternatives to Microsoft Office and what each is best at | 1.46, 1.51 | LibreOffice and OnlyOffice with their strengths, cited, in 18 turns (49 min); the third, FreeOffice, is free but not open source, and LibreOffice was recorded four times in different words; on 1.51: LibreOffice, OnlyOffice and Apache OpenOffice, all open source, in 7 turns (16 min) |
 | Step-by-step how-to | Make a sourdough starter, with quantities and what to do each day | 1.36–1.37 | A clear plan in 7 turns (19 min), but it invented the first water amount; 1.37 stops guessing and flags unsupported numbers, and the agent then went looking for the missing details |
 | Step-by-step how-to | Repot a houseplant: what you need, the steps, and care for the first two weeks | 1.43–1.44, 1.51 | 1.43 recorded six steps it wrote itself before reading the part that had them; 1.44 recorded the guide's real steps but took 68 min and its result left one out. 1.51: every step from the page, in 7 turns (29 min), and it says that no care instructions beyond the first watering were found |
-| Explanation of a topic | How photosynthesis works: inputs and outputs, both stages and where each takes place | 1.41–1.42 | A complete, cited explanation in 18 turns (45 min); 1.41 read four parts of a relevant page without recording them and had not finished after 49 min |
+| Explanation of a topic | How photosynthesis works: inputs and outputs, both stages and where each takes place | 1.41–1.42, 1.52 | A complete, cited explanation in 18 turns (45 min); 1.41 read four parts of a relevant page without recording them and had not finished after 49 min. 1.52 recorded both stages and where each takes place, but did not finish within 65 min: it read the history sections of a long encyclopedia article twice and never recorded the inputs and outputs |
+
+### Regression suite
+
+Since 1.53, every change is measured on the same seven objectives, one of each kind above, run one after the other
+with a time limit each (`tools/regression.py`, see [Testing](#testing)). Single runs vary with the pages a search
+turns up, so the suite compares versions on all seven at once:
+
+| Task | 1.52 | 1.53 (objective checklist) |
+|---|---|---|
+| Factual question | Finished, 12 min | Finished, 14 min |
+| List of three office suites | Time limit (45 min) | Finished, 30 min; but one of the three (WPS Office) is not open source |
+| Comparison and recommendation | Time limit (45 min) | Finished, 39 min, every part covered |
+| How-to (repotting) | Time limit (45 min) | Time limit (45 min); materials and steps found, aftercare not |
+| Explanation (photosynthesis) | Time limit (60 min) | Finished, 13 min, but thin: one finding, no oxygen or water |
+| Opinion (owls) | Time limit (45 min) | Finished, 38 min, four cultures and an opinion |
+| Documentation (venv, official docs) | Finished, 15 min, from a tutorial site | Time limit (30 min); create and activate found, from python.org |
 
 ## How it works
 
@@ -233,6 +249,11 @@ flowchart LR
   (or has a sentence that does) is sent back once, asking for only what the page says in its own words, and
   one far from anything on the page is refused outright. A small model will otherwise "complete" a list from
   its own knowledge when a page says "follow these 6 steps" and the steps come later.
+- **An objective checklist.** At the start, one short model call splits the objective into a fixed checklist of
+  what the result must contain, keeping conditions like "free, open-source" or "three". Each finding names the
+  items it serves, the checklist with its findings is in every prompt, and every check goes through these items
+  instead of asking the model to split the objective again from memory, which drifted: it once blocked its own
+  finish over "migration strategies", which the objective never asked for.
 - **Numbered findings.** Findings are numbered in the prompt and listed with every check, and the model names
   the finding that covers each part of the objective ("covered by [2]") instead of just saying "covered".
 - **Knowing when it's done.** After every new finding the model is asked whether the objective is now fully
@@ -285,8 +306,9 @@ Measured on an Intel i5-9500 (6 cores, no GPU) with Qwen2.5-7B-Instruct Q4_K_M a
 | Fresh prompt after the context budget is reached | 160–380 s (longest with long pages) |
 | Writing the final result (once, at the end) | 50–190 s, depending on how much was found |
 
-Whole runs from the results table: a factual question in about 12 minutes of model time, a list of three items
-in 24, a comparison in 33. A GPU or a smaller model will be much faster.
+Whole runs vary with the pages a search turns up. In the results table, a factual question takes about 12
+minutes of model time, a list of three items 16 to 49, a comparison 25 to 38 and an explanation 45 or more. A
+GPU or a smaller model will be much faster.
 
 ## FAQ
 
@@ -321,9 +343,18 @@ turn that brings in a new page takes a few minutes. See [Performance](#performan
   research, an opinion from several sources, a factual question, a comparison, a how-to, a list and an
   explanation of a topic. Each one
   exposed something that did not generalise and was fixed; other kinds of tasks may expose more.
-- **Reading without recording.** The model used to read long pages part by part without recording the
-  relevant parts. Since 1.42 it must say what a part contains before reading on, which fixed this on an
-  explanation task; it is still the area most likely to need more work on other pages.
+- **Reading long pages and knowing when it has enough.** The model used to read long pages part by part
+  without recording the relevant parts; since 1.42 it must say what a part contains before reading on. On 1.52
+  the explanation task still lost about 20 minutes reading unrelated sections of a long article and did not
+  finish. On 1.53, with the objective checklist, it finished in 13 minutes, but from a single overview paragraph
+  that left out oxygen and water: the model can now claim one finding covers every item. Judging whether a
+  finding really covers an item is the weakest area.
+- **Conditions in the objective.** A small model tends to drop conditions such as "open source" or "using the
+  official documentation": a list of open-source office suites once included FreeOffice (free but not open
+  source), and a task that asked for the official Python documentation used a tutorial site. The objective
+  checklist keeps the conditions in view (the documentation task went to python.org on 1.53), but on the same
+  run the list task still named WPS Office, which is not open source. Check the result against any conditions
+  you set.
 - **A small model's judgement is the ceiling.** It sometimes drifts to loosely related pages, prefers
   secondary sources or stays with one source too long, and results vary with the first pages a search turns
   up; every finding is in `research_report.md` regardless.
@@ -348,6 +379,9 @@ turn that brings in a new page takes a few minutes. See [Performance](#performan
 - [x] Finish as soon as the objective is covered (1.35)
 - [x] Get the model to weigh each part of a long page before reading on (1.42)
 - [ ] Let very strict objectives end with the gaps named instead of searching indefinitely (they finish since 1.50; naming the gaps is still open)
+- [x] Measure every change on a regression suite of seven kinds of task (1.53)
+- [x] Keep a fixed checklist of the objective's parts instead of re-splitting it at every check (1.53)
+- [ ] Check that a finding really covers its checklist item, including conditions like "open source"
 - [ ] Try more kinds of objectives (seven so far) and adjust the rules that don't generalise
 - [ ] Make fresh prompts (the slowest kind of turn) cheaper still (31% cheaper since 1.34)
 - [ ] Interacting with pages (clicking, forms) for tasks beyond reading
@@ -372,6 +406,19 @@ call:
 python tools/live_test.py "Your objective here" --minutes 25
 ```
 
+To measure a change on all seven kinds of task (about four hours on a CPU), compared with an earlier run:
+
+```bash
+python tools/regression.py --baseline regression_runs/<earlier run>/summary.json
+```
+
+Each run gets a folder in `regression_runs/` with every task's report, journal, console output and prompt
+trace, and a `summary.md` that is updated after each task: whether it finished, turns, model time, how many
+checklist items got findings, which checks fired, and problems found by simple word checks (missing answers,
+missed conditions such as a non-open-source item, uncited paragraphs, numbers missing from the findings, a
+missing opinion). `--suite all` adds five more tasks, `--only` runs some of them, and `--rescore` scores an
+earlier run again without the model.
+
 ## Contributing
 
 Contributions are welcome! See [CONTRIBUTING.md](https://github.com/SnowFXXX/LocalAgentBrowser/blob/main/CONTRIBUTING.md) for setup and guidelines, and please follow

{localagentbrowser-1.52 → localagentbrowser-1.53}/README.md

@@ -24,8 +24,8 @@ and tuned to work well with a **small 7B model on a CPU, no GPU required**.
 - ⚡ **Built for slow hardware.** It reuses LM Studio's prompt cache, so most turns process only what changed.
 - 📝 **A readable report every time.** `research_report.md` holds the final result and every finding with its
   source, written however the run ends; `research_log.txt` keeps every action and reason.
-- ✅ **Tested.** 112 tests on Linux and Windows, Python 3.10–3.13, run in about a second with no model or browser,
-  plus live runs on seven different kinds of task (see [results](#results-on-different-kinds-of-tasks)).
+- ✅ **Tested.** 136 tests on Linux and Windows, Python 3.10–3.13, run in about a second with no model or browser,
+  plus a [regression suite](#regression-suite) of live runs on seven different kinds of task.
 
 📖 **The story behind it:** [Teaching a 7B model on a CPU to finish what it starts](https://dev.to/snowfxxx/teaching-a-7b-model-on-a-cpu-to-finish-what-it-starts-1amc),
 six lessons from 30 versions of keeping a small local model on track.
@@ -161,7 +161,23 @@ does not generalise; the [changelog](https://github.com/SnowFXXX/LocalAgentBrows
 | List of items | Three free, open-source alternatives to Microsoft Office and what each is best at | 1.46, 1.51 | LibreOffice and OnlyOffice with their strengths, cited, in 18 turns (49 min); the third, FreeOffice, is free but not open source, and LibreOffice was recorded four times in different words; on 1.51: LibreOffice, OnlyOffice and Apache OpenOffice, all open source, in 7 turns (16 min) |
 | Step-by-step how-to | Make a sourdough starter, with quantities and what to do each day | 1.36–1.37 | A clear plan in 7 turns (19 min), but it invented the first water amount; 1.37 stops guessing and flags unsupported numbers, and the agent then went looking for the missing details |
 | Step-by-step how-to | Repot a houseplant: what you need, the steps, and care for the first two weeks | 1.43–1.44, 1.51 | 1.43 recorded six steps it wrote itself before reading the part that had them; 1.44 recorded the guide's real steps but took 68 min and its result left one out. 1.51: every step from the page, in 7 turns (29 min), and it says that no care instructions beyond the first watering were found |
-| Explanation of a topic | How photosynthesis works: inputs and outputs, both stages and where each takes place | 1.41–1.42 | A complete, cited explanation in 18 turns (45 min); 1.41 read four parts of a relevant page without recording them and had not finished after 49 min |
+| Explanation of a topic | How photosynthesis works: inputs and outputs, both stages and where each takes place | 1.41–1.42, 1.52 | A complete, cited explanation in 18 turns (45 min); 1.41 read four parts of a relevant page without recording them and had not finished after 49 min. 1.52 recorded both stages and where each takes place, but did not finish within 65 min: it read the history sections of a long encyclopedia article twice and never recorded the inputs and outputs |
+
+### Regression suite
+
+Since 1.53, every change is measured on the same seven objectives, one of each kind above, run one after the other
+with a time limit each (`tools/regression.py`, see [Testing](#testing)). Single runs vary with the pages a search
+turns up, so the suite compares versions on all seven at once:
+
+| Task | 1.52 | 1.53 (objective checklist) |
+|---|---|---|
+| Factual question | Finished, 12 min | Finished, 14 min |
+| List of three office suites | Time limit (45 min) | Finished, 30 min; but one of the three (WPS Office) is not open source |
+| Comparison and recommendation | Time limit (45 min) | Finished, 39 min, every part covered |
+| How-to (repotting) | Time limit (45 min) | Time limit (45 min); materials and steps found, aftercare not |
+| Explanation (photosynthesis) | Time limit (60 min) | Finished, 13 min, but thin: one finding, no oxygen or water |
+| Opinion (owls) | Time limit (45 min) | Finished, 38 min, four cultures and an opinion |
+| Documentation (venv, official docs) | Finished, 15 min, from a tutorial site | Time limit (30 min); create and activate found, from python.org |
 
 ## How it works
 
@@ -201,6 +217,11 @@ flowchart LR
   (or has a sentence that does) is sent back once, asking for only what the page says in its own words, and
   one far from anything on the page is refused outright. A small model will otherwise "complete" a list from
   its own knowledge when a page says "follow these 6 steps" and the steps come later.
+- **An objective checklist.** At the start, one short model call splits the objective into a fixed checklist of
+  what the result must contain, keeping conditions like "free, open-source" or "three". Each finding names the
+  items it serves, the checklist with its findings is in every prompt, and every check goes through these items
+  instead of asking the model to split the objective again from memory, which drifted: it once blocked its own
+  finish over "migration strategies", which the objective never asked for.
 - **Numbered findings.** Findings are numbered in the prompt and listed with every check, and the model names
   the finding that covers each part of the objective ("covered by [2]") instead of just saying "covered".
 - **Knowing when it's done.** After every new finding the model is asked whether the objective is now fully
@@ -253,8 +274,9 @@ Measured on an Intel i5-9500 (6 cores, no GPU) with Qwen2.5-7B-Instruct Q4_K_M a
 | Fresh prompt after the context budget is reached | 160–380 s (longest with long pages) |
 | Writing the final result (once, at the end) | 50–190 s, depending on how much was found |
 
-Whole runs from the results table: a factual question in about 12 minutes of model time, a list of three items
-in 24, a comparison in 33. A GPU or a smaller model will be much faster.
+Whole runs vary with the pages a search turns up. In the results table, a factual question takes about 12
+minutes of model time, a list of three items 16 to 49, a comparison 25 to 38 and an explanation 45 or more. A
+GPU or a smaller model will be much faster.
 
 ## FAQ
 
@@ -289,9 +311,18 @@ turn that brings in a new page takes a few minutes. See [Performance](#performan
   research, an opinion from several sources, a factual question, a comparison, a how-to, a list and an
   explanation of a topic. Each one
   exposed something that did not generalise and was fixed; other kinds of tasks may expose more.
-- **Reading without recording.** The model used to read long pages part by part without recording the
-  relevant parts. Since 1.42 it must say what a part contains before reading on, which fixed this on an
-  explanation task; it is still the area most likely to need more work on other pages.
+- **Reading long pages and knowing when it has enough.** The model used to read long pages part by part
+  without recording the relevant parts; since 1.42 it must say what a part contains before reading on. On 1.52
+  the explanation task still lost about 20 minutes reading unrelated sections of a long article and did not
+  finish. On 1.53, with the objective checklist, it finished in 13 minutes, but from a single overview paragraph
+  that left out oxygen and water: the model can now claim one finding covers every item. Judging whether a
+  finding really covers an item is the weakest area.
+- **Conditions in the objective.** A small model tends to drop conditions such as "open source" or "using the
+  official documentation": a list of open-source office suites once included FreeOffice (free but not open
+  source), and a task that asked for the official Python documentation used a tutorial site. The objective
+  checklist keeps the conditions in view (the documentation task went to python.org on 1.53), but on the same
+  run the list task still named WPS Office, which is not open source. Check the result against any conditions
+  you set.
 - **A small model's judgement is the ceiling.** It sometimes drifts to loosely related pages, prefers
   secondary sources or stays with one source too long, and results vary with the first pages a search turns
   up; every finding is in `research_report.md` regardless.
@@ -316,6 +347,9 @@ turn that brings in a new page takes a few minutes. See [Performance](#performan
 - [x] Finish as soon as the objective is covered (1.35)
 - [x] Get the model to weigh each part of a long page before reading on (1.42)
 - [ ] Let very strict objectives end with the gaps named instead of searching indefinitely (they finish since 1.50; naming the gaps is still open)
+- [x] Measure every change on a regression suite of seven kinds of task (1.53)
+- [x] Keep a fixed checklist of the objective's parts instead of re-splitting it at every check (1.53)
+- [ ] Check that a finding really covers its checklist item, including conditions like "open source"
 - [ ] Try more kinds of objectives (seven so far) and adjust the rules that don't generalise
 - [ ] Make fresh prompts (the slowest kind of turn) cheaper still (31% cheaper since 1.34)
 - [ ] Interacting with pages (clicking, forms) for tasks beyond reading
@@ -340,6 +374,19 @@ call:
 python tools/live_test.py "Your objective here" --minutes 25
 ```
 
+To measure a change on all seven kinds of task (about four hours on a CPU), compared with an earlier run:
+
+```bash
+python tools/regression.py --baseline regression_runs/<earlier run>/summary.json
+```
+
+Each run gets a folder in `regression_runs/` with every task's report, journal, console output and prompt
+trace, and a `summary.md` that is updated after each task: whether it finished, turns, model time, how many
+checklist items got findings, which checks fired, and problems found by simple word checks (missing answers,
+missed conditions such as a non-open-source item, uncited paragraphs, numbers missing from the findings, a
+missing opinion). `--suite all` adds five more tasks, `--only` runs some of them, and `--rescore` scores an
+earlier run again without the model.
+
 ## Contributing
 
 Contributions are welcome! See [CONTRIBUTING.md](https://github.com/SnowFXXX/LocalAgentBrowser/blob/main/CONTRIBUTING.md) for setup and guidelines, and please follow