localagentbrowser 1.63__tar.gz → 1.64__tar.gz

{localagentbrowser-1.63 → localagentbrowser-1.64}/CHANGELOG.md

@@ -1,11 +1,67 @@
 # Changelog
 
-Every version of LocalAgentBrowser, newest first. Each released version is also a tagged commit (`v1.11` … `v1.62`); 1.54 to 1.56 and 1.58 to 1.60 were internal steps
+Every version of LocalAgentBrowser, newest first. Each released version is also a tagged commit (`v1.11` … `v1.63`); 1.54 to 1.56 and 1.58 to 1.60 were internal steps
 measured on the regression suite and released as part of 1.57 and 1.61 (see
 [docs/regression-history.md](docs/regression-history.md)).
 Timings are from live runs on the development machine (Intel i5-9500, 6 cores, no GPU, Qwen2.5-7B-Instruct
 Q4_K_M in LM Studio), where reading the prompt takes about 90% of every model call.
 
+## [1.64] - 2026-11-08
+The theme is **silent omission** ([issue #11](https://github.com/SnowFXXX/LocalAgentBrowser/issues/11)): a part of
+the objective dropping out of the final answer without being marked "not found". It was measured on two PCs (PC A
+and PC B, both Intel i5-9500, no GPU) and then on four models with identical code.
+
+### Added
+- **Windows launcher and shortcuts.** `tools/windows/LocalAgentBrowser.cmd` checks that LM Studio is answering,
+  runs the agent in its own folder under `Documents\LocalAgentBrowser runs`, and opens the report when the run ends.
+  `tools/windows/make_shortcuts.ps1` adds Start-menu and desktop shortcuts with the project icon (from a clone of
+  the repository).
+- **The report opens by itself** when a launcher run ends: `research_report.md` is written next to it as a dark web
+  page, in the project site's colors, with the final result, checklist and discoveries in separate cards and
+  citations shown as tags (`tools/windows/show_report.py`, standard library only). Windows has no program for `.md`
+  files by default.
+- Regression suite: every run folder gets `run_info.json` (PC label from `LAB_PC_LABEL`, commit, and a fingerprint
+  of the uncommitted changes), so runs on several PCs can be tied to the exact code they ran.
+- Regression suite: `LAB_BLOCKED_SITES` makes NAVIGATE refuse given sites, for benchmark runs that must not read the
+  benchmark's own answer pages.
+
+### Fixed: parts of the objective dropping out of the answer
+- **The coverage check asks for the quote first.** "Does this discovery answer the item?" is now answered by first
+  copying the words that answer it, with a rule that words on the same topic but a different aspect don't count.
+  On 30 hand-labelled real cases, 25 right instead of 16 (no false "not found", 5 false "found" left).
+- **A part credited to several items only counts where the item's own words appear**, so one quote can't silently
+  cover three parts; such an item is sent back as unanswered.
+- **Missing amounts and concrete missing terms send the agent back.** When an item asks for amounts and no discovery
+  for it gives one, or the check names a concrete missing term ("O2") that no discovery contains, the agent is told
+  to find it, instead of finishing without it (an explanation once finished in 4 turns without oxygen).
+- **Each fact stays under its own part.** A discovery keeps the page heading it came from, including headings inside
+  a discovery that crosses page sections; a heading the discovery starts with isn't replaced by the one above;
+  bylines, very long "headings" (numbered sentences) and the prompt's own section names don't count as headings or
+  names.
+- **Names no discovery gives are cut,** with a note: a short cited heading's name is checked too, word forms need
+  an 80% shared start ("PhotoScape" over a PhotoSplice source), and words in a source's web address count as given.
+- **A sentence under one label copied from another subject's sentence is cut** (SQLite's sentence under
+  PostgreSQL).
+- **Numbers in words are checked like digits** ("two weeks" needs a 2 in a source, "one cup" equals "1 cup"), unit
+  abbreviations count as their words ("5,895 m (19,341 ft)" answers "in meters" and "in feet"), and a time needs
+  the same number as a time in a source ("2 weeks" was taken from a "2L jar").
+- **Citations:** citation lines on their own are joined to the line they cite; citations are moved out of code, and
+  unclosed code blocks closed, so correct commands aren't cut as invented.
+- A **"Recommendation" section is labelled as the model's view**, like "My view:".
+- **NAVIGATE refuses text that isn't a web address** (a run lost 5 turns on it).
+
+### Fixed: the regression suite's scoring
+- The code's own "Left out, because…" notes are no longer read as the result's claims (commands and uncited lines).
+  4 of 78 earlier run folders were over-counted.
+- A bold label ("**1. Creating a virtual environment:**") is a heading, not an uncited claim.
+
+### Measured
+- 7B on two PCs: 7 of 7 on both PCs with v11 (297.8 and 269.6 model minutes), with no outside knowledge, invented
+  numbers or invented commands in any finished result.
+- Four models, same code (v12, 90 minutes per task): Qwen2.5-7B 6 of 7, Llama 3.1 8B 7 of 7, Qwen2.5-14B 5 of 7,
+  Qwen2.5-3B 3 of 7. On 1.63 they were 6, 2, 2 and 0 of 7. No finished result used outside knowledge. Details in
+  docs/regression-history.md.
+
 ## [1.63] - 2026-11-01
 ### Added
 - A model call the server fails is tried again after a wait (5, 15, 30, 60 and 120 seconds, about 4 minutes in all),

{localagentbrowser-1.63/localagentbrowser.egg-info → localagentbrowser-1.64}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: localagentbrowser
-Version: 1.63
+Version: 1.64
 Summary: An autonomous web agent driven by a local LLM (LM Studio) and Playwright. Runs entirely on your own machine.
 Author: SnowFX
 License-Expression: MIT
@@ -36,6 +36,7 @@ Dynamic: license-file
 decides when it's done, all on your own computer.**
 
 [![Tests](https://github.com/SnowFXXX/LocalAgentBrowser/actions/workflows/tests.yml/badge.svg)](https://github.com/SnowFXXX/LocalAgentBrowser/actions/workflows/tests.yml)
+[![PyPI](https://img.shields.io/pypi/v/localagentbrowser.svg)](https://pypi.org/project/localagentbrowser/)
 [![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](https://github.com/SnowFXXX/LocalAgentBrowser/blob/main/LICENSE)
 ![Python 3.10–3.13](https://img.shields.io/badge/python-3.10%E2%80%933.13-blue.svg)
 ![Runs locally](https://img.shields.io/badge/LLM-local%20(LM%20Studio)-green.svg)
@@ -58,9 +59,11 @@ and tuned to work well with a **small 7B model on a CPU, no GPU required**.
 - ⚡ **Built for slow hardware.** It reuses LM Studio's prompt cache, so most turns process only what changed.
 - 📝 **A readable report every time.** `research_report.md` holds the final result and every finding with its
   source, written however the run ends; `research_log.txt` keeps every action and reason.
-- ✅ **Tested.** 251 tests on Linux and Windows, Python 3.10–3.13, run in a few seconds with no model or browser,
+- ✅ **Tested.** 324 tests on Linux and Windows, Python 3.10–3.13, run in a few seconds with no model or browser,
   plus a [regression suite](#regression-suite) of live runs on seven different kinds of task.
 
+**New with 1.64:** parts of the objective no longer drop out of the answer unannounced as often (the coverage check asks for the words first; missing amounts and terms send the agent back), measured on two PCs and then four models with identical code: Qwen2.5-7B 6 of 7, Llama 3.1 8B 7 of 7, Qwen2.5-14B 5 of 7 and Qwen2.5-3B 3 of 7 tasks, with no outside knowledge in any finished result ([regression history](docs/regression-history.md)). On Windows, a launcher with Start-menu and desktop shortcuts opens the report by itself when a run ends.
+
 **New with 1.63:** the [project site](https://snowfxxx.github.io/LocalAgentBrowser/), an article,
 [We Can Do Better: A Small AI That Can't Cover Up](https://snowfxxx.github.io/LocalAgentBrowser/article.html), and a
 technical paper, [Verification over Trust](https://snowfxxx.github.io/LocalAgentBrowser/paper.html).
@@ -135,6 +138,11 @@ six lessons from 30 versions of keeping a small local model on track.
    When it finishes, read `research_report.md` in the folder you ran it from. Press Ctrl+C at any time to
    stop; the report is still written with everything found so far.
 
+**On Windows**, from a clone of the repository (below), `powershell -ExecutionPolicy Bypass -File
+tools\windows\make_shortcuts.ps1` adds LocalAgentBrowser to the Start menu and the desktop. The launcher checks that
+LM Studio is answering, runs each objective in its own folder under `Documents\LocalAgentBrowser runs`, and opens the
+report as a page in your browser when the run ends.
+
 To work on the code instead, clone the repository and install from it:
 
 ```bash
@@ -231,15 +239,15 @@ turns up, so the suite compares versions on all seven at once. 1.54 to 1.56 and
 measured this way and released with 1.57 and 1.61; [docs/regression-history.md](https://github.com/SnowFXXX/LocalAgentBrowser/blob/main/docs/regression-history.md)
 has every step's results and what each one showed.
 
-| Task | 1.52 | 1.53 (objective checklist) | 1.57 (coverage check) | 1.61 (verified send-backs, outline pointer) |
-|---|---|---|---|---|
-| Factual question | Finished, 12 min | Finished, 14 min | Finished, 13 min | Finished, 17 min |
-| List of three office suites | Time limit | Finished, 30 min; but WPS Office is not open source | Time limit: three correct open-source suites by turn 7, sent back because no words showed them to be "free" | Finished, 21 min: LibreOffice, WPS Office and OnlyOffice, each marked "not shown in the pages read to be free and open-source" |
-| Comparison and recommendation | Time limit | Finished, 39 min | Time limit, after a send-back for "official documentation", which the objective does not ask for | Finished, 26 min, setup and administration from the part of a long page the outline pointer chose |
-| How-to (cold brew from 1.55; repotting before) | Time limit | Time limit | Finished, 36 min; but "how long it keeps" answered with the steeping time | Finished, 33 min: every step, the 16-hour steep and the shelf life, cited |
-| Explanation (photosynthesis) | Time limit | Finished, 13 min, but thin: no oxygen or water | Finished, 59 min: inputs, outputs and both stages, cited | Finished, 39 min |
-| Opinion (owls) | Time limit | Finished, 38 min | Time limit: its first page covered two of the three cultures asked for | Finished, 42 min: four cultures, cited, with a view |
-| Documentation (venv, official docs) | Finished, 15 min, from a tutorial site | Time limit (30 min) | Finished, 37 min, all from python.org; but the Windows line repeats the create command instead of the activate one | Time limit: all four items found from python.org, then it read on |
+| Task | 1.52 | 1.53 (objective checklist) | 1.57 (coverage check) | 1.61 (verified send-backs, outline pointer) | 1.64 (quote-first coverage, send-backs for missing amounts and terms) |
+|---|---|---|---|---|---|
+| Factual question | Finished, 12 min | Finished, 14 min | Finished, 13 min | Finished, 17 min | Finished, 17 min |
+| List of three office suites | Time limit | Finished, 30 min; but WPS Office is not open source | Time limit: three correct open-source suites by turn 7, sent back because no words showed them to be "free" | Finished, 21 min: LibreOffice, WPS Office and OnlyOffice, each marked "not shown in the pages read to be free and open-source" | Finished, 34 min: GIMP, Krita and Darktable (the list task asks for Photoshop alternatives from 1.64) |
+| Comparison and recommendation | Time limit | Finished, 39 min | Time limit, after a send-back for "official documentation", which the objective does not ask for | Finished, 26 min, setup and administration from the part of a long page the outline pointer chose | Finished, 49 min |
+| How-to (cold brew from 1.55; repotting before) | Time limit | Time limit | Finished, 36 min; but "how long it keeps" answered with the steeping time | Finished, 33 min: every step, the 16-hour steep and the shelf life, cited | Finished, 50 min |
+| Explanation (photosynthesis) | Time limit | Finished, 13 min, but thin: no oxygen or water | Finished, 59 min: inputs, outputs and both stages, cited | Finished, 39 min | Finished, 73 min, oxygen included |
+| Opinion (owls) | Time limit | Finished, 38 min | Time limit: its first page covered two of the three cultures asked for | Finished, 42 min: four cultures, cited, with a view | Finished, 49 min |
+| Documentation (venv, official docs) | Finished, 15 min, from a tutorial site | Time limit (30 min) | Finished, 37 min, all from python.org; but the Windows line repeats the create command instead of the activate one | Time limit: all four items found from python.org, then it read on | Time limit: read the reference page part after part without recording (finished in 34 and 42 min in the two v11 runs) |
 
 ## How it works
 

{localagentbrowser-1.63 → localagentbrowser-1.64}/README.md

@@ -4,6 +4,7 @@
 decides when it's done, all on your own computer.**
 
 [![Tests](https://github.com/SnowFXXX/LocalAgentBrowser/actions/workflows/tests.yml/badge.svg)](https://github.com/SnowFXXX/LocalAgentBrowser/actions/workflows/tests.yml)
+[![PyPI](https://img.shields.io/pypi/v/localagentbrowser.svg)](https://pypi.org/project/localagentbrowser/)
 [![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](https://github.com/SnowFXXX/LocalAgentBrowser/blob/main/LICENSE)
 ![Python 3.10–3.13](https://img.shields.io/badge/python-3.10%E2%80%933.13-blue.svg)
 ![Runs locally](https://img.shields.io/badge/LLM-local%20(LM%20Studio)-green.svg)
@@ -26,9 +27,11 @@ and tuned to work well with a **small 7B model on a CPU, no GPU required**.
 - ⚡ **Built for slow hardware.** It reuses LM Studio's prompt cache, so most turns process only what changed.
 - 📝 **A readable report every time.** `research_report.md` holds the final result and every finding with its
   source, written however the run ends; `research_log.txt` keeps every action and reason.
-- ✅ **Tested.** 251 tests on Linux and Windows, Python 3.10–3.13, run in a few seconds with no model or browser,
+- ✅ **Tested.** 324 tests on Linux and Windows, Python 3.10–3.13, run in a few seconds with no model or browser,
   plus a [regression suite](#regression-suite) of live runs on seven different kinds of task.
 
+**New with 1.64:** parts of the objective no longer drop out of the answer unannounced as often (the coverage check asks for the words first; missing amounts and terms send the agent back), measured on two PCs and then four models with identical code: Qwen2.5-7B 6 of 7, Llama 3.1 8B 7 of 7, Qwen2.5-14B 5 of 7 and Qwen2.5-3B 3 of 7 tasks, with no outside knowledge in any finished result ([regression history](docs/regression-history.md)). On Windows, a launcher with Start-menu and desktop shortcuts opens the report by itself when a run ends.
+
 **New with 1.63:** the [project site](https://snowfxxx.github.io/LocalAgentBrowser/), an article,
 [We Can Do Better: A Small AI That Can't Cover Up](https://snowfxxx.github.io/LocalAgentBrowser/article.html), and a
 technical paper, [Verification over Trust](https://snowfxxx.github.io/LocalAgentBrowser/paper.html).
@@ -103,6 +106,11 @@ six lessons from 30 versions of keeping a small local model on track.
    When it finishes, read `research_report.md` in the folder you ran it from. Press Ctrl+C at any time to
    stop; the report is still written with everything found so far.
 
+**On Windows**, from a clone of the repository (below), `powershell -ExecutionPolicy Bypass -File
+tools\windows\make_shortcuts.ps1` adds LocalAgentBrowser to the Start menu and the desktop. The launcher checks that
+LM Studio is answering, runs each objective in its own folder under `Documents\LocalAgentBrowser runs`, and opens the
+report as a page in your browser when the run ends.
+
 To work on the code instead, clone the repository and install from it:
 
 ```bash
@@ -199,15 +207,15 @@ turns up, so the suite compares versions on all seven at once. 1.54 to 1.56 and
 measured this way and released with 1.57 and 1.61; [docs/regression-history.md](https://github.com/SnowFXXX/LocalAgentBrowser/blob/main/docs/regression-history.md)
 has every step's results and what each one showed.
 
-| Task | 1.52 | 1.53 (objective checklist) | 1.57 (coverage check) | 1.61 (verified send-backs, outline pointer) |
-|---|---|---|---|---|
-| Factual question | Finished, 12 min | Finished, 14 min | Finished, 13 min | Finished, 17 min |
-| List of three office suites | Time limit | Finished, 30 min; but WPS Office is not open source | Time limit: three correct open-source suites by turn 7, sent back because no words showed them to be "free" | Finished, 21 min: LibreOffice, WPS Office and OnlyOffice, each marked "not shown in the pages read to be free and open-source" |
-| Comparison and recommendation | Time limit | Finished, 39 min | Time limit, after a send-back for "official documentation", which the objective does not ask for | Finished, 26 min, setup and administration from the part of a long page the outline pointer chose |
-| How-to (cold brew from 1.55; repotting before) | Time limit | Time limit | Finished, 36 min; but "how long it keeps" answered with the steeping time | Finished, 33 min: every step, the 16-hour steep and the shelf life, cited |
-| Explanation (photosynthesis) | Time limit | Finished, 13 min, but thin: no oxygen or water | Finished, 59 min: inputs, outputs and both stages, cited | Finished, 39 min |
-| Opinion (owls) | Time limit | Finished, 38 min | Time limit: its first page covered two of the three cultures asked for | Finished, 42 min: four cultures, cited, with a view |
-| Documentation (venv, official docs) | Finished, 15 min, from a tutorial site | Time limit (30 min) | Finished, 37 min, all from python.org; but the Windows line repeats the create command instead of the activate one | Time limit: all four items found from python.org, then it read on |
+| Task | 1.52 | 1.53 (objective checklist) | 1.57 (coverage check) | 1.61 (verified send-backs, outline pointer) | 1.64 (quote-first coverage, send-backs for missing amounts and terms) |
+|---|---|---|---|---|---|
+| Factual question | Finished, 12 min | Finished, 14 min | Finished, 13 min | Finished, 17 min | Finished, 17 min |
+| List of three office suites | Time limit | Finished, 30 min; but WPS Office is not open source | Time limit: three correct open-source suites by turn 7, sent back because no words showed them to be "free" | Finished, 21 min: LibreOffice, WPS Office and OnlyOffice, each marked "not shown in the pages read to be free and open-source" | Finished, 34 min: GIMP, Krita and Darktable (the list task asks for Photoshop alternatives from 1.64) |
+| Comparison and recommendation | Time limit | Finished, 39 min | Time limit, after a send-back for "official documentation", which the objective does not ask for | Finished, 26 min, setup and administration from the part of a long page the outline pointer chose | Finished, 49 min |
+| How-to (cold brew from 1.55; repotting before) | Time limit | Time limit | Finished, 36 min; but "how long it keeps" answered with the steeping time | Finished, 33 min: every step, the 16-hour steep and the shelf life, cited | Finished, 50 min |
+| Explanation (photosynthesis) | Time limit | Finished, 13 min, but thin: no oxygen or water | Finished, 59 min: inputs, outputs and both stages, cited | Finished, 39 min | Finished, 73 min, oxygen included |
+| Opinion (owls) | Time limit | Finished, 38 min | Time limit: its first page covered two of the three cultures asked for | Finished, 42 min: four cultures, cited, with a view | Finished, 49 min |
+| Documentation (venv, official docs) | Finished, 15 min, from a tutorial site | Time limit (30 min) | Finished, 37 min, all from python.org; but the Windows line repeats the create command instead of the activate one | Time limit: all four items found from python.org, then it read on | Time limit: read the reference page part after part without recording (finished in 34 and 42 min in the two v11 runs) |
 
 ## How it works
 

{localagentbrowser-1.63 → localagentbrowser-1.64}/localagentbrowser/agent.py

@@ -7,7 +7,7 @@ from playwright.async_api import async_playwright
 from .config import API_KEY, BASE_URL, MAX_CHEAP_RETRIES, MAX_CONVERSATION_TOKENS, MAX_DISCOVERY_PROMPT_CHARS, MAX_FOLLOWUPS, MAX_JSON_RETRIES, MAX_QUERY_WORDS, MAX_VISITS_PER_URL, MODEL, MODEL_RETRY_WAITS, MODEL_TIMEOUT_SECONDS, TURN_MAX_TOKENS, OBJECTIVE_CHECK_EVERY, REPORT_FILE, STALL_TURNS, TURN_PAUSE_SECONDS
 from .prompts import CUT_OFF, FINAL_RESULT, FIRST_TURN_INSTRUCTION, GIVE_UP_RULE, READ_FIRST, RESPONSE_FORMAT, SYSTEM_PROMPT, TURN_INSTRUCTIONS
 from .text import extract_message_content, was_cut_off
-from .pages import build_environment, count_search_results, determine_url, is_search_page, load_page, normalize_url, read_more_target, render_part, search_query, where_you_are
+from .pages import build_environment, count_search_results, determine_url, is_search_page, load_page, normalize_url, read_more_target, render_part, search_query, section_heading, where_you_are, with_inner_headings
 from .checklist import after_discovery_advice, check_parts, checklist_lines, link_discovery, split_objective
 from .coverage import amounts_not_found, coverage_refusal, not_found_items
 from .checks import check_decision, escalation_message, numbered_discoveries, numbered_lines, refusal_correction, source_note
@@ -319,7 +319,8 @@ async def main():
                         missing, unproven = await not_found_items(client, objective, tried, journal)
                         no_amounts = amounts_not_found(tried.get("checklist"), journal, skip=missing)
                         final = await compose_final_result(client, objective, journal, tried.get("checklist"), missing,
-                                                           unproven, no_amounts, tried.get("evidence"))
+                                                           unproven, no_amounts, tried.get("evidence"),
+                                                           tried.get("short_counts"))
                         if final:
                             finish_entry.update(draft=parameter, finding=final)
                     except Exception as err:
@@ -398,8 +399,13 @@ async def main():
                     print(f"📝 Logging critical discovery element: {parameter}")
                     number = sum(1 for e in journal if e.get("action") == "DISCOVER") + 1
                     linked = link_discovery(tried["checklist"], decision.get("checklist_items"), number)
+                    # Recorded with its section's heading when the copied text does not name it (the heading is page
+                    # text too): every later check and the final result then see what the lines are about
+                    heading = section_heading(page_state, parameter)
+                    inner = with_inner_headings(page_state, parameter)
+                    recorded = f'(Under the heading "{heading}") {inner}' if heading else inner
                     journal.append({"turn": turn_counter, "thought": thought, "action": "DISCOVER", "source": current_url,
-                                    "extracted_data": parameter, "checklist_items": linked})
+                                    "extracted_data": recorded, "checklist_items": linked})
                     page_state["visit_discoveries"] = page_state.get("visit_discoveries", 0) + 1
                     page_state.setdefault("recorded_parts", set()).add(page_state["index"])
                     # A discovery is when the objective can become complete; on a one-answer question the model

{localagentbrowser-1.63 → localagentbrowser-1.64}/localagentbrowser/checks.py

@@ -4,7 +4,7 @@ import re
 from .config import GAP_REFUSAL_LIMIT, LONG_PAGE_UNREAD, MAX_CHECK_DISCOVERY_CHARS, MAX_VISITS_PER_URL, OBJECTIVE_ECHO_SHARE
 from .prompts import FINAL_RESULT, GIVE_UP_RULE
 from .text import MISSING_STOPWORDS, shorten
-from .pages import determine_url, is_search_page, normalize_query, normalize_url, outline_pointer, read_more_target, search_query, where_you_are
+from .pages import blocked_site, determine_url, is_search_page, normalize_query, normalize_url, outline_pointer, read_more_target, search_query, where_you_are
 from .grounding import DESCRIBES_PAGE, discovery_lines, grounded_share, grounding_check, is_known_discovery, page_hint
 from .checklist import check_parts, every_item_has_discoveries, open_items
 
@@ -187,6 +187,17 @@ def check_decision(action, parameter, journal, tried, page_state, current_url, t
     elif action == "NAVIGATE":
         if not parameter.strip():
             return "BLOCKED_NAVIGATE", "NAVIGATE needs a full URL in the parameter."
+        # A list run on Box 1 NAVIGATEd to "discovered", then to sentences ("the size of their budget..."), each opened
+        # as https://<text> and failing: four turns lost. Text with spaces or without a dot is no address
+        address = parameter.strip().strip("\"'")
+        if not re.search(r"https?://\S", address) and not address.startswith("#") and (
+                re.search(r"\s", address) or "." not in address):
+            return "BLOCKED_NAVIGATE", (f"'{shorten(address, 60)}' is not a URL. NAVIGATE needs a full URL from a link "
+                                        "or a search result (such as https://example.org/page); to look something up, "
+                                        "use SEARCH.")
+        if blocked_site(determine_url(action, parameter)):
+            return "BLOCKED_NAVIGATE", ("That site is excluded for this run (LAB_BLOCKED_SITES). Find the answer on "
+                                        "another page: SEARCH, or NAVIGATE to a different result.")
         target = normalize_url(determine_url(action, parameter))
         # Shown a long page's outline, the model kept trying to jump to a section with a made-up anchor
         # ("NAVIGATE ...#sqlite-vs-postgresql", then "NAVIGATE #sqlite-vs-postgresql")
@@ -308,8 +319,12 @@ def check_decision(action, parameter, journal, tried, page_state, current_url, t
             return "BLOCKED_DISCOVER_SNIPPET", ("That DISCOVER is the search result's title and description, not what "
                                                 "this page says. DISCOVER the facts the page text shown gives for the "
                                                 "OBJECTIVE, in its own wording, or READ_MORE if they are further on.")
-        return grounding_check(parameter, page_state, current_url, tried.get("objective", ""))
+        return grounding_check(parameter, page_state, current_url, tried.get("objective", ""),
+                               tried.get("finish_drafts", ()))
     elif action == "FINISH":
+        # Kept whole (the journal keeps only 120 characters of a refused one): a refused FINISH's recommendation
+        # came back three turns later inside a DISCOVER, appended to a real page line, and was cited as a finding
+        tried.setdefault("finish_drafts", []).append(parameter)
         return finish_check(tried, journal, page_state, current_url) or gap_check(thought, tried)
     else:
         return f"UNKNOWN_{action}", f"'{action}' is not a valid action."

{localagentbrowser-1.63 → localagentbrowser-1.64}/localagentbrowser/config.py

@@ -11,6 +11,10 @@ CONTEXT_LENGTH = int(os.environ.get("LMSTUDIO_CONTEXT_LENGTH", "8192"))
 
 SEARCH_URL = "https://html.duckduckgo.com/html/?q="
 
+# Sites a run must not read, comma-separated (a benchmark run excludes pages that republish its questions and answers):
+# "huggingface.co/datasets,github.com/openai/simple-evals". Empty, so off, unless set
+BLOCKED_SITES = [s.strip() for s in os.environ.get("LAB_BLOCKED_SITES", "").split(",") if s.strip()]
+
 REPORT_FILE = "research_report.md"
 
 JOURNAL_BATCH = 6        # The prompt shows 6-11 recent journal entries; old ones are dropped 6 at a time

{localagentbrowser-1.63 → localagentbrowser-1.64}/localagentbrowser/coverage.py

@@ -16,6 +16,9 @@ AMOUNT_ASKED = re.compile(r"\b(?:quantit(?:y|ies)|amounts?|how (?:much|many|long
 AMOUNT_GIVEN = re.compile(r"\d|\b(?:one|two|three|four|five|six|seven|eight|nine|ten|twelve|half|quarter|dozen|"
                           r"minutes?|hours?|days?|weeks?|months?|years?|overnight)\b", re.I)
 
+AMOUNT_GAP = ("no discovery recorded for it gives the amounts it asks for; DISCOVER the words that give them, as the "
+              "page writes them")
+
 def amounts_not_found(checklist, journal, skip=()):
     """The checklist items that ask for amounts while no discovery linked to them gives one, decided in code."""
     found = [e for e in journal if e.get("action") == "DISCOVER"]
@@ -38,6 +41,16 @@ def quoted_from(quote, text):
     pairs = word_pairs(" ".join(plain_words(quote)))
     return bool(pairs) and len(pairs & word_pairs(" ".join(plain_words(text)))) >= 0.8 * len(pairs)
 
+def names_copied(quote, text, item):
+    """Whether a quote of short lines (a name per line: 'GIMP', 'carbohydrates') is copied from the text: each line
+    of one or two words is in the text, and at least one of them is not a word of the ITEM itself."""
+    lines = [plain_words(line) for line in str(quote).splitlines()]
+    lines = [words for words in lines if words]
+    if not lines or any(len(words) > 2 for words in lines):
+        return False
+    known, asked = set(plain_words(text)), set(plain_words(item))
+    return all(set(words) <= known for words in lines) and any(set(words) - asked for words in lines)
+
 async def cited_answer(client, item, cited):
     """Words from the cited discoveries that say what the item asks for, by a yes-or-no question about them alone;
     '' when the answer is no, the reply is not valid, or its words are not in what was cited."""
@@ -53,7 +66,8 @@ async def cited_answer(client, item, cited):
         return ""
     quote = str(reply.get("quote", ""))
     said = str(reply.get("answer", "")).strip().lower() == "yes"
-    return quote if said and re.search(r"\w", quote) and any(quoted_from(quote, c) for c in cited) else ""
+    return quote if said and re.search(r"\w", quote) and any(
+        quoted_from(quote, c) or names_copied(quote, c, item) for c in cited) else ""
 
 def item_words(text, checklist=()):
     """An item's own words: content words (s-stripped) that no other checklist item has."""
@@ -67,6 +81,17 @@ def is_bare_citation(evidence):
     evidence = str(evidence)
     return bool(re.search(r"\d", evidence)) and not re.sub(r"\[\d+\]|[\s,;.]|\band\b", "", evidence)
 
+CONFIRMED_MISSING = ": named by no discovery; DISCOVER it where a page gives it"
+
+def concrete_term(part):
+    """Whether a missing part is a concrete thing a word check can confirm: at most three words, one of them with a
+    digit, an all-capitals abbreviation, or a capital inside the phrase ("O2", "GA3P", "PowerShell activation
+    command"), not a description ("the specific steps of the Calvin cycle")."""
+    words = part.replace(",", " ").split()
+    return 0 < len(words) <= 3 and any(
+        re.search(r"\d", w) or re.fullmatch(r"[A-Z][A-Z0-9]+", w) or (i > 0 and re.search(r"[A-Z]", w))
+        for i, w in enumerate(words))
+
 def really_missing(missing, found):
     """The parts of a "missing" verdict that the discoveries do not already contain.
 
@@ -109,7 +134,7 @@ async def coverage_gaps(client, objective, checklist, journal, only=None, eviden
     found = [e for e in journal if e.get("action") == "DISCOVER"]
     listing = "\n\n".join(f"[{n}] {e['extracted_data']}" for n, e in enumerate(found, 1))
     items = "\n".join(f"{n}. {item['text']}" for n, item in enumerate(checklist, 1))
-    gaps, unanswered, unproven = [], [], []
+    gaps, unanswered, unproven, answered_with = [], [], [], {}
     for n, item in enumerate(checklist, 1):
         if OPINION_REQUEST.search(item["text"]) or (only is not None and n not in only):
             continue
@@ -177,7 +202,15 @@ async def coverage_gaps(client, objective, checklist, journal, only=None, eviden
             unanswered.append(n)
             gaps.append(f"Item {n} ({item['text']}): {UNANSWERED_GAP}")
         elif missing:
-            gaps.append(f"Item {n} ({item['text']}): missing {missing}")
+            # A concrete thing ("O2", "GA3P", "Step 3") the check says is missing, and no discovery names, is confirmed
+            # by the code, so it sends the agent back: an explanation finished in 4 turns from one overview paragraph,
+            # without oxygen, while the check said "missing O2". Vague verdicts ("the specific steps") stay soft: about
+            # half were wrong. Of 117 saved "missing" verdicts, 2 were concrete (this one and "Step 3")
+            concrete = [p for p in missing.split(", ") if concrete_term(p)]
+            gaps.append(f"Item {n} ({item['text']}): missing {missing}"
+                        + (f" ({', '.join(concrete)}{CONFIRMED_MISSING})" if concrete else ""))
+        if n not in unanswered:
+            answered_with[n] = re.sub(r"\[\d+\]", " ", str(verdict.get("evidence", "")))
         # Words the check copied for an answered item, kept where they really are in the discoveries: the final
         # result is shown them, as it said "the discoveries do not specify what goes into photosynthesis" over a
         # discovery that gave water, sunlight and carbon dioxide
@@ -210,8 +243,63 @@ async def coverage_gaps(client, objective, checklist, journal, only=None, eviden
                 unproven.append((str(thing["name"]).strip(), condition))
                 gaps.append(f"Item {n}: {thing['name']}: no words in DISCOVERIES SO FAR show it is {condition}. SEARCH "
                             f"for '{thing['name']} {condition}' to check it, or replace it with one that is")
+    for n in shared_evidence_items(answered_with, checklist):
+        unanswered.append(n)
+        gaps.append(shared_gap(n, checklist[n - 1]["text"]))
     return gaps, unanswered, unproven
 
+def shared_gap(n, text):
+    """The gap for an item whose only evidence was another item's words. It ends with UNANSWERED_GAP, the mark of a
+    gap that sends the agent back: with the reason after it, v9 found such items on Box 1's comparison but did not
+    send the agent back, and the final result filled all three items from memory."""
+    return (f"Item {n} ({text}): the words given for it are the same as for another item and say nothing of its own, "
+            f"so {UNANSWERED_GAP}")
+
+# Units as pages abbreviate them after a number, by the word an item asks with (in item_words' s-stripped form)
+UNIT_WORDS = {"m": "meter", "metres": "metre", "km": "kilometer", "cm": "centimeter", "mm": "millimeter",
+              "ft": "feet", "mi": "mile", "in": "inche", "kg": "kilogram", "g": "gram", "lb": "pound", "lbs": "pound",
+              "oz": "ounce", "l": "liter", "ml": "milliliter", "min": "minute", "h": "hour", "hr": "hour",
+              "hrs": "hour", "c": "celsiu", "f": "fahrenheit"}
+UNIT_AFTER_NUMBER = re.compile(r"\d\s*(" + "|".join(sorted(UNIT_WORDS, key=len, reverse=True)) + r")\b", re.I)
+
+def units_written(text):
+    """The unit words a text gives by abbreviation after a number: "5,895 m (19,341 ft)" gives meter and feet. A
+    Kilimanjaro sentence answering both "in meters" and "in feet" was dropped for both, as if it said neither."""
+    found = {UNIT_WORDS[m.group(1).lower()] for m in UNIT_AFTER_NUMBER.finditer(str(text))}
+    return found | ({"foot"} if "feet" in found else set())
+
+def shared_evidence_items(answered_with, checklist):
+    """Items whose only evidence is the same words given for other items, none of its own words in them.
+
+    A comparison finished from one sentence ("I'd pick SQLite for any single-server app under roughly 10,000 daily
+    active users, and PostgreSQL the moment concurrent writes ... enter the picture"), given as the evidence for
+    setup, many users and features alike, and pasted under all three. Of items sharing the same words, those whose
+    own words appear in them keep them; the others count as unanswered. Items about the same thing ("Compare SQLite
+    and PostgreSQL features", "Name three features PostgreSQL has") may share words: they share a word of their own,
+    one that no more than half of the checklist items have ("compare", "sqlite" and "postgresql" in 3 of 4 do not
+    count)."""
+    def words_of(text):
+        return {w.rstrip("s") for w in plain_words(text) if len(w) > 2} - MISSING_STOPWORDS - ITEM_VERBS
+    texts = {n: words_of(item["text"]) for n, item in enumerate(checklist, 1)}
+    common = {w for w in set().union(*texts.values()) if sum(w in t for t in texts.values()) > len(texts) / 2}
+    dropped = set()
+    for n, words in answered_with.items():
+        same = [m for m, other in answered_with.items()
+                if m != n and quoted_from(words, other) and quoted_from(other, words)]
+        if not same or any((texts[n] & texts[m]) - common for m in same):
+            continue
+        given = item_words(words) | units_written(words)
+        overlap = {m: len(item_words(checklist[m - 1]["text"], checklist) & given) for m in [n] + same}
+        if overlap[n] == 0 or overlap[n] < max(overlap.values()):
+            dropped.add(n)
+    return sorted(dropped)
+
+SHORT_COUNT = re.compile(r"^Item (\d+) \(.*\): only (\d+) of (\d+) found \(")
+
+def short_counts(gaps):
+    """{item number: (things found, things asked for)} from the coverage gaps."""
+    return {int(m.group(1)): (int(m.group(2)), int(m.group(3))) for m in map(SHORT_COUNT.match, gaps) if m}
+
 async def coverage_refusal(client, objective, tried, journal):
     # Once per run, so a noisy verdict costs one turn at most and a gap that cannot be filled never blocks finishing
     checklist = tried.get("checklist")
@@ -222,9 +310,15 @@ async def coverage_refusal(client, objective, tried, journal):
         print("🔎 Checking each checklist item against the discoveries before finishing...")
         gaps, tried["unanswered"], unproven = await coverage_gaps(client, objective, checklist, journal,
                                                                     evidence=tried.setdefault("evidence", {}))
+        tried["short_counts"] = short_counts(gaps)
         # Items whose named things lacked proof of a condition are checked again before the final result
         tried["unproven_items"] = sorted({n for n, item in enumerate(checklist, 1) if any(
             g.startswith(f"Item {n}: ") and "no words in DISCOVERIES SO FAR show it is" in g for g in gaps)})
+        # Amounts asked for and given by no discovery of the item, decided in code: a cold brew run on Box 1 had the
+        # check's "missing quantities" (not sent back), its own FINISH drafts gave "1 cup (113 grams)", and the
+        # final result said the amounts were not found, because nothing made it record the card that gave them
+        gaps += [f"Item {n} ({checklist[n - 1]['text']}): {AMOUNT_GAP}"
+                 for n in amounts_not_found(checklist, journal, skip=set(tried["unanswered"]))]
     else:
         # A factual run finished with nothing recorded (its answer was refused as not on the page shown) and its
         # final result said nothing was found
@@ -235,7 +329,8 @@ async def coverage_refusal(client, objective, tried, journal):
     # four suite runs ("missing official documentation" on a comparison, "H2O, O2" already recorded, "free" on a
     # correct list of open-source suites) and cost a finish each; those verdicts now only shape the final result
     hard = [g for g in gaps if g.endswith(UNANSWERED_GAP) or re.search(r"\): only \d+ of \d+ found", g)
-            or g.startswith("Nothing has been recorded")]
+            or g.startswith("Nothing has been recorded") or g.endswith(AMOUNT_GAP)
+            or g.endswith(CONFIRMED_MISSING + ")")]
     if gaps:
         print("⚠️ Gaps found:\n" + "\n".join(f"   - {g}{'' if g in hard else ' (not sent back)'}" for g in gaps))
     gaps = hard
@@ -264,10 +359,14 @@ async def not_found_items(client, objective, tried, journal):
         return [], []
     candidates = sorted({n for n, item in enumerate(checklist, 1) if not item["discoveries"]
                          and not OPINION_REQUEST.search(item["text"])} | set(tried.get("unanswered") or []))
-    rechecked = sorted(set(candidates) | set(tried.get("unproven_items") or []))
+    # Items short of the number they ask for are counted again too: what the agent recorded after being sent back
+    # may complete them
+    short = tried.get("short_counts") or {}
+    rechecked = sorted(set(candidates) | set(tried.get("unproven_items") or []) | set(short))
     if not rechecked or not any(e.get("action") == "DISCOVER" for e in journal):
         return candidates, []
     print(f"🔎 Checking checklist item(s) {', '.join(map(str, rechecked))} once more before writing the final result...")
-    _, unanswered, unproven = await coverage_gaps(client, objective, checklist, journal, only=set(rechecked),
-                                                  evidence=tried.setdefault("evidence", {}))
+    gaps, unanswered, unproven = await coverage_gaps(client, objective, checklist, journal, only=set(rechecked),
+                                                     evidence=tried.setdefault("evidence", {}))
+    tried["short_counts"] = {n: count for n, count in short_counts(gaps).items() if n not in unanswered}
     return [n for n in unanswered if n in candidates or n in rechecked], unproven

{localagentbrowser-1.63 → localagentbrowser-1.64}/localagentbrowser/grounding.py

@@ -3,7 +3,7 @@ names never mentioned, quoted when one is refused."""
 import re
 import unicodedata
 from .config import GROUNDED_PAIR_SHARE, GROUNDED_SENTENCE_SHARE, KNOWN_LINE_THRESHOLD, MAX_CLOSEST_CHARS, MIN_GROUNDING_PAIRS, MIN_SENTENCE_PAIRS, UNGROUNDED_PAIR_SHARE, UNGROUNDED_SENTENCE_SHARE
-from .text import plain_words, shorten, word_pairs
+from .text import plain_words, shorten, with_digits, word_pairs
 from .pages import normalize_url
 
 QUOTE_STYLES = str.maketrans({"‘": "'", "’": "'", "“": "'", "”": "'", '"': "'", "`": "'"})
@@ -24,8 +24,9 @@ def discovery_lines(text):
 
 def grounded_share(text, parts):
     """The share of a discovery's consecutive word pairs that also appear in the given page parts."""
-    mine = word_pairs(str(text).lower())
-    return len(mine & word_pairs(" ".join(parts).lower())) / len(mine) if mine else 1.0
+    # "1 cup" and the page's "one cup" are the same words
+    mine = word_pairs(with_digits(text).lower())
+    return len(mine & word_pairs(with_digits(" ".join(parts)).lower())) / len(mine) if mine else 1.0
 
 def code_text(text):
     # Case, spacing, quote style and placeholder brackets ("<venv>/bin/activate") do not make code different
@@ -77,7 +78,31 @@ def command_lines(commands, parts):
     heads = [h for h in heads if len(h) > 3]
     return [shorten(line, MAX_CLOSEST_CHARS) for line in dict.fromkeys(x for x in lines if any(h in x.lower() for h in heads))][:2]
 
-def grounding_check(parameter, page_state, current_url, objective=""):
+OWN_DRAFT_SHARE = 0.6  # A DISCOVER sentence sharing this much of its word pairs with an earlier FINISH draft...
+OWN_PAGE_SHARE = 0.35  # ...and less than this with the page itself (without the objective) is the agent's own text
+
+
+def own_draft_sentence(parameter, page_parts, drafts):
+    """A sentence of the DISCOVER the agent wrote itself in an earlier FINISH, and the page does not give.
+
+    Its FINISH "For a hobby project with a few hundred users, PostgreSQL would be the recommended choice..." was
+    refused; three turns later that sentence came back appended to a real page line, as a DISCOVER, and the result
+    cited it for its recommendation. It scored as grounded because the objective counts as page text, and the
+    sentence is mostly the objective's own words; scored against the page alone, it is not there."""
+    if not drafts:
+        return None
+    mine = [word_pairs(" ".join(plain_words(d))) for d in drafts]
+    for s in re.split(r"(?<=[.!?])\s+|\n+", parameter):
+        pairs = word_pairs(" ".join(plain_words(s)))
+        if len(pairs) < MIN_SENTENCE_PAIRS:
+            continue
+        from_draft = any(len(pairs & d) >= OWN_DRAFT_SHARE * len(pairs) for d in mine)
+        if from_draft and grounded_share(s, page_parts) < OWN_PAGE_SHARE:
+            return s.strip()
+    return None
+
+
+def grounding_check(parameter, page_state, current_url, objective="", drafts=()):
     # Shown part 2 of a guide, which ended "Just follow these 6 easy steps", the model recorded six steps it
     # wrote itself (the real ones were in part 3), then a care routine the page never gave. Real discoveries
     # from earlier runs shared 64-100% of their word pairs with the page; these shared 11-44%, and a rerun's
@@ -100,6 +125,13 @@ def grounding_check(parameter, page_state, current_url, objective=""):
                                             f"write it. Commands are copied exactly as the page writes them{said}. "
                                             "DISCOVER that in the page's own words, or READ_MORE if a later part gives "
                                             "the command.")
+    # Refused outright, before the share checks: repeating it must not confirm it, since the agent wrote it
+    own = own_draft_sentence(parameter, shown[:-1], drafts)
+    if own:
+        return "BLOCKED_DISCOVER_OWN_WORDS", (f"\"{shorten(own, 160)}\" is what you wrote yourself when you tried to "
+                                              "FINISH, not what the page says. DISCOVER only the page's own words; your "
+                                              "conclusion or recommendation belongs in the final result, written at "
+                                              "FINISH.")
     long_enough = len(word_pairs(parameter)) >= MIN_GROUNDING_PAIRS
     whole = grounded_share(parameter, shown) if long_enough else 1.0
     # An invented sentence can hide behind copied ones: an opinion task's discovery was 75% copied, but its last