@tiagojct/subsub 0.6.1 → 0.8.0

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Tiago Jacinto
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -1,105 +1,243 @@
 # Sub-Sub
 
-A Zotero librarian and research assistant that runs on your computer, in your browser or in your terminal. It is built on the [pi](https://pi.dev) agent and works with your own Zotero library through Zotero's local API.
+[![npm version](https://img.shields.io/npm/v/@tiagojct/subsub.svg?style=flat-square&color=0b62cf)](https://www.npmjs.com/package/@tiagojct/subsub)
+[![Website](https://img.shields.io/badge/website-subsub.tiagojacinto.eu-e0832a.svg?style=flat-square)](https://subsub.tiagojacinto.eu)
+[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg?style=flat-square)](https://opensource.org/licenses/MIT)
+[![Node Version](https://img.shields.io/badge/node-%3E%3D22.19.0-339933.svg?style=flat-square)](https://nodejs.org)
+[![Platform: macOS | Linux | Windows](https://img.shields.io/badge/platform-macOS%20%7C%20Linux%20%7C%20Windows-lightgrey.svg?style=flat-square)](#requirements)
 
-> "This mere painstaking burrower and grub-worm of a poor devil of a Sub-Sub appears to have gone through the long Vaticans and street-stalls of the earth, picking up whatever random allusions to whales he could anyways find in any book whatsoever." (Moby-Dick, Extracts)
+A Zotero librarian and research assistant that runs on your computer, in your browser or in your terminal. It is built on the [pi](https://pi.dev) agent and connects directly to your local Zotero library through Zotero's local API.
 
-Sub-Sub has two modes:
+> *"This mere painstaking burrower and grub-worm of a poor devil of a Sub-Sub appears to have gone through the long Vaticans and street-stalls of the earth, picking up whatever random allusions to whales he could anyways find in any book whatsoever."*  
+> — Herman Melville, *Moby-Dick* (Extracts)
 
-- The librarian changes the library: tags from your own tag list, imports by DOI, PMID or ISBN, metadata repair, duplicates, retractions, open-access PDFs, notes and collections. It has no web access.
-- The researcher reads the library, searches PubMed and OpenAlex, and writes notes in your notes folder. It cannot change the library, except to add a short linked note to an item.
+---
 
-Every library change is shown as a preview and runs only after you approve it. The model cannot skip this: the check is in the code, not in the prompt. Every change is journaled and can be undone with `/undo`.
+## Highlights
+
+* **Code-Enforced Safety Gate**: The model **cannot** modify your library without your explicit approval. Every change is calculated first as a dry-run preview (+added in green, -removed in red) and executed only when you confirm.
+* **Full Journal & Reversible Edits**: Every library modification is recorded with a unique journal ID and can be reverted anytime using `/undo`.
+* **Two Dedicated Roles**:
+  * 📚 **The Librarian**: Cleans tags, imports references (DOI, PMID, ISBN), deduplicates, fixes metadata, retrieves open-access PDFs, creates collections, and checks retractions. *Runs completely offline with no web search access.*
+  * 🔬 **The Researcher**: Reads items, searches PubMed and OpenAlex, finds gaps, checks citations in manuscripts, and writes structured Markdown notes in your notes vault (e.g. Obsidian). *Cannot modify library records.*
+* **Local & Private**: Listens strictly on `127.0.0.1` protected by an authenticated single-origin session cookie. Your library contents remain on your computer.
+* **Modern Web Interface & Terminal UI**: Use Sub-Sub in your browser with desktop shortcuts (`~/Applications/Sub-Sub.app`, Start menu, or `.desktop`), or launch it right in your terminal with adaptive color themes.
+
+---
 
 ## Profiles
 
-A profile sets how much Sub-Sub does for you. Choose it in `subsub init` and change it with `/profile`.
+Profiles calibrate how much Sub-Sub drafts and executes for you. Choose your profile during `subsub init` or change it on the fly with `/profile <name>`.
 
-| Profile | What it does |
+| Profile | Description |
 |---|---|
-| Reader | Explains each step. Reading notes are quotes with page numbers, plus questions for you to answer. No summaries or syntheses. No bulk library changes. |
-| Scholar | Literature notes, syntheses, searches and alerts. The full librarian. |
-| Author | Scholar, plus manuscripts: citation check against the library, bibliography for Quarto or Pandoc, comments on the argument. It does not write your paragraphs. |
-| Editor | Everything, including drafting manuscript text when you ask. |
+| **Reader** | Guided and instructional. Reading notes are quotes with page numbers and reflective questions. Never generates unrequested summaries or syntheses. No bulk library edits. |
+| **Scholar** | Full research workflow: literature notes, conceptual syntheses, database searches, alerts, and the full librarian toolkit. |
+| **Author** | Scholar capabilities plus manuscript workflows: citation auditing against your library, automated Pandoc/Quarto bibliographies, and argument analysis. It critiques but does not write your paragraphs. |
+| **Editor** | Everything unlocked, including drafting full manuscript sections and abstracts upon request. |
 
-A profile is your own choice, not a lock.
+> [!NOTE]
+> A profile is your own operational baseline, not a rigid constraint. You can switch between profiles whenever your research focus shifts.
+
+---
 
 ## Requirements
 
-- Zotero 10 or later, running. In Zotero, open Settings > Advanced and turn on "Allow other applications on this computer to communicate with Zotero".
-- Node.js 22.19 or later.
-- uv (https://docs.astral.sh/uv/).
-- An account with a model provider that pi supports (for example OpenCode Go, OpenRouter, Anthropic, OpenAI, or a local Ollama).
-- macOS, Windows or Linux.
+* **Zotero 10 or later** running locally. In Zotero, enable:  
+  *Settings > Advanced > "Allow other applications on this computer to communicate with Zotero"*.
+* **Node.js 22.19** or later.
+* **[uv](https://docs.astral.sh/uv/)** fast Python package manager.
+* **Model Provider Account**: Any provider supported by pi (e.g., OpenCode Go, OpenRouter, Anthropic, OpenAI, Mistral, DeepSeek, Google Gemini, or local Ollama).
+* **OS**: macOS, Linux, or Windows.
+
+---
 
 ## Install
 
-The installers on [subsub.tiagojacinto.eu](https://subsub.tiagojacinto.eu/install/) do all of this, including Node.js and uv. By hand:
+### Quick Installers
+
+Automated installers handle Node.js, uv, and desktop shortcuts:
+
+* **macOS & Linux**:
+  ```sh
+  curl -fsSL https://subsub.tiagojacinto.eu/install.sh | sh
+  ```
+* **Windows (PowerShell)**:
+  ```powershell
+  powershell -ExecutionPolicy ByPass -c "irm https://subsub.tiagojacinto.eu/install.ps1 | iex"
+  ```
+
+### Manual Installation
+
+1. Install the global package:
+   ```sh
+   npm install -g @tiagojct/subsub
+   ```
+2. Run setup:
+   ```sh
+   subsub init
+   ```
+   Answer the interactive prompts (press <kbd>Enter</kbd> to accept defaults). This creates your notes folder (`Inbox/` and `Systems/`), sets up starter tag vocabularies, and writes note templates (`Systems/Zotero agent.md`).
+3. Verify your environment:
+   ```sh
+   subsub doctor
+   ```
+   Resolve any items marked with `FIX`.
+4. Install system shortcuts:
+   ```sh
+   subsub shortcut
+   ```
+   Creates `~/Applications/Sub-Sub.app` on macOS, Start menu & desktop shortcuts on Windows, or an application launcher on Linux.
+5. Launch Sub-Sub:
+   Open the shortcut or run `subsub web`. Connect your model provider via the **Model** dialog with an API key (or in the terminal via `subsub`, then `/login`).
+
+If you already use [pi](https://pi.dev), you can add Sub-Sub as an extension:
+```sh
+pi install npm:@tiagojct/subsub
+```
+
+The underlying Python servers ([zotero-local-mcp](https://github.com/tiagojct/zotero-local-mcp)) run via `uv` from PyPI. To run from a local checkout, specify `"serverDir": "/path/to/zotero-local-mcp"` in your configuration.
 
-1. Type `npm install -g @tiagojct/subsub`. The command is `subsub`.
-2. Type `subsub init`. Answer the questions. Press Enter to keep a default.
-3. Type `subsub doctor`. Fix each line marked FIX.
-4. Type `subsub shortcut` to add a Sub-Sub shortcut (Applications on macOS, the Start menu and the desktop on Windows, the applications menu on Linux).
-5. Open Sub-Sub with the shortcut, or type `subsub web`. Select the model button and connect a provider with an API key (or, in the terminal, type `subsub`, then `/login`).
+---
 
-`subsub init` creates a notes folder with Inbox/ and Systems/, a starter tag list (health sciences, health informatics, or any field) and a file with the note formats (Systems/Zotero agent.md). You can edit both files. It never replaces a file that exists.
+## Web View
 
-If you already use pi, you can also add Sub-Sub to plain pi: `pi install npm:@tiagojct/subsub`.
+Starting Sub-Sub with `subsub web` (or `subsub serve`) launches the browser application:
 
-The Zotero servers (zotero-local-mcp) run from PyPI through uv. To run them from a copy of the repository, add `"serverDir": "/path/to/zotero-local-mcp"` to the settings file.
+* **Approval Previews**: Shows server-computed dry-run diffs directly in the UI. Changes are applied only after clicking **Yes** or pressing <kbd>Y</kbd>.
+* **Keyboard-First Review**: In approval dialogs, press <kbd>Y</kbd> or <kbd>Cmd</kbd>/<kbd>Ctrl</kbd>+<kbd>Enter</kbd> to approve, or <kbd>N</kbd>/<kbd>Esc</kbd> to decline.
+* **Theme Switcher**: Header toggle between **Light**, **Dark**, and **System** themes, with tailored styling for each mode (Try-Works amber for Librarian, Glauca blue for Researcher).
+* **One-Click Copy**: Built-in copy buttons on all code snippets, BibTeX records, and assistant literature notes.
+* **Session Search & Markdown Export**: Filter conversation history in real time and download full discussions as Markdown notes.
+* **Local Security**: Binds strictly to `127.0.0.1`. The initial link carries a random token that converts into an `HttpOnly`, `SameSite=Strict` cookie. Protected with strict Content Security Policy (CSP), origin validation, and sanitized output.
+* **Auto-Idle Management**: A single daemon serves requests and stops automatically after 10 minutes of inactivity when no pages remain open. Use `--stay` to keep running indefinitely, or `--idle N` to change the timeout.
 
-## Web view
+---
 
-`subsub web` (or `subsub serve`) opens Sub-Sub in your browser. It has the same modes, profiles, tools and approvals as the terminal:
+## Usage
+
+### Interactive Modes & Commands
+
+```sh
+subsub             # Start in researcher mode
+subsub --librarian # Start in librarian mode
+subsub web         # Start browser view
+subsub -c          # Continue the previous session
+subsub -r          # Resume an older session
+```
+
+Commands available within Sub-Sub:
+
+* `/librarian`: Switch to the librarian (modify library, tag, import; offline).
+* `/researcher`: Switch to the researcher (read library, search literature, write notes).
+* `/profile [name]`: View or change active profile (`reader`, `scholar`, `author`, `editor`).
+* `/subsub`: Display Zotero connection health and library statistics.
+* `/history`: List recent library changes with journal IDs.
+* `/undo [id]`: Revert the last change (or a specific journal ID).
+
+### Built-in Prompt Templates
+
+Type the command followed by arguments:
+
+* **Librarian**:
+  * `/tag-batch [size]`: Tag untagged items using your tag vocabulary.
+  * `/clean-tags`: Find irregular tags and reconcile with your vocabulary.
+  * `/import-queue [path]`: Import pending references from an inbox note.
+* **Researcher**:
+  * `/lit <question>`: Literature review: the library first, then PubMed, Europe PMC and OpenAlex in one merged search; reads before it summarises; an evidence table, agreement, disagreement and open questions; references checked; one note in `Research/` (Scholar, Author, Editor).
+  * `/compare <question or citekeys>`: Source matrix (claim, evidence type, caveats, confidence), agreements and disagreements (Scholar, Author, Editor).
+  * `/review <path>`: Critical review of your own manuscript: FATAL, MAJOR and MINOR issues against the reporting guideline that applies (CONSORT, STROBE, PRISMA, STARD, TRIPOD), with quotations and a revision plan. Comments only (Author, Editor).
+  * `/digest [days] [topic]`: Digest of the last days (default 7): what the new notes and alerts change, works worth reading, open questions, decisions for you (Scholar, Author, Editor).
+  * `/request-copy <citekey, DOI or PMID>`: Draft an email to the corresponding author asking for a copy of a paper that is not open access. The address comes from the PubMed record; you send the email yourself.
+  * `/lit-note <citekey>`: Generate an in-depth literature note from item fulltext.
+  * `/synthesis <topic>`: Synthesize library findings on a specific topic.
+  * `/gaps <topic>`: Identify unanswered questions and literature gaps.
+  * `/manuscript <path>`: Cross-check manuscript citations against your library.
+  * `/verify <path> [claims]`: Check that each cited work exists, matches its citation and was not retracted (needs the Starbuck add-on). With `claims`, also check whether each source supports the sentence that cites it.
+  * `/alert`: Check PubMed and OpenAlex for new publications matching research topics.
+
+### Batch Tag Review
+
+To preview or apply batch tag reviews generated in your `Inbox/`:
+```sh
+subsub review preview 13-31
+subsub review apply 13-31
+```
 
-- pi runs in RPC mode with the Sub-Sub extension. Sub-Sub's approval dialog (the server preview) appears in the page, and the change runs only after you select Yes.
-- The server listens only on 127.0.0.1. The address that opens the page carries a random key, which becomes an HttpOnly, SameSite=Strict cookie; other requests need that cookie, the right Host header and a same-origin JSON request. The page has a strict content security policy and shows model output without raw HTML.
-- One server per user: a second `subsub web` opens the running one. Without an open page for 10 minutes (and with nothing waiting for approval), the server stops. `--stay` keeps it running; `--idle N` sets the minutes; `--no-open` does not open a browser; `--librarian` starts in the librarian mode.
-- Labels are in European Portuguese when the language setting is Portuguese, otherwise in English. A theme button in the header toggles light, dark, and system themes.
-- Fast approval shortcuts (`y` to apply, `n` to decline), one-click copy buttons on code blocks and notes, conversation search, and export to Markdown.
-- The model button lists the models you can use and saves an API key for a provider in the same file as `/login`. Sign-ins through a provider's website stay in the terminal (`/login`).
-- A log is kept in `~/.subsub/web.log`.
+---
 
-## Use
+## Configuration
 
-- `subsub` starts in the researcher mode. `subsub --librarian` starts in the librarian mode. `/librarian` and `/researcher` change the mode.
-- `/profile` shows or changes the profile. `/subsub` shows the Zotero connection. `/history` lists recent library changes. `/undo` reverts one.
-- Templates: `/tag-batch`, `/clean-tags`, `/import-queue` (librarian); `/lit-note <citekey>`, `/synthesis <topic>`, `/gaps <topic>`, `/manuscript <file>`, `/alert` (researcher).
-- `subsub review preview 13-31` and `subsub review apply 13-31` preview or apply tag review notes by number.
-- `subsub -c` continues the last session. `subsub -r` selects an older one.
+`subsub init` writes settings to `~/.config/subsub/config.json` (or `$SUBSUB_CONFIG`):
 
-## Settings
+| Setting | Type | Description | Default |
+|---|---|---|---|
+| `profile` | string | `reader`, `scholar`, `author`, or `editor` | `"scholar"` |
+| `userName` | string | Name used by the agent in replies | `"the user"` |
+| `about` | string | Brief background context about your research domain | `null` |
+| `language` | string | Response language, or `"auto"` to match your input | `"English"` |
+| `vault` | string | Path to your Markdown notes folder / Obsidian vault | `$ZOTERO_VAULT` |
+| `models` | object | Model overrides per mode (e.g. `{"librarian": "opencode-go/glm-5.3-flash"}`) | Current model |
+| `defaultMode`| string | Default launch mode (`"researcher"` or `"librarian"`) | `"researcher"` |
+| `serverDir` | string | Local development path for `zotero-local-mcp` | PyPI package |
+| `look` | boolean| Show header banners and status lines in terminal | `true` |
+| `quotes` | boolean| Display quotes from *Moby-Dick* in terminal banner | `true` |
+| `themes` | object | Mode-specific terminal themes | `true` |
+| `addons` | array | Optional servers; `["starbuck"]` adds reference checks | `[]` |
+| `starbuckDir` | string | Local Starbuck folder (used when it has a `pyproject.toml`) | `~/Projects/starbuck` |
 
-`subsub init` writes `~/.config/subsub/config.json` (or the file in `SUBSUB_CONFIG`):
+Server-level configurations (notes folder path, tag vocabulary path, Unpaywall email) reside in `~/.config/zotero-local-mcp/env`.
 
-| Setting | Meaning | Default |
-|---|---|---|
-| `profile` | reader, scholar, author or editor | scholar |
-| `userName` | how Sub-Sub names you | "the user" |
-| `about` | one line about you | none |
-| `language` | language of replies and notes, or `auto` | English |
-| `vault` | notes folder | `ZOTERO_VAULT` from the server settings |
-| `models` | model per mode, e.g. `{ "librarian": "opencode-go/glm-5.3-flash" }` | pi's current model |
-| `defaultMode` | librarian or researcher | researcher |
-| `serverDir` | a local copy of zotero-local-mcp | the PyPI release |
-| `look`, `quotes`, `themes` | banner, Moby-Dick line, themes per mode | on |
+### Papers that are not open access
 
-The server settings (notes folder, tag list, contact email for Unpaywall) are in `~/.config/zotero-local-mcp/env`.
+Sub-Sub finds open-access copies (Unpaywall, Europe PMC). For other papers, use Sub-Sub on your institution's network (at FMUP: the U.Porto network), where the DOI link opens the publisher's version through the library's subscriptions, or use `/request-copy` to ask the author. Sub-Sub does not use unlicensed sources.
 
-## Model test
+### Reference checks (Starbuck add-on)
 
-`subsub-bench prepare`, `subsub-bench run` and `subsub-bench score` run the same tasks for several models against your library. Every library change is recorded and blocked, and files are written only in the run folder. The results show tag accuracy, invented references, tokens, cost and time.
+[Starbuck](https://github.com/tiagojct/starbuck) checks references: does each cited work exist, do the title, first author and year match, was it retracted or corrected, and (on request) does the source support the sentence that cites it.
 
-## Tests
+1. Add the add-on to the settings file:
+   ```json
+   "addons": ["starbuck"]
+   ```
+2. Optional: set `"starbuckDir"` to a local copy of Starbuck. Without it, Sub-Sub runs the pinned Starbuck release with uv.
+3. Start Sub-Sub again and run `subsub doctor`. The line `Starbuck` must show `ok`.
+
+The researcher gets four tools: `verify_check_manuscript`, `verify_check_references`, `verify_prepare_claims` and `verify_record_claims`. Starbuck writes its report to a `_starbuck` folder next to the manuscript. Inside the notes folder or the working folder this needs no approval; elsewhere Sub-Sub asks first. Starbuck gets the contact email from the zotero-local-mcp env file. For the claim check, Sub-Sub's own model judges; Starbuck checks that every quoted passage is in the source.
+
+---
+
+## Model Evaluation (`subsub-bench`)
+
+Evaluate how accurately different LLMs tag, import, and synthesize literature against your own library without applying changes:
 
 ```sh
+subsub-bench prepare    # Sample realistic test tasks from your library
+subsub-bench run        # Run models in parallel against safety-isolated sandbox
+subsub-bench score      # Generate objective evaluation scorecard (results.md)
+```
+
+Measures tag accuracy, adherence to vocabulary, hallucinated citations, token usage, cost, and latency.
+
+---
+
+## Development & Testing
+
+```sh
+git clone https://github.com/tiagojct/subsub.git
+cd subsub
 npm install
 npm test
+npm run build
+```
+
+Run end-to-end integration tests (real agent, Python MCP servers, fake Zotero, scripted LLM):
+```sh
 ZLM_DIR=/path/to/zotero-local-mcp npm run test:e2e
 ```
 
-The end-to-end tests run the real pi with Sub-Sub, the real Python servers, a fake Zotero and a scripted model.
+---
 
-## Licence
+## License
 
-MIT.
+[MIT](LICENSE) © Tiago Jacinto. The `/lit`, `/compare` and `/review` prompts adapt text from [Feynman](https://github.com/Companion-Inc/feynman) (MIT); see [THIRD_PARTY_NOTICES](THIRD_PARTY_NOTICES).

package/THIRD_PARTY_NOTICES

@@ -0,0 +1,153 @@
+Third-party notices
+
+prompts/lit.md, prompts/compare.md, prompts/review.md and the integrity rules in
+roles/researcher.md adapt text from Feynman (prompts/lit.md, prompts/deepresearch.md,
+prompts/compare.md, prompts/review.md and .feynman/agents/researcher.md),
+https://github.com/Companion-Inc/feynman, commit 39f3ece. Its licence:
+
+MIT License
+
+Copyright (c) 2026 Companion, Inc.
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.
+
+
+prompts/digest.md takes its section structure ("What changed in our understanding",
+"Questions for the user") from alberto-research, src/alberto_research/digest.py,
+https://github.com/gabriel-affonso/alberto-research. Its licence:
+
+MIT License
+
+Copyright (c) 2025 Gabriel Affonso
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.
+
+
+web/fonts/ and the website use IBM Plex Sans, IBM Plex Serif and IBM Plex Mono,
+https://github.com/IBM/plex. Their licence:
+
+Copyright © 2017 IBM Corp. with Reserved Font Name "Plex"
+
+This Font Software is licensed under the SIL Open Font License, Version 1.1.
+
+This license is copied below, and is also available with a FAQ at: http://scripts.sil.org/OFL
+
+
+-----------------------------------------------------------
+SIL OPEN FONT LICENSE Version 1.1 - 26 February 2007
+-----------------------------------------------------------
+
+PREAMBLE
+The goals of the Open Font License (OFL) are to stimulate worldwide
+development of collaborative font projects, to support the font creation
+efforts of academic and linguistic communities, and to provide a free and
+open framework in which fonts may be shared and improved in partnership
+with others.
+
+The OFL allows the licensed fonts to be used, studied, modified and
+redistributed freely as long as they are not sold by themselves. The
+fonts, including any derivative works, can be bundled, embedded, 
+redistributed and/or sold with any software provided that any reserved
+names are not used by derivative works. The fonts and derivatives,
+however, cannot be released under any other type of license. The
+requirement for fonts to remain under this license does not apply
+to any document created using the fonts or their derivatives.
+
+DEFINITIONS
+"Font Software" refers to the set of files released by the Copyright
+Holder(s) under this license and clearly marked as such. This may
+include source files, build scripts and documentation.
+
+"Reserved Font Name" refers to any names specified as such after the
+copyright statement(s).
+
+"Original Version" refers to the collection of Font Software components as
+distributed by the Copyright Holder(s).
+
+"Modified Version" refers to any derivative made by adding to, deleting,
+or substituting -- in part or in whole -- any of the components of the
+Original Version, by changing formats or by porting the Font Software to a
+new environment.
+
+"Author" refers to any designer, engineer, programmer, technical
+writer or other person who contributed to the Font Software.
+
+PERMISSION & CONDITIONS
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of the Font Software, to use, study, copy, merge, embed, modify,
+redistribute, and sell modified and unmodified copies of the Font
+Software, subject to the following conditions:
+
+1) Neither the Font Software nor any of its individual components,
+in Original or Modified Versions, may be sold by itself.
+
+2) Original or Modified Versions of the Font Software may be bundled,
+redistributed and/or sold with any software, provided that each copy
+contains the above copyright notice and this license. These can be
+included either as stand-alone text files, human-readable headers or
+in the appropriate machine-readable metadata fields within text or
+binary files as long as those fields can be easily viewed by the user.
+
+3) No Modified Version of the Font Software may use the Reserved Font
+Name(s) unless explicit written permission is granted by the corresponding
+Copyright Holder. This restriction only applies to the primary font name as
+presented to the users.
+
+4) The name(s) of the Copyright Holder(s) or the Author(s) of the Font
+Software shall not be used to promote, endorse or advertise any
+Modified Version, except to acknowledge the contribution(s) of the
+Copyright Holder(s) and the Author(s) or with their explicit written
+permission.
+
+5) The Font Software, modified or unmodified, in part or in whole,
+must be distributed entirely under this license, and must not be
+distributed under any other license. The requirement for fonts to
+remain under this license does not apply to any document created
+using the Font Software.
+
+TERMINATION
+This license becomes null and void if any of the above conditions are
+not met.
+
+DISCLAIMER
+THE FONT SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT
+OF COPYRIGHT, PATENT, TRADEMARK, OR OTHER RIGHT. IN NO EVENT SHALL THE
+COPYRIGHT HOLDER BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
+INCLUDING ANY GENERAL, SPECIAL, INDIRECT, INCIDENTAL, OR CONSEQUENTIAL
+DAMAGES, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+FROM, OUT OF THE USE OR INABILITY TO USE THE FONT SOFTWARE OR FROM
+OTHER DEALINGS IN THE FONT SOFTWARE.

package/dist/bench.js

@@ -15,7 +15,7 @@ import { spawn, spawnSync } from "node:child_process";
 import { existsSync, mkdirSync, readdirSync, readFileSync, writeFileSync } from "node:fs";
 import { dirname, join, resolve } from "node:path";
 import { fileURLToPath } from "node:url";
-import { expand, loadConfig } from "./config.js";
+import { expand, loadConfig, starbuckCommand, starbuckEnabled } from "./config.js";
 import { findUv } from "./subsub.js";
 export const PACKAGE_DIR = resolve(dirname(fileURLToPath(import.meta.url)), "..");
 export const DEFAULT_MODELS = {
@@ -24,7 +24,7 @@ export const DEFAULT_MODELS = {
 };
 export const TASKS = {
     librarian: ["tagging", "facet_fix", "import"],
-    researcher: ["lit_note", "synthesis", "search"],
+    researcher: ["lit_note", "synthesis", "search", "lit"],
 };
 // ---------------------------------------------------------------- prompts
 function about(t) {
@@ -60,6 +60,12 @@ export function promptFor(task, t, code, out) {
                 ? `Write a synthesis of what my library says about ${about(t)}, following the synthesis format in the shared rules. ` +
                     `This is a model test: save it as ${join(out, "Synthesis.md")}.`
                 : undefined;
+        case "lit":
+            // The /lit template, with the model-test paths in its arguments.
+            return t.synthesis
+                ? `/lit ${about(t)}. This is a model test: write the review as ${join(out, "Lit.md")}, the plan and the screening log ` +
+                    `in ${join(out, ".plans")}, nothing in Research/, and do not queue imports.`
+                : undefined;
         case "search":
             return t.synthesis
                 ? `Find up to 5 important papers published from ${t.search.year_from} onwards on ${about(t)} that are not yet in my library. ` +
@@ -242,8 +248,30 @@ export function citekeysIn(text) {
         out.add(m[1]);
     return [...out];
 }
-// ---------------------------------------------------------------- score one run
-export function scoreRun(task, dir, t, index, blindRef) {
+export function starbuckScorer(cfg = loadConfig(), env = process.env) {
+    if (!starbuckEnabled(cfg))
+        return undefined;
+    const cmd = starbuckCommand(cfg, env);
+    return (file) => {
+        const claims = env.STARBUCK_JUDGE_URL && env.STARBUCK_JUDGE_MODEL ? ["--claims"] : [];
+        const args = [...cmd.args.slice(0, -1), "starbuck", "check", file, "--to", "", "--quiet", ...claims];
+        const r = spawnSync(cmd.command, args, { env, encoding: "utf8", timeout: 20 * 60_000 });
+        const report = join(dirname(file), "_starbuck", `${file.split(/[\\/]/).pop().replace(/\.[^.]+$/, "")}-references.json`);
+        if (!existsSync(report))
+            return { error: (r.stderr || r.error?.message || "no report").trim().slice(0, 300) };
+        const res = JSON.parse(readFileSync(report, "utf8"));
+        const run = res.claims_run ?? {};
+        return {
+            references: res.summary?.references_checked,
+            fail: res.summary?.fail,
+            check: res.summary?.warn,
+            uncited_sentences: res.summary?.uncited_sentences,
+            citation_recall: run.citation_recall?.value,
+            citation_precision: run.citation_precision?.value,
+        };
+    };
+}
+export function scoreRun(task, dir, t, index, blindRef, starbuck) {
     const events = readEvents(dir);
     const calls = toolCalls(events);
     const entries = benchEntries(events);
@@ -359,6 +387,29 @@ export function scoreRun(task, dir, t, index, blindRef) {
         }
         return res;
     }
+    if (task === "lit") {
+        const text = readFile("Lit.md") ?? "";
+        const seen = calls.map((c) => c.result).join("\n").toLowerCase();
+        const dois = [...new Set([...text.matchAll(DOI_RE)].map((m) => normDoi(m[0])))];
+        const pmids = [...new Set([...text.matchAll(PMID_RE)].map((m) => m[1]))];
+        const rows = text.split("\n").filter((l) => /^\|\s*\[?\d+\]?\s*\|/.test(l));
+        const readCol = rows.map((l) => l.split("|").map((c) => c.trim().toLowerCase())).map((c) => c.find((x) => /^(full text|abstract|metadata)$/.test(x)));
+        const plans = join(dir, ".plans");
+        return {
+            ...base,
+            written: !!text,
+            words: text ? text.split(/\s+/).filter(Boolean).length : 0,
+            evidence_rows: rows.length,
+            evidence_labelled: readCol.filter(Boolean).length,
+            dois: dois.length,
+            pmids: pmids.length,
+            ungrounded: [...dois.filter((d) => !seen.includes(d)), ...pmids.filter((p) => !seen.includes(p))],
+            screening_log: existsSync(plans) && readdirSync(plans).some((f) => /screening/i.test(f)),
+            multi_searches: calls.filter((c) => c.name === "scholar_search_multi").length,
+            fulltext_reads: calls.filter((c) => c.name === "scholar_read_oa_fulltext" || c.name === "zotero_get_fulltext").length,
+            starbuck: text && starbuck ? starbuck(join(dir, "Lit.md")) : undefined,
+        };
+    }
     if (task === "search") {
         const text = readFile("Search.md") ?? "";
         const seen = calls.map((c) => c.result).join("\n").toLowerCase();
@@ -401,6 +452,9 @@ export function renderResults(rows) {
         lit_note: ["written", "words", "read_fulltext", "cites_itself", "citekeys_unknown"],
         synthesis: ["written", "words", "topic_items", "topic_items_cited", "citekeys_unknown"],
         search: ["written", "dois", "pmids", "ungrounded", "already_in_library", "queued", "searches"],
+        lit: ["written", "words", "evidence_rows", "evidence_labelled", "dois", "pmids", "ungrounded", "screening_log",
+            "multi_searches", "fulltext_reads", "starbuck.fail", "starbuck.check", "starbuck.uncited_sentences",
+            "starbuck.citation_recall", "starbuck.citation_precision"],
     };
     const common = ["seconds", "tool_calls", "tool_errors", "wrong_mode", "tokens_in", "tokens_out", "cost", "error"];
     for (const task of Object.keys(cols)) {
@@ -516,13 +570,14 @@ export async function main(argv = process.argv.slice(2)) {
         const key = JSON.parse(readFileSync(join(dir, "key.json"), "utf8"));
         const blind = existsSync(join(dir, "reference.json")) ? JSON.parse(readFileSync(join(dir, "reference.json"), "utf8")) : {};
         const rows = [];
+        const scorer = starbuckScorer(cfg);
         for (const code of Object.keys(key).sort()) {
             for (const role of ["librarian", "researcher"])
                 for (const task of TASKS[role]) {
                     const d = join(dir, "runs", code, task);
                     if (!existsSync(join(d, "summary.json")))
                         continue;
-                    rows.push({ model: key[code], role, task, s: scoreRun(task, d, t, index, blind) });
+                    rows.push({ model: key[code], role, task, s: scoreRun(task, d, t, index, blind, scorer) });
                 }
         }
         writeFileSync(join(dir, "results.json"), JSON.stringify(rows, null, 1));