lyrenth-research 0.1.0__tar.gz

lyrenth_research-0.1.0/.gitignore

@@ -0,0 +1,7 @@
+.venv/
+__pycache__/
+*.egg-info/
+dist/
+build/
+.env
+context.md

lyrenth_research-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Aleksma AI, Inc.
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

lyrenth_research-0.1.0/PKG-INFO

@@ -0,0 +1,181 @@
+Metadata-Version: 2.5
+Name: lyrenth-research
+Version: 0.1.0
+Summary: A research agent that reads its sources: give it a question and URLs, get an answer with numbered citations.
+Project-URL: Homepage, https://lyrenth.com
+Project-URL: Documentation, https://lyrenth.com/docs/quickstart
+Author: Lyrenth
+License-Expression: MIT
+License-File: LICENSE
+Keywords: agent,aidocument,citations,llm,lyrenth,rag,research,web
+Classifier: Environment :: Console
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Requires-Python: >=3.9
+Requires-Dist: lyrenth>=0.1.1
+Description-Content-Type: text/markdown
+
+# lyrenth-research
+
+A research agent that actually reads its sources.
+
+Every AI answer that reads the web pays for reading pages, and Lyrenth
+makes that reading several times cheaper. This agent is the easiest way to
+see it.
+
+Give it a question and a list of URLs. It reads every page through
+[Lyrenth](https://lyrenth.com) as clean text, drops duplicates, stays inside
+a token budget, tells you which pages it could not read, and asks your model
+for an answer with numbered citations. Then it checks that every citation
+points at a source it really read.
+
+```bash
+pip install lyrenth-research
+export LYRENTH_API_KEY=...            # free key: https://lyrenth.com/signup
+export LLM_BASE_URL=http://localhost:11434/v1   # any OpenAI-compatible endpoint
+export LLM_MODEL=your-model-name
+
+lyrenth-research "What is the difference between a web crawler and web indexing?" \
+  https://en.wikipedia.org/wiki/Web_indexing \
+  https://en.wikipedia.org/wiki/Web_crawler
+```
+
+## Why
+
+Most research agents get the finding and the writing right and treat the
+reading as one line: fetch the page, strip some tags, paste it into the
+prompt. That line decides more about the answer than the prompt does. An
+agent that reads badly pays for navigation menus and cookie banners, reads
+the same article twice under two URLs, cites pages it never really read,
+and quietly answers from nothing when a source fails.
+
+This agent does five things on purpose:
+
+1. **Reads clean text, not HTML.** Every page arrives as an AIDocument:
+   Markdown content, the canonical URL, when it was read, and how many
+   tokens it costs.
+2. **Keeps provenance.** Each source keeps its title, canonical URL and
+   read time, and all three go into the prompt.
+3. **Reads each document once.** A mobile copy, a tracking parameter or an
+   old redirect resolves to the same canonical URL and is dropped.
+4. **Stays inside a budget.** Sources are added in the order you give them
+   until the next one would exceed the budget. Put the ones you trust most
+   first.
+5. **Fails visibly.** Every page that was skipped is listed with the reason,
+   and a citation to a source number that does not exist is flagged.
+
+## A real run
+
+Reading step only (`--sources-only`), against the live API on September 19,
+2026, unedited. The progress report goes to stderr, the numbered context to
+stdout (here, `context.md`).
+
+```text
+$ lyrenth-research "What is the difference between a web crawler and web indexing?" \
+    https://en.wikipedia.org/wiki/Web_indexing \
+    https://en.wikipedia.org/wiki/Web_crawler \
+    https://en.wikipedia.org/wiki/Robots.txt \
+    https://en.wikipedia.org/wiki/No_such_page_for_this_example \
+    https://en.m.wikipedia.org/wiki/Web_indexing \
+    --sources-only > context.md
+skipped  https://en.wikipedia.org/wiki/Robots.txt: over budget (16,719 tokens would exceed 30,000)
+skipped  https://en.wikipedia.org/wiki/No_such_page_for_this_example: origin returned 404
+skipped  https://en.m.wikipedia.org/wiki/Web_indexing: duplicate of https://en.wikipedia.org/wiki/Web_indexing
+read [1] Web indexing - Wikipedia  3,119 tokens  https://en.wikipedia.org/wiki/Web_indexing
+read [2] Web crawler - Wikipedia  22,410 tokens  https://en.wikipedia.org/wiki/Web_crawler
+context  25,529 tokens from 2 sources (raw HTML would be 111,672)
+```
+
+The same question with a model configured, run on September 19, 2026 with
+a hosted model through its OpenAI-compatible endpoint. The reading report is
+identical; this is the end of the answer and the source list, shortened for
+length and otherwise as printed:
+
+```text
+**Key Difference:**
+- The web crawler is the tool or agent that collects web content by navigating the web.
+- Web indexing is the process that takes the content gathered by the crawler and organizes it into a searchable index.
+
+In summary:
+**Web crawling** is about collecting web pages, while **web indexing** is about organizing and making sense of the collected data for efficient search and retrieval[1][2].
+
+Sources
+  [1] Web indexing - Wikipedia  https://en.wikipedia.org/wiki/Web_indexing
+  [2] Web crawler - Wikipedia  https://en.wikipedia.org/wiki/Web_crawler
+```
+
+Every claim carries a source number, both sources were cited, and neither
+is marked `(not cited)`.
+
+## Two keys, and why
+
+- **`LYRENTH_API_KEY`** reads the pages. The free tier includes 2,000 reads
+  a month; one question with five sources uses five.
+- **Your model** writes the answer. Anything that speaks the
+  OpenAI-compatible chat completions protocol works: the large hosted
+  providers, most gateways, and local servers such as Ollama or llama.cpp.
+  Set `LLM_BASE_URL` and `LLM_MODEL`, plus `LLM_API_KEY` if your endpoint
+  needs one. Nothing is sent anywhere else.
+
+Only need the reading? `--sources-only` prints the numbered, cited context
+without calling a model, ready to drop into your own pipeline. Add `--json`
+for structured output.
+
+## Options
+
+```text
+lyrenth-research QUESTION [URL ...] [options]
+
+  -f, --file FILE     more URLs, one per line (lines starting with # are ignored)
+  --budget N          token budget for all sources together (default 30000)
+  --fresh             ask for a fresh fetch instead of the stored page
+  --sources-only      read and print the context, no model call
+  --json              JSON output
+  --model-url URL     OpenAI-compatible base URL (default: LLM_BASE_URL)
+  --model NAME        model name (default: LLM_MODEL)
+```
+
+## As a library
+
+```python
+from lyrenth_research import gather, answer
+
+gathered = gather(
+    ["https://en.wikipedia.org/wiki/Web_indexing",
+     "https://en.wikipedia.org/wiki/Web_crawler"],
+    token_budget=30_000,
+)
+for skipped in gathered.skipped:
+    print("skipped", skipped.url, skipped.reason)
+
+result = answer("How do crawlers and indexes relate?", gathered.sources)
+print(result.text)
+print("cited:", result.cited, "unknown:", result.unknown_citations)
+```
+
+`gather` returns the sources it read, the ones it skipped with a reason, and
+the tokens used. `answer` returns the model's text, the source numbers it
+cited, and any cited numbers that do not exist.
+
+## Where the URLs come from
+
+On purpose, not from here. Your agent may get them from a search provider,
+from the user, from links inside pages it already read, or from a curated
+list of trusted sites. Finding and reading are separate jobs; keeping them
+separate means you can change one without breaking the other.
+
+## Tests
+
+```bash
+pip install -e .
+python tests/test_research.py
+```
+
+No network and no keys needed: reading is tested against a fake client, and
+the answer step against a local HTTP server that speaks the chat
+completions protocol.
+
+## License
+
+MIT

lyrenth_research-0.1.0/README.md

@@ -0,0 +1,163 @@
+# lyrenth-research
+
+A research agent that actually reads its sources.
+
+Every AI answer that reads the web pays for reading pages, and Lyrenth
+makes that reading several times cheaper. This agent is the easiest way to
+see it.
+
+Give it a question and a list of URLs. It reads every page through
+[Lyrenth](https://lyrenth.com) as clean text, drops duplicates, stays inside
+a token budget, tells you which pages it could not read, and asks your model
+for an answer with numbered citations. Then it checks that every citation
+points at a source it really read.
+
+```bash
+pip install lyrenth-research
+export LYRENTH_API_KEY=...            # free key: https://lyrenth.com/signup
+export LLM_BASE_URL=http://localhost:11434/v1   # any OpenAI-compatible endpoint
+export LLM_MODEL=your-model-name
+
+lyrenth-research "What is the difference between a web crawler and web indexing?" \
+  https://en.wikipedia.org/wiki/Web_indexing \
+  https://en.wikipedia.org/wiki/Web_crawler
+```
+
+## Why
+
+Most research agents get the finding and the writing right and treat the
+reading as one line: fetch the page, strip some tags, paste it into the
+prompt. That line decides more about the answer than the prompt does. An
+agent that reads badly pays for navigation menus and cookie banners, reads
+the same article twice under two URLs, cites pages it never really read,
+and quietly answers from nothing when a source fails.
+
+This agent does five things on purpose:
+
+1. **Reads clean text, not HTML.** Every page arrives as an AIDocument:
+   Markdown content, the canonical URL, when it was read, and how many
+   tokens it costs.
+2. **Keeps provenance.** Each source keeps its title, canonical URL and
+   read time, and all three go into the prompt.
+3. **Reads each document once.** A mobile copy, a tracking parameter or an
+   old redirect resolves to the same canonical URL and is dropped.
+4. **Stays inside a budget.** Sources are added in the order you give them
+   until the next one would exceed the budget. Put the ones you trust most
+   first.
+5. **Fails visibly.** Every page that was skipped is listed with the reason,
+   and a citation to a source number that does not exist is flagged.
+
+## A real run
+
+Reading step only (`--sources-only`), against the live API on September 19,
+2026, unedited. The progress report goes to stderr, the numbered context to
+stdout (here, `context.md`).
+
+```text
+$ lyrenth-research "What is the difference between a web crawler and web indexing?" \
+    https://en.wikipedia.org/wiki/Web_indexing \
+    https://en.wikipedia.org/wiki/Web_crawler \
+    https://en.wikipedia.org/wiki/Robots.txt \
+    https://en.wikipedia.org/wiki/No_such_page_for_this_example \
+    https://en.m.wikipedia.org/wiki/Web_indexing \
+    --sources-only > context.md
+skipped  https://en.wikipedia.org/wiki/Robots.txt: over budget (16,719 tokens would exceed 30,000)
+skipped  https://en.wikipedia.org/wiki/No_such_page_for_this_example: origin returned 404
+skipped  https://en.m.wikipedia.org/wiki/Web_indexing: duplicate of https://en.wikipedia.org/wiki/Web_indexing
+read [1] Web indexing - Wikipedia  3,119 tokens  https://en.wikipedia.org/wiki/Web_indexing
+read [2] Web crawler - Wikipedia  22,410 tokens  https://en.wikipedia.org/wiki/Web_crawler
+context  25,529 tokens from 2 sources (raw HTML would be 111,672)
+```
+
+The same question with a model configured, run on September 19, 2026 with
+a hosted model through its OpenAI-compatible endpoint. The reading report is
+identical; this is the end of the answer and the source list, shortened for
+length and otherwise as printed:
+
+```text
+**Key Difference:**
+- The web crawler is the tool or agent that collects web content by navigating the web.
+- Web indexing is the process that takes the content gathered by the crawler and organizes it into a searchable index.
+
+In summary:
+**Web crawling** is about collecting web pages, while **web indexing** is about organizing and making sense of the collected data for efficient search and retrieval[1][2].
+
+Sources
+  [1] Web indexing - Wikipedia  https://en.wikipedia.org/wiki/Web_indexing
+  [2] Web crawler - Wikipedia  https://en.wikipedia.org/wiki/Web_crawler
+```
+
+Every claim carries a source number, both sources were cited, and neither
+is marked `(not cited)`.
+
+## Two keys, and why
+
+- **`LYRENTH_API_KEY`** reads the pages. The free tier includes 2,000 reads
+  a month; one question with five sources uses five.
+- **Your model** writes the answer. Anything that speaks the
+  OpenAI-compatible chat completions protocol works: the large hosted
+  providers, most gateways, and local servers such as Ollama or llama.cpp.
+  Set `LLM_BASE_URL` and `LLM_MODEL`, plus `LLM_API_KEY` if your endpoint
+  needs one. Nothing is sent anywhere else.
+
+Only need the reading? `--sources-only` prints the numbered, cited context
+without calling a model, ready to drop into your own pipeline. Add `--json`
+for structured output.
+
+## Options
+
+```text
+lyrenth-research QUESTION [URL ...] [options]
+
+  -f, --file FILE     more URLs, one per line (lines starting with # are ignored)
+  --budget N          token budget for all sources together (default 30000)
+  --fresh             ask for a fresh fetch instead of the stored page
+  --sources-only      read and print the context, no model call
+  --json              JSON output
+  --model-url URL     OpenAI-compatible base URL (default: LLM_BASE_URL)
+  --model NAME        model name (default: LLM_MODEL)
+```
+
+## As a library
+
+```python
+from lyrenth_research import gather, answer
+
+gathered = gather(
+    ["https://en.wikipedia.org/wiki/Web_indexing",
+     "https://en.wikipedia.org/wiki/Web_crawler"],
+    token_budget=30_000,
+)
+for skipped in gathered.skipped:
+    print("skipped", skipped.url, skipped.reason)
+
+result = answer("How do crawlers and indexes relate?", gathered.sources)
+print(result.text)
+print("cited:", result.cited, "unknown:", result.unknown_citations)
+```
+
+`gather` returns the sources it read, the ones it skipped with a reason, and
+the tokens used. `answer` returns the model's text, the source numbers it
+cited, and any cited numbers that do not exist.
+
+## Where the URLs come from
+
+On purpose, not from here. Your agent may get them from a search provider,
+from the user, from links inside pages it already read, or from a curated
+list of trusted sites. Finding and reading are separate jobs; keeping them
+separate means you can change one without breaking the other.
+
+## Tests
+
+```bash
+pip install -e .
+python tests/test_research.py
+```
+
+No network and no keys needed: reading is tested against a fake client, and
+the answer step against a local HTTP server that speaks the chat
+completions protocol.
+
+## License
+
+MIT

lyrenth_research-0.1.0/pyproject.toml

@@ -0,0 +1,30 @@
+[build-system]
+requires = ["hatchling"]
+build-backend = "hatchling.build"
+
+[project]
+name = "lyrenth-research"
+version = "0.1.0"
+description = "A research agent that reads its sources: give it a question and URLs, get an answer with numbered citations."
+readme = "README.md"
+requires-python = ">=3.9"
+license = "MIT"
+authors = [{ name = "Lyrenth" }]
+keywords = ["research", "agent", "citations", "rag", "llm", "web", "lyrenth", "aidocument"]
+classifiers = [
+  "Programming Language :: Python :: 3",
+  "License :: OSI Approved :: MIT License",
+  "Operating System :: OS Independent",
+  "Environment :: Console",
+]
+dependencies = ["lyrenth>=0.1.1"]
+
+[project.scripts]
+lyrenth-research = "lyrenth_research.cli:main"
+
+[project.urls]
+Homepage = "https://lyrenth.com"
+Documentation = "https://lyrenth.com/docs/quickstart"
+
+[tool.hatch.build.targets.wheel]
+packages = ["src/lyrenth_research"]

lyrenth_research-0.1.0/src/lyrenth_research/__init__.py

@@ -0,0 +1,30 @@
+"""A research agent that reads its sources.
+
+    from lyrenth_research import gather, answer
+
+    gathered = gather(["https://example.com/a", "https://example.com/b"])
+    result = answer("What does the page say about X?", gathered.sources)
+    print(result.text)
+
+Reading goes through Lyrenth (LYRENTH_API_KEY, free key at
+https://lyrenth.com/signup). The answer comes from any OpenAI-compatible
+model endpoint you configure with LLM_BASE_URL and LLM_MODEL.
+"""
+
+__version__ = "0.1.0"
+
+from .answer import Answer, ModelError, answer, build_context, check_citations  # noqa: E402
+from .sources import Gathered, Skipped, Source, gather  # noqa: E402
+
+__all__ = [
+    "Answer",
+    "Gathered",
+    "ModelError",
+    "Skipped",
+    "Source",
+    "answer",
+    "build_context",
+    "check_citations",
+    "gather",
+    "__version__",
+]

lyrenth_research-0.1.0/src/lyrenth_research/answer.py

@@ -0,0 +1,111 @@
+"""Turning sources into an answer with numbered citations.
+
+The model is yours. Anything that speaks the OpenAI-compatible chat
+completions protocol works: the large hosted providers, most gateways, and
+local servers such as Ollama or llama.cpp. Configure it with three
+environment variables (or the matching CLI flags):
+
+    LLM_BASE_URL   e.g. http://localhost:11434/v1 for a local Ollama
+    LLM_MODEL      the model name your endpoint expects
+    LLM_API_KEY    optional; local servers usually need none
+"""
+
+from __future__ import annotations
+
+import json
+import os
+import re
+import urllib.error
+import urllib.request
+from dataclasses import dataclass, field
+from typing import List, Optional
+
+from .sources import Source
+
+SYSTEM_PROMPT = (
+    "You answer questions using only the numbered sources you are given. "
+    "Cite every claim with the number of the source it comes from, like [1] or [2][3]. "
+    "If the sources do not contain the answer, say so plainly instead of guessing. "
+    "Do not use knowledge that is not in the sources."
+)
+
+
+class ModelError(RuntimeError):
+    """The model endpoint was not configured, unreachable, or answered badly."""
+
+
+@dataclass
+class Answer:
+    text: str
+    sources: List[Source]
+    cited: List[int] = field(default_factory=list)
+    unknown_citations: List[int] = field(default_factory=list)
+
+
+def build_context(sources: List[Source]) -> str:
+    """Number the sources and keep where and when each was read."""
+    parts = []
+    for i, s in enumerate(sources, 1):
+        header = f"[{i}] {s.title or s.url}\nURL: {s.url}"
+        if s.fetched_at:
+            header += f"\nRead: {s.fetched_at}"
+        parts.append(f"{header}\n\n{s.text}")
+    return "\n\n---\n\n".join(parts)
+
+
+def build_messages(question: str, sources: List[Source]) -> list:
+    user = f"Question: {question}\n\nSources:\n\n{build_context(sources)}"
+    return [
+        {"role": "system", "content": SYSTEM_PROMPT},
+        {"role": "user", "content": user},
+    ]
+
+
+def check_citations(text: str, n_sources: int):
+    """Return (cited, unknown): source numbers used, and numbers that do not exist."""
+    found = sorted({int(m) for m in re.findall(r"\[(\d+)\]", text)})
+    cited = [n for n in found if 1 <= n <= n_sources]
+    unknown = [n for n in found if n < 1 or n > n_sources]
+    return cited, unknown
+
+
+def call_model(
+    messages: list,
+    base_url: Optional[str] = None,
+    model: Optional[str] = None,
+    api_key: Optional[str] = None,
+    timeout: float = 180.0,
+) -> str:
+    base_url = (base_url or os.environ.get("LLM_BASE_URL", "")).rstrip("/")
+    model = model or os.environ.get("LLM_MODEL", "")
+    api_key = api_key if api_key is not None else os.environ.get("LLM_API_KEY", "")
+    if not base_url or not model:
+        raise ModelError(
+            "no model configured: set LLM_BASE_URL and LLM_MODEL "
+            "(or pass --model-url and --model), or use --sources-only"
+        )
+    headers = {"Content-Type": "application/json"}
+    if api_key:
+        headers["Authorization"] = f"Bearer {api_key}"
+    body = json.dumps({"model": model, "messages": messages, "temperature": 0}).encode("utf-8")
+    req = urllib.request.Request(f"{base_url}/chat/completions", data=body, headers=headers, method="POST")
+    try:
+        with urllib.request.urlopen(req, timeout=timeout) as resp:
+            payload = json.loads(resp.read().decode("utf-8"))
+    except urllib.error.HTTPError as e:
+        detail = e.read().decode("utf-8", "replace")[:300]
+        raise ModelError(f"model endpoint returned HTTP {e.code}: {detail}") from None
+    except urllib.error.URLError as e:
+        raise ModelError(f"could not reach the model endpoint: {e.reason}") from None
+    try:
+        return payload["choices"][0]["message"]["content"].strip()
+    except (KeyError, IndexError, TypeError, AttributeError):
+        raise ModelError("model endpoint answered in an unexpected shape") from None
+
+
+def answer(question: str, sources: List[Source], **model_kwargs) -> Answer:
+    if not sources:
+        return Answer(text="None of the sources could be read, so there is nothing to answer from.", sources=[])
+    text = call_model(build_messages(question, sources), **model_kwargs)
+    cited, unknown = check_citations(text, len(sources))
+    return Answer(text=text, sources=sources, cited=cited, unknown_citations=unknown)

lyrenth_research-0.1.0/src/lyrenth_research/cli.py

@@ -0,0 +1,111 @@
+"""Command line: lyrenth-research "question" URL [URL ...]"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import sys
+from dataclasses import asdict
+
+from lyrenth import Lyrenth, LyrenthError
+
+from . import __version__
+from .answer import ModelError, answer, build_context
+from .sources import gather
+
+
+def _parser() -> argparse.ArgumentParser:
+    p = argparse.ArgumentParser(
+        prog="lyrenth-research",
+        description="Read the given URLs through Lyrenth and answer the question with numbered citations.",
+    )
+    p.add_argument("question", help="the question to answer")
+    p.add_argument("urls", nargs="*", help="source URLs, most trusted first")
+    p.add_argument("-f", "--file", help="read more URLs from a file, one per line")
+    p.add_argument("--budget", type=int, default=30_000, help="token budget for all sources together (default 30000)")
+    p.add_argument("--fresh", action="store_true", help="ask Lyrenth for a fresh fetch instead of the stored page")
+    p.add_argument("--sources-only", action="store_true", help="read the sources and print the cited context, without calling a model")
+    p.add_argument("--json", action="store_true", help="print the result as JSON")
+    p.add_argument("--model-url", help="OpenAI-compatible base URL (default: LLM_BASE_URL)")
+    p.add_argument("--model", help="model name (default: LLM_MODEL)")
+    p.add_argument("--version", action="version", version=f"%(prog)s {__version__}")
+    return p
+
+
+def _urls(args) -> list:
+    urls = list(args.urls)
+    if args.file:
+        with open(args.file, encoding="utf-8") as fh:
+            urls += [line.strip() for line in fh if line.strip() and not line.startswith("#")]
+    return urls
+
+
+def _report(gathered, out=None) -> None:
+    # Looked up at call time, so a caller that redirects stderr captures it.
+    out = out or sys.stderr
+    for s in gathered.skipped:
+        print(f"skipped  {s.url}: {s.reason}", file=out)
+    for i, s in enumerate(gathered.sources, 1):
+        print(f"read [{i}] {s.title}  {s.tokens:,} tokens  {s.url}", file=out)
+    raw = gathered.raw_html_tokens
+    line = f"context  {gathered.tokens:,} tokens from {len(gathered.sources)} sources"
+    if raw:
+        line += f" (raw HTML would be {raw:,})"
+    print(line, file=out)
+
+
+def main(argv=None) -> int:
+    args = _parser().parse_args(argv)
+    urls = _urls(args)
+    if not urls:
+        print("give at least one URL, or a file of URLs with --file", file=sys.stderr)
+        return 2
+
+    try:
+        gathered = gather(urls, client=Lyrenth(), token_budget=args.budget, fresh=args.fresh)
+    except LyrenthError as e:
+        print(f"error: {e}", file=sys.stderr)
+        return 1
+
+    _report(gathered)
+
+    if args.sources_only:
+        if args.json:
+            print(json.dumps({"sources": [asdict(s) for s in gathered.sources],
+                              "skipped": [asdict(s) for s in gathered.skipped]}, indent=2))
+        else:
+            print(build_context(gathered.sources))
+        return 0
+
+    try:
+        result = answer(args.question, gathered.sources, base_url=args.model_url, model=args.model)
+    except ModelError as e:
+        print(f"error: {e}", file=sys.stderr)
+        return 1
+
+    if args.json:
+        print(json.dumps({
+            "question": args.question,
+            "answer": result.text,
+            "cited": result.cited,
+            "unknown_citations": result.unknown_citations,
+            "sources": [{"n": i, "title": s.title, "url": s.url, "read": s.fetched_at}
+                        for i, s in enumerate(result.sources, 1)],
+            "skipped": [asdict(s) for s in gathered.skipped],
+        }, indent=2))
+        return 0
+
+    print(result.text)
+    print()
+    print("Sources")
+    for i, s in enumerate(result.sources, 1):
+        mark = "" if i in result.cited else "  (not cited)"
+        print(f"  [{i}] {s.title}  {s.url}{mark}")
+    if result.unknown_citations:
+        nums = ", ".join(f"[{n}]" for n in result.unknown_citations)
+        print(f"\nwarning: the answer cites {nums}, which is not one of the sources above", file=sys.stderr)
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

lyrenth_research-0.1.0/src/lyrenth_research/sources.py

@@ -0,0 +1,125 @@
+"""Reading sources properly: once each, with provenance, inside a budget.
+
+This is the part of a research agent that decides what the model gets to
+see. It reads every URL through Lyrenth, which returns each page as a clean
+AIDocument with its canonical URL and a measured token count, and then:
+
+- drops a page that was already read under another URL (mobile copies,
+  tracking parameters, redirects all resolve to one canonical URL);
+- stops adding sources once the token budget would be exceeded, so one
+  long page cannot crowd out several short ones;
+- records every page it could not read, with the reason, instead of
+  quietly leaving a gap.
+"""
+
+from __future__ import annotations
+
+from dataclasses import dataclass, field
+from typing import Iterable, List, Optional
+
+from lyrenth import Lyrenth
+
+# The batch endpoint takes up to 20 URLs per call.
+BATCH_SIZE = 20
+
+
+@dataclass
+class Source:
+    """One page the agent read, with everything a citation needs."""
+
+    url: str
+    title: str
+    text: str
+    fetched_at: str = ""
+    tokens: int = 0
+    raw_html_tokens: int = 0
+
+
+@dataclass
+class Skipped:
+    """A URL that did not make it into the sources, and why."""
+
+    url: str
+    reason: str
+
+
+@dataclass
+class Gathered:
+    sources: List[Source] = field(default_factory=list)
+    skipped: List[Skipped] = field(default_factory=list)
+    tokens: int = 0
+
+    @property
+    def raw_html_tokens(self) -> int:
+        return sum(s.raw_html_tokens for s in self.sources)
+
+
+def _estimate_tokens(text: str) -> int:
+    # Only used when a response carries no economics block. Four characters
+    # per token is the usual rough figure for English text.
+    return max(1, len(text) // 4)
+
+
+def _to_source(doc) -> Source:
+    raw = doc.raw or {}
+    economics = raw.get("economics") or {}
+    source = raw.get("source") or {}
+    tokens = int(economics.get("output_tokens_approx") or 0) or _estimate_tokens(doc.markdown)
+    return Source(
+        url=doc.url,
+        title=doc.title,
+        text=doc.markdown,
+        fetched_at=source.get("fetched_at") or "",
+        tokens=tokens,
+        raw_html_tokens=int(economics.get("raw_html_tokens_approx") or 0),
+    )
+
+
+def _dedupe_input(urls: Iterable[str]) -> List[str]:
+    seen, out = set(), []
+    for u in urls:
+        u = u.strip()
+        if u and u not in seen:
+            seen.add(u)
+            out.append(u)
+    return out
+
+
+def gather(
+    urls: Iterable[str],
+    client: Optional[Lyrenth] = None,
+    token_budget: int = 30_000,
+    fresh: bool = False,
+) -> Gathered:
+    """Read `urls` in order and return the sources that fit the budget.
+
+    Order matters: earlier URLs are preferred when the budget runs out, so
+    put the sources you trust most first.
+    """
+    client = client or Lyrenth()
+    urls = _dedupe_input(urls)
+    out = Gathered()
+    seen_canonical = set()
+
+    for start in range(0, len(urls), BATCH_SIZE):
+        batch = urls[start : start + BATCH_SIZE]
+        for result in client.read_batch(batch, fresh=fresh):
+            if not result.ok or result.document is None:
+                out.skipped.append(Skipped(result.url, result.error or "could not be read"))
+                continue
+            src = _to_source(result.document)
+            if not src.text.strip():
+                out.skipped.append(Skipped(result.url, "page has no readable text"))
+                continue
+            if src.url in seen_canonical:
+                out.skipped.append(Skipped(result.url, f"duplicate of {src.url}"))
+                continue
+            if out.tokens + src.tokens > token_budget:
+                out.skipped.append(
+                    Skipped(result.url, f"over budget ({src.tokens:,} tokens would exceed {token_budget:,})")
+                )
+                continue
+            seen_canonical.add(src.url)
+            out.sources.append(src)
+            out.tokens += src.tokens
+    return out

lyrenth_research-0.1.0/tests/test_research.py

@@ -0,0 +1,236 @@
+"""Tests for lyrenth-research. No network and no keys needed.
+
+    python tests/test_research.py      # plain, no pytest needed
+    pytest tests/                     # if pytest is installed
+
+Reading is tested against a fake Lyrenth client. The answer step is tested
+against a real HTTP server on localhost that speaks the OpenAI-compatible
+chat completions protocol, so the request the agent sends is checked for real.
+"""
+
+import io
+import json
+import os
+import sys
+import threading
+from contextlib import redirect_stderr, redirect_stdout
+from http.server import BaseHTTPRequestHandler, HTTPServer
+
+HERE = os.path.dirname(__file__)
+sys.path.insert(0, os.path.join(HERE, "..", "src"))
+# Prefer the SDK from this monorepo when it is next door, else the installed one.
+sdk = os.path.join(HERE, "..", "..", "lyrenth-python", "src")
+if os.path.isdir(sdk):
+    sys.path.insert(0, sdk)
+
+from lyrenth import AIDocument, BatchResult  # noqa: E402
+
+import lyrenth_research.cli as cli  # noqa: E402
+from lyrenth_research import (  # noqa: E402
+    ModelError,
+    Source,
+    answer,
+    build_context,
+    check_citations,
+    gather,
+)
+
+
+def doc(canonical, title, text, tokens, raw=0, fetched="2026-09-19T12:00:00Z"):
+    return AIDocument(
+        url=canonical,
+        title=title,
+        markdown=text,
+        raw={
+            "source": {"url": canonical, "canonical_url": canonical, "fetched_at": fetched},
+            "economics": {"output_tokens_approx": tokens, "raw_html_tokens_approx": raw},
+        },
+    )
+
+
+class FakeClient:
+    """Answers read_batch from a dict of url -> AIDocument or error string."""
+
+    def __init__(self, pages):
+        self.pages = pages
+        self.calls = []
+
+    def read_batch(self, urls, fresh=False, max_tokens=None):
+        self.calls.append(list(urls))
+        out = []
+        for u in urls:
+            v = self.pages.get(u, "upstream_not_found")
+            if isinstance(v, str):
+                out.append(BatchResult(url=u, ok=False, error=v))
+            else:
+                out.append(BatchResult(url=u, ok=True, document=v))
+        return out
+
+
+A = "https://example.com/a"
+A_MOBILE = "https://m.example.com/a"
+B = "https://example.com/b"
+C = "https://example.com/c"
+MISSING = "https://example.com/missing"
+
+
+def pages():
+    return {
+        A: doc(A, "Page A", "Alpha text.", 3000, 25000),
+        A_MOBILE: doc(A, "Page A", "Alpha text.", 3000, 25000),
+        B: doc(B, "Page B", "Beta text.", 20000, 80000),
+        C: doc(C, "Page C", "Gamma text.", 16000, 40000),
+    }
+
+
+# ---------------------------------------------------------------- gather
+
+def test_gather_every_branch():
+    g = gather([A, B, C, MISSING, A_MOBILE, A], client=FakeClient(pages()), token_budget=30000)
+    assert [s.url for s in g.sources] == [A, B], g.sources
+    assert g.tokens == 23000
+    assert g.raw_html_tokens == 105000
+    reasons = {s.url: s.reason for s in g.skipped}
+    assert "over budget" in reasons[C], reasons
+    assert reasons[MISSING] == "upstream_not_found"
+    assert reasons[A_MOBILE] == f"duplicate of {A}"
+    # The exact same URL given twice is read once, not reported as skipped.
+    assert len(g.skipped) == 3
+
+
+def test_gather_prefers_earlier_urls_when_budget_is_tight():
+    g = gather([C, A, B], client=FakeClient(pages()), token_budget=20000)
+    assert [s.url for s in g.sources] == [C, A]
+
+
+def test_gather_batches_by_twenty():
+    many = {f"https://example.com/{i}": doc(f"https://example.com/{i}", str(i), "x", 10) for i in range(45)}
+    client = FakeClient(many)
+    g = gather(list(many), client=client, token_budget=10**6)
+    assert [len(c) for c in client.calls] == [20, 20, 5]
+    assert len(g.sources) == 45
+
+
+def test_gather_skips_empty_pages():
+    p = {A: doc(A, "Empty", "   \n", 5)}
+    g = gather([A], client=FakeClient(p))
+    assert g.sources == [] and g.skipped[0].reason == "page has no readable text"
+
+
+def test_gather_estimates_tokens_without_economics():
+    d = AIDocument(url=A, title="A", markdown="x" * 400, raw={"source": {"url": A}})
+    g = gather([A], client=FakeClient({A: d}))
+    assert g.sources[0].tokens == 100
+
+
+# ---------------------------------------------------------------- answer helpers
+
+def test_build_context_numbers_and_keeps_provenance():
+    ctx = build_context([Source(A, "Page A", "Alpha.", "2026-09-19T12:00:00Z"), Source(B, "", "Beta.")])
+    assert ctx.startswith("[1] Page A\nURL: https://example.com/a\nRead: 2026-09-19T12:00:00Z\n\nAlpha.")
+    assert "[2] https://example.com/b\nURL: https://example.com/b\n\nBeta." in ctx
+
+
+def test_check_citations_flags_numbers_that_do_not_exist():
+    cited, unknown = check_citations("Alpha [1]. Beta [2][2]. Nonsense [7]. Zero [0].", 2)
+    assert cited == [1, 2] and unknown == [0, 7]
+
+
+def test_answer_without_sources_does_not_call_a_model():
+    r = answer("q", [], base_url="http://127.0.0.1:9", model="m")
+    assert r.sources == [] and "nothing to answer from" in r.text
+
+
+def test_missing_model_config_is_a_clear_error():
+    saved = {k: os.environ.pop(k, None) for k in ("LLM_BASE_URL", "LLM_MODEL")}
+    try:
+        answer("q", [Source(A, "A", "Alpha.")])
+    except ModelError as e:
+        assert "--sources-only" in str(e)
+    else:
+        raise AssertionError("expected ModelError")
+    finally:
+        for k, v in saved.items():
+            if v is not None:
+                os.environ[k] = v
+
+
+# ---------------------------------------------------------------- a real HTTP model endpoint
+
+class _Model(BaseHTTPRequestHandler):
+    received = []
+    reply = "Alpha says hello [1]. Beta agrees [2]."
+
+    def do_POST(self):
+        body = json.loads(self.rfile.read(int(self.headers["Content-Length"])))
+        _Model.received.append({"path": self.path, "auth": self.headers.get("Authorization"), "body": body})
+        out = json.dumps({"choices": [{"message": {"role": "assistant", "content": _Model.reply}}]}).encode()
+        self.send_response(200)
+        self.send_header("Content-Type", "application/json")
+        self.send_header("Content-Length", str(len(out)))
+        self.end_headers()
+        self.wfile.write(out)
+
+    def log_message(self, *a):
+        pass
+
+
+def _serve():
+    srv = HTTPServer(("127.0.0.1", 0), _Model)
+    threading.Thread(target=srv.serve_forever, daemon=True).start()
+    return srv, f"http://127.0.0.1:{srv.server_address[1]}/v1"
+
+
+def test_answer_calls_an_openai_compatible_endpoint():
+    srv, url = _serve()
+    try:
+        _Model.received.clear()
+        _Model.reply = "Alpha says hello [1]. Beta agrees [2]. Also [5]."
+        r = answer("What do they say?", [Source(A, "Page A", "Alpha."), Source(B, "Page B", "Beta.")],
+                   base_url=url, model="test-model", api_key="secret")
+        sent = _Model.received[0]
+        assert sent["path"] == "/v1/chat/completions"
+        assert sent["auth"] == "Bearer secret"
+        assert sent["body"]["model"] == "test-model" and sent["body"]["temperature"] == 0
+        user = sent["body"]["messages"][1]["content"]
+        assert user.startswith("Question: What do they say?") and "[2] Page B" in user
+        assert r.cited == [1, 2] and r.unknown_citations == [5]
+    finally:
+        srv.shutdown()
+
+
+def test_cli_end_to_end_with_fake_reader_and_local_model():
+    srv, url = _serve()
+    real = cli.Lyrenth
+    cli.Lyrenth = lambda: FakeClient(pages())
+    try:
+        _Model.reply = "Alpha is first [1]."
+        out, err = io.StringIO(), io.StringIO()
+        with redirect_stdout(out), redirect_stderr(err):
+            code = cli.main(["What is first?", A, MISSING, B, "--model-url", url, "--model", "m"])
+        assert code == 0, err.getvalue()
+        assert "skipped  https://example.com/missing: upstream_not_found" in err.getvalue()
+        assert "Alpha is first [1]." in out.getvalue()
+        assert "[2] Page B  https://example.com/b  (not cited)" in out.getvalue()
+
+        out = io.StringIO()
+        with redirect_stdout(out), redirect_stderr(io.StringIO()):
+            assert cli.main(["q", A, B, "--sources-only", "--json"]) == 0
+        payload = json.loads(out.getvalue())
+        assert [s["url"] for s in payload["sources"]] == [A, B]
+    finally:
+        cli.Lyrenth = real
+        srv.shutdown()
+
+
+def test_cli_needs_a_url():
+    with redirect_stderr(io.StringIO()):
+        assert cli.main(["question only"]) == 2
+
+
+if __name__ == "__main__":
+    tests = [v for k, v in sorted(globals().items()) if k.startswith("test_") and callable(v)]
+    for t in tests:
+        t()
+        print(f"ok  {t.__name__}")
+    print(f"{len(tests)} passed")