passageway-connector-llmstxt 0.3.0__tar.gz

passageway_connector_llmstxt-0.3.0/.gitignore

@@ -0,0 +1,11 @@
+docs/superpowers/
+.superpowers/
+.env
+.DS_Store
+.venv/
+__pycache__/
+.pytest_cache/
+.models/
+dist/
+site/
+.playwright-mcp/

passageway_connector_llmstxt-0.3.0/PKG-INFO

@@ -0,0 +1,94 @@
+Metadata-Version: 2.5
+Name: passageway-connector-llmstxt
+Version: 0.3.0
+Summary: passageway connector for the llms-full.txt of a list of websites
+License-Expression: PolyForm-Noncommercial-1.0.0
+Requires-Python: >=3.12
+Requires-Dist: passageway-sdk[web]~=0.3.0
+Description-Content-Type: text/markdown
+
+# passageway-connector-llmstxt
+
+A [passageway](https://github.com/getpassageway/passageway#readme) connector that indexes the `llms-full.txt` of a list of websites, a single markdown file such as a `SKILL.md`, or the pages an `llms.txt` index links to. Most documentation sites publish one of these for language models, and this connector makes them searchable without crawling.
+
+## What it does
+
+- **Each top-level heading is one item.** A `llms-full.txt` that concatenates a whole site becomes one item per page, with the sections below each heading as passages. Its id is the file's address plus the heading (`https://docs.example.org/llms-full.txt#install`), and its version is a hash of the text, so an edit re-indexes only that page. A `#` inside a code block is not a heading, and front matter at the top of a file is not indexed.
+- **A front matter block with a `url:` key starts a page, wherever it is in the file.** Some `llms-full.txt` files (the VitePress llms plugin writes them) open every page with `---`, `url: /howto/install.md` and `---`. Each such page is one item that runs to the next block, however many headings it has. Its hit links to the page itself (`https://docs.example.org/howto/install`, the `.md` dropped), its title is the page's first heading (else the last part of the path), and its `path` is the host and the folders of the page, so a `path:` filter works. Its id is the file's address plus the `url:` value. A file without such blocks is split by heading, as above.
+- **A site address reads its `llms-full.txt`.** `https://docs.example.org` is read as `https://docs.example.org/llms-full.txt`.
+- **A file address is read as it is:** `https://docs.example.org/llms-full.txt`, `https://passageway.marceltov.de/SKILL.md`, any markdown or text file.
+- **An `llms.txt` index is followed.** Every link in it that stays on the same host is fetched and indexed as a file of its own. Links to other hosts are not followed.
+- **Any failure fails the whole listing.** A page that returns an error, or a server that is down, stops the sync with a message that names it, so a half-read site never deletes what was indexed before.
+
+## Install
+
+Add the connector to the environment that runs passageway (see [Install](https://github.com/getpassageway/passageway/blob/main/docs/install.md)):
+
+```bash
+uv tool install \
+  --with 'passageway-connector-llmstxt @ git+https://github.com/getpassageway/passageway#subdirectory=connectors/llmstxt' \
+  git+https://github.com/getpassageway/passageway
+```
+
+## An example
+
+A file `https://docs.example.org/llms-full.txt`:
+
+```markdown
+# Install
+
+Run the installer.
+
+## Linux
+
+Use the package manager.
+```
+
+becomes the item:
+
+```json
+{
+  "title": "Install",
+  "sections": [
+    {
+      "heading": null,
+      "text": "Run the installer."
+    },
+    {
+      "heading": "Linux",
+      "text": "Use the package manager."
+    }
+  ]
+}
+```
+
+and is configured as:
+
+```toml
+[[connector]]
+name = "docs"
+command = ["/path/to/python", "-m", "passageway_connector_llmstxt"]
+env = { URLS = "https://docs.example.org|https://passageway.marceltov.de/SKILL.md" }
+```
+
+For a site whose `llms-full.txt` opens every page with a `url:` front matter block, and whose release notes should rank below its how-to pages:
+
+```toml
+[[connector]]
+name = "docs"
+command = ["/path/to/python", "-m", "passageway_connector_llmstxt"]
+env = { URLS = "https://docs.example.org", WEIGHTS = "changelog=0.5" }
+```
+
+## Settings
+
+All settings are strings in the connector's `env`.
+
+| Setting | Meaning | Default |
+|---|---|---|
+| `URLS` | Required. Site addresses, file addresses and `llms.txt` addresses, separated by `\|`. | |
+| `MAX_BYTES` | The largest file that is read, in bytes. A larger one fails the sync. | `20000000` |
+| `WEIGHTS` | Lowers or raises the weight of the pages whose path starts with a prefix: `changelog=0.5,archive/=0.3` (comma-separated `prefix=weight`, the weight above 0, the first match wins). Release notes repeat the vocabulary of how-to pages and push them down the results; a weight below 1 ranks them lower. The path is the page's address without the host. | all pages 1.0 |
+| `MAX_PAGES` | The most pages followed from one `llms.txt` index. The rest are skipped with a warning. | `500` |
+
+Requests carry the user agent `passageway-connector-llmstxt`, follow redirects and verify HTTPS against the operating system's trust store.

passageway_connector_llmstxt-0.3.0/README.md

@@ -0,0 +1,85 @@
+# passageway-connector-llmstxt
+
+A [passageway](https://github.com/getpassageway/passageway#readme) connector that indexes the `llms-full.txt` of a list of websites, a single markdown file such as a `SKILL.md`, or the pages an `llms.txt` index links to. Most documentation sites publish one of these for language models, and this connector makes them searchable without crawling.
+
+## What it does
+
+- **Each top-level heading is one item.** A `llms-full.txt` that concatenates a whole site becomes one item per page, with the sections below each heading as passages. Its id is the file's address plus the heading (`https://docs.example.org/llms-full.txt#install`), and its version is a hash of the text, so an edit re-indexes only that page. A `#` inside a code block is not a heading, and front matter at the top of a file is not indexed.
+- **A front matter block with a `url:` key starts a page, wherever it is in the file.** Some `llms-full.txt` files (the VitePress llms plugin writes them) open every page with `---`, `url: /howto/install.md` and `---`. Each such page is one item that runs to the next block, however many headings it has. Its hit links to the page itself (`https://docs.example.org/howto/install`, the `.md` dropped), its title is the page's first heading (else the last part of the path), and its `path` is the host and the folders of the page, so a `path:` filter works. Its id is the file's address plus the `url:` value. A file without such blocks is split by heading, as above.
+- **A site address reads its `llms-full.txt`.** `https://docs.example.org` is read as `https://docs.example.org/llms-full.txt`.
+- **A file address is read as it is:** `https://docs.example.org/llms-full.txt`, `https://passageway.marceltov.de/SKILL.md`, any markdown or text file.
+- **An `llms.txt` index is followed.** Every link in it that stays on the same host is fetched and indexed as a file of its own. Links to other hosts are not followed.
+- **Any failure fails the whole listing.** A page that returns an error, or a server that is down, stops the sync with a message that names it, so a half-read site never deletes what was indexed before.
+
+## Install
+
+Add the connector to the environment that runs passageway (see [Install](https://github.com/getpassageway/passageway/blob/main/docs/install.md)):
+
+```bash
+uv tool install \
+  --with 'passageway-connector-llmstxt @ git+https://github.com/getpassageway/passageway#subdirectory=connectors/llmstxt' \
+  git+https://github.com/getpassageway/passageway
+```
+
+## An example
+
+A file `https://docs.example.org/llms-full.txt`:
+
+```markdown
+# Install
+
+Run the installer.
+
+## Linux
+
+Use the package manager.
+```
+
+becomes the item:
+
+```json
+{
+  "title": "Install",
+  "sections": [
+    {
+      "heading": null,
+      "text": "Run the installer."
+    },
+    {
+      "heading": "Linux",
+      "text": "Use the package manager."
+    }
+  ]
+}
+```
+
+and is configured as:
+
+```toml
+[[connector]]
+name = "docs"
+command = ["/path/to/python", "-m", "passageway_connector_llmstxt"]
+env = { URLS = "https://docs.example.org|https://passageway.marceltov.de/SKILL.md" }
+```
+
+For a site whose `llms-full.txt` opens every page with a `url:` front matter block, and whose release notes should rank below its how-to pages:
+
+```toml
+[[connector]]
+name = "docs"
+command = ["/path/to/python", "-m", "passageway_connector_llmstxt"]
+env = { URLS = "https://docs.example.org", WEIGHTS = "changelog=0.5" }
+```
+
+## Settings
+
+All settings are strings in the connector's `env`.
+
+| Setting | Meaning | Default |
+|---|---|---|
+| `URLS` | Required. Site addresses, file addresses and `llms.txt` addresses, separated by `\|`. | |
+| `MAX_BYTES` | The largest file that is read, in bytes. A larger one fails the sync. | `20000000` |
+| `WEIGHTS` | Lowers or raises the weight of the pages whose path starts with a prefix: `changelog=0.5,archive/=0.3` (comma-separated `prefix=weight`, the weight above 0, the first match wins). Release notes repeat the vocabulary of how-to pages and push them down the results; a weight below 1 ranks them lower. The path is the page's address without the host. | all pages 1.0 |
+| `MAX_PAGES` | The most pages followed from one `llms.txt` index. The rest are skipped with a warning. | `500` |
+
+Requests carry the user agent `passageway-connector-llmstxt`, follow redirects and verify HTTPS against the operating system's trust store.

passageway_connector_llmstxt-0.3.0/passageway_connector_llmstxt/__main__.py

@@ -0,0 +1,3 @@
+from passageway_connector_llmstxt.connector import main
+
+main()

passageway_connector_llmstxt-0.3.0/passageway_connector_llmstxt/connector.py

@@ -0,0 +1,209 @@
+"""passageway connector for the llms-full.txt (and llms.txt) of a list of websites."""
+
+import hashlib
+import re
+import time
+from collections.abc import Callable, Iterator, Mapping
+from dataclasses import dataclass
+from urllib.parse import urldefrag, urljoin, urlparse
+
+import httpx
+from passageway_sdk import Connector, SettingsError, SourceError, split_markdown
+from passageway_sdk.settings import whole
+from passageway_sdk.web import TooLarge, Web
+
+FRONT_MATTER = re.compile(r"\A---[ \t]*\r?\n.*?\r?\n(?:---|\.\.\.)[ \t]*(?:\r?\n|\Z)", re.S)
+LINK = re.compile(r"\]\(\s*<?([^)\s>]+)")
+FENCE = re.compile(r"^\s*(```|~~~)")
+PAGE_URL = re.compile(r"url:[ \t]*(\S+)[ \t]*$")
+BLOCK_LINES = 20  # a front matter block longer than this is not one
+
+
+def _weights(text: str) -> list[tuple[str, float]]:
+    """`changelog=0.5,archive/=0.3`: a weight for every page whose path starts with a prefix; the first match wins."""
+    out = []
+    for part in (p.strip() for p in text.split(",") if p.strip()):
+        prefix, _, value = part.partition("=")
+        try:
+            weight = float(value)
+        except ValueError:
+            weight = 0.0
+        if not prefix.strip() or weight <= 0:
+            raise SettingsError(f"WEIGHTS must be prefix=weight pairs with a weight above 0, got {part!r}")
+        out.append((prefix.strip().lstrip("/"), weight))
+    return out
+
+
+def _slug(title: str) -> str:
+    return re.sub(r"[^a-z0-9]+", "-", title.casefold()).strip("-") or "page"
+
+
+@dataclass(frozen=True)
+class Page:
+    id: str
+    title: str
+    file: str
+    text: str
+    url: str = ""  # the address a hit opens; the file's own when the page has none of its own
+    path: tuple[str, ...] = ()
+
+
+def split_pages(file: str, text: str) -> list[Page]:
+    """Pages of a file: at every front matter block with a `url:` key (the layout of the VitePress llms plugin), else at every
+    top-level heading. A block or a `#` inside a code fence does not count; text before the first block is split by heading."""
+    lines = text.splitlines()
+    starts = _page_blocks(lines)
+    if not starts:
+        return _by_headings(file, FRONT_MATTER.sub("", text, count=1))
+    pages = _by_headings(file, "\n".join(lines[:starts[0][0]]))
+    used = {p.id for p in pages}
+    for n, (start, end, url) in enumerate(starts):
+        stop = starts[n + 1][0] if n + 1 < len(starts) else len(lines)
+        body = "\n".join(lines[end:stop]).strip()
+        if body:
+            pages.append(_page_of_url(file, url, body, used))
+    return pages
+
+
+def _page_blocks(lines: list[str]) -> list[tuple[int, int, str]]:
+    """(first line, line after the last, url) of every front matter block with a `url:` key, outside code fences."""
+    blocks, fenced, i = [], False, 0
+    while i < len(lines):
+        if FENCE.match(lines[i]):
+            fenced = not fenced
+        elif not fenced and lines[i].strip() == "---":
+            close = next((j for j in range(i + 1, min(i + BLOCK_LINES, len(lines))) if lines[j].strip() == "---"), None)
+            found = close and next((m.group(1) for m in map(PAGE_URL.match, (l.strip() for l in lines[i + 1:close])) if m), None)
+            if found:
+                blocks.append((i, close + 1, found))
+                i = close
+        i += 1
+    return blocks
+
+
+def _page_of_url(file: str, url: str, body: str, used: set[str]) -> Page:
+    path = urlparse(urljoin(file, url)).path
+    target = urljoin(file, url[:-3] if url.endswith(".md") else url)  # the HTML page, not the raw markdown
+    heading = next((l[2:].strip() for l in body.splitlines() if l.startswith("# ") and l[2:].strip()), "")
+    title = heading or (path.rstrip("/").rsplit("/", 1)[-1].removesuffix(".md") or urlparse(file).netloc)
+    page_id, n = f"{file}#{url}", 1
+    while page_id in used:
+        n += 1
+        page_id = f"{file}#{url}-{n}"
+    used.add(page_id)
+    folders = tuple(p for p in path.split("/")[1:-1] if p)
+    return Page(page_id, title, file, body, target, (urlparse(file).netloc, *folders))
+
+
+def _by_headings(file: str, text: str) -> list[Page]:
+    """One page per top-level heading; text before the first goes to a page of its own."""
+    host = urlparse(file).netloc
+    chunks: list[tuple[str, list[str]]] = [(host, [])]
+    fenced = False
+    for line in text.splitlines():
+        if FENCE.match(line):
+            fenced = not fenced
+        if not fenced and line.startswith("# "):
+            chunks.append((line[2:].strip() or host, [line]))
+        else:
+            chunks[-1][1].append(line)
+    pages, used = [], set()
+    for title, lines in chunks:
+        body = "\n".join(lines).strip()
+        if not body:
+            continue
+        slug, n = _slug(title), 1
+        while slug in used:
+            n += 1
+            slug = f"{_slug(title)}-{n}"
+        used.add(slug)
+        pages.append(Page(f"{file}#{slug}", title, file, body, file, (host,)))
+    return pages
+
+
+class LlmsTxt(Connector):
+    name = "llmstxt connector"
+
+    def __init__(self, env: Mapping[str, str], *, transport: httpx.BaseTransport | None = None,
+                 sleep: Callable[[float], None] = time.sleep):
+        self.urls = [u.strip() for u in env.get("URLS", "").split("|") if u.strip()]
+        if not self.urls:
+            raise SettingsError("URLS must name at least one site or file")
+        for url in self.urls:
+            if urlparse(url).scheme not in ("http", "https") or not urlparse(url).netloc:
+                raise SettingsError(f"URLS must be http(s) addresses, got {url!r}")
+        self.max_bytes = whole(env, "MAX_BYTES", 20_000_000)
+        self.max_pages = whole(env, "MAX_PAGES", 500)
+        self.weights = _weights(env.get("WEIGHTS", ""))
+        self._web = Web("passageway-connector-llmstxt", max_bytes=self.max_bytes, transport=transport, sleep=sleep)
+
+    # -- the protocol ------------------------------------------------------------------------
+
+    def versions(self) -> dict[str, str]:
+        return {p.id: self._version(p) for p in self._pages()}
+
+    def fetch(self, ids: list[str]) -> Iterator[dict]:
+        wanted = set(ids)
+        for file in dict.fromkeys(i.split("#")[0] for i in ids if urlparse(i).scheme in ("http", "https")):  # an id that is no address names nothing
+            for page in split_pages(file, self._get(file)):
+                if page.id in wanted:
+                    yield self._item(page)
+
+    # -- the sites ---------------------------------------------------------------------------
+
+    def _files(self) -> list[str]:
+        files: list[str] = []
+        for url in self.urls:
+            parsed = urlparse(url)
+            if parsed.path in ("", "/"):
+                files.append(f"{parsed.scheme}://{parsed.netloc}/llms-full.txt")
+            elif parsed.path.rsplit("/", 1)[-1] == "llms.txt":
+                files.extend(self._linked(url))
+            else:
+                files.append(urldefrag(url)[0])
+        return list(dict.fromkeys(files))
+
+    def _linked(self, index: str) -> list[str]:
+        host, links = urlparse(index).netloc, []
+        for target in LINK.findall(self._get(index)):
+            link = urldefrag(urljoin(index, target))[0]
+            if urlparse(link).scheme in ("http", "https") and urlparse(link).netloc == host and link != index:
+                links.append(link)
+        links = list(dict.fromkeys(links))
+        if len(links) > self.max_pages:
+            self.warn(f"{index} links to {len(links)} pages; only the first {self.max_pages} are indexed (MAX_PAGES)")
+        return links[:self.max_pages]
+
+    def _pages(self) -> list[Page]:
+        return [page for file in self._files() for page in split_pages(file, self._get(file))]
+
+    def _get(self, url: str) -> str:
+        try:
+            return self._web.get(url, required=True).text
+        except TooLarge:
+            raise SourceError(f"{url}: larger than MAX_BYTES ({self.max_bytes})") from None
+
+    def _weight(self, page: Page) -> float:
+        where = urlparse(page.url).path.lstrip("/")
+        return next((w for prefix, w in self.weights if where.startswith(prefix)), 1.0)
+
+    def _version(self, page: Page) -> str:
+        weight = self._weight(page)  # a changed weight is a changed item, so the index follows it
+        return hashlib.sha256((page.text if weight == 1.0 else f"{page.text}\0{weight}").encode()).hexdigest()[:16]
+
+    def _item(self, page: Page) -> dict:
+        item = {
+            "id": page.id,
+            "title": page.title,
+            "url": page.url,
+            "version": self._version(page),
+            "sections": split_markdown(page.text, skip_first_h1=True),
+            "path": list(page.path),
+        }
+        if self._weight(page) != 1.0:
+            item["weight"] = self._weight(page)
+        return item
+
+
+def main() -> None:
+    LlmsTxt.main()

passageway_connector_llmstxt-0.3.0/pyproject.toml

@@ -0,0 +1,22 @@
+[build-system]
+requires = ["hatchling==1.33.0"]
+build-backend = "hatchling.build"
+
+[project]
+name = "passageway-connector-llmstxt"
+version = "0.3.0"
+description = "passageway connector for the llms-full.txt of a list of websites"
+readme = "README.md"
+requires-python = ">=3.12"
+license = "PolyForm-Noncommercial-1.0.0"
+dependencies = ["passageway-sdk[web]~=0.3.0"]
+
+[tool.hatch.build.targets.wheel]
+packages = ["passageway_connector_llmstxt"]
+
+[dependency-groups]
+dev = ["jsonschema>=4.23", "pytest>=8.3", "passageway"]
+
+[tool.uv.sources]
+passageway = { workspace = true }
+passageway-sdk = { workspace = true }

passageway_connector_llmstxt-0.3.0/tests/conftest.py

@@ -0,0 +1,54 @@
+import io
+import json
+
+import httpx
+import pytest
+
+from passageway_connector_llmstxt.connector import LlmsTxt
+
+DOCS = """# Install
+
+Run the installer.
+
+## Linux
+
+Use the package manager.
+
+# Configure
+
+Set the token in the settings file.
+
+```
+# not a heading
+```
+"""
+
+
+class Sites:
+    """A fake web: path -> (status, body); and a way to run the connector against it in-process."""
+
+    def __init__(self):
+        self.pages = {"https://docs.example/llms-full.txt": DOCS}
+        self.requests = []
+
+    def handler(self, request: httpx.Request) -> httpx.Response:
+        self.requests.append(str(request.url))
+        body = self.pages.get(str(request.url))
+        return httpx.Response(404) if body is None else httpx.Response(200, text=body)
+
+    def call(self, mode, stdin="", **settings):
+        out, err = io.StringIO(), io.StringIO()
+        env = {"URLS": "https://docs.example", **settings}
+        code = LlmsTxt.run(["llmstxt", mode], env, io.StringIO(stdin), out, err, transport=httpx.MockTransport(self.handler),
+                           sleep=lambda s: None)
+        return code, [json.loads(line) for line in out.getvalue().splitlines()], err.getvalue()
+
+    def fetch(self, *ids, **settings):
+        code, records, err = self.call("fetch", "".join(json.dumps({"id": i}) + "\n" for i in ids), **settings)
+        assert code == 0, err
+        return {r["id"]: r for r in records}
+
+
+@pytest.fixture
+def sites():
+    return Sites()

passageway_connector_llmstxt-0.3.0/tests/test_llmstxt_connector.py

@@ -0,0 +1,183 @@
+import json
+
+
+def test_each_top_level_heading_is_an_item_of_the_file(sites):
+    code, records, err = sites.call("list")
+    assert code == 0, err
+    ids = [r["id"] for r in records]
+    assert ids == ["https://docs.example/llms-full.txt#install", "https://docs.example/llms-full.txt#configure"]
+
+
+def test_a_page_has_its_title_sections_and_the_file_as_address(sites):
+    page = sites.fetch("https://docs.example/llms-full.txt#install")["https://docs.example/llms-full.txt#install"]
+    assert page["title"] == "Install" and page["url"] == "https://docs.example/llms-full.txt"
+    assert page["path"] == ["docs.example"]
+    assert {"heading": "Linux", "text": "Use the package manager."} in page["sections"]
+
+
+def test_a_hash_in_a_code_block_does_not_start_a_page(sites):
+    page = sites.fetch("https://docs.example/llms-full.txt#configure")["https://docs.example/llms-full.txt#configure"]
+    assert "not a heading" in json.dumps(page["sections"])
+    assert len(sites.call("list")[1]) == 2
+
+
+def test_the_version_changes_only_when_a_page_changes(sites):
+    before = {r["id"]: r["version"] for r in sites.call("list")[1]}
+    sites.pages["https://docs.example/llms-full.txt"] = sites.pages["https://docs.example/llms-full.txt"].replace("installer", "setup tool")
+    after = {r["id"]: r["version"] for r in sites.call("list")[1]}
+    assert before["https://docs.example/llms-full.txt#configure"] == after["https://docs.example/llms-full.txt#configure"]
+    assert before["https://docs.example/llms-full.txt#install"] != after["https://docs.example/llms-full.txt#install"]
+
+
+def test_a_direct_url_and_several_sites_are_read(sites):
+    sites.pages["https://other.example/docs/llms-full.txt"] = "# Other\n\nText."
+    code, records, err = sites.call("list", URLS="https://docs.example|https://other.example/docs/llms-full.txt")
+    assert code == 0, err
+    assert len(records) == 3
+
+
+def test_a_file_without_a_heading_is_one_item_named_after_the_host(sites):
+    sites.pages["https://docs.example/llms-full.txt"] = "Just text."
+    page = next(iter(sites.fetch("https://docs.example/llms-full.txt#docs-example").values()))
+    assert page["title"] == "docs.example"
+
+
+def test_a_missing_file_fails_the_whole_listing_so_nothing_is_deleted(sites):
+    code, records, err = sites.call("list", URLS="https://docs.example|https://gone.example")
+    assert code == 1 and records == [] and "gone.example" in err
+
+
+def test_settings_are_checked(sites):
+    assert sites.call("list", URLS="")[0] == 2
+    assert sites.call("list", URLS="ftp://x.example")[0] == 2
+    assert sites.call("list", MAX_BYTES="0")[0] == 2
+
+
+def test_a_file_over_the_size_limit_fails_the_listing(sites):
+    code, _, err = sites.call("list", MAX_BYTES="10")
+    assert code == 1 and "MAX_BYTES" in err
+
+
+def test_a_single_markdown_file_with_front_matter_is_one_item(sites):
+    sites.pages["https://docs.example/SKILL.md"] = "---\nname: passageway\n---\n# Passageway\n\nAsk the sources.\n"
+    code, records, err = sites.call("list", URLS="https://docs.example/SKILL.md")
+    assert code == 0, err
+    assert [r["id"] for r in records] == ["https://docs.example/SKILL.md#passageway"]
+    page = sites.fetch("https://docs.example/SKILL.md#passageway", URLS="https://docs.example/SKILL.md")
+    assert "name: passageway" not in json.dumps(page)  # the front matter is not indexed as text
+
+
+def test_an_llms_txt_index_is_followed_page_by_page_on_the_same_host(sites):
+    sites.pages["https://docs.example/llms.txt"] = (
+        "# Example\n\n> A site.\n\n## Docs\n\n- [Start](https://docs.example/start.md): begin here\n"
+        "- [Relative](/guide/setup.md)\n- [Elsewhere](https://evil.example/x.md)\n- [Start again](https://docs.example/start.md#top)\n")
+    sites.pages["https://docs.example/start.md"] = "# Start\n\nBegin here."
+    sites.pages["https://docs.example/guide/setup.md"] = "# Setup\n\nInstall it."
+    code, records, err = sites.call("list", URLS="https://docs.example/llms.txt")
+    assert code == 0, err
+    assert [r["id"] for r in records] == ["https://docs.example/start.md#start", "https://docs.example/guide/setup.md#setup"]
+    assert not any("evil.example" in url for url in sites.requests)
+
+
+def test_a_broken_link_in_an_index_fails_the_listing(sites):
+    sites.pages["https://docs.example/llms.txt"] = "# Example\n\n- [Gone](https://docs.example/gone.md)\n"
+    code, records, err = sites.call("list", URLS="https://docs.example/llms.txt")
+    assert code == 1 and records == [] and "gone.md" in err
+
+
+def test_the_number_of_pages_followed_from_an_index_is_capped(sites):
+    sites.pages["https://docs.example/llms.txt"] = "".join(f"- [P{n}](https://docs.example/p{n}.md)\n" for n in range(5))
+    for n in range(5):
+        sites.pages[f"https://docs.example/p{n}.md"] = f"# P{n}\n\ntext"
+    code, records, err = sites.call("list", URLS="https://docs.example/llms.txt", MAX_PAGES="3")
+    assert code == 0 and len(records) == 3 and "MAX_PAGES" in err
+
+
+TEAMSCALE = """---
+url: /changelog/v1/patch-releases.md
+---
+
+# Patch Releases
+
+Fixes.
+
+## Details
+
+# Not a new page
+
+Still the same page.
+
+---
+url: /howto/install.md
+---
+
+# Install
+
+Run it.
+
+---
+url: /howto/no-heading.md
+---
+
+Text without any heading.
+"""
+FULL = "https://docs.example/llms-full.txt"
+
+
+def test_front_matter_with_a_url_starts_a_page_wherever_it_is_in_the_file(sites):
+    sites.pages[FULL] = TEAMSCALE
+    ids = [r["id"] for r in sites.call("list")[1]]
+    assert ids == [f"{FULL}#/changelog/v1/patch-releases.md", f"{FULL}#/howto/install.md", f"{FULL}#/howto/no-heading.md"]
+
+
+def test_such_a_page_links_to_its_own_address_and_carries_its_folders(sites):
+    sites.pages[FULL] = TEAMSCALE
+    page = sites.fetch(f"{FULL}#/howto/install.md")[f"{FULL}#/howto/install.md"]
+    assert page["url"] == "https://docs.example/howto/install"  # the .md of the file's path is dropped
+    assert page["title"] == "Install"
+    assert page["path"] == ["docs.example", "howto"]
+    assert "url:" not in json.dumps(page["sections"])  # the front matter block is not text
+
+
+def test_a_page_runs_to_the_next_block_even_over_its_own_headings(sites):
+    sites.pages[FULL] = TEAMSCALE
+    page = sites.fetch(f"{FULL}#/changelog/v1/patch-releases.md")[f"{FULL}#/changelog/v1/patch-releases.md"]
+    assert "Still the same page" in json.dumps(page["sections"])
+    assert page["path"] == ["docs.example", "changelog", "v1"]
+
+
+def test_a_page_without_a_heading_is_titled_by_the_last_part_of_its_path(sites):
+    sites.pages[FULL] = TEAMSCALE
+    page = sites.fetch(f"{FULL}#/howto/no-heading.md")[f"{FULL}#/howto/no-heading.md"]
+    assert page["title"] == "no-heading"
+
+
+def test_text_before_the_first_block_is_still_indexed(sites):
+    sites.pages[FULL] = "# Intro\n\nWelcome.\n\n" + TEAMSCALE
+    ids = [r["id"] for r in sites.call("list")[1]]
+    assert ids[0] == f"{FULL}#intro" and len(ids) == 4
+
+
+def test_front_matter_in_a_code_block_is_not_a_page(sites):
+    sites.pages[FULL] = "# Guide\n\n```\n---\nurl: /fake.md\n---\n```\n"
+    assert [r["id"] for r in sites.call("list")[1]] == [f"{FULL}#guide"]
+
+
+def test_a_weight_per_path_prefix_lowers_a_part_of_a_site(sites):
+    sites.pages[FULL] = TEAMSCALE
+    pages = sites.fetch(f"{FULL}#/changelog/v1/patch-releases.md", f"{FULL}#/howto/install.md", WEIGHTS="changelog=0.5")
+    assert pages[f"{FULL}#/changelog/v1/patch-releases.md"]["weight"] == 0.5
+    assert "weight" not in pages[f"{FULL}#/howto/install.md"]
+
+
+def test_a_changed_weight_changes_the_version_so_the_index_follows(sites):
+    sites.pages[FULL] = TEAMSCALE
+    before = {r["id"]: r["version"] for r in sites.call("list")[1]}
+    after = {r["id"]: r["version"] for r in sites.call("list", WEIGHTS="changelog=0.5")[1]}
+    assert before[f"{FULL}#/changelog/v1/patch-releases.md"] != after[f"{FULL}#/changelog/v1/patch-releases.md"]
+    assert before[f"{FULL}#/howto/install.md"] == after[f"{FULL}#/howto/install.md"]
+
+
+def test_weights_are_checked(sites):
+    for bad in ("changelog", "changelog=heavy", "changelog=0", "=0.5"):
+        assert sites.call("list", WEIGHTS=bad)[0] == 2, bad

passageway_connector_llmstxt-0.3.0/tests/test_llmstxt_contract.py

@@ -0,0 +1,87 @@
+import functools
+import http.server
+import json
+import os
+import re
+import subprocess
+import sys
+import threading
+from pathlib import Path
+
+import jsonschema
+import pytest
+
+from passageway.protocol import Connector
+from passageway.testing.contract import check_connector
+
+README = Path(__file__).resolve().parents[1] / "README.md"
+SCHEMA = Path(__file__).resolve().parents[3] / "docs" / "reference" / "connector-protocol.schema.json"
+DOCS = "# Install\n\nRun the installer.\n\n## Linux\n\nUse the package manager.\n\n# Configure\n\nSet the token.\n"
+
+
+class Site(str):
+    def __new__(cls, url, root):
+        site = super().__new__(cls, url)
+        site.root = root
+        return site
+
+
+@pytest.fixture
+def site(tmp_path):
+    """A real web server on localhost serving a llms-full.txt, so the connector runs as a real subprocess."""
+    (tmp_path / "llms-full.txt").write_text(DOCS)
+    handler = functools.partial(http.server.SimpleHTTPRequestHandler, directory=str(tmp_path))
+    handler.log_message = lambda *args: None
+    server = http.server.ThreadingHTTPServer(("127.0.0.1", 0), handler)
+    threading.Thread(target=server.serve_forever, daemon=True).start()
+    yield Site(f"http://127.0.0.1:{server.server_port}", tmp_path)
+    server.shutdown()
+
+
+def command():
+    return [sys.executable, "-m", "passageway_connector_llmstxt"]
+
+
+def test_the_connector_passes_the_shared_contract_as_a_real_subprocess(site):
+    connector = Connector("llmstxt", command(), env={"URLS": site})
+    check_connector(connector)
+    items = {i.id: i for i in connector.fetch([f"{site}/llms-full.txt#install"])}
+    assert items[f"{site}/llms-full.txt#install"].title == "Install"
+
+
+def test_the_output_validates_against_the_published_json_schema(site):
+    schema = json.loads(SCHEMA.read_text())
+
+    def valid(name, record):
+        return jsonschema.Draft202012Validator({"$ref": f"#/$defs/{name}", "$defs": schema["$defs"]}).is_valid(record)
+
+    def lines(mode, stdin=""):
+        out = subprocess.run(command() + [mode], env={**os.environ, "URLS": site}, input=stdin,
+                             capture_output=True, text=True, check=True).stdout
+        return [json.loads(line) for line in out.splitlines()]
+
+    assert lines("check") and all(valid("check", r) for r in lines("check"))
+    assert lines("list") and all(valid("listRecord", r) for r in lines("list"))
+    items = lines("fetch", f'{{"id": "{site}/llms-full.txt#install"}}\n')
+    assert len(items) == 1 and valid("item", items[0])
+
+
+def test_every_setting_is_in_the_readme():
+    code = (Path(__file__).resolve().parents[1] / "passageway_connector_llmstxt" / "connector.py").read_text()
+    names = sorted(set(re.findall(r'(?:env\.get|names|whole|flag|seconds)\(\s*(?:env,\s*)?"([A-Z_]+)"', code)) | {"URLS"})
+    assert len(names) >= 3, names  # the pattern must actually find the settings
+    readme = README.read_text()
+    for name in names:
+        assert f"`{name}`" in readme, f"{name} is read by the llmstxt connector but missing from its README"
+
+
+def test_the_example_in_the_readme_is_what_the_connector_really_produces(site):
+    readme = README.read_text()
+    text = re.search(r"```markdown\n(.*?)```", readme, re.S).group(1)
+    shown = json.loads(re.search(r"```json\n(.*?)```", readme, re.S).group(1))
+    (site.root / "example.md").write_text(text)
+    out = subprocess.run(command() + ["fetch"], env={**os.environ, "URLS": f"{site}/example.md"},
+                         input=f'{{"id": "{site}/example.md#install"}}\n', capture_output=True, text=True, check=True).stdout
+    real = json.loads(out)
+    for key, value in shown.items():
+        assert real[key] == value, f"the README shows {key} = {value!r} but the connector prints {real[key]!r}"