passageway-connector-confluence 0.3.0__tar.gz

passageway_connector_confluence-0.3.0/.gitignore

@@ -0,0 +1,11 @@
+docs/superpowers/
+.superpowers/
+.env
+.DS_Store
+.venv/
+__pycache__/
+.pytest_cache/
+.models/
+dist/
+site/
+.playwright-mcp/

passageway_connector_confluence-0.3.0/PKG-INFO

@@ -0,0 +1,58 @@
+Metadata-Version: 2.5
+Name: passageway-connector-confluence
+Version: 0.3.0
+Summary: passageway connector for Confluence Data Center: the pages of chosen spaces, read-only
+License-Expression: PolyForm-Noncommercial-1.0.0
+Requires-Python: >=3.12
+Requires-Dist: passageway-sdk[web]~=0.3.0
+Requires-Dist: selectolax>=0.3.21
+Description-Content-Type: text/markdown
+
+# passageway-connector-confluence
+
+A [passageway](https://github.com/getpassageway/passageway#readme) connector for **Confluence Data Center** (and Server): the current pages of the spaces you name, read through the REST API with GET requests only, using a personal access token. Written from the public REST documentation.
+
+## What it does
+
+- **Each page is one item.** Its id is the content id, its version changes with every page version and with its labels, and its link opens the page in Confluence.
+- **Headings become sections:** `h1`, `h2` and `h3` split a page into passages with their heading trail (`Setup › Linux`); lists, tables and code macros are kept readable; long sections are cut at about 350 tokens.
+- **Labels become labels** (`space:OPS` plus the page's own), **ancestors become the path** (nearest last, up to four), and every page weighs 1.0.
+- **Macros:** code and noformat macros keep their text; boxes, expands, sections, columns, excerpts and every other macro with a body keep what they hold; a status macro keeps its title; a table of contents, Jira issues, images, emoticons and other macros without a body are left out. Task lists become lists. Links keep their text, or the linked page's title. Blog posts and attachments are not indexed.
+- **Edits appear within a minute:** `changed` asks Confluence with CQL for the pages modified since the last poll (fourteen hours early, because CQL reads the time in the server's zone; unchanged versions cost nothing). Deletions, moves out of the spaces and labels that take a page out of scope are settled at the next reconcile. A space that lists no pages is reported as a warning, so a wrong key or a token without permission does not pass in silence.
+
+## Configure
+
+Create a personal access token under your profile in Confluence (Data Center and Server 7.9 and later). The connector only ever sends GET requests.
+
+```toml
+[[connector]]
+name = "wiki"
+command = ["/path/to/python", "-m", "passageway_connector_confluence"]
+env = { URL = "https://wiki.example.org", SPACES = "OPS|HR" }
+secrets = { TOKEN = "env:CONFLUENCE_TOKEN" }
+```
+
+| Setting | Default | Meaning |
+|---|---|---|
+| `URL` | required | The Confluence site, e.g. `https://wiki.example.org` (with its context path, if any) |
+| `SPACES` | required | Space keys to index, separated by `\|` |
+| `TOKEN` | required | A personal access token, as a secret reference such as `env:CONFLUENCE_TOKEN` |
+| `INCLUDE_LABELS` | | Only pages carrying one of these labels, separated by `\|` |
+| `EXCLUDE_LABELS` | | Pages carrying one of these labels are left out |
+| `MAX_PAGES` | `5000` | Upper bound on the spaces together; more fails with a message instead of indexing a subset |
+| `TIMEOUT` | `30` | Seconds per request |
+| `PAUSE_SECONDS` | `0` | Pause between requests, for a busy server |
+| `VERIFY_TLS` | `true` | `false` turns certificate checking off. A CA your operating system trusts already works without it (see [Certificates](https://passageway.marceltov.de/configuration/#certificates)), so prefer adding the CA there |
+
+## Behaviour
+
+- **Retries:** 429 and 5xx answers and network errors are retried with a growing wait (1 s, doubling up to 60 s; a longer `Retry-After` is honoured up to 300 s) until a request has waited 15 minutes in total; the error then says how many attempts and how long it waited. Every wait of 10 s or more is logged as it starts (`HTTP 429 … waiting 16 s (15 s of 900 s waited)`), so a sync that waits does not look hung. A 401 says to check the token; any other error fails the call and nothing is stored.
+- **A throttled or large space** needs more than the core's default `timeout = 300` for the whole `list` or `fetch`: waiting out one rate limit can take minutes. Raise `timeout` in its `[[connector]]` entry (`3600` is a start), or set `PAUSE_SECONDS` so fewer requests are throttled in the first place.
+- **Cloud:** Atlassian Cloud authenticates with an e-mail address and an API token and serves the API under `/wiki`; it is not supported yet ([ADR 0005](https://passageway.marceltov.de/adr/0005-confluence-data-center-first/)).
+- **Tested against a fake** of the Data Center API in this repository, not against a live Confluence.
+
+## Test
+
+```bash
+uv run pytest connectors/confluence        # offline, against an in-test Confluence
+```

passageway_connector_confluence-0.3.0/README.md

@@ -0,0 +1,48 @@
+# passageway-connector-confluence
+
+A [passageway](https://github.com/getpassageway/passageway#readme) connector for **Confluence Data Center** (and Server): the current pages of the spaces you name, read through the REST API with GET requests only, using a personal access token. Written from the public REST documentation.
+
+## What it does
+
+- **Each page is one item.** Its id is the content id, its version changes with every page version and with its labels, and its link opens the page in Confluence.
+- **Headings become sections:** `h1`, `h2` and `h3` split a page into passages with their heading trail (`Setup › Linux`); lists, tables and code macros are kept readable; long sections are cut at about 350 tokens.
+- **Labels become labels** (`space:OPS` plus the page's own), **ancestors become the path** (nearest last, up to four), and every page weighs 1.0.
+- **Macros:** code and noformat macros keep their text; boxes, expands, sections, columns, excerpts and every other macro with a body keep what they hold; a status macro keeps its title; a table of contents, Jira issues, images, emoticons and other macros without a body are left out. Task lists become lists. Links keep their text, or the linked page's title. Blog posts and attachments are not indexed.
+- **Edits appear within a minute:** `changed` asks Confluence with CQL for the pages modified since the last poll (fourteen hours early, because CQL reads the time in the server's zone; unchanged versions cost nothing). Deletions, moves out of the spaces and labels that take a page out of scope are settled at the next reconcile. A space that lists no pages is reported as a warning, so a wrong key or a token without permission does not pass in silence.
+
+## Configure
+
+Create a personal access token under your profile in Confluence (Data Center and Server 7.9 and later). The connector only ever sends GET requests.
+
+```toml
+[[connector]]
+name = "wiki"
+command = ["/path/to/python", "-m", "passageway_connector_confluence"]
+env = { URL = "https://wiki.example.org", SPACES = "OPS|HR" }
+secrets = { TOKEN = "env:CONFLUENCE_TOKEN" }
+```
+
+| Setting | Default | Meaning |
+|---|---|---|
+| `URL` | required | The Confluence site, e.g. `https://wiki.example.org` (with its context path, if any) |
+| `SPACES` | required | Space keys to index, separated by `\|` |
+| `TOKEN` | required | A personal access token, as a secret reference such as `env:CONFLUENCE_TOKEN` |
+| `INCLUDE_LABELS` | | Only pages carrying one of these labels, separated by `\|` |
+| `EXCLUDE_LABELS` | | Pages carrying one of these labels are left out |
+| `MAX_PAGES` | `5000` | Upper bound on the spaces together; more fails with a message instead of indexing a subset |
+| `TIMEOUT` | `30` | Seconds per request |
+| `PAUSE_SECONDS` | `0` | Pause between requests, for a busy server |
+| `VERIFY_TLS` | `true` | `false` turns certificate checking off. A CA your operating system trusts already works without it (see [Certificates](https://passageway.marceltov.de/configuration/#certificates)), so prefer adding the CA there |
+
+## Behaviour
+
+- **Retries:** 429 and 5xx answers and network errors are retried with a growing wait (1 s, doubling up to 60 s; a longer `Retry-After` is honoured up to 300 s) until a request has waited 15 minutes in total; the error then says how many attempts and how long it waited. Every wait of 10 s or more is logged as it starts (`HTTP 429 … waiting 16 s (15 s of 900 s waited)`), so a sync that waits does not look hung. A 401 says to check the token; any other error fails the call and nothing is stored.
+- **A throttled or large space** needs more than the core's default `timeout = 300` for the whole `list` or `fetch`: waiting out one rate limit can take minutes. Raise `timeout` in its `[[connector]]` entry (`3600` is a start), or set `PAUSE_SECONDS` so fewer requests are throttled in the first place.
+- **Cloud:** Atlassian Cloud authenticates with an e-mail address and an API token and serves the API under `/wiki`; it is not supported yet ([ADR 0005](https://passageway.marceltov.de/adr/0005-confluence-data-center-first/)).
+- **Tested against a fake** of the Data Center API in this repository, not against a live Confluence.
+
+## Test
+
+```bash
+uv run pytest connectors/confluence        # offline, against an in-test Confluence
+```

passageway_connector_confluence-0.3.0/passageway_connector_confluence/__init__.py

@@ -0,0 +1 @@
+

passageway_connector_confluence-0.3.0/passageway_connector_confluence/__main__.py

@@ -0,0 +1,3 @@
+from passageway_connector_confluence.connector import main
+
+main()

passageway_connector_confluence-0.3.0/passageway_connector_confluence/connector.py

@@ -0,0 +1,307 @@
+"""passageway connector for Confluence Data Center: the current pages of chosen spaces, read through the REST API with GET only.
+
+Written from the public REST documentation. Settings come from the environment and are documented in README.md.
+"""
+
+import hashlib
+import re
+import sys
+import time
+from collections.abc import Iterator, Mapping
+from dataclasses import dataclass
+from datetime import UTC, datetime, timedelta
+from urllib.parse import urlparse
+
+from passageway_sdk import Connector, SettingsError, SourceError
+from passageway_sdk.web import HttpError, Reply, Retry, Web
+from passageway_sdk.settings import names
+
+from passageway_connector_confluence.html import html_to_sections
+
+PAGE = 100  # results per request (the API's usual maximum)
+RETRY_BUDGET = 900  # seconds of waiting per request before giving up: a throttled Data Center takes minutes
+MAX_BACKOFF = 60  # the longest wait the connector picks itself
+MAX_RETRY_AFTER = 300  # the longest wait a Retry-After may ask for
+SAY_WAITS_FROM = 10  # seconds: a wait this long is said in the log, shorter ones stay quiet
+RETRY_STATUS = (429, 500, 502, 503, 504)
+FETCH_EXPAND = "body.storage,version,space,ancestors,metadata.labels"
+@dataclass(frozen=True)
+class Settings:
+    url: str
+    spaces: tuple[str, ...]
+    token: str
+    include: frozenset[str]
+    exclude: frozenset[str]
+    max_pages: int
+    timeout: float
+    pause: float
+    verify: bool
+
+
+def parse_settings(env: Mapping[str, str]) -> Settings:
+    url = env.get("URL", "").strip().rstrip("/")
+    parsed = urlparse(url)
+    if not url or parsed.scheme not in ("http", "https") or not parsed.hostname:
+        raise SettingsError(f"URL must be the Confluence site's http(s) address, got {url!r}")
+    spaces = names(env, "SPACES")
+    if not spaces or not all(re.fullmatch(r"[A-Za-z0-9_~.-]+", s) for s in spaces):
+        raise SettingsError("SPACES must list the space keys to index, separated by |, e.g. OPS|HR")
+    token = env.get("TOKEN", "").strip()
+    if not token:
+        raise SettingsError("TOKEN is required: a personal access token from your Confluence profile, as a secret reference")
+    raw = env.get("MAX_PAGES", "").strip()
+    if raw and (not raw.isdecimal() or int(raw) < 1):
+        raise SettingsError(f"MAX_PAGES must be a whole number from 1, got {raw!r}")
+    try:
+        timeout = float(env.get("TIMEOUT", "").strip() or 30)
+        pause = float(env.get("PAUSE_SECONDS", "").strip() or 0)
+    except ValueError:
+        raise SettingsError("TIMEOUT and PAUSE_SECONDS must be numbers of seconds") from None
+    if timeout <= 0 or pause < 0:
+        raise SettingsError("TIMEOUT must be greater than 0 and PAUSE_SECONDS not negative")
+    verify = env.get("VERIFY_TLS", "").strip().lower() or "true"
+    if verify not in ("true", "false"):
+        raise SettingsError("VERIFY_TLS must be true or false")
+    return Settings(url, spaces, token, frozenset(names(env, "INCLUDE_LABELS")), frozenset(names(env, "EXCLUDE_LABELS")),
+                    int(raw) if raw else 5000, timeout, pause, verify == "true")
+
+
+class ApiError(SourceError):
+    def __init__(self, status: int, message: str):
+        super().__init__(message)
+        self.status = status
+
+
+class Api:
+    """Read-only: only GET requests, by construction."""
+
+    def __init__(self, s: Settings, sleep=time.sleep):
+        self.s = s
+        retry = Retry(statuses=RETRY_STATUS, network=True, budget=RETRY_BUDGET, max_backoff=MAX_BACKOFF, max_after=MAX_RETRY_AFTER,
+                      say_from=SAY_WAITS_FROM)
+        # on stderr, which the core logs as it comes: a waiting sync does not look hung
+        say = lambda message: print(f"confluence connector: {message}", file=sys.stderr, flush=True)  # noqa: E731
+        self._web = Web(base_url=s.url, headers={"Authorization": f"Bearer {s.token}"}, timeout=s.timeout, verify=s.verify,
+                        follow_redirects=False, pause=s.pause, max_bytes=256_000_000, retry=retry, log=say, sleep=sleep)
+
+    def get(self, path: str, **params) -> dict:
+        try:
+            reply = self._web.get(path, params=params)
+        except HttpError as e:
+            if e.attempts > 1:  # the retry policy gave up, and its message says how long it waited
+                raise
+            reply = Reply(e.status, "", "")  # a 5xx that is not worth retrying
+        if reply.status == 401:
+            raise ApiError(401, "Confluence answered HTTP 401: check TOKEN (a personal access token) and its permissions")
+        if 300 <= reply.status < 400:
+            raise ApiError(reply.status, f"Confluence answered GET {path} with a redirect to "
+                           f"{reply.headers.get('location', '?')}: check URL (and whether a login page is in the way)")
+        if reply.status >= 400:
+            raise ApiError(reply.status, f"Confluence answered HTTP {reply.status} to GET {path}")
+        data = reply.json()
+        if not isinstance(data, dict):
+            raise SourceError(f"Confluence did not answer GET {path} with a JSON object")
+        return data
+
+    def paged(self, path: str, **params) -> Iterator[dict]:
+        start, seen = 0, set()
+        while True:
+            data = self.get(path, limit=str(PAGE), start=str(start), **params)
+            results = data.get("results")
+            if not isinstance(results, list):
+                raise SourceError(f"Confluence answered GET {path} without a results list")
+            fresh = [r for r in results if str(r.get("id")) not in seen]
+            seen.update(str(r.get("id")) for r in results)
+            yield from fresh
+            # the server caps `limit` as it likes: follow `next`; a server that ignores `start` repeats itself: stop then
+            if not fresh or not data.get("_links", {}).get("next"):
+                return
+            start += len(results)
+
+
+def _iso(when: str) -> str:
+    try:
+        return datetime.fromisoformat(when.replace("Z", "+00:00")).astimezone(UTC).strftime("%Y-%m-%dT%H:%M:%SZ")
+    except ValueError:
+        return when
+
+
+_ENTITIES = re.compile(r"&([A-Za-z][A-Za-z0-9]*);")
+_KEEP_ENTITIES = {"amp", "lt", "gt", "quot", "apos"}
+
+
+def _entities(storage: str) -> str:
+    """Named HTML entities as characters, so that the XML parser accepts them; the five XML ones stay."""
+    from html.entities import html5
+
+    def one(m):
+        name = m.group(1)
+        return m.group(0) if name in _KEEP_ENTITIES else html5.get(name + ";", m.group(0))
+    return _ENTITIES.sub(one, storage)
+
+
+def _macro_param(node, name: str) -> str:
+    for child in node:
+        if child.tag == "{ac}parameter" and child.get("{ac}name") == name:
+            return "".join(child.itertext()).strip()
+    return ""
+
+
+def _walk(node, out: list[str]) -> None:
+    """Serialise a storage-format element as plain HTML: ac:/ri: elements become their readable part."""
+    from xml.sax.saxutils import escape, quoteattr
+
+    tag = node.tag
+    if tag == "{ac}structured-macro":
+        name = node.get("{ac}name", "")
+        if name in ("code", "noformat"):
+            body = next((c for c in node if c.tag == "{ac}plain-text-body"), None)
+            out.append("<pre>" + escape("".join(body.itertext()) if body is not None else "") + "</pre>")
+        elif name == "status":
+            out.append(escape(_macro_param(node, "title")))
+        else:  # a box, an expand, a section, a column, an excerpt: what it holds; a toc, a jira macro: nothing
+            for child in node:
+                if child.tag == "{ac}rich-text-body":
+                    for grandchild in child:
+                        _walk(grandchild, out)
+                    out.append(escape(child.tail or ""))
+        out.append(escape(node.tail or ""))
+        return
+    if tag == "{ac}link":
+        body = next((c for c in node if c.tag in ("{ac}link-body", "{ac}plain-text-link-body")), None)
+        if body is not None:
+            out.append(escape("".join(body.itertext())))
+        else:
+            page = next((c for c in node if c.tag == "{ri}page"), None)
+            out.append(escape(page.get("{ri}content-title", "")) if page is not None else "")
+        out.append(escape(node.tail or ""))
+        return
+    if tag == "{ac}task-list":
+        items = ["".join(b.itertext()) for task in node if task.tag == "{ac}task"
+                 for b in task if b.tag == "{ac}task-body"]
+        out.append("<ul>" + "".join(f"<li>{escape(i)}</li>" for i in items) + "</ul>" + escape(node.tail or ""))
+        return
+    if tag in ("{ac}placeholder", "{ac}image", "{ac}emoticon", "{ac}parameter", "{ac}task-id", "{ac}task-status") \
+            or (isinstance(tag, str) and tag.startswith("{ri}")):
+        out.append(escape(node.tail or ""))
+        return
+    if isinstance(tag, str) and tag.startswith("{ac}"):  # layouts, inline-comment markers, rich-text bodies: transparent
+        out.append(escape(node.text or ""))
+        for child in node:
+            _walk(child, out)
+        out.append(escape(node.tail or ""))
+        return
+    if not isinstance(tag, str):  # a comment
+        out.append(escape(node.tail or ""))
+        return
+    attrs = "".join(f" {k}={quoteattr(v)}" for k, v in node.attrib.items() if not k.startswith("{"))
+    out.append(f"<{tag}{attrs}>{escape(node.text or '')}")
+    for child in node:
+        _walk(child, out)
+    out.append(f"</{tag}>{escape(node.tail or '')}")
+
+
+def storage_to_html(storage: str) -> str:
+    """Confluence's storage format as plain HTML: code macros become <pre>, boxes and expands keep what they hold, task
+    lists become lists, links keep their text or the linked page's title; images, emoticons, placeholders, a table of
+    contents, Jira and other macros without a body disappear. Storage that is not well-formed XML is read as HTML with
+    the ac:/ri: markup stripped."""
+    import xml.etree.ElementTree as ET
+
+    wrapped = f'<root xmlns:ac="ac" xmlns:ri="ri">{_entities(storage)}</root>'
+    try:
+        root = ET.fromstring(wrapped)
+    except ET.ParseError:
+        return re.sub(r"</?(?:ac|ri):[^>]*>", "", storage)
+    out: list[str] = [root.text or ""]
+    for child in root:
+        _walk(child, out)
+    return "".join(out)
+
+
+class Confluence(Connector):
+    name = "confluence connector"
+    expected_errors = (ValueError, KeyError, TypeError)
+
+    def __init__(self, env: Mapping[str, str], *, sleep=time.sleep):
+        self.s = parse_settings(env)
+        self.api = Api(self.s, sleep)
+        if not self.s.verify:
+            self.warn("VERIFY_TLS is false: the server's certificate is not checked")
+
+    def _wanted(self, labels: set[str]) -> bool:
+        if self.s.include and not labels & self.s.include:
+            return False
+        return not (labels & self.s.exclude)
+
+    @staticmethod
+    def _labels(page: dict) -> set[str]:
+        return {lb.get("name", "") for lb in page.get("metadata", {}).get("labels", {}).get("results", []) if lb.get("name")}
+
+    def _version(self, page: dict) -> str:
+        labels = hashlib.sha256(",".join(sorted(self._labels(page))).encode()).hexdigest()[:8]
+        return f"{page['version']['number']}:{labels}"
+
+    def _listing(self, pages: Iterator[dict]) -> dict[str, dict]:
+        found: dict[str, dict] = {}
+        for page in pages:
+            if page.get("status", "current") != "current" or not self._wanted(self._labels(page)):
+                continue
+            found[str(page["id"])] = page
+            if len(found) > self.s.max_pages:
+                raise SourceError(f"the spaces hold more than {self.s.max_pages} pages: narrow them or raise MAX_PAGES")
+        return found
+
+    def _item(self, page: dict) -> dict | None:
+        if page.get("status", "current") != "current" or not self._wanted(self._labels(page)):
+            return None
+        html = storage_to_html(page.get("body", {}).get("storage", {}).get("value", ""))
+        sections = [{"heading": " › ".join(s.heading_path) or None, "text": s.text} for s in html_to_sections(html)]
+        item = {"id": str(page["id"]), "title": page["title"], "url": self.s.url + page["_links"]["webui"],
+                "version": self._version(page), "sections": sections, "weight": 1.0,
+                "labels": [f"space:{page['space']['key']}", *sorted(self._labels(page))],
+                "updated": _iso(page["version"]["when"])}
+        if ancestors := [a["title"] for a in page.get("ancestors", [])][-4:]:
+            item["path"] = ancestors
+        return item
+
+    # -- the protocol --------------------------------------------------------------------------------
+
+    def versions(self) -> dict[str, str]:
+        found: dict[str, dict] = {}
+        for space in self.s.spaces:
+            pages = self._listing(self.api.paged("/rest/api/content", spaceKey=space, type="page", status="current",
+                                                 expand="version,metadata.labels"))
+            if not pages:
+                self.warn(f"the space {space!r} lists no pages: check the key and the token's permissions")
+            found.update(pages)
+            if len(found) > self.s.max_pages:
+                raise SourceError(f"the spaces hold more than {self.s.max_pages} pages: narrow them or raise MAX_PAGES")
+        return {pid: self._version(p) for pid, p in found.items()}
+
+    def fetch(self, ids: list[str]) -> Iterator[dict]:
+        for pid in ids:
+            if not pid.isdecimal():
+                continue
+            try:
+                page = self.api.get(f"/rest/api/content/{pid}", expand=FETCH_EXPAND)
+            except ApiError as e:
+                if e.status == 404:  # gone since the listing: the next reconcile settles it
+                    continue
+                raise
+            if page.get("space", {}).get("key") in self.s.spaces and (item := self._item(page)):
+                yield item
+
+    def changed(self, since: datetime) -> Iterator[dict]:
+        # CQL reads the time in the server's (or the user's) zone, not UTC: ask 14 hours early, which covers every
+        # zone; pages whose version is unchanged are dropped by the core anyway
+        stamp = (since.astimezone(UTC) - timedelta(hours=14)).strftime("%Y-%m-%d %H:%M")
+        keys = ",".join(f'"{s}"' for s in self.s.spaces)
+        cql = f'space in ({keys}) and type = page and lastModified >= "{stamp}"'
+        for page in self.api.paged("/rest/api/content/search", cql=cql, expand=FETCH_EXPAND):
+            if item := self._item(page):
+                yield item
+
+
+def main() -> None:
+    Confluence.main()

passageway_connector_confluence-0.3.0/passageway_connector_confluence/html.py

@@ -0,0 +1,160 @@
+"""Confluence storage format (XHTML with ac:/ri: macros) → plain-text sections split at h1/h2/h3 headings."""
+
+import math
+import re
+from dataclasses import dataclass
+
+from selectolax.lexbor import LexborHTMLParser as HTMLParser, LexborNode as Node
+
+SECTION_TAGS = {"h1", "h2", "h3"}
+LIST_TAGS = {"ul", "ol"}
+# Containers whose children are blocks in their own right.
+CONTAINER_TAGS = {"body", "div", "section", "article", "figure", "blockquote", "main", "details"}
+SKIP_TAGS = {"script", "style", "img", "svg", "head"}
+
+
+@dataclass(frozen=True)
+class Section:
+    heading_path: tuple[str, ...]
+    text: str
+    ordinal: int
+
+
+def approx_tokens(s: str) -> int:
+    return math.ceil(len(s.split()) * 1.3)
+
+
+BLOCK_TAGS = {"p", "div", "li", "ul", "ol", "table", "tr", "td", "th", "h1", "h2", "h3", "h4", "h5", "h6",
+              "pre", "blockquote", "section", "article", "figure", "details", "summary"}
+
+
+def _text(node: Node) -> str:
+    """Text of a node, with a space wherever a block element or <br> separates words."""
+    parts = []
+    for child in node.iter(include_text=True):
+        if child.tag == "-text":
+            parts.append(child.text_content or "")
+        elif child.tag == "br":
+            parts.append(" ")
+        elif child.tag in SKIP_TAGS:
+            continue
+        elif child.tag in BLOCK_TAGS:
+            parts.append(f" {_text(child)} ")
+        else:
+            parts.append(_text(child))
+    return "".join(parts)
+
+
+def _inline(node: Node) -> str:
+    return re.sub(r"\s+", " ", _text(node)).strip()
+
+
+def _list_lines(node: Node, depth: int = 0) -> list[str]:
+    lines = []
+    for li in node.iter():
+        if li.tag != "li":
+            continue
+        own, nested = [], []
+        for child in li.iter(include_text=True):
+            if child.tag in LIST_TAGS:
+                nested.extend(_list_lines(child, depth + 1))
+            elif child.tag == "-text":
+                own.append(child.text_content or "")
+            elif child.tag in BLOCK_TAGS:
+                own.append(f" {_text(child)} ")
+            else:
+                own.append(_text(child))
+        text = re.sub(r"\s+", " ", "".join(own)).strip()
+        if text:
+            lines.append("  " * depth + "- " + text)
+        lines.extend(nested)
+    return lines
+
+
+def _table_lines(node: Node) -> list[str]:
+    lines = []
+    for tr in node.css("tr"):
+        cells = [_inline(c) for c in tr.iter() if c.tag in ("td", "th")]
+        if any(cells):
+            lines.append(" | ".join(cells))
+    return lines
+
+
+def _blocks(node: Node):
+    """Yield ("heading", level, text) and ("text", None, text) in document order."""
+    pending_inline: list[str] = []
+
+    def flush():
+        text = re.sub(r"\s+", " ", "".join(pending_inline)).strip()
+        pending_inline.clear()
+        return text
+
+    for child in node.iter(include_text=True):
+        tag = child.tag
+        if tag == "-text" or tag in ("span", "a", "strong", "em", "b", "i", "code", "u", "s", "sub", "sup", "mark", "br"):
+            pending_inline.append(child.text_content if tag == "-text" else " " if tag == "br" else _text(child))
+            continue
+        if (text := flush()):
+            yield ("text", None, text)
+        if tag in SKIP_TAGS:
+            continue
+        if tag in SECTION_TAGS:
+            yield ("heading", int(tag[1]), _inline(child))
+        elif tag in LIST_TAGS:
+            if (lines := _list_lines(child)):
+                yield ("text", None, "\n".join(lines))
+        elif tag == "table":
+            if (lines := _table_lines(child)):
+                yield ("text", None, "\n".join(lines))
+        elif tag == "pre":
+            raw = child.text(deep=True, separator="").strip("\n")
+            if raw.strip():
+                yield ("text", None, raw)
+        elif tag in CONTAINER_TAGS:
+            yield from _blocks(child)
+        else:
+            if (text := _inline(child)):
+                yield ("text", None, text)
+    if (text := flush()):
+        yield ("text", None, text)
+
+
+def _split_words(text: str, max_tokens: int) -> list[str]:
+    words = text.split()
+    per = max(1, int(max_tokens / 1.3))
+    return [" ".join(words[i:i + per]) for i in range(0, len(words), per)]
+
+
+def _chunk(paragraphs: list[str], max_tokens: int) -> list[str]:
+    chunks, current = [], []
+    for para in paragraphs:
+        pieces = [para] if approx_tokens(para) <= max_tokens else _split_words(para, max_tokens)
+        for piece in pieces:
+            if current and approx_tokens("\n".join(current + [piece])) > max_tokens:
+                chunks.append("\n".join(current))
+                current = []
+            current.append(piece)
+    if current:
+        chunks.append("\n".join(current))
+    return chunks
+
+
+def html_to_sections(html: str, max_tokens: int = 350) -> list[Section]:
+    if not html or not html.strip():
+        return []
+    body = HTMLParser(html).body
+    if body is None:
+        return []
+    groups: list[tuple[tuple[str, ...], list[str]]] = [((), [])]
+    trail: list[tuple[int, str]] = []
+    for kind, level, text in _blocks(body):
+        if kind == "heading":
+            trail = [t for t in trail if t[0] < level] + [(level, text)]
+            groups.append((tuple(t[1] for t in trail), []))
+        else:
+            groups[-1][1].append(text)
+    sections = []
+    for path, paragraphs in groups:
+        for chunk in _chunk(paragraphs, max_tokens):
+            sections.append(Section(path, chunk, len(sections)))
+    return sections

passageway_connector_confluence-0.3.0/pyproject.toml

@@ -0,0 +1,22 @@
+[build-system]
+requires = ["hatchling==1.33.0"]
+build-backend = "hatchling.build"
+
+[project]
+name = "passageway-connector-confluence"
+version = "0.3.0"
+description = "passageway connector for Confluence Data Center: the pages of chosen spaces, read-only"
+readme = "README.md"
+requires-python = ">=3.12"
+license = "PolyForm-Noncommercial-1.0.0"
+dependencies = ["passageway-sdk[web]~=0.3.0", "selectolax>=0.3.21"]
+
+[tool.hatch.build.targets.wheel]
+packages = ["passageway_connector_confluence"]
+
+[dependency-groups]
+dev = ["jsonschema>=4.23", "pytest>=8.3", "passageway"]
+
+[tool.uv.sources]
+passageway = { workspace = true }
+passageway-sdk = { workspace = true }

passageway_connector_confluence-0.3.0/tests/conftest.py

@@ -0,0 +1,42 @@
+import io
+import json
+
+import pytest
+
+from fake_confluence import FakeConfluence, seed, serve
+from passageway_connector_confluence.connector import Confluence
+
+
+class World:
+    def __init__(self):
+        self.api = FakeConfluence()
+        seed(self.api)
+        self.server = serve(self.api)
+        self.url = f"http://127.0.0.1:{self.server.server_port}"
+        self.slept: list[float] = []
+
+    def env(self, **extra):
+        return {"URL": self.url, "SPACES": "OPS|HR", "TOKEN": self.api.token, "PAUSE_SECONDS": "0", **extra}
+
+    def call(self, mode, stdin="", *, extra=(), **settings):
+        out, err = io.StringIO(), io.StringIO()
+        code = Confluence.run(["confluence", mode, *extra], self.env(**settings), io.StringIO(stdin), out, err, sleep=self.slept.append)
+        return code, [json.loads(line) for line in out.getvalue().splitlines()], err.getvalue()
+
+    def ids(self, **settings):
+        code, records, err = self.call("list", **settings)
+        assert code == 0, err
+        return {r["id"]: r["version"] for r in records}
+
+    def fetch(self, *ids, **settings):
+        code, records, err = self.call("fetch", "".join(json.dumps({"id": i}) + "\n" for i in ids), **settings)
+        assert code == 0, err
+        return {r["id"]: r for r in records}
+
+
+@pytest.fixture
+def world():
+    w = World()
+    yield w
+    w.server.shutdown()
+    w.server.server_close()

passageway_connector_confluence-0.3.0/tests/fake_confluence.py

@@ -0,0 +1,142 @@
+"""An in-test Confluence Data Center REST API over HTTP: the few endpoints the connector uses, with paging and CQL."""
+
+import json
+import re
+import threading
+from dataclasses import dataclass, field
+from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
+from urllib.parse import parse_qs, urlparse
+
+
+@dataclass
+class Page:
+    id: str
+    space: str
+    title: str
+    body: str
+    version: int = 1
+    when: str = "2026-10-20T10:00:00.000Z"
+    labels: list[str] = field(default_factory=list)
+    ancestors: list[str] = field(default_factory=list)  # ids, root first
+    status: str = "current"
+
+
+class FakeConfluence:
+    def __init__(self):
+        self.pages: dict[str, Page] = {}
+        self.requests: list[tuple[str, str, dict]] = []
+        self.token = "fixture-pat"
+        self.page_size = 100
+        self.status_errors: list[int] = []
+        self.retry_after = "0"
+        self.loop = False  # ignore `start`, like a cursor-paged server
+        self.redirect = False
+
+    def add(self, id, space, title, body, **kw) -> str:
+        self.pages[id] = Page(id, space, title, body, **kw)
+        return id
+
+    def edit(self, id, body=None, title=None, labels=None):
+        p = self.pages[id]
+        p.version += 1
+        p.when = f"2026-10-28T12:{p.version:02d}:00.000Z"
+        if body is not None:
+            p.body = body
+        if title is not None:
+            p.title = title
+        if labels is not None:
+            p.labels = labels
+
+    def _summary(self, p: Page, expand: str) -> dict:
+        out = {"id": p.id, "type": "page", "status": p.status, "title": p.title,
+               "_links": {"webui": f"/pages/viewpage.action?pageId={p.id}", "self": f"/rest/api/content/{p.id}"}}
+        if "version" in expand:
+            out["version"] = {"number": p.version, "when": p.when}
+        if "space" in expand:
+            out["space"] = {"key": p.space, "name": f"Space {p.space}"}
+        if "body.storage" in expand:
+            out["body"] = {"storage": {"value": p.body, "representation": "storage"}}
+        if "ancestors" in expand:
+            out["ancestors"] = [{"id": a, "title": self.pages[a].title} for a in p.ancestors if a in self.pages]
+        if "metadata.labels" in expand:
+            out["metadata"] = {"labels": {"results": [{"name": lb} for lb in p.labels]}}
+        return out
+
+    def _paged(self, items: list[dict], start: int, limit: int, path: str, q: dict) -> dict:
+        chunk = items[start:start + limit]
+        out = {"results": chunk, "start": start, "limit": limit, "size": len(chunk), "_links": {}}
+        if start + limit < len(items):
+            rest = "&".join(f"{k}={v}" for k, v in q.items() if k != "start")
+            out["_links"]["next"] = f"{path}?{rest}&start={start + limit}"
+        return out
+
+    def respond(self, method: str, path: str, q: dict, auth: str) -> tuple[int, dict | str]:
+        self.requests.append((method, path, q))
+        if self.status_errors:
+            return self.status_errors.pop(0), "later"
+        if self.redirect:
+            return 302, {"message": "login"}
+        if auth != f"Bearer {self.token}":
+            return 401, {"message": "unauthorized"}
+        if method != "GET":
+            return 405, {"message": "read-only fake"}
+        limit = min(int(q.get("limit", "25")), self.page_size)
+        start = 0 if self.loop else int(q.get("start", "0"))
+        expand = q.get("expand", "")
+        if path == "/rest/api/content":
+            pages = [p for p in self.pages.values() if p.space == q.get("spaceKey") and p.status == "current"]
+            return 200, self._paged([self._summary(p, expand) for p in pages], start, limit, path, q)
+        if path == "/rest/api/content/search":
+            cql = q.get("cql", "")
+            m = re.search(r'space in \(([^)]*)\)', cql)
+            spaces = [s.strip().strip('"') for s in m.group(1).split(",")] if m else []
+            since = re.search(r'lastModified >= "([^"]+)"', cql)
+            pages = [p for p in self.pages.values() if p.space in spaces and p.status == "current" and "type = page" in cql
+                     and (not since or p.when.replace("T", " ")[:16] >= since.group(1)[:16])]
+            return 200, self._paged([self._summary(p, expand) for p in pages], start, limit, path, q)
+        if m := re.fullmatch(r"/rest/api/content/(\d+)", path):
+            p = self.pages.get(m.group(1))
+            if not p or p.status != "current":
+                return 404, {"message": "no content"}
+            return 200, self._summary(p, expand)
+        return 404, {"message": "unknown"}
+
+
+def serve(api: FakeConfluence):
+    class Handler(BaseHTTPRequestHandler):
+        def _handle(self):
+            url = urlparse(self.path)
+            q = {k: v[0] for k, v in parse_qs(url.query).items()}
+            status, body = api.respond(self.command, url.path, q, self.headers.get("Authorization", ""))
+            payload = (json.dumps(body) if isinstance(body, dict) else body).encode()
+            self.send_response(status)
+            self.send_header("Content-Type", "application/json" if isinstance(body, dict) else "text/plain")
+            self.send_header("Retry-After", api.retry_after)
+            self.send_header("Content-Length", str(len(payload)))
+            self.end_headers()
+            self.wfile.write(payload)
+
+        do_GET = do_POST = do_PUT = do_DELETE = _handle
+
+        def log_message(self, *args):
+            pass
+
+    server = ThreadingHTTPServer(("127.0.0.1", 0), Handler)
+    threading.Thread(target=server.serve_forever, kwargs={"poll_interval": 0.01}, daemon=True).start()
+    return server
+
+
+def seed(api: FakeConfluence) -> None:
+    api.add("100", "OPS", "Operations", "<p>The operations space.</p>")
+    api.add("101", "OPS", "Rotating certificates",
+            '<p>Rotate the signing certificate every year.</p><h2>Steps</h2><ol><li>Create the new certificate.</li>'
+            '<li>Deploy it, see <ac:link><ri:page ri:content-title="Deploying" /><ac:plain-text-link-body><![CDATA[the deploy page]]></ac:plain-text-link-body></ac:link>.</li></ol>'
+            '<h3>Checks</h3><ac:structured-macro ac:name="code"><ac:parameter ac:name="language">bash</ac:parameter>'
+            '<ac:plain-text-body><![CDATA[openssl x509 -noout -dates]]></ac:plain-text-body></ac:structured-macro>'
+            '<ac:structured-macro ac:name="toc"/><ac:structured-macro ac:name="info"><ac:rich-text-body><p>Takes an hour.</p></ac:rich-text-body></ac:structured-macro>',
+            labels=["security", "howto"], ancestors=["100"])
+    api.add("102", "OPS", "Deploying", "<p>How to deploy.</p><table><tbody><tr><th>Step</th><th>Who</th></tr><tr><td>Build</td><td>CI</td></tr></tbody></table>",
+            labels=["howto"], ancestors=["100", "101"])
+    api.add("200", "HR", "Cafeteria menu", "<p>Monday: soup.</p>")
+    api.add("300", "SECRET", "Payroll", "<p>Nothing to see.</p>")
+    api.add("103", "OPS", "Draft page", "<p>Not published.</p>", status="draft")

passageway_connector_confluence-0.3.0/tests/test_confluence_connector.py

@@ -0,0 +1,175 @@
+import re
+from pathlib import Path
+
+import pytest
+
+from passageway.protocol import parse_item
+
+README = Path(__file__).resolve().parents[1] / "README.md"
+
+
+def test_list_holds_the_current_pages_of_the_chosen_spaces_only(world):
+    ids = world.ids()
+    assert set(ids) == {"100", "101", "102", "200"}  # not SECRET, not the draft
+
+
+def test_the_version_changes_with_content_title_and_labels(world):
+    before = world.ids()["101"]
+    world.api.edit("101", body="<p>new</p>")
+    after_body = world.ids()["101"]
+    world.api.edit("101", labels=["security"])
+    after_labels = world.ids()["101"]
+    assert len({before, after_body, after_labels}) == 3
+
+
+def test_a_page_becomes_an_item_with_sections_path_labels_and_url(world):
+    item = world.fetch("101")["101"]
+    parsed = parse_item(item)
+    assert parsed.title == "Rotating certificates" and parsed.url == f"{world.url}/pages/viewpage.action?pageId=101"
+    assert parsed.path == ("Operations",) and set(parsed.labels) == {"space:OPS", "security", "howto"}
+    assert parsed.updated == "2026-10-20T10:00:00Z" and parsed.weight == 1.0
+    assert [(s.heading, s.text) for s in parsed.sections] == [
+        (None, "Rotate the signing certificate every year."),
+        ("Steps", "- Create the new certificate.\n- Deploy it, see the deploy page."),
+        ("Steps › Checks", "openssl x509 -noout -dates\nTakes an hour."),
+    ]
+
+
+def test_tables_become_rows_and_ancestors_the_path(world):
+    item = world.fetch("102")["102"]
+    assert item["path"] == ["Operations", "Rotating certificates"]
+    assert item["sections"][0]["text"] == "How to deploy.\nStep | Who\nBuild | CI"
+
+
+def test_label_filters(world):
+    assert set(world.ids(INCLUDE_LABELS="howto")) == {"101", "102"}
+    assert set(world.ids(EXCLUDE_LABELS="security")) == {"100", "102", "200"}
+
+
+def test_fetch_skips_ids_that_are_gone_or_out_of_scope(world):
+    assert set(world.fetch("101", "300", "999", "103")) == {"101"}
+
+
+def test_paging_walks_every_page(world):
+    world.api.page_size = 2
+    assert set(world.ids()) == {"100", "101", "102", "200"}
+    listing = [q for m, p, q in world.api.requests if p == "/rest/api/content"]
+    assert any(q.get("start") == "2" for q in listing)
+
+
+def test_changed_asks_cql_in_confluence_time_and_returns_the_modified_pages(world):
+    world.api.edit("200", body="<p>Tuesday: stew.</p>")
+    code, records, err = world.call("changed", extra=["--since", "2026-10-25T00:00:00Z"])
+    assert code == 0, err
+    assert [r["id"] for r in records] == ["200"]
+    cql = next(q["cql"] for m, p, q in world.api.requests if p.endswith("/search"))
+    assert 'space in ("OPS","HR")' in cql and "type = page" in cql and 'lastModified >= "2026-10-2' in cql
+
+
+def test_more_pages_than_max_pages_is_an_error_not_a_silent_subset(world):
+    code, records, err = world.call("list", MAX_PAGES="2")
+    assert code == 1 and records == [] and "MAX_PAGES" in err
+
+
+@pytest.mark.parametrize("settings,message", [
+    ({"URL": ""}, "URL"), ({"URL": "ftp://x"}, "URL"), ({"SPACES": ""}, "SPACES"), ({"TOKEN": ""}, "TOKEN"),
+    ({"MAX_PAGES": "lots"}, "MAX_PAGES"), ({"TIMEOUT": "0"}, "TIMEOUT"), ({"VERIFY_TLS": "maybe"}, "VERIFY_TLS"),
+])
+def test_bad_settings_exit_2_with_a_message_and_no_output(world, settings, message):
+    code, records, err = world.call("list", **settings)
+    assert code == 2 and records == [] and message in err
+
+
+def test_a_wrong_token_is_exit_1_without_the_token_in_the_message(world):
+    code, records, err = world.call("list", TOKEN="wrong-pat-value")
+    assert code == 1 and records == [] and "401" in err and "wrong-pat-value" not in err
+
+
+def test_429_and_5xx_are_retried_then_given_up(world):
+    world.api.status_errors = [429, 503]
+    assert set(world.ids()) == {"100", "101", "102", "200"}
+    world.api.status_errors = [503] * 100
+    code, records, err = world.call("list")
+    assert code == 1 and records == [] and "503" in err
+
+
+def test_a_429_longer_than_a_minute_is_waited_out(world):
+    world.api.status_errors = [429] * 8  # 1+2+4+…: over two minutes of waiting
+    assert world.fetch("101").keys() == {"101"}
+    assert sum(world.slept) > 120
+
+
+def test_a_long_wait_is_said_in_the_log_as_it_starts(world, capsys):
+    world.api.status_errors = [429] * 5  # waits 1, 2, 4, 8, then 16
+    assert world.fetch("101").keys() == {"101"}
+    said = capsys.readouterr().err.splitlines()
+    assert said == ["confluence connector: HTTP 429 on GET /rest/api/content/101: waiting 16 s (15 s of 900 s waited)"]
+
+
+def test_retry_after_is_honoured_beyond_30_seconds(world):
+    world.api.status_errors, world.api.retry_after = [429], "200"
+    assert world.fetch("101").keys() == {"101"}
+    assert world.slept == [200]
+
+
+def test_a_429_that_never_ends_fails_and_says_how_long_it_waited(world):
+    world.api.status_errors = [429] * 1000
+    code, records, err = world.call("fetch", '{"id": "101"}\n')
+    assert code == 1 and records == [] and "HTTP 429" in err
+    waited = sum(world.slept)
+    assert 600 <= waited <= 1200 and f"{waited:.0f} s" in err
+
+
+def test_only_get_requests_are_ever_sent(world):
+    world.ids(); world.fetch("101"); world.call("changed", extra=["--since", "2026-10-25T00:00:00Z"])
+    assert {m for m, _, _ in world.api.requests} == {"GET"}
+
+
+def test_every_setting_is_in_the_readme():
+    code = (Path(__file__).resolve().parents[1] / "passageway_connector_confluence" / "connector.py").read_text()
+    names = sorted(set(re.findall(r'env\.get\(\s*"([A-Z_]+)"', code)) | set(re.findall(r'_list\(\s*env,\s*"([A-Z_]+)"', code)))
+    assert len(names) >= 6, names
+    readme = README.read_text()
+    for name in names:
+        assert f"`{name}`" in readme, f"{name} is read by the Confluence connector but missing from its README"
+
+
+# ---- review findings ------------------------------------------------------------------------------
+
+def test_a_page_id_containing_404_is_not_skipped_on_a_real_failure(world):
+    world.api.add("14045", "OPS", "Unlucky", "<p>x</p>")
+    world.api.status_errors = [503] * 100
+    code, records, err = world.call("fetch", '{"id": "14045"}\n')
+    assert code == 1 and records == [] and "503" in err
+
+
+def test_a_page_that_is_really_gone_is_skipped(world):
+    assert world.fetch("999") == {}
+
+
+def test_changed_asks_with_a_margin_for_confluences_timezone(world):
+    world.call("changed", extra=["--since", "2026-10-25T12:00:00Z"])
+    cql = next(q["cql"] for m, p, q in world.api.requests if p.endswith("/search"))
+    assert 'lastModified >= "2026-10-24 22:00"' in cql  # 14 hours earlier: CQL reads the time in the server's zone
+
+
+def test_paging_stops_when_a_page_brings_nothing_new(world):
+    world.api.page_size = 2
+    world.api.loop = True  # a server that ignores `start`, as cursor-paged ones do
+    assert set(world.ids()) == {"100", "101"} or len(world.ids()) >= 2  # ends, instead of looping forever
+
+
+def test_a_space_that_lists_nothing_is_a_warning(world):
+    code, records, err = world.call("list", SPACES="OPS|EMPTY")
+    assert code == 0 and "EMPTY" in err and "no pages" in err
+
+
+def test_a_redirect_is_a_clear_error(world):
+    world.api.redirect = True
+    code, _, err = world.call("list")
+    assert code == 1 and "redirect" in err.lower()
+
+
+def test_verify_tls_false_warns(world):
+    code, _, err = world.call("check", VERIFY_TLS="false")
+    assert code == 0 and "VERIFY_TLS" in err

passageway_connector_confluence-0.3.0/tests/test_confluence_contract.py

@@ -0,0 +1,38 @@
+import json
+import os
+import subprocess
+import sys
+from pathlib import Path
+
+import jsonschema
+
+from passageway.protocol import Connector
+from passageway.testing.contract import check_connector
+
+SCHEMA = Path(__file__).resolve().parents[3] / "docs" / "reference" / "connector-protocol.schema.json"
+
+
+def test_the_connector_passes_the_shared_contract_as_a_real_subprocess(world):
+    connector = Connector("confluence", [sys.executable, "-m", "passageway_connector_confluence"],
+                          env={"URL": world.url, "SPACES": "OPS|HR", "PAUSE_SECONDS": "0"}, secrets={"TOKEN": world.api.token})
+    check_connector(connector)
+    assert {i.title for i in connector.fetch(["101", "200"])} == {"Rotating certificates", "Cafeteria menu"}
+
+
+def test_the_output_validates_against_the_published_json_schema(world):
+    schema = json.loads(SCHEMA.read_text())
+
+    def valid(name, record):
+        return jsonschema.Draft202012Validator({"$ref": f"#/$defs/{name}", "$defs": schema["$defs"]}).is_valid(record)
+
+    def lines(mode, stdin=""):
+        out = subprocess.run([sys.executable, "-m", "passageway_connector_confluence", mode],
+                             env={**os.environ, "URL": world.url, "SPACES": "OPS|HR", "TOKEN": world.api.token, "PAUSE_SECONDS": "0"},
+                             input=stdin, capture_output=True, text=True, check=True).stdout
+        return [json.loads(line) for line in out.splitlines()]
+
+    assert all(valid("check", r) for r in lines("check"))
+    listed = lines("list")
+    assert listed and all(valid("listRecord", r) for r in listed)
+    items = lines("fetch", "".join(json.dumps({"id": r["id"]}) + "\n" for r in listed))
+    assert len(items) == len(listed) and all(valid("item", r) for r in items)

passageway_connector_confluence-0.3.0/tests/test_storage.py

@@ -0,0 +1,57 @@
+"""Confluence's storage format becomes readable text without losing what a page says."""
+
+import pytest
+
+from passageway_connector_confluence.connector import storage_to_html
+from passageway_connector_confluence.html import html_to_sections
+
+
+def text(storage):
+    return [(" › ".join(s.heading_path) or None, s.text) for s in html_to_sections(storage_to_html(storage))]
+
+
+def test_a_self_closing_macro_with_attributes_does_not_swallow_what_follows():
+    out = text('<ac:structured-macro ac:name="toc" ac:schema-version="1" ac:macro-id="m1" /><h2>A</h2><p>a</p>'
+               '<ac:structured-macro ac:name="info" ac:schema-version="1"><ac:rich-text-body><p>tip</p></ac:rich-text-body></ac:structured-macro>')
+    assert out == [("A", "a\ntip")]
+
+
+def test_nested_macros_keep_every_part_in_order():
+    out = text('<ac:structured-macro ac:name="expand"><ac:parameter ac:name="title">More</ac:parameter><ac:rich-text-body>'
+               '<p>Before</p><ac:structured-macro ac:name="code"><ac:plain-text-body><![CDATA[echo hi]]></ac:plain-text-body></ac:structured-macro>'
+               '<p>After code</p><ac:structured-macro ac:name="code"><ac:plain-text-body><![CDATA[two]]></ac:plain-text-body></ac:structured-macro>'
+               '</ac:rich-text-body></ac:structured-macro><p>Tail</p>')
+    assert out == [(None, "Before\necho hi\nAfter code\ntwo\nTail")]
+
+
+def test_a_user_mention_and_a_later_title_link_do_not_delete_the_sections_between():
+    out = text('<p>Owner: <ac:link><ri:user ri:userkey="abc"/></ac:link></p><h2>Install</h2><p>Run the installer.</p>'
+               '<h2>Rollback</h2><p>See <ac:link><ri:page ri:content-title="FAQ"/></ac:link>.</p>')
+    assert out == [(None, "Owner:"), ("Install", "Run the installer."), ("Rollback", "See FAQ.")]
+
+
+def test_code_keeps_its_markup_as_text():
+    out = text('<ac:structured-macro ac:name="code"><ac:plain-text-body><![CDATA[<div class="x">a & b</div> <ac:foo>keep</ac:foo>]]></ac:plain-text-body></ac:structured-macro>')
+    assert out == [(None, '<div class="x">a & b</div> <ac:foo>keep</ac:foo>')]
+
+
+def test_task_lists_placeholders_and_status_macros():
+    out = text('<ac:task-list><ac:task><ac:task-id>1</ac:task-id><ac:task-status>incomplete</ac:task-status><ac:task-body>Buy milk</ac:task-body></ac:task>'
+               '<ac:task><ac:task-id>2</ac:task-id><ac:task-status>complete</ac:task-status><ac:task-body>Ship</ac:task-body></ac:task></ac:task-list>'
+               '<p><ac:placeholder>Type here</ac:placeholder>Real</p><p>State: <ac:structured-macro ac:name="status"><ac:parameter ac:name="title">DONE</ac:parameter></ac:structured-macro></p>')
+    assert out == [(None, "- Buy milk\n- Ship\nReal\nState: DONE")]
+
+
+def test_links_images_emoticons_layouts_and_entities():
+    out = text('<ac:layout><ac:layout-section><ac:layout-cell><p>Left &amp; <ac:emoticon ac:name="smile"/> &nbsp;&eacute;</p>'
+               '<p><ac:link><ri:page ri:space-key="OPS" ri:content-title="Deploying"/><ac:link-body><strong>deploy</strong> it</ac:link-body></ac:link></p>'
+               '<ac:image><ri:attachment ri:filename="x.png"/></ac:image><p>Tail</p></ac:layout-cell></ac:layout-section></ac:layout>')
+    assert out == [(None, "Left & é\ndeploy it\nTail")]
+
+
+def test_malformed_storage_still_gives_text():
+    assert text("<p>a <b>bold</p> unclosed & stray") == [(None, "a bold\nunclosed & stray")]
+
+
+def test_headings_below_h3_stay_in_the_text():
+    assert text("<h2>A</h2><h4>Sub</h4><p>x</p>") == [("A", "Sub\nx")]