confluence-knowledge 0.3.1__tar.gz

confluence_knowledge-0.3.1/PKG-INFO

@@ -0,0 +1,77 @@
+Metadata-Version: 2.4
+Name: confluence-knowledge
+Version: 0.3.1
+Summary: Local, GET-only Confluence subtree indexer + grounded-Q&A MCP service
+License: MIT
+Project-URL: Repository, https://github.com/nkunal1992/agents
+Project-URL: Documentation, https://github.com/nkunal1992/agents/blob/main/docs/confluence-knowledge.md
+Project-URL: Issues, https://github.com/nkunal1992/agents/issues
+Classifier: Operating System :: OS Independent
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3 :: Only
+Requires-Python: >=3.11
+Description-Content-Type: text/markdown
+Requires-Dist: httpx<0.29,>=0.27
+Requires-Dist: beautifulsoup4>=4.12
+Requires-Dist: numpy>=1.26
+Requires-Dist: keyring>=25.2
+Requires-Dist: mcp<2,>=1.6
+Requires-Dist: pip-system-certs>=5.0
+Provides-Extra: models
+Requires-Dist: sentence-transformers<4.0,>=3.0; extra == "models"
+Requires-Dist: huggingface-hub>=0.26; extra == "models"
+Provides-Extra: dev
+Requires-Dist: pytest>=8.0; extra == "dev"
+Requires-Dist: pytest-cov>=5.0; extra == "dev"
+Requires-Dist: ruff>=0.6; extra == "dev"
+
+# confluence-knowledge (local MCP service)
+
+Local companion service for the `confluence-knowledge` Copilot agent.
+See `../../docs/confluence-knowledge.md` for the full guide.
+
+- GET-only `ConfluenceClient` — non-GET raises `SecurityException` pre-flight.
+- Recursive subtree crawl (Cloud v2 + Data Center), resumable.
+- Local embeddings (`all-MiniLM-L6-v2`) + local reranker
+  (`ms-marco-MiniLM-L6-v2`) — Apache-2.0, pinned in `models.lock.json`,
+  never a cloud embedding call.
+- SQLite + FTS5 + vector-blob index under `.conf_knowledge_index/`.
+- Credentials via OS keyring / env var / chmod-600 `.conf_auth.json`.
+
+## Dev
+
+```bash
+python3.11+ -m venv .venv && .venv/bin/pip install -e '.[dev]'
+.venv/bin/python -m pytest        # all offline — no network needed
+.venv/bin/python -m ruff check src tests
+```
+
+Before every release, run the end-to-end smoke test. It spawns the real stdio
+server through the MCP client SDK and uses the real models against a fake
+Confluence on localhost, with simulated network latency:
+
+```bash
+.venv/bin/pip install -e '.[models]'
+CONF_KNOWLEDGE_HOME=/tmp/ck/home .venv/bin/confluence-knowledge prepare-models
+CONF_KNOWLEDGE_HOME=/tmp/ck/home .venv/bin/python scripts/e2e_smoke.py   # ends with "E2E OK"
+```
+
+`e2e_bootstrap.py` goes further — it replays the agent's entire install flow
+in a sandboxed $HOME (python discovery → venv → `pip install` from the
+shipped `dist/` bundle → models → credentials → `register` → the real server
+over stdio → a full Q&A round-trip). Run it before every release:
+
+```bash
+CK_MODEL_CACHE=/tmp/ck/home/models .venv/bin/python scripts/e2e_bootstrap.py  # "E2E BOOTSTRAP OK"
+```
+
+## CLI
+
+| Command | Purpose |
+|---|---|
+| `confluence-knowledge setup` | guided env/site/token setup (hidden input, keyring) |
+| `confluence-knowledge prepare-models` | fetch pinned local models once |
+| `confluence-knowledge doctor` | verify config + heartbeat + models |
+| `confluence-knowledge serve` | run the stdio MCP server (what the IDE launches) |
+| `confluence-knowledge clean` | delete the local index (`~/.confluence-knowledge/work` unless `CONF_KNOWLEDGE_WORKDIR` is set) |

confluence_knowledge-0.3.1/README.md

@@ -0,0 +1,49 @@
+# confluence-knowledge (local MCP service)
+
+Local companion service for the `confluence-knowledge` Copilot agent.
+See `../../docs/confluence-knowledge.md` for the full guide.
+
+- GET-only `ConfluenceClient` — non-GET raises `SecurityException` pre-flight.
+- Recursive subtree crawl (Cloud v2 + Data Center), resumable.
+- Local embeddings (`all-MiniLM-L6-v2`) + local reranker
+  (`ms-marco-MiniLM-L6-v2`) — Apache-2.0, pinned in `models.lock.json`,
+  never a cloud embedding call.
+- SQLite + FTS5 + vector-blob index under `.conf_knowledge_index/`.
+- Credentials via OS keyring / env var / chmod-600 `.conf_auth.json`.
+
+## Dev
+
+```bash
+python3.11+ -m venv .venv && .venv/bin/pip install -e '.[dev]'
+.venv/bin/python -m pytest        # all offline — no network needed
+.venv/bin/python -m ruff check src tests
+```
+
+Before every release, run the end-to-end smoke test. It spawns the real stdio
+server through the MCP client SDK and uses the real models against a fake
+Confluence on localhost, with simulated network latency:
+
+```bash
+.venv/bin/pip install -e '.[models]'
+CONF_KNOWLEDGE_HOME=/tmp/ck/home .venv/bin/confluence-knowledge prepare-models
+CONF_KNOWLEDGE_HOME=/tmp/ck/home .venv/bin/python scripts/e2e_smoke.py   # ends with "E2E OK"
+```
+
+`e2e_bootstrap.py` goes further — it replays the agent's entire install flow
+in a sandboxed $HOME (python discovery → venv → `pip install` from the
+shipped `dist/` bundle → models → credentials → `register` → the real server
+over stdio → a full Q&A round-trip). Run it before every release:
+
+```bash
+CK_MODEL_CACHE=/tmp/ck/home/models .venv/bin/python scripts/e2e_bootstrap.py  # "E2E BOOTSTRAP OK"
+```
+
+## CLI
+
+| Command | Purpose |
+|---|---|
+| `confluence-knowledge setup` | guided env/site/token setup (hidden input, keyring) |
+| `confluence-knowledge prepare-models` | fetch pinned local models once |
+| `confluence-knowledge doctor` | verify config + heartbeat + models |
+| `confluence-knowledge serve` | run the stdio MCP server (what the IDE launches) |
+| `confluence-knowledge clean` | delete the local index (`~/.confluence-knowledge/work` unless `CONF_KNOWLEDGE_WORKDIR` is set) |

confluence_knowledge-0.3.1/pyproject.toml

@@ -0,0 +1,56 @@
+[build-system]
+requires = ["setuptools>=68"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "confluence-knowledge"
+version = "0.3.1"
+description = "Local, GET-only Confluence subtree indexer + grounded-Q&A MCP service"
+readme = "README.md"
+requires-python = ">=3.11"
+license = { text = "MIT" }
+classifiers = [
+    "Operating System :: OS Independent",
+    "Intended Audience :: Developers",
+    "License :: OSI Approved :: MIT License",
+    "Programming Language :: Python :: 3 :: Only",
+]
+dependencies = [
+    "httpx>=0.27,<0.29",
+    "beautifulsoup4>=4.12",
+    "numpy>=1.26",
+    "keyring>=25.2",
+    "mcp>=1.6,<2",
+    "pip-system-certs>=5.0",  # OS cert store → works behind TLS-inspecting proxies
+]
+
+[project.urls]
+Repository = "https://github.com/nkunal1992/agents"
+Documentation = "https://github.com/nkunal1992/agents/blob/main/docs/confluence-knowledge.md"
+Issues = "https://github.com/nkunal1992/agents/issues"
+
+[project.optional-dependencies]
+models = [
+    "sentence-transformers>=3.0,<4.0",
+    "huggingface-hub>=0.26",
+]
+dev = [
+    "pytest>=8.0",
+    "pytest-cov>=5.0",
+    "ruff>=0.6",
+]
+
+[project.scripts]
+confluence-knowledge = "confluence_knowledge.cli:main"
+
+[tool.setuptools.packages.find]
+where = ["src"]
+
+[tool.setuptools.package-data]
+confluence_knowledge = ["models.lock.json"]
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]
+
+[tool.ruff]
+line-length = 110

confluence_knowledge-0.3.1/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

confluence_knowledge-0.3.1/src/confluence_knowledge/__init__.py

@@ -0,0 +1,3 @@
+"""confluence-knowledge: local GET-only Confluence subtree index + grounded Q&A."""
+
+__version__ = "0.1.0"

confluence_knowledge-0.3.1/src/confluence_knowledge/auth.py

@@ -0,0 +1,83 @@
+"""Credential storage: OS keyring → env var → protected file (explicit choice).
+
+Tokens are never accepted via chat, MCP tool arguments, or command-line flags —
+the setup wizard prompts with getpass in the user's own terminal.
+"""
+
+from __future__ import annotations
+
+import json
+import os
+import stat
+from pathlib import Path
+
+from .settings import APP_NAME, AUTH_FILE_NAME, ensure_gitignore, reject_unsafe_path
+
+SERVICE = APP_NAME
+ENV_TOKEN = "CONF_KNOWLEDGE_TOKEN"
+ENV_EMAIL = "CONF_KNOWLEDGE_EMAIL"
+
+
+def _keyring_store(ref: str, secret: str) -> bool:
+    try:
+        import keyring
+        from keyring.backends.fail import Keyring as FailBackend
+        if isinstance(keyring.get_keyring(), FailBackend):
+            return False
+        keyring.set_password(SERVICE, ref, secret)
+        return True
+    except Exception:  # noqa: BLE001 — keyring backends fail in many ways; fallback is intentional
+        return False
+
+
+def _keyring_get(ref: str) -> str | None:
+    try:
+        import keyring
+        return keyring.get_password(SERVICE, ref)
+    except Exception:  # noqa: BLE001 — unavailable keyring → try next provider
+        return None
+
+
+def store_credential(workdir: Path, home: Path, ref: str, token: str) -> str:
+    """Persist token: keyring preferred, protected file as explicit fallback.
+
+    Returns a human description of where it went (never the token).
+    """
+    if _keyring_store(ref, token):
+        return "OS credential store"
+    # file fallback — gitignore BEFORE writing
+    ensure_gitignore(workdir, [AUTH_FILE_NAME, ".conf_knowledge_index/"])
+    p = reject_unsafe_path(workdir / AUTH_FILE_NAME)
+    data = json.loads(p.read_text()) if p.exists() else {}
+    data[ref] = token
+    tmp = p.with_suffix(".tmp")
+    tmp.write_text(json.dumps(data))
+    os.replace(tmp, p)
+    if os.name != "nt":
+        os.chmod(p, stat.S_IRUSR | stat.S_IWUSR)
+    return f"protected file {p} (chmod 600)"
+
+
+def get_credential(workdir: Path, ref: str) -> str | None:
+    """Lookup order: env var (CI/scripts) → keyring → protected file."""
+    t = os.environ.get(ENV_TOKEN)
+    if t:
+        return t
+    t = _keyring_get(ref)
+    if t:
+        return t
+    p = workdir / AUTH_FILE_NAME
+    if p.exists():
+        data = json.loads(p.read_text())
+        return data.get(ref)
+    p2 = Path.home() / ".confluence-knowledge" / AUTH_FILE_NAME
+    if p2.exists():
+        return json.loads(p2.read_text()).get(ref)
+    return None
+
+
+def redact(text: str, secrets: list[str]) -> str:
+    for s in secrets:
+        if s:
+            text = text.replace(s, "***")
+    return text

confluence_knowledge-0.3.1/src/confluence_knowledge/cli.py

@@ -0,0 +1,180 @@
+"""confluence-knowledge CLI: setup / doctor / prepare-models / serve / clean.
+
+The setup wizard prompts for tokens via getpass in the user's own terminal —
+tokens never travel through chat or command-line arguments.
+"""
+
+from __future__ import annotations
+
+import argparse
+import getpass
+import json
+import os
+import sys
+from pathlib import Path
+
+from .auth import store_credential
+from .client import AccessError, make_client
+from .retrieval import prepare_models
+from .settings import (
+    AUTH_FILE_NAME,
+    INDEX_DIR_NAME,
+    SetupError,
+    SiteProfile,
+    default_home,
+    ensure_gitignore,
+    resolve_workdir,
+)
+
+
+def _wizard() -> int:
+    print("confluence-knowledge setup\n")
+    env = ""
+    while env not in ("1", "2"):
+        env = input("Confluence environment — [1] Cloud (atlassian.net) "
+                    "or [2] Data Center (corporate URL)? ").strip()
+    env = "cloud" if env == "1" else "datacenter"
+
+    if env == "cloud":
+        print("\nCreate a token: sign in to https://id.atlassian.com → "
+              "Security → Create and manage API tokens → Create API token "
+              "(choose Confluence read scopes if scoped tokens are offered).")
+        site = input("\nSite URL (e.g. https://acme.atlassian.net): ").strip()
+        email = input("Atlassian account email: ").strip()
+    else:
+        print("\nCreate a token: Confluence → your profile picture → Settings → "
+              "Personal Access Tokens → Create token (read access is enough).")
+        site = input("\nSite URL incl. context path (e.g. https://conf.acme.corp/confluence): ").strip()
+        email = None
+
+    try:
+        base = SiteProfile.canonical_base(env, site)
+    except SetupError as e:
+        print(f"error: {e}", file=sys.stderr)
+        return 2
+
+    workdir = resolve_workdir()
+    workdir.mkdir(parents=True, exist_ok=True)
+    home = default_home()
+    # gitignore protections BEFORE any secret/cache write
+    try:
+        ensure_gitignore(workdir, [AUTH_FILE_NAME, ".conf_knowledge_index/"])
+    except SetupError as e:
+        print(f"error: {e}", file=sys.stderr)
+        return 2
+
+    ref = f"{env}:{base}:{email or '-'}"
+    token = getpass.getpass("Token (input hidden): ")
+    if not token:
+        print("error: empty token", file=sys.stderr)
+        return 2
+
+    where = store_credential(workdir, home, ref, token)
+    SiteProfile(env, base, email, ref).save(home)
+
+    # heartbeat
+    try:
+        c = make_client(SiteProfile(env, base, email, ref), token)
+        ep = "wiki/rest/api/user/current" if env == "cloud" else "rest/api/user/current"
+        who = c.get(ep).json()
+        print(f"\nconnected as: {who.get('displayName') or who.get('username') or 'OK'}")
+    except Exception as e:  # noqa: BLE001 — any connect failure (auth/network/TLS) exits non-zero
+        print(f"\nconnection check failed: {e}", file=sys.stderr)
+        return 3
+
+    print(f"credential stored: {where}")
+    print(f"profile: {home / 'profile.json'}")
+    print("\nNext: `confluence-knowledge prepare-models` to fetch the local "
+          "models (Apache-2.0), then point your IDE's MCP config at:\n"
+          f"  command: {sys.executable}\n"
+          f"  args:    [\"-m\", \"confluence_knowledge.server\"]\n"
+          f"  env:     CONF_KNOWLEDGE_WORKDIR={workdir}")
+    return 0
+
+
+def _doctor() -> int:
+    from .server import status
+    try:
+        s = status()
+        print(json.dumps(s, indent=2))
+        if s.get("configured"):
+            from .server import _client_for  # internal but fine in CLI
+            profile = SiteProfile.load(default_home())
+            try:
+                c = _client_for(profile)
+                ep = ("wiki/rest/api/user/current" if profile.env == "cloud"
+                      else "rest/api/user/current")
+                who = c.get(ep).json()
+                print("heartbeat: OK —",
+                      who.get("displayName") or who.get("username"))
+            except (AccessError, SetupError) as e:
+                print(f"heartbeat: FAILED — {e}", file=sys.stderr)
+                return 3
+        return 0
+    except SetupError as e:
+        print(f"setup error: {e}", file=sys.stderr)
+        return 2
+
+
+def main() -> None:
+    ap = argparse.ArgumentParser(prog="confluence-knowledge")
+    sub = ap.add_subparsers(dest="cmd", required=True)
+    sub.add_parser("setup", help="guided credential + config setup")
+    sub.add_parser("doctor", help="verify config, credentials, models")
+    sub.add_parser("prepare-models", help="download the pinned local models")
+    reg = sub.add_parser(
+        "register",
+        help="write this server into the IDE's MCP config (no manual paste)")
+    reg.add_argument("--ide", default="auto",
+                     choices=["auto", "all", "vscode", "jetbrains"],
+                     help="which config files to write (default: all that apply)")
+    reg.add_argument("--workspace", type=Path, default=None,
+                     help="also write <workspace>/.vscode/mcp.json")
+    reg.add_argument("--remove", action="store_true",
+                     help="remove our entry instead of adding it")
+    sub.add_parser("serve", help="run the stdio MCP server")
+    clean = sub.add_parser("clean", help="delete the local index for this repo")
+    clean.add_argument("--yes", action="store_true")
+    args = ap.parse_args()
+
+    if args.cmd == "setup":
+        sys.exit(_wizard())
+    if args.cmd == "doctor":
+        sys.exit(_doctor())
+    if args.cmd == "prepare-models":
+        for line in prepare_models(default_home()):
+            print(line)
+        return
+    if args.cmd == "register":
+        from .registration import register
+        try:
+            wd = (Path(os.environ["CONF_KNOWLEDGE_WORKDIR"]).expanduser().resolve()
+                  if os.environ.get("CONF_KNOWLEDGE_WORKDIR") else None)
+            results = register(ide=args.ide, workspace=args.workspace,
+                               workdir=wd, remove=args.remove)
+        except SetupError as e:
+            print(f"error: {e}", file=sys.stderr)
+            sys.exit(2)
+        for path, action in results:
+            print(f"{action}: {path}")
+        if not args.remove:
+            print("\nReload your IDE so Copilot picks up the server, "
+                  "then re-ask the agent.")
+        return
+    if args.cmd == "serve":
+        from .server import main as serve
+        serve()
+        return
+    if args.cmd == "clean":
+        import shutil
+
+        target = resolve_workdir() / INDEX_DIR_NAME
+        if target.exists() and (args.yes or input(
+                f"delete {target}? [y/N] ").strip().lower() == "y"):
+            shutil.rmtree(target)
+            print("index deleted")
+        return
+
+
+if __name__ == "__main__":
+    main()

confluence_knowledge-0.3.1/src/confluence_knowledge/client.py

@@ -0,0 +1,152 @@
+"""GET-only Confluence REST client.
+
+Every transport call passes through request(): a non-GET method is rejected
+with SecurityException before any network I/O, and paths are validated against
+a per-adapter endpoint allowlist. No convenience method can bypass this.
+"""
+
+from __future__ import annotations
+
+import re
+from dataclasses import dataclass
+from urllib.parse import urljoin, urlparse
+
+import httpx
+
+
+class SecurityException(RuntimeError):
+    """Raised when a write/delete/mutation HTTP operation is attempted."""
+
+
+class AccessError(RuntimeError):
+    """Authentication, permission, or reachability failure (actionable)."""
+
+
+@dataclass(frozen=True)
+class EndpointRule:
+    pattern: re.Pattern  # matched against the API path
+    purpose: str
+
+
+class ConfluenceClient:
+    """GET-only client bound to one site base URL and one route allowlist."""
+
+    def __init__(
+        self,
+        base_url: str,
+        headers: dict[str, str],
+        rules: list[EndpointRule],
+        timeout: float = 20.0,
+        max_bytes: int = 8 * 1024 * 1024,
+        session: httpx.Client | None = None,
+    ):
+        u = urlparse(base_url)
+        if u.scheme != "https" and u.hostname not in ("localhost", "127.0.0.1"):
+            raise AccessError("Confluence base URL must be https")
+        self._base = base_url.rstrip("/") + "/"
+        self._base_origin = f"{u.scheme}://{u.netloc}"
+        self._base_path = u.path.rstrip("/")
+        self._rules = rules
+        self._http = session or httpx.Client(
+            headers={"Accept": "application/json", **headers},
+            timeout=timeout,
+            follow_redirects=False,  # never carry credentials through redirects
+        )
+        self._max_bytes = max_bytes
+
+    # ---- the GET-only wrapper ----
+
+    def request(self, method: str, path: str, params: dict | None = None) -> httpx.Response:
+        if method.upper() != "GET":
+            raise SecurityException(
+                "Error: This agent is strictly read-only. "
+                "Write/Delete operations are forbidden."
+            )
+        api_path = self._validate_path(path)
+        resp = self._http.get(urljoin(self._base, api_path.lstrip("/")), params=params)
+        if resp.status_code >= 400:
+            raise AccessError(self._describe(resp))
+        if resp.status_code >= 300:
+            raise AccessError(
+                f"unexpected redirect ({resp.status_code}) — check the base URL/context path"
+            )
+        if len(resp.content) > self._max_bytes:
+            raise AccessError(f"response exceeds {self._max_bytes} bytes — refusing to buffer")
+        return resp
+
+    def get(self, path: str, params: dict | None = None) -> httpx.Response:
+        return self.request("GET", path, params)
+
+    # ---- validation ----
+
+    def _validate_path(self, path: str) -> str:
+        """Normalize to an API-relative path; enforce site + endpoint bounds."""
+        u = urlparse(path)
+        if u.scheme or u.netloc:
+            # absolute URL (e.g. pagination next-link) must stay inside
+            # the configured origin + API base path
+            full_base = self._base_origin + self._base_path
+            full = f"{u.scheme}://{u.netloc}{u.path}"
+            if not (full == full_base or full.startswith(full_base + "/")):
+                raise SecurityException(
+                    "Error: request escapes the configured Confluence site — refused."
+                )
+            rel = u.path[len(self._base_path):].lstrip("/")
+        elif path.startswith("/"):
+            # site-relative path must live under the API base path
+            if self._base_path and not path.startswith(self._base_path + "/"):
+                raise SecurityException(
+                    "Error: request escapes the configured Confluence site — refused."
+                )
+            rel = path[len(self._base_path):].lstrip("/")
+        else:
+            rel = path
+        if ".." in rel:
+            raise SecurityException("Error: path traversal rejected.")
+        if not any(r.pattern.match(rel) for r in self._rules):
+            raise SecurityException(
+                "Error: This agent is strictly read-only. "
+                f"Endpoint not on the allowed GET list: {rel}"
+            )
+        return rel
+
+    @staticmethod
+    def _describe(resp: httpx.Response) -> str:
+        s = resp.status_code
+        return {
+            401: "authentication failed (401) — check the token and account",
+            403: "permission denied (403) — your account can't view this content",
+            404: "not found (404) — check the page ID/space and base URL",
+            429: "rate limited (429) — retry after the server's Retry-After",
+        }.get(s, f"HTTP {s} from Confluence")
+
+
+# ---- adapters ----
+
+RULES_V2 = [
+    EndpointRule(re.compile(r"wiki/api/v2/pages/\d+/?$"), "page body"),
+    EndpointRule(re.compile(r"wiki/api/v2/pages/\d+/direct-children/?$"), "children"),
+    EndpointRule(re.compile(r"wiki/api/v2/folders/\d+/direct-children/?$"), "folder children"),
+    EndpointRule(re.compile(r"wiki/rest/api/user/current/?$"), "identity"),
+]
+
+RULES_DC = [
+    EndpointRule(re.compile(r"(?:confluence/)?rest/api/content/\d+/?$"), "page body"),
+    EndpointRule(re.compile(r"(?:confluence/)?rest/api/content/\d+/child/page/?$"), "children"),
+    EndpointRule(re.compile(r"(?:confluence/)?rest/api/user/current/?$"), "identity"),
+]
+
+def make_client(profile, token: str, cloud_id: str | None = None) -> ConfluenceClient:
+    """Build a client for the configured profile. Token only in headers."""
+    import base64
+
+    if profile.env == "cloud":
+        auth = base64.b64encode(f"{profile.email}:{token}".encode()).decode()
+        # scoped tokens go through the Atlassian gateway bound to one cloudId;
+        # the same v2 route shapes apply relative to that base path
+        base = (f"https://api.atlassian.com/ex/confluence/{cloud_id}"
+                if cloud_id else profile.base_url)
+        return ConfluenceClient(base, {"Authorization": f"Basic {auth}"}, RULES_V2)
+    return ConfluenceClient(
+        profile.base_url, {"Authorization": f"Bearer {token}"}, RULES_DC
+    )