kiwime-compiler 0.1.0__tar.gz

kiwime_compiler-0.1.0/.gitignore

@@ -0,0 +1,78 @@
+# OS
+.DS_Store
+Thumbs.db
+
+# Editors
+.vscode/*
+!.vscode/settings.json
+.idea/
+*.swp
+*.swo
+
+# Rust
+target/
+Cargo.lock.bak
+
+# Node
+node_modules/
+dist/
+.pnpm-store/
+*.log
+
+# Python
+__pycache__/
+*.py[cod]
+*.egg-info/
+.venv/
+.pytest_cache/
+.mypy_cache/
+.ruff_cache/
+.coverage
+htmlcov/
+
+# Tauri
+apps/desktop/src-tauri/target/
+
+# Frozen Python sidecars (build artifacts, see ADR 0006)
+apps/desktop/src-tauri/binaries/
+target/pyinstaller/
+
+# uv
+.python-version
+
+# Frontend build copied into the host package by the publish workflow
+apps/host/src/kiwime_host/_ui/
+
+# Local data
+*.sqlite
+*.sqlite-journal
+*.db
+data/
+blobs/
+
+# Secrets
+.env
+.env.local
+*.pem
+
+# Per-user Claude Code settings (personal permission allowlists, local paths)
+.claude/settings.local.json
+
+# TypeScript incremental build state
+*.tsbuildinfo
+apps/desktop/dist-types-node/
+
+# Internal planning notes (kept out of the public tree; see docs/ for the
+# published roadmap, ADRs, and release docs)
+.internal-notes/
+docs/release/C_NOTES.md
+docs/release/D_NOTES.md
+docs/release/oss-readiness-review.md
+docs/release/implementation-plan-*.md
+docs/release/webification-plan.md
+
+# Tool caches / scratch
+.mypy_cache/
+.pytest_cache/
+.ruff_cache/
+.a1/

kiwime_compiler-0.1.0/PKG-INFO

@@ -0,0 +1,19 @@
+Metadata-Version: 2.4
+Name: kiwime-compiler
+Version: 0.1.0
+Summary: kiwiMe compiler sidecar — LLM and embedding-driven compilation.
+Project-URL: Homepage, https://github.com/kiwiberry-ai/kiwime
+License-Expression: Apache-2.0
+Requires-Python: >=3.11
+Requires-Dist: boto3>=1.35
+Requires-Dist: httpx>=0.27
+Requires-Dist: kiwime-sdk==0.1.0
+Requires-Dist: kiwime-secret==0.1.0
+Requires-Dist: numpy>=2
+Requires-Dist: scikit-learn>=1.5
+Requires-Dist: sqlite-vec>=0.1.9
+Provides-Extra: dev
+Requires-Dist: mypy>=1.10; extra == 'dev'
+Requires-Dist: pytest-asyncio>=0.23; extra == 'dev'
+Requires-Dist: pytest-httpx>=0.30; extra == 'dev'
+Requires-Dist: pytest>=8.0; extra == 'dev'

kiwime_compiler-0.1.0/README.md

@@ -0,0 +1,92 @@
+# workers/compiler
+
+The Python sidecar that runs LLM- and embedding-driven compilation for kiwiMe. It reads ingested `Document` and `ActivityEvent` rows from `workers/store`, calls a generator (Bedrock Claude) and an embedder (Ollama `nomic-embed-text`), and writes back the compilation-layer objects defined in `docs/schema/core.md`: memories, capabilities, topics, cases, graph nodes/edges.
+
+## Pipeline
+
+`pipeline.run(...)` runs seven tasks in order, serially:
+
+1. **identity** — single Identity Memory from top-signal documents (pinned READMEs, "about me" pages).
+2. **topic** — k-means clustering over document embeddings, LLM-labelled.
+3. **semantic** — per-topic stable claims about capabilities and methodologies.
+4. **episodic** — last 90 days of activity events, bucketed by source/kind.
+5. **capability** — per-topic capability hints with `recency` derived from supporting-doc timestamps.
+6. **case** — `pr_merged + linked_issue + outcome_doc` triples drafted into Case cards.
+7. **graph** — projects everything into `GraphNode` / `GraphEdge` with kinds `expresses`, `demonstrates`, `derived_from`, `co_occurs_with`.
+
+Every compilation output carries a non-empty `provenance` (or `evidence`) array. Empty provenance is treated as a bug per `docs/schema/core.md`.
+
+## Provider configuration (per-profile)
+
+The compiler reads `provider_config` rows from the store. Three roles are recognised:
+
+| role | used for | required? |
+| --- | --- | --- |
+| `generator` | compile pipeline (memory / topic / capability extraction) | yes |
+| `embedder` | semantic vectors over the corpus | yes |
+| `assistant` | user-facing chat (Constellation Ask), summarization, AI skill drafting | optional |
+
+`generator` and `assistant` are kept distinct so a user can run a small local model for compile (cheap, batch-friendly) while still using Claude / GPT-4 for interactive chat.
+
+```jsonc
+// generator (compile pipeline)
+{ "role": "generator", "provider": "bedrock", "config": {
+    "region": "us-east-1",
+    "model_id": "anthropic.claude-sonnet-4-20250514-v1:0"
+}}
+// embedder (locked to Ollama nomic-embed-text in v1 — UI doesn't expose it)
+{ "role": "embedder", "provider": "ollama", "config": {
+    "endpoint": "http://localhost:11434",
+    "model": "nomic-embed-text"
+}}
+// assistant (user-facing LLM). api_key is a $secret_ref pointing at the OS
+// keychain — the registry resolves it at load time; plaintext keys are never
+// stored in provider_config.
+{ "role": "assistant", "provider": "openai_compat", "config": {
+    "base_url": "https://api.deepseek.com/v1",
+    "api_key": {"$secret_ref": {"namespace": "provider:assistant", "key": "api_key"}},
+    "model": "deepseek-chat"
+}}
+// or:
+{ "role": "assistant", "provider": "anthropic", "config": {
+    "api_key": {"$secret_ref": {"namespace": "provider:assistant", "key": "api_key"}},
+    "model": "claude-sonnet-4-20250514"
+}}
+```
+
+Generator providers
+: `bedrock`, `ollama`, `openai_compat` (OpenAI / DeepSeek / Together / Groq / vLLM / any chat-completions-compatible endpoint), `anthropic` (native Messages API). Both `generator` and `assistant` roles accept any of these.
+
+There are **no defaults** for Bedrock — missing `region` or `model_id` raises immediately. AWS credentials are not read by the compiler; the host fetches them via `secret.aws_credentials` (env → `~/.aws` → keychain) and injects them at registry-load time. `openai_compat` and `anthropic` likewise refuse to start without an explicit `model` and `api_key`; the UI must collect them.
+
+## Prompt design notes
+
+- **Capability tag.** Every prompt is currently sized for a Sonnet-4-class generator. Each call site is implicitly tagged `needs_strong_reasoning`. Local 7B-class models will need different prompts (shorter context, smaller output, more explicit format scaffolding); when we add an `OllamaGenerator`, we will branch prompts on the active provider's capability tag rather than text-substitute. See ADR 0004.
+- **JSON-coerced output.** Structured output is implemented by injecting a `Respond with JSON matching this schema` preamble plus one parse retry on malformed JSON. Empirically ~5% of responses are malformed on first attempt; if both attempts fail we raise `ProviderError` rather than recovering silently.
+- **Provenance discipline.** Tasks drop any LLM-suggested object whose `supporting_doc_ids` are empty or all unknown. The compiler will not invent text without source.
+- **Capability mismatch.** If a configured provider can't meet a task's bar (e.g. small local model returning unparseable JSON twice), the task fails loudly with `CapabilityMismatchError`. This is per ADR 0004 — we do not silently degrade.
+
+## Idempotency
+
+IDs for memories, capabilities, topics, cases, nodes, and edges are content-derived (`profile_id` + kind + canonical content hashed to 12 hex chars). Re-running `compile.run` over the same inputs produces the same ids, so the `INSERT OR REPLACE` paths in the store are safe.
+
+## Direct mode (temporary)
+
+The compiler is a sidecar; in the target architecture it talks to `workers/store` through the host. The host doesn't yet route worker→worker calls, so until it does the compiler supports a direct-mode shortcut:
+
+```bash
+kiwime-compiler --direct /path/to/kiwime.db
+# or
+KIWIME_COMPILER_DIRECT_DB=/path/to/kiwime.db kiwime-compiler
+```
+
+In direct mode the compiler opens the SQLite database itself read-write, bypassing the host entirely. This is explicitly tagged as temporary in `store_client.py`; `RoutedStoreClient` is a stub that raises `NotImplementedError` until the host routing lands. Tests run against an in-memory `InMemoryStore` fake and never touch SQLite.
+
+## Tests
+
+```bash
+cd workers/compiler
+python -m pytest
+```
+
+Mocks: `boto3` is mocked via `MagicMock`; Ollama is mocked via `pytest-httpx`; tasks and pipeline use the in-memory `FakeGenerator` / `FakeEmbedder` / `InMemoryStore` from `tests/fakes.py`.

kiwime_compiler-0.1.0/pyproject.toml

@@ -0,0 +1,46 @@
+[project]
+name = "kiwime-compiler"
+version = "0.1.0"
+description = "kiwiMe compiler sidecar — LLM and embedding-driven compilation."
+requires-python = ">=3.11"
+license = "Apache-2.0"
+dependencies = [
+    "kiwime-sdk==0.1.0",
+    # Keychain access for resolving $secret_ref entries in provider_config
+    # (API keys are never stored in SQLite; see registry._keychain_resolver).
+    "kiwime-secret==0.1.0",
+    "boto3>=1.35",
+    "httpx>=0.27",
+    "numpy>=2",
+    "scikit-learn>=1.5",
+    "sqlite-vec>=0.1.9",
+]
+
+[project.optional-dependencies]
+dev = [
+    "pytest>=8.0",
+    "pytest-asyncio>=0.23",
+    "pytest-httpx>=0.30",
+    "mypy>=1.10",
+]
+
+[project.scripts]
+kiwime-compiler = "kiwime_compiler.__main__:main"
+
+[project.urls]
+Homepage = "https://github.com/kiwiberry-ai/kiwime"
+
+[build-system]
+requires = ["hatchling"]
+build-backend = "hatchling.build"
+
+[tool.hatch.build.targets.wheel]
+packages = ["src/kiwime_compiler"]
+
+[tool.uv.sources]
+kiwime-sdk = { workspace = true }
+kiwime-secret = { workspace = true }
+
+[tool.pytest.ini_options]
+asyncio_mode = "auto"
+testpaths = ["tests"]

kiwime_compiler-0.1.0/src/kiwime_compiler/__init__.py

@@ -0,0 +1,8 @@
+"""kiwiMe compiler sidecar.
+
+Compiles ingested Documents and ActivityEvents into Memory / Capability /
+Topic / Case / GraphNode / GraphEdge objects. See docs/architecture/ipc-protocol.md
+section 4 and docs/schema/core.md.
+"""
+
+__version__ = "0.0.0"

kiwime_compiler-0.1.0/src/kiwime_compiler/__main__.py

@@ -0,0 +1,57 @@
+"""Sidecar entry point.
+
+Two run modes:
+
+    --direct PATH       Open the SQLite DB at PATH read-write. Temporary
+                         shortcut while the host doesn't yet route
+                         worker→worker calls. See store_client.py.
+
+    (no flag)           Routed mode — talk to the host. Not implemented yet.
+
+Always reads JSON-RPC frames on stdin and writes them to stdout. Logs go
+to stderr.
+"""
+
+from __future__ import annotations
+
+import argparse
+import os
+import sys
+
+from .providers.registry import ProviderRegistry
+from .server import CompilerServer
+from .store_client import DirectStoreClient, RoutedStoreClient
+
+
+def main() -> int:
+    p = argparse.ArgumentParser(prog="kiwime-compiler")
+    p.add_argument(
+        "--direct",
+        metavar="DB_PATH",
+        default=os.environ.get("KIWIME_COMPILER_DIRECT_DB"),
+        help="Open the SQLite DB directly. Temporary shortcut.",
+    )
+    p.add_argument("--profile", help="Profile id to compile for. Defaults to first in DB.")
+    args = p.parse_args()
+
+    if args.direct:
+        store = DirectStoreClient(args.direct)
+        profile_id = args.profile or store.query_default_profile()
+    else:
+        # When the host can route, this becomes the default path.
+        store = RoutedStoreClient()  # raises NotImplementedError
+        profile_id = args.profile or ""
+
+    providers = ProviderRegistry()
+    try:
+        providers.load(store, profile_id)  # type: ignore[arg-type]
+    except Exception as e:
+        print(f"warning: provider registry not loaded: {e}", file=sys.stderr)
+
+    server = CompilerServer(store=store, providers=providers, profile_id=profile_id)
+    server.serve()
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

kiwime_compiler-0.1.0/src/kiwime_compiler/context.py

@@ -0,0 +1,30 @@
+"""Pipeline Context — the per-run handle passed to every task."""
+
+from __future__ import annotations
+
+from collections.abc import Callable
+from dataclasses import dataclass, field
+from typing import Any
+
+from .providers.registry import ProviderRegistry
+from .store_client import StoreClient
+
+
+@dataclass
+class Context:
+    profile_id: str
+    store: StoreClient
+    providers: ProviderRegistry
+    documents: list[dict[str, Any]] = field(default_factory=list)
+    events: list[dict[str, Any]] = field(default_factory=list)
+    # Cached cross-task outputs.
+    topics: list[dict[str, Any]] = field(default_factory=list)
+    capabilities: list[dict[str, Any]] = field(default_factory=list)
+    cases: list[dict[str, Any]] = field(default_factory=list)
+    memories: list[dict[str, Any]] = field(default_factory=list)
+    # Document embeddings cache, keyed by document id.
+    doc_embeddings: dict[str, list[float]] = field(default_factory=dict)
+    # Progress reporter; the pipeline injects a real one, tasks call it freely.
+    progress: Callable[[str, int, int, str | None], None] = field(
+        default=lambda phase, done, total, msg=None: None
+    )

kiwime_compiler-0.1.0/src/kiwime_compiler/ids.py

@@ -0,0 +1,43 @@
+"""Deterministic content-derived ids.
+
+See docs/schema/core.md "Common conventions / IDs". Every compiled object's
+id is derived from ``(profile_id, kind-specific salt, content-hash)`` so that
+re-running compile produces the same ids — that is what makes
+``compile.run`` idempotent.
+"""
+
+from __future__ import annotations
+
+import hashlib
+
+
+def _h(*parts: str, n: int = 12) -> str:
+    h = hashlib.sha256()
+    for p in parts:
+        h.update(p.encode("utf-8"))
+        h.update(b"\x1f")
+    return h.hexdigest()[:n]
+
+
+def memory_id(profile_id: str, kind: str, content: str) -> str:
+    return f"mem_{kind}_{_h(profile_id, kind, content)}"
+
+
+def capability_id(profile_id: str, name: str) -> str:
+    return f"cap_{_h(profile_id, name.lower())}"
+
+
+def topic_id(profile_id: str, label: str) -> str:
+    return f"topic_{_h(profile_id, label.lower())}"
+
+
+def case_id(profile_id: str, title: str) -> str:
+    return f"case_{_h(profile_id, title.lower())}"
+
+
+def node_id(profile_id: str, kind: str, ref: str) -> str:
+    return f"node_{kind}_{_h(profile_id, kind, ref)}"
+
+
+def edge_id(profile_id: str, from_node: str, to_node: str, kind: str) -> str:
+    return f"edge_{_h(profile_id, from_node, to_node, kind)}"

kiwime_compiler-0.1.0/src/kiwime_compiler/pipeline.py

@@ -0,0 +1,91 @@
+"""Compilation pipeline orchestrator.
+
+Serial execution (first version). Order matters:
+
+    1. topic        — embeds + clusters documents; downstream tasks reuse
+                      ``ctx.doc_embeddings``
+    2. identity     — picks centroid-closest docs from those embeddings and
+                      synthesizes one identity statement
+    3. semantic     — per-topic
+    4. episodic     — independent of topics, driven by activity events
+    5. capability   — per-topic
+    6. case         — uses capabilities for cross-link
+    7. graph        — projects everything into nodes/edges
+
+Identity used to run first based on filename heuristics, but it now needs
+the embedding cache that ``topic`` populates, so it moved.
+
+The pipeline emits ``compile:progress`` notifications via the supplied
+callback. Callers (the JSON-RPC server) wire those into the host.
+"""
+
+from __future__ import annotations
+
+from collections.abc import Callable
+from typing import Any
+
+from .context import Context
+from .providers.registry import ProviderRegistry
+from .store_client import StoreClient
+from .tasks import capability as t_capability
+from .tasks import case as t_case
+from .tasks import episodic as t_episodic
+from .tasks import graph as t_graph
+from .tasks import identity as t_identity
+from .tasks import semantic as t_semantic
+from .tasks import topic as t_topic
+
+ProgressFn = Callable[[str, int, int, str | None], None]
+
+_TASK_ORDER = [
+    ("topic", t_topic.run),
+    ("identity", t_identity.run),
+    ("semantic", t_semantic.run),
+    ("episodic", t_episodic.run),
+    ("capability", t_capability.run),
+    ("case", t_case.run),
+    ("graph", t_graph.run),
+]
+
+
+def _noop_progress(phase: str, done: int, total: int, msg: str | None = None) -> None:
+    pass
+
+
+def run(
+    *,
+    profile_id: str,
+    store: StoreClient,
+    providers: ProviderRegistry,
+    since: str | None = None,
+    progress: ProgressFn | None = None,
+) -> dict[str, int]:
+    """Run the seven tasks in order; return summary counts."""
+    progress = progress or _noop_progress
+
+    pending = store.list_pending_for_compile(since=since)
+    ctx = Context(
+        profile_id=profile_id,
+        store=store,
+        providers=providers,
+        documents=pending["documents"],
+        events=pending["events"],
+        progress=progress,
+    )
+
+    progress("start", 0, len(_TASK_ORDER), f"docs={len(ctx.documents)} events={len(ctx.events)}")
+
+    results: dict[str, Any] = {}
+    for i, (name, fn) in enumerate(_TASK_ORDER):
+        progress(name, i, len(_TASK_ORDER), "starting")
+        results[name] = fn(ctx)
+        progress(name, i + 1, len(_TASK_ORDER), "done")
+
+    return {
+        "memories": len(ctx.memories),
+        "capabilities": len(ctx.capabilities),
+        "topics": len(ctx.topics),
+        "cases": len(ctx.cases),
+        "edges": int(results["graph"].get("edge_count", 0)) if results.get("graph") else 0,
+        "nodes": int(results["graph"].get("node_count", 0)) if results.get("graph") else 0,
+    }

kiwime_compiler-0.1.0/src/kiwime_compiler/providers/__init__.py

@@ -0,0 +1 @@
+"""Provider layer — Generator / Embedder / Reranker (see ADR 0004)."""

kiwime_compiler-0.1.0/src/kiwime_compiler/providers/anthropic.py

@@ -0,0 +1,189 @@
+"""Anthropic Messages API generator.
+
+Why a dedicated class instead of folding it into ``openai_compat``
+    Anthropic's ``/v1/messages`` is *not* OpenAI-compatible: ``system``
+    is a top-level field (not a message), ``max_tokens`` is required,
+    response shape is ``{content: [{type:"text", text:"..."}], ...}``,
+    auth is ``x-api-key`` instead of bearer, and a separate
+    ``anthropic-version`` header is required. Forcing it into the
+    chat-completions schema would just produce a maze of ifs.
+
+JSON coercion
+    Anthropic doesn't expose a JSON-mode toggle (yet). We emulate the
+    same recipe as the other providers: schema preamble in ``system``,
+    parse, retry once on failure with an explicit error turn.
+"""
+
+from __future__ import annotations
+
+import json
+from typing import Any
+
+import httpx
+
+from .base import ProviderError
+
+_DEFAULT_BASE_URL = "https://api.anthropic.com"
+_DEFAULT_VERSION = "2023-06-01"
+
+
+class AnthropicGenerator:
+    def __init__(
+        self,
+        *,
+        api_key: str,
+        model: str,
+        base_url: str = _DEFAULT_BASE_URL,
+        anthropic_version: str = _DEFAULT_VERSION,
+        timeout: float = 120.0,
+        extra_headers: dict[str, str] | None = None,
+    ) -> None:
+        if not model:
+            raise ProviderError(
+                "Anthropic generator requires a model id.", kind="permanent"
+            )
+        self.api_key = api_key
+        self.model = model
+        self.base_url = base_url.rstrip("/")
+        self.anthropic_version = anthropic_version
+        self.timeout = timeout
+        self.extra_headers = dict(extra_headers or {})
+
+    # ------------------------------------------------------------------ Generator
+    def generate(
+        self,
+        prompt: str,
+        system: str | None = None,
+        *,
+        max_tokens: int = 1024,
+        temperature: float = 0.2,
+        json_schema: dict[str, Any] | None = None,
+    ) -> str:
+        effective_system = system or ""
+        if json_schema is not None:
+            schema_preamble = (
+                "Respond with JSON matching this schema. Output JSON only, "
+                "no prose, no markdown fences.\n\nSchema:\n"
+                + json.dumps(json_schema, ensure_ascii=False)
+            )
+            effective_system = (
+                effective_system + "\n\n" + schema_preamble
+                if effective_system
+                else schema_preamble
+            )
+
+        messages: list[dict[str, str]] = [{"role": "user", "content": prompt}]
+        text = self._messages(
+            system=effective_system,
+            messages=messages,
+            temperature=temperature,
+            max_tokens=max_tokens,
+        )
+
+        if json_schema is None:
+            return text
+
+        stripped = _strip_fences(text)
+        try:
+            json.loads(stripped)
+            return stripped
+        except json.JSONDecodeError as first_err:
+            retry_messages = [
+                *messages,
+                {"role": "assistant", "content": text},
+                {
+                    "role": "user",
+                    "content": (
+                        f"Your last response was not valid JSON, here is the "
+                        f"error: {first_err}. Output ONLY a JSON object "
+                        f"matching the schema, with no other text."
+                    ),
+                },
+            ]
+            text2 = self._messages(
+                system=effective_system,
+                messages=retry_messages,
+                temperature=temperature,
+                max_tokens=max_tokens,
+            )
+            stripped2 = _strip_fences(text2)
+            try:
+                json.loads(stripped2)
+            except json.JSONDecodeError as second_err:
+                raise ProviderError(
+                    f"Anthropic JSON coercion failed after retry: {second_err}",
+                    kind="permanent",
+                ) from second_err
+            return stripped2
+
+    # ------------------------------------------------------------------ internals
+    def _messages(
+        self,
+        *,
+        system: str,
+        messages: list[dict[str, str]],
+        temperature: float,
+        max_tokens: int,
+    ) -> str:
+        url = f"{self.base_url}/v1/messages"
+        payload: dict[str, Any] = {
+            "model": self.model,
+            "messages": messages,
+            "max_tokens": int(max_tokens),
+            "temperature": float(temperature),
+        }
+        if system:
+            payload["system"] = system
+
+        headers = {
+            "x-api-key": self.api_key,
+            "anthropic-version": self.anthropic_version,
+            "Content-Type": "application/json",
+            **self.extra_headers,
+        }
+
+        try:
+            with httpx.Client(timeout=self.timeout) as client:
+                r = client.post(url, json=payload, headers=headers)
+        except httpx.HTTPError as e:
+            raise ProviderError(
+                f"Anthropic messages call failed: {e}", kind="transient"
+            ) from e
+
+        if r.status_code >= 400:
+            raise ProviderError(
+                f"Anthropic messages returned {r.status_code}: {r.text[:200]}",
+                kind="transient" if r.status_code >= 500 else "permanent",
+            )
+        try:
+            data = r.json()
+        except ValueError as e:
+            raise ProviderError(
+                f"Anthropic messages returned non-JSON envelope: {e}"
+            ) from e
+
+        # ``content`` is a list of blocks; we concatenate every text block.
+        # Tool-use blocks (rare in our flow) get skipped — generators only
+        # need the text part.
+        content = data.get("content") or []
+        parts: list[str] = []
+        for block in content:
+            if isinstance(block, dict) and block.get("type") == "text":
+                t = block.get("text")
+                if isinstance(t, str):
+                    parts.append(t)
+        if not parts:
+            raise ProviderError(
+                f"Anthropic messages returned no text content: {data!r}"
+            )
+        return "".join(parts)
+
+
+def _strip_fences(text: str) -> str:
+    s = text.strip()
+    if s.startswith("```"):
+        s = s.split("\n", 1)[1] if "\n" in s else s[3:]
+        if s.endswith("```"):
+            s = s[:-3]
+        s = s.strip()
+    return s