codemap-semantic-index 0.1.0__tar.gz

codemap_semantic_index-0.1.0/.gitignore

@@ -0,0 +1,43 @@
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+*$py.class
+
+# Build artifacts
+build/
+dist/
+*.egg-info/
+*.egg
+.eggs/
+
+# Test / coverage
+.pytest_cache/
+.coverage
+.coverage.*
+htmlcov/
+coverage.xml
+.tox/
+.mypy_cache/
+.ruff_cache/
+.benchmarks/
+
+# Virtualenv
+.venv/
+venv/
+env/
+
+# uv / pdm lockfiles (commit uv.lock once we settle)
+# uv.lock
+
+# IDE
+.idea/
+.vscode/
+*.swp
+*.swo
+
+# OS
+.DS_Store
+Thumbs.db
+
+# CodeMap own index when dogfooding
+.codemap/

codemap_semantic_index-0.1.0/CHANGELOG.md

@@ -0,0 +1,86 @@
+# Changelog — codemap-semantic-index
+
+This plugin's version is **independent** of `codemap-core` lockstep —
+it's an opt-in semantic ranker, not part of the L1 indexing core.
+
+## 0.1.0 (2026-06-27)
+
+First release. Closes AI-EDS roadmap **P1-3**.
+
+### What it does
+
+Registers an embedding-based ranker into `codemap recall` via the
+`codemap.recall_hooks` entry-point group (introduced in `codemap-core`
+0.4.1). `codemap-aimemory` RRF-fuses our ranking with its token
+ranking and multiplies by freshness (P4-2), so installing this plugin
+upgrades recall from token-only to hybrid semantic + token + freshness
+with zero user code change.
+
+### Modules
+
+- `chunker` — markdown → chunks. Splits on `##` headings; over-long
+  sections re-split with sliding windows (500 tokens / 50 overlap).
+  Every chunk text is prefixed `"<knowledge_id> / <h2_title>\n\n..."`
+  so embeddings have an anchor to the source doc.
+- `store` — atomic on-disk store under `.ai-memory/_semantic/`:
+  `chunks.json` (model-independent metadata) + `vectors.npy`
+  (model-specific 1024-dim float32) + `model_id.txt` (active
+  backend fingerprint) + `manifest.json` (text_hash → chunk_id for
+  incremental).
+- `config` — `~/.config/codemap/embedding.yaml` reader/writer; chmod 600.
+- `embedding/local.py` — sentence-transformers wrapper, default
+  `Qwen/Qwen3-Embedding-0.6B` (1024-dim, 32k context, same-source as
+  Qwen cloud text-embedding-v3). Lazy-imports `sentence_transformers`
+  so plain `--help` doesn't pay torch boot cost.
+- `embedding/openai_compat.py` — `POST {base_url}/embeddings` over
+  httpx. Handles 4 preset providers (Qwen / OpenAI / Zhipu / Voyage)
+  + custom (self-hosted vLLM / Ollama / TEI / Jina).
+- `indexer` — `rebuild_index` (full) + `incremental_index` (hash-diff;
+  only re-encode chunks whose text changed); refuses on model mismatch.
+- `recall_hook` — entry-point function. Loads the on-disk store, encodes
+  the query, computes cosine similarities (vectors are L2-normalised so
+  dot-product), aggregates chunks → knowledge_id (best chunk wins),
+  returns hook-contract-shaped candidates with freshness already
+  computed. Failure modes (no store / model mismatch / network down)
+  all silently return `[]` so recall never crashes.
+
+### CLI — `codemap embed`
+
+11 sub-commands organised in two groups:
+
+- `codemap embed [--rebuild | --incremental | --dry-run | --project P]`
+  — main embed pipeline; default is incremental.
+- `codemap embed install [<model_id>]` — interactive picker (3 preset
+  candidates + custom) or direct install.
+- `codemap embed list` — show locally downloaded HF models, mark active.
+- `codemap embed use <model_id>` — switch active local model; prints
+  rebuild hint.
+- `codemap embed backend set [--provider P --api-key K --base-url U
+  --model M --dimensions N]` — configure local or cloud backend.
+  Interactive picker when no `--provider`. Auto-fills base_url / model /
+  dimensions from preset.
+- `codemap embed backend show` — print effective config (api key masked).
+- `codemap embed backend reset` — back to local defaults.
+- `codemap embed backend path` — print config file location.
+
+### Dependencies
+
+- `codemap-core>=0.4.1` (entry-point group)
+- `codemap-aimemory>=0.4.1` (freshness + recall infrastructure reused)
+- `numpy>=1.24` (vector math)
+- `httpx>=0.27` (cloud backend HTTP)
+- `pyyaml>=6.0`, `typer>=0.12`
+- `sentence-transformers>=3.0` — default install includes this (pulls
+  torch ~200MB) so `codemap embed install` works out of the box
+
+### Tests
+
+66 unit tests covering chunker (10) + store (12) + config (9) +
+openai_compat backend (8) + indexer (8) + recall_hook (8) + cli (11).
+All deterministic — no network, no real embedding model download in
+tests (uses a hashing fake backend that produces stable 4-dim unit
+vectors).
+
+### Design doc
+
+`Obsidian/Notes/07-Ideas/AI-Enterprise-Delivery-System/2026-06-27-p1-3-codemap-semantic-index-设计方案.md`

codemap_semantic_index-0.1.0/PKG-INFO

@@ -0,0 +1,76 @@
+Metadata-Version: 2.4
+Name: codemap-semantic-index
+Version: 0.1.0
+Summary: Embedding-based semantic recall hook for CodeMap — registers into `codemap recall` via the recall_hooks entry-point group and adds vector search over .ai-memory/knowledge/*.yml
+Project-URL: Homepage, https://github.com/qxbyte/codemap
+Author: CodeMap Contributors
+License: MIT
+Keywords: ai-memory,codemap,embedding,rag,semantic-search
+Classifier: Development Status :: 3 - Alpha
+Classifier: Programming Language :: Python :: 3
+Classifier: Topic :: Software Development
+Requires-Python: >=3.11
+Requires-Dist: codemap-aimemory>=0.4.1
+Requires-Dist: codemap-core>=0.4.1
+Requires-Dist: httpx>=0.27
+Requires-Dist: numpy>=1.24
+Requires-Dist: pyyaml>=6.0
+Requires-Dist: sentence-transformers>=3.0
+Requires-Dist: typer>=0.12
+Provides-Extra: dev
+Requires-Dist: pytest-cov>=6.0; extra == 'dev'
+Requires-Dist: pytest>=8.0; extra == 'dev'
+Description-Content-Type: text/markdown
+
+# codemap-semantic-index
+
+Embedding-based semantic recall plugin for [codemap](https://github.com/qxbyte/codemap).
+
+Registers an embedding ranker into `codemap recall` via the `codemap.recall_hooks` entry-point group (introduced in `codemap-core` 0.4.1). `codemap-aimemory` automatically RRF-fuses the embedding ranking with its token ranking and applies freshness decay (P4-2).
+
+## Install
+
+```bash
+pipx inject codemap codemap-semantic-index
+# pulls sentence-transformers + torch (~200MB)
+
+# Pick + download a local model (1.2GB default)
+codemap embed install                   # interactive picker
+codemap embed install BAAI/bge-m3       # direct
+
+# First embed (writes <project>/.ai-memory/_semantic/)
+codemap embed
+
+# Now `codemap recall` does double-path (token + embedding) + RRF + freshness
+codemap recall '<query>' --with-content
+```
+
+## Default model
+
+`Qwen/Qwen3-Embedding-0.6B` (1024 dim, 32k context, 1.2GB). Same-source training as the Qwen cloud `text-embedding-v3`, so switching to cloud preserves recall "feel".
+
+## Cloud backend (any OpenAI-compatible embedding API)
+
+```bash
+codemap embed backend set    # interactive picker (qwen / openai / zhipu / voyage / custom)
+codemap embed backend show
+codemap embed backend reset  # back to local
+```
+
+Four preset providers + `custom` for self-hosted vLLM / Ollama / TEI. Config persists to `~/.config/codemap/embedding.yaml` (chmod 600).
+
+## Storage
+
+```
+<project_root>/.ai-memory/_semantic/
+├── chunks.json    chunked text + metadata (model-independent)
+├── vectors.npy    (n_chunks, 1024) float32 (model-specific)
+├── model_id.txt   active backend fingerprint
+└── manifest.json  text_hash → chunk_id (drives incremental)
+```
+
+Switching models requires `codemap embed --rebuild` (different vector spaces are not comparable).
+
+## Design doc
+
+`Obsidian/Notes/07-Ideas/AI-Enterprise-Delivery-System/2026-06-27-p1-3-codemap-semantic-index-设计方案.md` — full spec including chunker / RRF / freshness integration.

codemap_semantic_index-0.1.0/README.md

@@ -0,0 +1,52 @@
+# codemap-semantic-index
+
+Embedding-based semantic recall plugin for [codemap](https://github.com/qxbyte/codemap).
+
+Registers an embedding ranker into `codemap recall` via the `codemap.recall_hooks` entry-point group (introduced in `codemap-core` 0.4.1). `codemap-aimemory` automatically RRF-fuses the embedding ranking with its token ranking and applies freshness decay (P4-2).
+
+## Install
+
+```bash
+pipx inject codemap codemap-semantic-index
+# pulls sentence-transformers + torch (~200MB)
+
+# Pick + download a local model (1.2GB default)
+codemap embed install                   # interactive picker
+codemap embed install BAAI/bge-m3       # direct
+
+# First embed (writes <project>/.ai-memory/_semantic/)
+codemap embed
+
+# Now `codemap recall` does double-path (token + embedding) + RRF + freshness
+codemap recall '<query>' --with-content
+```
+
+## Default model
+
+`Qwen/Qwen3-Embedding-0.6B` (1024 dim, 32k context, 1.2GB). Same-source training as the Qwen cloud `text-embedding-v3`, so switching to cloud preserves recall "feel".
+
+## Cloud backend (any OpenAI-compatible embedding API)
+
+```bash
+codemap embed backend set    # interactive picker (qwen / openai / zhipu / voyage / custom)
+codemap embed backend show
+codemap embed backend reset  # back to local
+```
+
+Four preset providers + `custom` for self-hosted vLLM / Ollama / TEI. Config persists to `~/.config/codemap/embedding.yaml` (chmod 600).
+
+## Storage
+
+```
+<project_root>/.ai-memory/_semantic/
+├── chunks.json    chunked text + metadata (model-independent)
+├── vectors.npy    (n_chunks, 1024) float32 (model-specific)
+├── model_id.txt   active backend fingerprint
+└── manifest.json  text_hash → chunk_id (drives incremental)
+```
+
+Switching models requires `codemap embed --rebuild` (different vector spaces are not comparable).
+
+## Design doc
+
+`Obsidian/Notes/07-Ideas/AI-Enterprise-Delivery-System/2026-06-27-p1-3-codemap-semantic-index-设计方案.md` — full spec including chunker / RRF / freshness integration.

codemap_semantic_index-0.1.0/pyproject.toml

@@ -0,0 +1,46 @@
+[build-system]
+requires = ["hatchling>=1.21"]
+build-backend = "hatchling.build"
+
+[project]
+name = "codemap-semantic-index"
+version = "0.1.0"
+description = "Embedding-based semantic recall hook for CodeMap — registers into `codemap recall` via the recall_hooks entry-point group and adds vector search over .ai-memory/knowledge/*.yml"
+readme = "README.md"
+requires-python = ">=3.11"
+license = { text = "MIT" }
+authors = [{ name = "CodeMap Contributors" }]
+keywords = ["codemap", "embedding", "semantic-search", "ai-memory", "rag"]
+classifiers = [
+    "Development Status :: 3 - Alpha",
+    "Programming Language :: Python :: 3",
+    "Topic :: Software Development",
+]
+dependencies = [
+    "codemap-core>=0.4.1",
+    "codemap-aimemory>=0.4.1",
+    "numpy>=1.24",
+    "httpx>=0.27",
+    "pyyaml>=6.0",
+    "typer>=0.12",
+    # sentence-transformers (and torch) is the default local backend; pulled
+    # by default so `codemap embed install` works out of the box. Users who
+    # only want the cloud backend can drop torch with --no-deps gymnastics
+    # at their own risk.
+    "sentence-transformers>=3.0",
+]
+
+[project.optional-dependencies]
+dev = ["pytest>=8.0", "pytest-cov>=6.0"]
+
+[project.entry-points."codemap.cli_commands"]
+embed = "codemap_semantic_index.cli:register"
+
+[project.entry-points."codemap.recall_hooks"]
+semantic = "codemap_semantic_index.recall_hook:rank"
+
+[project.urls]
+Homepage = "https://github.com/qxbyte/codemap"
+
+[tool.hatch.build.targets.wheel]
+packages = ["src/codemap_semantic_index"]

codemap_semantic_index-0.1.0/src/codemap_semantic_index/__init__.py

@@ -0,0 +1,13 @@
+"""codemap-semantic-index — embedding-based semantic recall for codemap.
+
+Registers via two entry-point groups:
+
+* ``codemap.cli_commands.embed`` → :func:`codemap_semantic_index.cli.register`
+  adds the ``codemap embed`` subcommand tree.
+* ``codemap.recall_hooks.semantic`` → :func:`codemap_semantic_index.
+  recall_hook.rank` plugs an embedding-based ranker into ``codemap recall``;
+  ``codemap-aimemory>=0.4.1`` discovers it automatically and RRF-fuses
+  the result with token recall + freshness.
+"""
+
+__version__ = "0.1.0"

codemap_semantic_index-0.1.0/src/codemap_semantic_index/chunker.py

@@ -0,0 +1,283 @@
+"""Markdown → chunks for the semantic index.
+
+Source: ``<project_root>/knowledge-base/{rules,business,modules,cases,
+pitfalls}/*.md`` (written by ``specode-distill`` 3.0+ and ``task-swarm``
+0.6+ — see specode-distill's ``references/doc-template.md`` for the
+human-readable templates these files follow).
+
+Algorithm (regex-only — no markdown lib so the chunker stays a
+dependency-free wheel of its own):
+
+1. Strip YAML frontmatter (``---`` ... ``---``)
+2. Read the H1 (``# ...``) as the document title
+3. Split the body on ``^## `` headings; each section = ``(h2_title, body)``
+4. Body sections whose token count exceeds ``MAX_TOKENS`` are split with
+   a sliding window (``WINDOW_TOKENS`` / ``WINDOW_OVERLAP``)
+5. Each emitted chunk's text is prefixed with the title path
+   ``"<knowledge_id> / <h2_title>\\n\\n<body>"`` so embedding models
+   anchor on the right doc even when the body is a generic snippet.
+
+Token counting is approximate: 1 token ≈ 4 characters for English /
+2 characters for Chinese. The whole pipeline tolerates being slightly
+off — a longer chunk gets one extra sliding-window slice; nothing
+breaks."""
+
+from __future__ import annotations
+
+import hashlib
+import re
+from collections.abc import Iterator
+from dataclasses import dataclass
+from pathlib import Path
+
+__all__ = [
+    "MAX_TOKENS",
+    "WINDOW_OVERLAP",
+    "WINDOW_TOKENS",
+    "Chunk",
+    "approx_token_count",
+    "chunk_knowledge_base",
+    "chunk_markdown",
+]
+
+#: Body sections longer than this get split into sliding windows.
+MAX_TOKENS = 1000
+#: Sliding window size when splitting an over-long section.
+WINDOW_TOKENS = 500
+#: Token overlap between adjacent windows (preserves boundary context).
+WINDOW_OVERLAP = 50
+
+#: Categories under ``knowledge-base/`` recognised by spec-distill v3.
+KNOWLEDGE_CATEGORIES: tuple[str, ...] = (
+    "rules",
+    "business",
+    "modules",
+    "cases",
+    "pitfalls",
+)
+
+
+@dataclass
+class Chunk:
+    """One unit of text fed to the embedding model.
+
+    ``chunk_id`` is stable across re-runs (knowledge_id + h2 slug +
+    optional window index) so incremental embedding can hash-compare and
+    only re-encode the chunks whose ``text`` changed.
+    """
+
+    chunk_id: str
+    knowledge_id: str
+    category: str  # rules / business / modules / cases / pitfalls
+    title: str  # the H1 of the md doc
+    h2_title: str  # the H2 of the section this chunk came from
+    text: str  # prefixed text fed to the embedder
+    source_md: str  # path relative to project_root
+    source_yml: str  # twin yml path under .ai-memory/knowledge/
+    text_hash: str  # sha1 of text — incremental diff key
+
+    def to_dict(self) -> dict[str, str]:
+        return {
+            "chunk_id": self.chunk_id,
+            "knowledge_id": self.knowledge_id,
+            "category": self.category,
+            "title": self.title,
+            "h2_title": self.h2_title,
+            "text": self.text,
+            "source_md": self.source_md,
+            "source_yml": self.source_yml,
+            "text_hash": self.text_hash,
+        }
+
+
+# ---------- core algorithm ----------
+
+
+_FRONTMATTER_RE = re.compile(r"\A---\s*\n.*?\n---\s*\n", re.DOTALL)
+_H1_RE = re.compile(r"^#\s+(.+?)\s*$", re.MULTILINE)
+# A heading line is "## ..." OR "### ..." (we split at the same depth as
+# H2 only; H3 stays inside its parent section).
+_H2_SPLIT_RE = re.compile(r"^##\s+(.+?)\s*$", re.MULTILINE)
+
+
+def _strip_frontmatter(text: str) -> str:
+    return _FRONTMATTER_RE.sub("", text, count=1)
+
+
+def _extract_h1(text: str) -> str:
+    m = _H1_RE.search(text)
+    return m.group(1).strip() if m else ""
+
+
+def _split_h2_sections(body: str) -> list[tuple[str, str]]:
+    """Return ``[(h2_title, section_body), ...]``. Content before the
+    first H2 lands as ``("", preamble)``; sections without a body are
+    dropped."""
+    # Find all H2 positions; iterate to build slices.
+    matches = list(_H2_SPLIT_RE.finditer(body))
+    if not matches:
+        stripped = body.strip()
+        return [("", stripped)] if stripped else []
+
+    out: list[tuple[str, str]] = []
+    # Preamble (text before first H2).
+    preamble = body[: matches[0].start()].strip()
+    if preamble:
+        out.append(("", preamble))
+    for i, m in enumerate(matches):
+        end = matches[i + 1].start() if i + 1 < len(matches) else len(body)
+        section_body = body[m.end() : end].strip()
+        if section_body:
+            out.append((m.group(1).strip(), section_body))
+    return out
+
+
+def approx_token_count(text: str) -> int:
+    """Conservative ≈ token estimator: 1 token per 2 CJK chars,
+    1 per 4 ASCII chars. Off by ~20% vs real BPE but consistent."""
+    cjk = sum(1 for ch in text if "一" <= ch <= "鿿")
+    other = len(text) - cjk
+    return max(1, cjk // 2 + other // 4)
+
+
+def _sliding_split(text: str) -> Iterator[str]:
+    """Split an over-long section into windows of ~WINDOW_TOKENS each.
+
+    Uses character indices proportional to the token estimator above so a
+    pure-CJK section yields 2x as many chars per window as a pure-ASCII
+    one (the inverse of the token math)."""
+    cjk_ratio = sum(1 for ch in text if "一" <= ch <= "鿿") / max(1, len(text))
+    chars_per_token = 2 if cjk_ratio > 0.5 else 4
+    window_chars = WINDOW_TOKENS * chars_per_token
+    overlap_chars = WINDOW_OVERLAP * chars_per_token
+    step = max(1, window_chars - overlap_chars)
+    i = 0
+    while i < len(text):
+        yield text[i : i + window_chars]
+        if i + window_chars >= len(text):
+            return
+        i += step
+
+
+def chunk_markdown(
+    md_text: str,
+    *,
+    knowledge_id: str,
+    category: str,
+    source_md: str,
+    source_yml: str,
+) -> list[Chunk]:
+    """Turn one md document into a list of :class:`Chunk` ready for
+    embedding."""
+    stripped = _strip_frontmatter(md_text)
+    title = _extract_h1(stripped)
+    # Remove the H1 line itself before sectioning so the preamble doesn't
+    # carry the heading text twice.
+    if title:
+        stripped = _H1_RE.sub("", stripped, count=1).lstrip("\n")
+    sections = _split_h2_sections(stripped)
+
+    out: list[Chunk] = []
+    for h2_title, section_body in sections:
+        h2_slug = _slug(h2_title) if h2_title else "_preamble"
+        if approx_token_count(section_body) <= MAX_TOKENS:
+            out.append(
+                _build_chunk(
+                    chunk_id=f"{knowledge_id}::{h2_slug}",
+                    knowledge_id=knowledge_id,
+                    category=category,
+                    title=title,
+                    h2_title=h2_title,
+                    section_body=section_body,
+                    source_md=source_md,
+                    source_yml=source_yml,
+                )
+            )
+            continue
+        # Over-long → sliding-window split
+        for w_idx, window in enumerate(_sliding_split(section_body)):
+            out.append(
+                _build_chunk(
+                    chunk_id=f"{knowledge_id}::{h2_slug}::w{w_idx}",
+                    knowledge_id=knowledge_id,
+                    category=category,
+                    title=title,
+                    h2_title=h2_title,
+                    section_body=window,
+                    source_md=source_md,
+                    source_yml=source_yml,
+                )
+            )
+    return out
+
+
+_SLUG_RE = re.compile(r"[^a-z0-9一-鿿]+")
+
+
+def _slug(text: str) -> str:
+    return _SLUG_RE.sub("-", text.lower()).strip("-") or "section"
+
+
+def _build_chunk(
+    *,
+    chunk_id: str,
+    knowledge_id: str,
+    category: str,
+    title: str,
+    h2_title: str,
+    section_body: str,
+    source_md: str,
+    source_yml: str,
+) -> Chunk:
+    # Prefix: title path so the embedding has the "which doc / which
+    # section" anchor even when the body is a generic sentence.
+    prefix = f"{knowledge_id} / {h2_title}" if h2_title else knowledge_id
+    text = f"{prefix}\n\n{section_body}"
+    text_hash = hashlib.sha1(text.encode("utf-8"), usedforsecurity=False).hexdigest()[:16]
+    return Chunk(
+        chunk_id=chunk_id,
+        knowledge_id=knowledge_id,
+        category=category,
+        title=title,
+        h2_title=h2_title,
+        text=text,
+        source_md=source_md,
+        source_yml=source_yml,
+        text_hash=text_hash,
+    )
+
+
+# ---------- knowledge-base traversal ----------
+
+
+def chunk_knowledge_base(project_root: Path) -> list[Chunk]:
+    """Walk ``<project_root>/knowledge-base/{5 categories}/*.md`` and
+    chunk every file. Missing dirs / files are silently tolerated
+    (consistent with the rest of codemap's "missing inputs degrade
+    gracefully" stance)."""
+    kb_root = project_root / "knowledge-base"
+    if not kb_root.is_dir():
+        return []
+    out: list[Chunk] = []
+    for category in KNOWLEDGE_CATEGORIES:
+        cat_dir = kb_root / category
+        if not cat_dir.is_dir():
+            continue
+        for md_file in sorted(cat_dir.glob("*.md")):
+            try:
+                md_text = md_file.read_text(encoding="utf-8")
+            except OSError:
+                continue
+            knowledge_id = md_file.stem
+            source_md = str(md_file.relative_to(project_root))
+            source_yml = f".ai-memory/knowledge/{category}/{knowledge_id}.yml"
+            out.extend(
+                chunk_markdown(
+                    md_text,
+                    knowledge_id=knowledge_id,
+                    category=category,
+                    source_md=source_md,
+                    source_yml=source_yml,
+                )
+            )
+    return out