mempalace-code 1.6.0__tar.gz → 1.6.2__tar.gz

{mempalace_code-1.6.0 → mempalace_code-1.6.2}/CHANGELOG.md

@@ -1,5 +1,28 @@
 # Changelog
 
+## v1.6.2 — 2026-05-01
+
+### Added
+
+- Shared language catalog for miner detection, `code_search` validation, and MCP language hints.
+- `code_search(language=...)` now accepts Kotlin, XML project files, and Perl shebang-detected files, matching mined language labels from the catalog.
+
+### Changed
+
+- The `mempalace_code_search` MCP language description is generated from the same catalog used by search validation, reducing future drift when language support changes.
+- PR #4's scan-exclude proposal is split into backlog item `MINE-APP-SCAN-EXCLUDES-PR4` instead of being merged with the catalog refactor.
+
+## v1.6.1 — 2026-04-30
+
+### Added
+
+- Markdown section metadata in mined drawers: heading, heading level, heading path, document section type, and flags for Mermaid diagrams, fenced code blocks, and Markdown tables.
+- `search_memories` now returns Markdown section context with each result when available.
+
+### Changed
+
+- Markdown prose chunking treats `#` through `######` headings as section boundaries and preserves section metadata through small-section merges and oversized-section splits.
+
 ## v1.6.0 — 2026-04-27
 
 ### Added

{mempalace_code-1.6.0 → mempalace_code-1.6.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: mempalace-code
-Version: 1.6.0
+Version: 1.6.2
 Summary: Developer memory tool — mine codebases and conversations into a LanceDB-backed searchable palace. No API key required.
 Project-URL: Homepage, https://github.com/rergards/mempalace-code
 Project-URL: Repository, https://github.com/rergards/mempalace-code
@@ -69,14 +69,14 @@ No cloud. No API keys. No subscription. Nothing leaves your machine.
 
 <table>
 <tr>
-<td align="center"><strong>Tree-sitter AST Parsing</strong><br><sub>Chunks at function boundaries<br>not arbitrary line counts</sub></td>
+<td align="center"><strong>Language-Aware Mining</strong><br><sub>AST, regex, and adaptive chunking<br>matched to each file type</sub></td>
 <td align="center"><strong>27 MCP Tools</strong><br><sub>Native Claude Code integration<br>search, store, traverse</sub></td>
 <td align="center"><strong>Temporal Knowledge Graph</strong><br><sub>Facts that change over time<br>with validity windows</sub></td>
 </tr>
 <tr>
 <td align="center"><strong>595x Token Savings</strong><br><sub>measured peak · median 80x<br><a href="docs/BENCH_TOKEN_DELTA.md">scales with project size</a></sub></td>
 <td align="center"><strong>Cross-Project Tunnels</strong><br><sub>Search <code>auth</code> in one project<br>find it everywhere</sub></td>
-<td align="center"><strong>1312 Tests · $0 Cost</strong><br><sub>Every feature acceptance-gated<br>fully offline after install</sub></td>
+<td align="center"><strong>1322 Tests · $0 Cost</strong><br><sub>Every feature acceptance-gated<br>fully offline after install</sub></td>
 </tr>
 </table>
 
@@ -150,7 +150,7 @@ You write code. You make decisions. You debug things. Between sessions, all that
 mempalace-code **indexes it once** into a local vector store, then your AI finds it in milliseconds — using [595x fewer tokens](docs/BENCH_TOKEN_DELTA.md) than grep + read at measured peak (median 80x on a 19k-chunk project, and it keeps scaling). Think of it as `git log` for everything that *isn't* in the code: the *why*, the discussions, the dead ends, the decisions.
 
 **What gets indexed:**
-- Code files — functions, classes, modules (Python, TypeScript/JS, Go, Rust, C/C++, C#, F#, VB.NET, XAML, Java, Kotlin, Scala, Swift, Dart, PHP, Markdown) plus Kubernetes manifests
+- Code files — structural chunks for Python, TypeScript/JS/TSX/JSX, Go, Rust, Java, Kotlin, C#, F#, VB.NET, XAML, Swift, PHP, Scala, Dart, Terraform/HCL, Markdown, and Kubernetes manifests; adaptive chunks for C/C++, Ruby, shell, SQL, HTML/CSS, JSON/YAML/TOML, CSV, Dockerfile, Make, templates, and config files
 - .NET solutions — `.sln`/`.csproj` project graphs, cross-project symbol relationships, interface implementations
 - Conversation exports — Claude, ChatGPT, Slack
 - Architecture notes, decisions, anything you store manually
@@ -163,14 +163,14 @@ mempalace-code **indexes it once** into a local vector store, then your AI finds
 
 ### Language-Aware Code Mining
 
-`mempalace mine` walks your source tree and chunks at **structural boundaries** — functions, classes, methods — not arbitrary line counts. Leading comments and docstrings stay attached to their declarations.
+`mempalace mine` walks your source tree and chooses the best chunker for each file type: AST boundaries where optional tree-sitter grammars are available, regex structural boundaries for supported languages, YAML-aware Kubernetes resource splits, Markdown/prose sections, or adaptive line-count chunks for formats without reliable declarations. Leading comments and docstrings stay attached to declarations where structural chunking is active; Markdown drawers keep heading path, section type, and Mermaid/code/table flags in search metadata.
 
 | Language | Strategy | AST Support |
 |----------|----------|:-----------:|
-| Python | Functions, classes, methods, decorators | Tree-sitter |
-| TypeScript / JavaScript / TSX / JSX | Functions, classes, exports, imports | Tree-sitter |
-| Go | Functions, types, methods, interfaces | Tree-sitter |
-| Rust | Functions, structs, enums, traits, impls | Tree-sitter |
+| Python | Functions, classes, methods, decorators | Optional tree-sitter |
+| TypeScript / JavaScript / TSX / JSX | Functions, classes, exports, imports | Optional tree-sitter |
+| Go | Functions, types, methods, interfaces | Optional tree-sitter |
+| Rust | Functions, structs, enums, traits, impls | Optional tree-sitter |
 | Java | Classes, interfaces, methods, annotations | Regex |
 | Kotlin | Classes, objects, functions, extensions | Regex |
 | Scala | Classes, case classes, objects, traits, enums, functions, implicits, type aliases, generics | Regex |
@@ -180,12 +180,20 @@ mempalace-code **indexes it once** into a local vector store, then your AI finds
 | C# | Classes, interfaces, records, methods, properties | Regex |
 | F# / VB.NET | Modules, types, functions | Regex |
 | XAML | Controls, resources, code-behind linking | Regex |
-| C / C++ | Functions, structs, enums, classes | Regex |
-| Kubernetes manifests | Deployments, Services, ConfigMaps, Secrets, Ingresses, CRDs (indexed by kind/namespace/labels) | YAML-aware |
-| Markdown / plain text | Heading sections, paragraphs | — |
-| YAML / JSON / TOML | Adaptive line-count | — |
+| Terraform / HCL | Terraform/HCL top-level blocks (`resource`, `module`, `variable`, `moved`, `import`, `check`, etc.) | Regex |
+| Kubernetes manifests | Deployments, Services, ConfigMaps, Secrets, Ingresses, CRDs (indexed by kind/name) | YAML-aware |
+| Markdown / plain text | Heading sections (`#`-`######`), heading paths, section metadata, paragraphs | — |
+| C / C++ | Indexed and searchable with best-effort symbol metadata; chunked adaptively today | — |
+| Ruby / shell / SQL | Indexed and searchable; chunked adaptively today | — |
+| HTML / CSS / CSV | Indexed and searchable; chunked adaptively today | — |
+| YAML / JSON / TOML | Adaptive line-count; Kubernetes YAML auto-detected separately | — |
+| Dockerfile / Make / templates / config | Dockerfile, Containerfile, Makefile, GNUmakefile, Vagrantfile, Go templates, Jinja2, `.conf`, `.cfg`, `.ini` | — |
 
-Tree-sitter is optional (`pip install "mempalace-code[treesitter]"`). Without it, all languages fall back to regex boundary detection — still structural, just less precise.
+The `mempalace_code_search` language filter is generated from the same language
+catalog as the miner. If a file type is mined with a language label, the MCP
+schema and unsupported-language hints stay aligned with that catalog.
+
+Tree-sitter is optional (`pip install "mempalace-code[treesitter]"`). When a grammar is missing, Python, TypeScript/JavaScript/TSX/JSX, Go, and Rust fall back to regex structural chunking. Other recognized formats use their regex, YAML-aware, prose, or adaptive chunker as listed above.
 
 ```bash
 mempalace mine ~/projects/myapp                  # all supported file types
@@ -307,7 +315,7 @@ claude mcp add mempalace -- python -m mempalace.mcp_server
 | `mempalace_list_wings` | All wings with drawer counts |
 | `mempalace_list_rooms` | Rooms within a wing |
 | `mempalace_get_taxonomy` | Full wing → room → count tree |
-| `mempalace_search` | Semantic search with optional wing/room filters |
+| `mempalace_search` | Semantic search with optional wing/room filters; Markdown hits include heading path and section metadata |
 | `mempalace_code_search` | Filter by language, symbol name/type, file glob |
 | `mempalace_file_context` | All indexed chunks for a source file, ordered by chunk_index |
 | `mempalace_check_duplicate` | Similarity check before filing (0.9 threshold) |
@@ -504,7 +512,7 @@ This is a code-first fork of [milla-jovovich/mempalace](https://github.com/milla
 | No backup, no recovery | `backup` / `restore` / `export` / `import` |
 | No incremental mining | Content-hash incremental: only changed files re-chunked |
 | No code-search | `code_search` — filter by language, symbol, glob |
-| Line-count chunking | Tree-sitter AST + regex structural chunking |
+| Line-count chunking | Language-aware mining: tree-sitter AST for supported grammars, regex structural chunking, YAML-aware Kubernetes splits, prose sections, and adaptive chunks for configs/data |
 
 Full audit: [`docs/UPSTREAM_HARDENING.md`](docs/UPSTREAM_HARDENING.md).
 
@@ -657,9 +665,9 @@ python -m pytest tests/ -x -q    # full suite, all local, no network
 Apache 2.0 — see [LICENSE](LICENSE) and [NOTICE](NOTICE).
 
 <!-- Link Definitions -->
-[version-shield]: https://img.shields.io/badge/version-1.6.0-4dc9f6?style=flat-square&labelColor=0a0e14
+[version-shield]: https://img.shields.io/badge/version-1.6.2-4dc9f6?style=flat-square&labelColor=0a0e14
 [release-link]: https://github.com/rergards/mempalace-code/releases
-[python-shield]: https://img.shields.io/badge/python-3.9+-7dd8f8?style=flat-square&labelColor=0a0e14&logo=python&logoColor=7dd8f8
+[python-shield]: https://img.shields.io/badge/python-3.11+-7dd8f8?style=flat-square&labelColor=0a0e14&logo=python&logoColor=7dd8f8
 [python-link]: https://www.python.org/
 [license-shield]: https://img.shields.io/badge/license-Apache_2.0-b0e8ff?style=flat-square&labelColor=0a0e14
 [license-link]: https://github.com/rergards/mempalace-code/blob/main/LICENSE

{mempalace_code-1.6.0 → mempalace_code-1.6.2}/README.md

@@ -22,14 +22,14 @@ No cloud. No API keys. No subscription. Nothing leaves your machine.
 
 <table>
 <tr>
-<td align="center"><strong>Tree-sitter AST Parsing</strong><br><sub>Chunks at function boundaries<br>not arbitrary line counts</sub></td>
+<td align="center"><strong>Language-Aware Mining</strong><br><sub>AST, regex, and adaptive chunking<br>matched to each file type</sub></td>
 <td align="center"><strong>27 MCP Tools</strong><br><sub>Native Claude Code integration<br>search, store, traverse</sub></td>
 <td align="center"><strong>Temporal Knowledge Graph</strong><br><sub>Facts that change over time<br>with validity windows</sub></td>
 </tr>
 <tr>
 <td align="center"><strong>595x Token Savings</strong><br><sub>measured peak · median 80x<br><a href="docs/BENCH_TOKEN_DELTA.md">scales with project size</a></sub></td>
 <td align="center"><strong>Cross-Project Tunnels</strong><br><sub>Search <code>auth</code> in one project<br>find it everywhere</sub></td>
-<td align="center"><strong>1312 Tests · $0 Cost</strong><br><sub>Every feature acceptance-gated<br>fully offline after install</sub></td>
+<td align="center"><strong>1322 Tests · $0 Cost</strong><br><sub>Every feature acceptance-gated<br>fully offline after install</sub></td>
 </tr>
 </table>
 
@@ -103,7 +103,7 @@ You write code. You make decisions. You debug things. Between sessions, all that
 mempalace-code **indexes it once** into a local vector store, then your AI finds it in milliseconds — using [595x fewer tokens](docs/BENCH_TOKEN_DELTA.md) than grep + read at measured peak (median 80x on a 19k-chunk project, and it keeps scaling). Think of it as `git log` for everything that *isn't* in the code: the *why*, the discussions, the dead ends, the decisions.
 
 **What gets indexed:**
-- Code files — functions, classes, modules (Python, TypeScript/JS, Go, Rust, C/C++, C#, F#, VB.NET, XAML, Java, Kotlin, Scala, Swift, Dart, PHP, Markdown) plus Kubernetes manifests
+- Code files — structural chunks for Python, TypeScript/JS/TSX/JSX, Go, Rust, Java, Kotlin, C#, F#, VB.NET, XAML, Swift, PHP, Scala, Dart, Terraform/HCL, Markdown, and Kubernetes manifests; adaptive chunks for C/C++, Ruby, shell, SQL, HTML/CSS, JSON/YAML/TOML, CSV, Dockerfile, Make, templates, and config files
 - .NET solutions — `.sln`/`.csproj` project graphs, cross-project symbol relationships, interface implementations
 - Conversation exports — Claude, ChatGPT, Slack
 - Architecture notes, decisions, anything you store manually
@@ -116,14 +116,14 @@ mempalace-code **indexes it once** into a local vector store, then your AI finds
 
 ### Language-Aware Code Mining
 
-`mempalace mine` walks your source tree and chunks at **structural boundaries** — functions, classes, methods — not arbitrary line counts. Leading comments and docstrings stay attached to their declarations.
+`mempalace mine` walks your source tree and chooses the best chunker for each file type: AST boundaries where optional tree-sitter grammars are available, regex structural boundaries for supported languages, YAML-aware Kubernetes resource splits, Markdown/prose sections, or adaptive line-count chunks for formats without reliable declarations. Leading comments and docstrings stay attached to declarations where structural chunking is active; Markdown drawers keep heading path, section type, and Mermaid/code/table flags in search metadata.
 
 | Language | Strategy | AST Support |
 |----------|----------|:-----------:|
-| Python | Functions, classes, methods, decorators | Tree-sitter |
-| TypeScript / JavaScript / TSX / JSX | Functions, classes, exports, imports | Tree-sitter |
-| Go | Functions, types, methods, interfaces | Tree-sitter |
-| Rust | Functions, structs, enums, traits, impls | Tree-sitter |
+| Python | Functions, classes, methods, decorators | Optional tree-sitter |
+| TypeScript / JavaScript / TSX / JSX | Functions, classes, exports, imports | Optional tree-sitter |
+| Go | Functions, types, methods, interfaces | Optional tree-sitter |
+| Rust | Functions, structs, enums, traits, impls | Optional tree-sitter |
 | Java | Classes, interfaces, methods, annotations | Regex |
 | Kotlin | Classes, objects, functions, extensions | Regex |
 | Scala | Classes, case classes, objects, traits, enums, functions, implicits, type aliases, generics | Regex |
@@ -133,12 +133,20 @@ mempalace-code **indexes it once** into a local vector store, then your AI finds
 | C# | Classes, interfaces, records, methods, properties | Regex |
 | F# / VB.NET | Modules, types, functions | Regex |
 | XAML | Controls, resources, code-behind linking | Regex |
-| C / C++ | Functions, structs, enums, classes | Regex |
-| Kubernetes manifests | Deployments, Services, ConfigMaps, Secrets, Ingresses, CRDs (indexed by kind/namespace/labels) | YAML-aware |
-| Markdown / plain text | Heading sections, paragraphs | — |
-| YAML / JSON / TOML | Adaptive line-count | — |
+| Terraform / HCL | Terraform/HCL top-level blocks (`resource`, `module`, `variable`, `moved`, `import`, `check`, etc.) | Regex |
+| Kubernetes manifests | Deployments, Services, ConfigMaps, Secrets, Ingresses, CRDs (indexed by kind/name) | YAML-aware |
+| Markdown / plain text | Heading sections (`#`-`######`), heading paths, section metadata, paragraphs | — |
+| C / C++ | Indexed and searchable with best-effort symbol metadata; chunked adaptively today | — |
+| Ruby / shell / SQL | Indexed and searchable; chunked adaptively today | — |
+| HTML / CSS / CSV | Indexed and searchable; chunked adaptively today | — |
+| YAML / JSON / TOML | Adaptive line-count; Kubernetes YAML auto-detected separately | — |
+| Dockerfile / Make / templates / config | Dockerfile, Containerfile, Makefile, GNUmakefile, Vagrantfile, Go templates, Jinja2, `.conf`, `.cfg`, `.ini` | — |
 
-Tree-sitter is optional (`pip install "mempalace-code[treesitter]"`). Without it, all languages fall back to regex boundary detection — still structural, just less precise.
+The `mempalace_code_search` language filter is generated from the same language
+catalog as the miner. If a file type is mined with a language label, the MCP
+schema and unsupported-language hints stay aligned with that catalog.
+
+Tree-sitter is optional (`pip install "mempalace-code[treesitter]"`). When a grammar is missing, Python, TypeScript/JavaScript/TSX/JSX, Go, and Rust fall back to regex structural chunking. Other recognized formats use their regex, YAML-aware, prose, or adaptive chunker as listed above.
 
 ```bash
 mempalace mine ~/projects/myapp                  # all supported file types
@@ -260,7 +268,7 @@ claude mcp add mempalace -- python -m mempalace.mcp_server
 | `mempalace_list_wings` | All wings with drawer counts |
 | `mempalace_list_rooms` | Rooms within a wing |
 | `mempalace_get_taxonomy` | Full wing → room → count tree |
-| `mempalace_search` | Semantic search with optional wing/room filters |
+| `mempalace_search` | Semantic search with optional wing/room filters; Markdown hits include heading path and section metadata |
 | `mempalace_code_search` | Filter by language, symbol name/type, file glob |
 | `mempalace_file_context` | All indexed chunks for a source file, ordered by chunk_index |
 | `mempalace_check_duplicate` | Similarity check before filing (0.9 threshold) |
@@ -457,7 +465,7 @@ This is a code-first fork of [milla-jovovich/mempalace](https://github.com/milla
 | No backup, no recovery | `backup` / `restore` / `export` / `import` |
 | No incremental mining | Content-hash incremental: only changed files re-chunked |
 | No code-search | `code_search` — filter by language, symbol, glob |
-| Line-count chunking | Tree-sitter AST + regex structural chunking |
+| Line-count chunking | Language-aware mining: tree-sitter AST for supported grammars, regex structural chunking, YAML-aware Kubernetes splits, prose sections, and adaptive chunks for configs/data |
 
 Full audit: [`docs/UPSTREAM_HARDENING.md`](docs/UPSTREAM_HARDENING.md).
 
@@ -610,9 +618,9 @@ python -m pytest tests/ -x -q    # full suite, all local, no network
 Apache 2.0 — see [LICENSE](LICENSE) and [NOTICE](NOTICE).
 
 <!-- Link Definitions -->
-[version-shield]: https://img.shields.io/badge/version-1.6.0-4dc9f6?style=flat-square&labelColor=0a0e14
+[version-shield]: https://img.shields.io/badge/version-1.6.2-4dc9f6?style=flat-square&labelColor=0a0e14
 [release-link]: https://github.com/rergards/mempalace-code/releases
-[python-shield]: https://img.shields.io/badge/python-3.9+-7dd8f8?style=flat-square&labelColor=0a0e14&logo=python&logoColor=7dd8f8
+[python-shield]: https://img.shields.io/badge/python-3.11+-7dd8f8?style=flat-square&labelColor=0a0e14&logo=python&logoColor=7dd8f8
 [python-link]: https://www.python.org/
 [license-shield]: https://img.shields.io/badge/license-Apache_2.0-b0e8ff?style=flat-square&labelColor=0a0e14
 [license-link]: https://github.com/rergards/mempalace-code/blob/main/LICENSE

{mempalace_code-1.6.0 → mempalace_code-1.6.2}/docs/BACKLOG-archived.yaml

@@ -308,3 +308,8 @@ items:
       resolution: '2026-04-26: completed in pipeline run'
       archived_date: "2026-04-26"
       done_summary: completed in pipeline run
+    - key: CODE-LANGUAGE-CATALOG-FRESH-PR4
+      summary: 'Rebuild PR #4 language catalog on top of current mining/search code'
+      resolution: '2026-05-01: completed in pipeline run'
+      archived_date: "2026-05-01"
+      done_summary: completed in pipeline run

{mempalace_code-1.6.0 → mempalace_code-1.6.2}/docs/BACKLOG.yaml

@@ -43,6 +43,36 @@ sections:
 # OPEN ITEMS
 # ============================================================
 items:
+  - key: MINE-APP-SCAN-EXCLUDES-PR4
+    summary: "Evaluate PR #4 app-level scan excludes as a separate mining feature"
+    type: task
+    priority: medium
+    status: open
+    size: M
+    section_id: coding_first
+    labels: [mining, watcher, config, pr4]
+    description: |-
+      ## Problem
+      PR #4 includes app-level scan exclusions (`scan_skip_dirs`, `scan_skip_files`,
+      `scan_skip_globs`) alongside its language-catalog refactor. The catalog part was
+      rebuilt on current main, but global scan excludes are a separate behavior change and
+      should not be merged accidentally through the catalog task.
+
+      ## Scope
+      Re-evaluate the PR #4 scan-exclude idea on current main and implement it only if the
+      behavior is still desirable. The feature should exclude noisy generated artifacts such
+      as `.kotlin-lsp/` and `workspace.json` consistently from `scan_project()` and watcher
+      relevance checks, while preserving explicit `--include-ignored` overrides.
+
+      ## Acceptance criteria
+      - Global scan-exclude config keys are documented and loaded from
+        `~/.mempalace/config.json`.
+      - `scan_project()` and watcher relevance filtering use the same app-level rules.
+      - Explicit `--include-ignored` paths override app-level excludes.
+      - Defaults are conservative and do not hide common source files.
+      - Tests cover config loading, miner scanning, watcher filtering, and override behavior.
+      - README/docs describe how to remove previously indexed noise by re-mining.
+    resolution: null
   - key: CLEAN-ONBOARDING
     summary: "Replace interactive onboarding with config-file-first setup"
     type: task

{mempalace_code-1.6.0 → mempalace_code-1.6.2}/docs/COMPARISON_GRAPHIFY.md

@@ -19,7 +19,7 @@ If you want to answer "what did we decide about auth last quarter?" or "find the
 | Dimension | Graphify | mempalace-code |
 |-----------|----------|-----------|
 | Core data structure | NetworkX MultiDiGraph | LanceDB columnar vector store + SQLite KG |
-| Code understanding | tree-sitter AST, 20 languages | regex-based structural chunking (def/class/export) + language detection |
+| Code understanding | tree-sitter AST, 20 languages | language-aware mining: optional tree-sitter chunks for Python/JS/TS/TSX/JSX/Go/Rust, regex structural chunks for supported languages, YAML-aware Kubernetes, adaptive config/prose chunks |
 | Semantic layer | Claude subagent extracts concepts into graph nodes | `all-MiniLM-L6-v2` embeddings (384d, local) |
 | Graph clustering | **Leiden community detection** (produces "god nodes" + clusters) | none — query-time ranked retrieval only |
 | Search primitive | graph traversal, BFS with hop limits | cosine distance over vectors, filtered by wing/room |
@@ -28,7 +28,7 @@ If you want to answer "what did we decide about auth last quarter?" or "find the
 | Conversation mining | none | `convo_miner.py` ingests Claude/ChatGPT/Slack exports |
 | Multimodal | **PDFs, images, videos, YouTube links** (via host LLM API) | text only |
 | Visualization | **interactive HTML graph** (pyvis) | none |
-| Incremental rebuild | **SHA256 file-level cache** | not yet (planned: CODE-INCREMENTAL) |
+| Incremental rebuild | **SHA256 file-level cache** | content-hash incremental mining; only changed files are re-chunked |
 | Privacy on ingest | code stays local; **docs/PDFs/images sent to host LLM API** | **nothing leaves the host, ever** (fully offline) |
 | Embedding dependency | none | 80 MB `all-MiniLM-L6-v2` model downloaded once |
 | MCP surface | `/graphify query`, `/graphify path`, `/graphify explain` | 27 MCP tools (search, traverse, diary, KG, arch-retrieval, stats, …) |
@@ -85,11 +85,11 @@ Graphify is per-project — each repo has its own `graphify-out/` directory and
 
 mempalace-code has no visualization layer. Vector spaces do not visualize well; graph structures do.
 
-### 2. Tree-sitter AST, 20 languages
+### 2. Full AST graph precision across more languages
 
 Graphify uses tree-sitter for parsing, covering 20 languages precisely. Function calls, imports, class references, and type usages are captured at AST fidelity.
 
-mempalace-code uses regex-based structural chunking. It handles Python, JS, TS, Go, Rust reasonably well but it is not an AST — it cannot track `foo()` → function definition of `foo` across files. Symbol metadata is per-chunk only, not cross-referenced.
+mempalace-code uses tree-sitter for chunk boundaries when optional grammars are installed for Python, TypeScript/JavaScript/TSX/JSX, Go, and Rust. It also uses regex structural chunking for Java, Kotlin, .NET languages, XAML, Swift, PHP, Scala, Dart, and Terraform/HCL, YAML-aware splitting for Kubernetes manifests, and adaptive chunking for configs/data/prose. That is still not a call graph: it cannot track `foo()` → function definition of `foo` across files. Symbol metadata is per-chunk only, not cross-referenced.
 
 **Consequence**: for "find all call sites of this function" graphify is the right tool. mempalace-code will not answer that precisely.
 
@@ -105,11 +105,11 @@ For projects that include research papers, architecture diagrams as PNGs, or rec
 
 mempalace-code is text-only. No PDF parsing, no image captioning, no video transcription.
 
-### 5. Shipped SHA256 incremental rebuild
+### 5. Incremental rebuild is no longer a Graphify-only win
 
 Graphify caches parsed AST by file SHA256. Re-running on an unchanged file is a cache hit; only changed files are re-processed.
 
-mempalace-code's incremental re-mine is on the pre_release backlog (`CODE-INCREMENTAL`) but not yet shipped. Today, `mempalace mine` against a large repo is full-rebuild.
+mempalace-code now also mines incrementally by content hash: unchanged drawers are skipped and only changed files are re-chunked unless `--full` is passed. Graphify still wins on full structural graph analysis, but the basic "do not rebuild every unchanged file" capability is now table stakes for both tools.
 
 ### 6. 10-platform reach via installer
 
@@ -175,12 +175,12 @@ These are genuinely good ideas from graphify that mempalace can incorporate with
 
 | Idea | Cost | Value | Status |
 |------|------|-------|--------|
-| **SHA256 file cache for incremental re-mine** | M | high | Already in pre_release as `CODE-INCREMENTAL` |
+| **Broader AST coverage / call graph extraction** | L | high | Post-launch candidate — current tree-sitter support is chunk-boundary only for Python/JS/TS/TSX/JSX/Go/Rust |
 | **Explicit per-edge / per-drawer provenance label** | S | medium | New (not in backlog yet) — e.g. `confidence`, `extractor_version` |
 | **`benchmarks/TOKEN_DELTA.md` with one public number** | S | high | Filed as `LAUNCH-BENCH-TOKEN-DELTA` (owner task) |
 | **Minimal static HTML visualization** of palace structure (wings × rooms × drawer counts) | M | medium | New candidate for post-launch |
 | **Per-platform installer** (`mempalace install --platform codex\|cursor\|gemini`) | L | low | Not urgent — Claude Code + Codex both have native MCP; per-platform hooks are maintenance burden |
-| **Tree-sitter backend for structural chunking** | L | medium | Not urgent — current regex chunker scores R@5 = 0.95 on the internal bench |
+| **Tree-sitter grammars beyond Python/JS/TS/Go/Rust** | M | medium | Not urgent — current regex/adaptive chunkers cover the launch languages, but not full AST semantics |
 
 Note: the always-on PreToolUse hook is intentionally absent from this list. See the preceding section for why.
 
@@ -194,7 +194,7 @@ Note: the always-on PreToolUse hook is intentionally absent from this list. See
 - crash-safe LanceDB (survives `Ctrl+C`)
 
 **Do not claim**:
-- AST precision — mempalace uses regex chunking
+- full AST/code-graph precision — mempalace uses AST chunk boundaries for a subset, regex structural chunks for many languages, and adaptive chunks for configs/data, but does not build call graphs
 - multimodal ingest — mempalace is text-only
 - visualization — mempalace has none
 - community detection — different problem, different algorithm, not mempalace's game

{mempalace_code-1.6.0 → mempalace_code-1.6.2}/docs/HOW_SEARCH_WORKS.md

@@ -4,7 +4,7 @@ mempalace-code does **semantic vector search** — it finds content by *meaning*
 
 ## The Algorithm in 5 Steps
 
-1. **During mining** (`mempalace mine`), every source file is split into chunks. Each chunk is passed through the `all-MiniLM-L6-v2` model, which converts the text into a **384-dimensional vector** — a numeric fingerprint of its meaning. The vector is stored in LanceDB alongside metadata (`wing`, `room`, `source_file`, `language`, `symbol_name`, `symbol_type`).
+1. **During mining** (`mempalace mine`), every source file is split into chunks. Each chunk is passed through the `all-MiniLM-L6-v2` model, which converts the text into a **384-dimensional vector** — a numeric fingerprint of its meaning. The vector is stored in LanceDB alongside metadata (`wing`, `room`, `source_file`, `language`, `symbol_name`, `symbol_type`). Markdown drawers also store section metadata (`heading`, `heading_level`, `heading_path`, `doc_section_type`) and flags for Mermaid diagrams, fenced code blocks, and tables.
 
 2. **At query time**, the query string (e.g. `"detect language file extension"`) goes through the same model and produces another 384-dimensional vector in the same semantic space.
 
@@ -12,7 +12,7 @@ mempalace-code does **semantic vector search** — it finds content by *meaning*
 
 4. **Optional `wing` / `room` filters** are applied as standard SQL `WHERE` predicates. LanceDB decides whether to pre-filter before the vector search or post-filter after it.
 
-5. **Top-N results are returned** with a `similarity = 1 - distance` score (1.0 = perfect match, 0.0 = unrelated).
+5. **Top-N results are returned** with a `similarity = 1 - distance` score (1.0 = perfect match, 0.0 = unrelated). Programmatic search returns the stored metadata with each hit so agents can cite the file, symbol, language, and Markdown section path when available.
 
 ## ASCII Diagram
 
@@ -62,9 +62,10 @@ mempalace-code does **semantic vector search** — it finds content by *meaning*
                                     │     source: miner.py   sim: 0.396    │
                                     │     def detect_language(path): ...   │
                                     │                                      │
-                                    │ [2] mempalace / miner                │
-                                    │     source: miner.py   sim: 0.351    │
-                                    │     EXTENSION_LANG_MAP = { ... }     │
+                                    │ [2] mempalace / language_catalog     │
+                                    │     source: language_catalog.py       │
+                                    │     sim: 0.351                        │
+                                    │     _EXTENSION_LANG_MAP = { ... }     │
                                     │                                      │
                                     │ [3] ...                              │
                                     └──────────────────────────────────────┘
@@ -78,9 +79,11 @@ mempalace-code does **semantic vector search** — it finds content by *meaning*
 - **The ANN index is approximate.** LanceDB uses IVF-PQ, which trades a tiny amount of recall for a massive speedup. On palaces with ~20k rows, the difference between the ANN search and exact brute force is negligible.
 - **Similarity is not a probability.** A score of 0.396 does not mean "40% match". Scores are only comparable *within the same query* — 0.4 beats 0.3 for the same query, but a 0.4 on one query and a 0.4 on another are not the same thing.
 - **`wing` / `room` filters are cheap.** They are plain columns in LanceDB, evaluated as SQL predicates.
+- **Language filters share the miner catalog.** `code_search(language=...)` validates against the same language labels the miner emits, and the MCP schema hint is generated from that catalog.
+- **Markdown location survives retrieval.** For `.md` files, `search_memories()` results include `heading`, `heading_level`, `heading_path`, `doc_section_type`, `contains_mermaid`, `contains_code`, and `contains_table` when the drawer came from a headed section.
 
 ## Where the Code Lives
 
-- `mempalace/searcher.py:21-90` — high-level `search()` and `search_memories()` functions.
+- `mempalace/searcher.py` — high-level `search()` and `search_memories()` functions.
 - `mempalace/storage.py` — `LanceStore.query()`, which owns the embedding model, the LanceDB handle, and the actual vector search call.
 - `mempalace/miner.py` — smart chunker, language detection, symbol extraction, and the batch embedding loop used during `mempalace mine`.

{mempalace_code-1.6.0 → mempalace_code-1.6.2}/mempalace/README.md

@@ -6,14 +6,15 @@ The Python package that powers mempalace-code. All modules, all logic.
 
 | Module | What it does |
 |--------|-------------|
-| `cli.py` | CLI entry point — routes to mine, search, init, compress, wake-up |
+| `cli.py` | CLI entry point — routes to init, mine, search, watch, backup/restore, export/import, health, and wake-up |
 | `config.py` | Configuration loading — `~/.mempalace/config.json`, env vars, defaults |
+| `language_catalog.py` | Shared language metadata for miner detection, `code_search` validation, and MCP language hints |
 | `normalize.py` | Converts 5 chat formats (Claude Code JSONL, Claude.ai JSON, ChatGPT JSON, Slack JSON, plain text) to standard transcript format |
-| `miner.py` | Project file ingest — scans directories, chunks by paragraph, stores to ChromaDB |
+| `miner.py` | Project file ingest — scans directories, detects languages, chunks code/prose/config, stores drawers; Markdown chunks keep heading path and section metadata |
 | `convo_miner.py` | Conversation ingest — chunks by exchange pair (Q+A), detects rooms from content |
-| `searcher.py` | Semantic search via ChromaDB vectors — filters by wing/room, returns verbatim + scores |
+| `searcher.py` | Semantic search via LanceDB vectors — filters by wing/room/language/symbol, returns verbatim text, scores, and stored metadata such as Markdown heading path |
 | `layers.py` | 4-layer memory stack: L0 (identity), L1 (critical facts), L2 (room recall), L3 (deep search) |
-| `dialect.py` | AAAK compression — entity codes, emotion markers, 30x lossless ratio |
+| `dialect.py` | AAAK lossy summary dialect — entity codes, topic markers, and token-saving estimates |
 | `knowledge_graph.py` | Temporal entity-relationship graph — SQLite, time-filtered queries, fact invalidation |
 | `palace_graph.py` | Room-based navigation graph — BFS traversal, tunnel detection across wings |
 | `mcp_server.py` | MCP server — 27 tools, AAAK auto-teach, Palace Protocol, agent diary |
@@ -28,7 +29,7 @@ The Python package that powers mempalace-code. All modules, all logic.
 ## Architecture
 
 ```
-User → CLI → miner/convo_miner → ChromaDB (palace)
+User → CLI → miner/convo_miner → LanceDB (palace)
                                      ↕
                               knowledge_graph (SQLite)
                                      ↕
@@ -37,4 +38,4 @@ User → MCP Server → searcher → results
                   → diary    → agent journal
 ```
 
-The palace (ChromaDB) stores verbatim content. The knowledge graph (SQLite) stores structured relationships. The MCP server exposes both to any AI tool.
+The palace (LanceDB) stores verbatim drawer content and vector metadata. The knowledge graph (SQLite) stores structured relationships. The MCP server exposes both to any AI tool. ChromaDB is a deprecated optional legacy backend only.