conexus 3.2.2__tar.gz → 3.2.4__tar.gz

{conexus-3.2.2 → conexus-3.2.4}/.claude-plugin/marketplace.json

@@ -9,13 +9,13 @@
       "name": "nx",
       "source": "./nx",
       "description": "Self-hosted three-tier knowledge management with 17 specialized agents, analytical query pipelines, semantic search, and RDR decision tracking for Claude Code.",
-      "version": "3.2.2"
+      "version": "3.2.4"
     },
     {
       "name": "sn",
       "source": "./sn",
       "description": "Injects Serena and Context7 MCP tool usage guidance into subagents via SubagentStart hook.",
-      "version": "3.2.2"
+      "version": "3.2.4"
     }
   ]
 }

{conexus-3.2.2 → conexus-3.2.4}/CHANGELOG.md

@@ -6,6 +6,16 @@ Versioning follows [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
 ## [Unreleased]
 
+## [3.2.4] - 2026-04-07
+
+### Fixed
+- **Chunk boundary overlap** (RDR-054) — wired dead `overlap_chars` into `SemanticMarkdownChunker._split_large_section`, which previously had zero overlap between sub-chunks. Bumped `PDFChunker` default overlap from 15% to 20% (225 → 300 chars). Fixed header duplication bug when overlap exceeds emitted content length. Guarded Python `[-0:]` edge case.
+
+## [3.2.3] - 2026-04-07
+
+### Fixed
+- **Pagination completeness** — all list-returning MCP tools and CLI commands now include pagination footers when results are truncated. Tools fixed: `query`, `scratch` (search/list), `plan_search`, `catalog_search`, `catalog_list`, `catalog_link_query`. CLI commands fixed: `nx catalog list`, `nx catalog search`, `nx catalog links`. Docstrings updated to document pagination behavior.
+
 ## [3.2.2] - 2026-04-07
 
 ### Fixed

{conexus-3.2.2 → conexus-3.2.4}/CLAUDE.md

@@ -62,15 +62,15 @@ Nexus is a Python 3.12+ CLI + persistent server for semantic search and knowledg
 
 **Catalog (T3 metadata layer)**: Git-backed document registry that tracks *what* is indexed and *how documents relate*. JSONL files are the source of truth; SQLite + FTS5 is the query cache (rebuilt automatically on mtime change). Tumblers (hierarchical addresses like `1.2.5`) identify documents. Every indexing pathway (`index repo`, `index pdf`, `index rdr`, MCP `store_put`) auto-registers entries. `nx catalog setup` creates and populates the catalog in one step.
 
-**Link graph**: Typed edges between documents — agents create these during their work:
-- `cites` — citation (auto from `nx enrich` via Semantic Scholar, or by research agents)
-- `implements-heuristic` — code→RDR (auto from indexer, title substring match)
-- `supersedes` — replacement (by rdr-close, knowledge-tidier)
-- `relates` — related findings (by debugger, deep-analyst, codebase-analyzer, architect-planner)
-- `implements` — manual or by developer agent linking insights to RDRs
-- `created_by` tracks provenance: which agent or user created each link
+**Link graph**: Typed edges between documents (`cites`, `implements`, `implements-heuristic`, `supersedes`, `relates`, or custom). `created_by` tracks provenance. Three creation paths:
 
-**Two graph views**: `catalog_links` MCP tool returns live-document links only. `catalog_link_query` returns all links including orphans (for audit). The `query` MCP tool has catalog-aware routing (`author`, `content_type`, `subtree`, `follow_links`, `depth` params) for scoped search in a single call. The `/nx:query` skill orchestrates multi-step analytical queries via three-path dispatch (single query → template match → planner).
+1. **Post-hoc** (batch, after indexing): `generate_citation_links()`, `generate_code_rdr_links()`, `generate_rdr_filepath_links()` in `link_generator.py`
+2. **Auto-linker** (`auto_linker.py`): fires on every `store_put` MCP call, reads `link-context` from T1 scratch (tag: `link-context`), creates links to seeded targets. Skills seed before dispatch; agents self-seed from their task prompt when no context exists.
+3. **Agent-direct**: agents call `catalog_link` MCP tool during work for precise typed links
+
+**Two graph views**: `catalog_links` returns live-document links only. `catalog_link_query` returns all including orphans. The `query` MCP tool has catalog-aware routing (`author`, `content_type`, `subtree`, `follow_links`, `depth`) for scoped search.
+
+**Pagination**: All list-returning tools include footers when truncated — `offset=N` for next page.
 
 **T3 expire guard**: always filter `ttl_days > 0 AND expires_at != "" AND expires_at < now` — the `expires_at != ""` guard is mandatory: permanent entries use `expires_at=""` which sorts before ISO timestamps and would be incorrectly deleted by a 2-condition guard.
 
@@ -84,7 +84,8 @@ src/nexus/           # Core package
     catalog.py       # Core: link(), link_query(), graph(), delete_document(), link_audit(), descendants(), resolve_chunk()
     catalog_db.py    # SQLite schema + FTS5 + UNIQUE link constraint + descendants() SQL helper
     tumbler.py       # Hierarchical addresses (depth, ancestors, lca) + JSONL readers with resilience
-    link_generator.py # Auto-generate citation + code-RDR links from metadata
+    auto_linker.py    # Storage-boundary auto-linking from T1 scratch link-context
+    link_generator.py # Post-hoc batch linkers: citation, code-RDR heuristic, RDR file-path
   db/                # t1.py, t2.py, t3.py — tier implementations; local_ef.py — local ONNX embeddings
   indexer.py         # Repo indexing pipeline (classify → chunk → embed → store)
   classifier.py      # File classification: CODE / PROSE / PDF / SKIP

{conexus-3.2.2 → conexus-3.2.4}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: conexus
-Version: 3.2.2
+Version: 3.2.4
 Summary: Self-hosted semantic search and knowledge management for LLM-driven development
 Project-URL: Homepage, https://github.com/Hellblazer/nexus
 Project-URL: Repository, https://github.com/Hellblazer/nexus

{conexus-3.2.2 → conexus-3.2.4}/docs/rdr/README.md

@@ -75,6 +75,7 @@ An RDR (Research-Design-Review) is a short document that records a technical dec
 | [RDR-051](rdr-051-link-lifecycle.md) | Link Lifecycle: Full CRUD, Queryable Links, Bulk Operations | Architecture | Closed | 2026-04-05 |
 | [RDR-052](rdr-052-catalog-first-query-routing.md) | Catalog-First Query Routing — Push Planning into MCP | Architecture | Closed | 2026-04-05 |
 | [RDR-053](rdr-053-xanadu-fidelity.md) | Xanadu Fidelity — Tumbler Arithmetic and Content-Addressed Spans | Architecture | Closed | 2026-04-05 |
+| [RDR-054](rdr-054-chunk-boundary-equation-splitting.md) | Chunk Boundary Equation Splitting — Information Loss at Chunk Boundaries | Bug | Closed | 2026-04-07 |
 
 ---
 

{conexus-3.2.2 → conexus-3.2.4}/docs/rdr/rdr-006-chunk-size-configuration.md

@@ -239,7 +239,7 @@ nx index repo PATH
 ```
 
 Markdown prose uses `SemanticMarkdownChunker(chunk_size=512, chunk_overlap=50)` (tokens).
-PDFs use `PDFChunker(chunk_chars=1500, overlap_percent=0.15)`.
+PDFs use `PDFChunker(chunk_chars=1500, overlap_percent=0.20)`.
 Both are out of scope — this RDR addresses code files only.
 
 ### R3: Scope of change (Confirmed)

{conexus-3.2.2 → conexus-3.2.4}/docs/rdr/rdr-048-streaming-pdf-pipeline.md

@@ -311,7 +311,7 @@ Three concurrent threads need coordinated shutdown when any stage fails. Standar
 **Current pipeline memory for 771-page CMRB book**:
 - Extraction: `md_parts` list — 771 page strings, ~2-5 MB total
 - `"\n".join(md_parts)` — creates a second copy, ~2-5 MB
-- `PDFChunker.chunk()` — returns ~2000 `TextChunk` objects, each holding a copy of its text. Total: ~3-5 MB (overlapping chunks mean ~15% extra text)
+- `PDFChunker.chunk()` — returns ~2000 `TextChunk` objects, each holding a copy of its text. Total: ~3-5 MB (overlapping chunks mean ~20% extra text)
 - `_embed_with_fallback` — embedding vectors: 2000 × 1024 floats × 4 bytes = ~8 MB
 - `_index_document` / `_index_pdf_incremental` — IDs, documents, metadatas lists: ~5-10 MB
 - **Peak concurrent**: ~20-30 MB for the 771-page book (extraction + chunks + embeddings all in memory simultaneously before the first upsert)

conexus-3.2.4/docs/rdr/rdr-054-chunk-boundary-equation-splitting.md

@@ -0,0 +1,128 @@
+---
+title: "Chunk Boundary Equation Splitting — Information Loss at Chunk Boundaries"
+id: RDR-054
+type: Bug
+status: closed
+close_reason: implemented
+closed_date: 2026-04-07
+priority: high
+author: Hal Hildebrand
+reviewed-by: self
+created: 2026-04-07
+accepted_date: 2026-04-07
+related_issues: []
+---
+
+# RDR-054: Chunk Boundary Equation Splitting — Information Loss at Chunk Boundaries
+
+> Revise during planning; lock at implementation.
+> If wrong, abandon code and iterate RDR.
+
+## Problem Statement
+
+When indexing PDF papers with mathematical equations, the chunker splits equation definitions across chunk boundaries such that:
+- Chunk N ends with the prose leading to the definition (e.g., "The sigmoid excitatory feedback signal: f_u(u_ij) =")
+- Chunk N+1 starts with the formula body, WITHOUT the variable name from Chunk N
+
+This makes the formula **unsearchable by its variable name**. Searching for "f_u" finds Chunk N (truncated before the formula). Searching for the formula content finds Chunk N+1 but without context about what function it defines.
+
+### Reproduction
+
+- Paper: GnaGroSovereign2007.pdf (109 pages, 537 chunks, MinerU extractor)
+- Section: Appendix A4.5, Eq.56 (page 88)
+- Search: "sigmoid excitatory feedback signal f_u" → returns chunk ending before the formula
+- Re-indexing with `--extractor mineru --force` produces identical truncation
+
+The actual equation (from direct PDF read):
+```
+f_u(u_ij) = α^N · ([u_ij]⁺)^N / (α^N + θ · ([u_ij]⁺)^N)
+where α=1.7, N=5, θ=9.8
+```
+
+### Impact
+
+- HIGH for mathematical/scientific papers where equation definitions are primary content
+- The f_u function was a Hill sigmoid, NOT the ([u]⁺)² inferred from analogy. The inference was wrong — 3 independent lines of analogical evidence all pointed to the wrong answer because the actual definition was trapped in a chunk boundary.
+- Multiple parameter values (D₁=18 vs assumed 1.0, S_j={1/6,1/3,1/2} vs assumed {1.0,0.75,0.5}) were also in this truncated region.
+- 17 parallel agents across 4 rounds of deep research (851+ chunk searches) all failed to find this equation via T3 search. Only resolved by reading the PDF directly with the Read tool at the page number.
+
+## Proposed Mitigations
+
+### M1: Chunk overlap (recommended, general-purpose)
+
+**Priority 1 (CRITICAL):** Wire SemanticMarkdownChunker's dead `overlap_chars` (165 chars) into `_split_large_section` which currently has zero overlap. Affects: `md_chunker.py` lines 288–309. Cost: +5-8% corpus-level (only split sections pay).
+
+**Priority 2 (HIGH):** Increase PDFChunker overlap from 225→300 chars (15%→20%). Affects: `pdf_chunker.py` line 12 (`_DEFAULT_OVERLAP`). Cost: +6.3% chunks (<$0.001/paper embedding).
+
+**Migration cost:** Existing indexed documents retain old chunk boundaries until re-indexed. Full corpus re-indexing (`nx index pdf --force` per paper) required to benefit from the fix. For large corpora, estimate total embedding cost as documents × avg_chunks × embedding_price_per_token before running.
+
+300 chars covers all tested equation types including worst case (Long Boltzmann at 202 chars) for the overlap window. However, overlap only ensures the *leading context* is repeated — it does not prevent mid-equation splits for equations longer than the overlap window.
+
+### M2: Equation-aware chunking
+
+Detect LaTeX equation environments (`\begin{equation}...\end{equation}`, `$$...$$`, `\tag{N}`) and ensure they are never split across chunks.
+
+Affects: `pdf_chunker.py` primarily (LaTeX in extracted text).
+
+Trade-off: requires equation boundary detection in extracted text, which varies by extractor. **Not yet verified:** whether MinerU output for GnaGroSovereign2007.pdf actually contains `$$`, `\begin{equation}`, or `\tag{N}` markers. If MinerU renders equations as plain text/Unicode, M2's LaTeX boundary detection has zero coverage. Needs an RF verifying the actual extraction format before implementing.
+
+### M3: Paragraph-boundary chunking
+
+Split only at paragraph boundaries (double newline), never mid-paragraph. The f_u definition was mid-paragraph — M3 addresses this failure mode by construction.
+
+Affects: `pdf_chunker.py` (sentence snapper could prefer `\n\n` over `. `), `md_chunker.py`.
+
+Trade-off: may produce very large chunks for papers with long prose paragraphs. Needs a max-size fallback. However, scientific appendix equations are typically in short display-math paragraphs — paragraph length distribution in the reproduction document (GnaGroSovereign2007.pdf) should be measured to validate this concern. If equation-bearing paragraphs are short, M3 + max-size fallback may be simpler and more correct than overlap alone.
+
+### M4: Post-extraction equation linking
+
+After chunking, scan for equation references (e.g., "Eq.56", "\tag{56}") and create cross-references between chunks that reference the same equation. Could use catalog links with `relates` type.
+
+Affects: new module or extension to `link_generator.py`.
+
+Trade-off: doesn't fix searchability — the equation is still split. But makes it discoverable via link traversal.
+
+## Recommendation
+
+Start with M1 (chunk overlap) — md_chunker first (CRITICAL: currently zero overlap), then PDFChunker bump to 20%. M1 reduces the probability of context loss at boundaries but does not prevent mid-equation splits for equations longer than ~300 chars. M2 (equation-aware snapper) remains necessary for the elimination case.
+
+M3 (paragraph boundaries) deserves evaluation alongside M1 for PDFChunker — it directly addresses the observed failure mode (mid-paragraph split). M4 is independent and complementary.
+
+## Key Discoveries
+
+### RF-1: Chunker boundary overlap audit (verified, source search)
+
+Three distinct overlap gaps identified:
+
+1. **PDFChunker** (`pdf_chunker.py`): Has 225-char overlap (15% of 1500), but boundary snapper at lines 59–63 only looks for `. ` (period-space) — no awareness of LaTeX `$$...$$` or equation environments. Will split mid-equation if the equation doesn't end with a sentence period.
+
+2. **SemanticMarkdownChunker** (`md_chunker.py`): `_split_large_section()` at lines 261–323 has **zero overlap** between sub-chunks. The `chunk_overlap` / `overlap_chars` fields are defined but never used by the semantic path — dead code. Only the naive fallback path has overlap (165 chars). This is the most significant gap.
+
+3. **`_enforce_byte_cap`** (`chunker.py:163–211`): Sub-splits of oversized AST nodes also have zero overlap between sub-chunks. Lower priority since code chunking uses AST boundaries.
+
+**Insertion points for fix:**
+- `md_chunker.py:288–309`: Before `chunks.append(...)` at line 288, capture `emitted_text = "\n\n".join(current_parts)`. After the flush, prepend `emitted_text[-overlap_chars:]` to the new `current_parts` (after the header, so section context is preserved). The naive fallback path at lines 354–376 is a working reference implementation — same overlap structure, already tested.
+- `pdf_chunker.py:59–63`: Extend snapper to check for unclosed `$$` in candidate window
+
+### RF-2: Prior art — RAG chunking overlap and equation-aware splitting (verified, source search)
+
+No mainstream chunking library has equation/math-aware splitting (verified across LlamaIndex, LangChain, Unstructured, Chonkie, semchunk). Default overlaps: LlamaIndex SentenceSplitter ~19.5%, LangChain 200 chars (5%), Unstructured 0. Literature consensus: 10-20% optimal (NVIDIA FinanceBench). Nexus PDFChunker at 15% is already in range — the bug is the sentence snapper and the dead md_chunker overlap code. Equation-boundary snapping would be novel.
+
+### RF-3: Storage cost analysis — overlap overhead measured empirically (verified, spike)
+
+PDFChunker 225→300 chars: +6.3% chunks, <$0.001/paper. 450 chars (doubled): +21.7% — excessive. md_chunker wiring 165ch overlap: +5-8% corpus-level. 200 chars is NOT enough for multi-line equations (Long Boltzmann at 202ch). 300 chars covers all tested types with margin. md_chunker overlap is CRITICAL priority (currently zero).
+
+## Finalization Gate
+
+- [x] Overlap size determined: 300 chars PDFChunker, 165 chars md_chunker (RF-3)
+- [x] Storage overhead measured: +6.3% PDF, +5-8% md corpus-level (RF-3)
+- [ ] Impact validated against real-world corpus (post-implementation)
+- [ ] No regression in existing chunker tests (post-implementation)
+- [ ] Search recall improvement measured for boundary-split cases (post-implementation)
+- [ ] MinerU extraction format verified for equation markers (RF needed for M2)
+
+## Revision History
+
+- 2026-04-07: Gate PASSED. 2 criticals resolved in-gate (insertion point corrected, overlap limitation acknowledged), 3 significants addressed (migration cost, M3 evaluation, M2 feasibility).
+- 2026-04-07: RF-1 (code audit), RF-2 (prior art), RF-3 (cost analysis) — all verified.
+- 2026-04-07: Created from T2 issue report (nexus_issues/chunker-splits-equations-at-boundary). Source: RDR-068 research session where 17 parallel agents failed to find f_u equation definition.

{conexus-3.2.2 → conexus-3.2.4}/nx/.claude-plugin/plugin.json

@@ -1,6 +1,6 @@
 {
   "name": "nx",
-  "version": "3.2.2",
+  "version": "3.2.3",
   "description": "Self-hosted three-tier knowledge management with specialized agents, analytical query pipelines, semantic search, and RDR decision tracking for Claude Code.",
   "author": {
     "name": "Hal Hildebrand",

{conexus-3.2.2 → conexus-3.2.4}/nx/CHANGELOG.md

@@ -6,6 +6,14 @@ Versioning follows [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
 ## [Unreleased]
 
+## [3.2.4] - 2026-04-07
+
+Plugin version aligned with Nexus CLI 3.2.4. No plugin-level functional changes.
+
+## [3.2.3] - 2026-04-07
+
+Plugin version aligned with Nexus CLI 3.2.3. No plugin-level functional changes.
+
 ## [3.2.2] - 2026-04-07
 
 ### Fixed

{conexus-3.2.2 → conexus-3.2.4}/nx/hooks/scripts/subagent-start.sh

@@ -109,6 +109,9 @@ T3 knowledge — permanent, semantic search:
     store_get(doc_id="...", collection="knowledge")
   Tool: mcp__plugin_nx_nexus__store_put
     store_put(content="...", collection="knowledge", title="...", tags="...")
+    AUTO-LINKING: store_put checks T1 scratch for link-context (tag: "link-context") and auto-creates catalog links.
+    If no link-context in scratch, self-seed: catalog_search your task references → scratch put with targets.
+    You MUST call store_put before returning — findings not stored are findings lost.
   Tool: mcp__plugin_nx_nexus__store_delete
     store_delete(doc_id="...", collection="knowledge")
   Tool: mcp__plugin_nx_nexus__collection_list

{conexus-3.2.2 → conexus-3.2.4}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "conexus"
-version = "3.2.2"
+version = "3.2.4"
 description = "Self-hosted semantic search and knowledge management for LLM-driven development"
 readme = { file = "README.md", content-type = "text/markdown" }
 requires-python = ">=3.12,<3.14"

{conexus-3.2.2 → conexus-3.2.4}/sn/.claude-plugin/plugin.json

@@ -1,5 +1,5 @@
 {
   "name": "sn",
-  "version": "3.2.2",
+  "version": "3.2.4",
   "description": "Injects Serena and Context7 MCP tool usage guidance into subagents via SubagentStart hook."
 }

{conexus-3.2.2 → conexus-3.2.4}/src/nexus/commands/catalog.py

@@ -300,23 +300,27 @@ def setup_cmd(remote: str) -> None:
 @click.option("--owner", default="")
 @click.option("--type", "content_type", default="")
 @click.option("--limit", "-n", default=50)
+@click.option("--offset", default=0)
 @click.option("--json", "as_json", is_flag=True)
-def list_cmd(owner: str, content_type: str, limit: int, as_json: bool) -> None:
+def list_cmd(owner: str, content_type: str, limit: int, offset: int, as_json: bool) -> None:
     """List catalog entries."""
     cat = _get_catalog()
     if owner:
         entries = cat.by_owner(Tumbler.parse(owner))
     else:
-        entries = cat.all_documents(limit=limit)
+        entries = cat.all_documents(limit=limit + offset + 1)
     if content_type:
         entries = [e for e in entries if e.content_type == content_type]
-    entries = entries[:limit]
+    total = len(entries)
+    entries = entries[offset:offset + limit]
 
     if as_json:
         click.echo(json.dumps([_entry_to_dict(e) for e in entries], indent=2))
     else:
         for e in entries:
             click.echo(f"{str(e.tumbler):<12} {e.content_type:<10} {e.title}")
+        if offset + limit < total:
+            click.echo(f"\n  Next page: --offset {offset + limit}")
 
 
 @catalog.command("show")
@@ -378,15 +382,18 @@ def show_cmd(tumbler_or_title: str, as_json: bool) -> None:
 @catalog.command("search")
 @click.argument("query")
 @click.option("--limit", "-n", default=20)
+@click.option("--offset", default=0)
 @click.option("--json", "as_json", is_flag=True)
-def search_cmd(query: str, limit: int, as_json: bool) -> None:
+def search_cmd(query: str, limit: int, offset: int, as_json: bool) -> None:
     """Find documents by title, author, corpus, or file path.
 
     Uses full-text search across document metadata. Faster than T3 semantic
     search for exact metadata lookups. Returns tumbler, type, and title.
     """
     cat = _get_catalog()
-    results = cat.find(query)[:limit]
+    all_results = cat.find(query)
+    total = len(all_results)
+    results = all_results[offset:offset + limit]
     if as_json:
         click.echo(json.dumps([_entry_to_dict(e) for e in results], indent=2))
     else:
@@ -395,6 +402,8 @@ def search_cmd(query: str, limit: int, as_json: bool) -> None:
             return
         for e in results:
             click.echo(f"{str(e.tumbler):<12} {e.content_type:<10} {e.title}")
+        if offset + limit < total:
+            click.echo(f"\n  Next page: --offset {offset + limit}")
 
 
 @catalog.command("register", hidden=True)
@@ -599,8 +608,10 @@ def links_cmd(
     links = cat.link_query(
         from_t=resolved_from, to_t=resolved_to,
         link_type=link_type, created_by=created_by,
-        limit=limit, offset=offset,
+        limit=limit + 1, offset=offset,
     )
+    has_more = len(links) > limit
+    links = links[:limit]
     if as_json:
         click.echo(json.dumps([_link_to_dict(lnk) for lnk in links], indent=2))
     else:
@@ -609,6 +620,8 @@ def links_cmd(
             return
         for edge in links:
             click.echo(f"{edge.from_tumbler} → {edge.to_tumbler} ({edge.link_type}) by {edge.created_by}")
+        if has_more:
+            click.echo(f"\n  Next page: --offset {offset + limit}")
 
 
 @catalog.command("link-bulk-delete", hidden=True)

{conexus-3.2.2 → conexus-3.2.4}/src/nexus/mcp_server.py

@@ -391,6 +391,9 @@ def query(
 ) -> str:
     """Document-level semantic search for analytical questions.
 
+    Results are capped at ``limit``. When more documents match, a footer line shows
+    the total count. Increase ``limit`` to see more.
+
     Unlike ``search`` which returns individual chunks, ``query`` groups results
     by source document and returns the best-matching snippet per document along
     with full metadata (title, year, citations, page count, extraction method).
@@ -558,7 +561,9 @@ def query(
                 docs[doc_key]["snippet"] = r.content[:300].replace("\n", " ")
 
         # Sort by best match distance, limit
-        sorted_docs = sorted(docs.values(), key=lambda d: d["distance"])[:limit]
+        all_docs = sorted(docs.values(), key=lambda d: d["distance"])
+        sorted_docs = all_docs[:limit]
+        total = len(all_docs)
 
         header = f"Found {len(sorted_docs)} documents (from {len(results)} chunks across {len(target)} collections)"
         lines: list[str] = [f"{routing_note}\n{header}" if routing_note else header]
@@ -598,6 +603,9 @@ def query(
             lines.append(f"   {d['snippet']}")
             lines.append("")
 
+        if total > limit:
+            lines.append(f"\n--- showing 1-{len(sorted_docs)} of {total} documents. Results are capped at limit={limit}.")
+
         return "\n".join(lines)
     except Exception as e:
         return f"Error: {e}"
@@ -936,6 +944,9 @@ def scratch(
 ) -> str:
     """T1 session scratch pad — ephemeral within-session storage.
 
+    For ``search`` and ``list``, results are capped at ``limit``. A footer
+    indicates when more entries exist.
+
     Args:
         action: One of "put", "search", "list", "get", "delete"
         content: Content to store (for "put")
@@ -964,17 +975,23 @@ def scratch(
             for r in results:
                 snippet = r["content"][:200].replace("\n", " ")
                 lines.append(f"{prefix}[{r['id'][:12]}] {snippet}")
+            if len(results) >= limit:
+                lines.append(f"\n--- showing {len(results)} results (limit={limit}). Increase limit to see more.")
             return "\n".join(lines)
 
         elif action == "list":
             entries = t1.list_entries()
             if not entries:
                 return f"{prefix}No scratch entries."
+            total = len(entries)
+            entries = entries[:limit]
             lines = []
             for e in entries:
                 snippet = e["content"][:80].replace("\n", " ")
                 tags_str = f"  [{e.get('tags', '')}]" if e.get("tags") else ""
                 lines.append(f"{prefix}[{e['id'][:12]}] {snippet}{tags_str}")
+            if total > limit:
+                lines.append(f"\n--- showing {limit} of {total} entries. Increase limit to see all.")
             return "\n".join(lines)
 
         elif action == "get":
@@ -1159,17 +1176,25 @@ def plan_save(
 
 
 @mcp.tool()
-def plan_search(query: str, project: str = "", limit: int = 5) -> str:
+def plan_search(query: str, project: str = "", limit: int = 5, offset: int = 0) -> str:
     """Search the T2 plan library for similar query plans.
 
+    Results are paged. Response footer shows ``offset=N`` for next page.
+
     Args:
         query: Search query (matched against plan query text and tags)
         project: Optional project filter (e.g. "nexus")
-        limit: Maximum results to return
+        limit: Maximum results to return (default 5)
+        offset: Skip this many results (default 0). Use for pagination.
     """
     try:
         with _t2_ctx() as db:
-            results = db.search_plans(query, limit=limit, project=project)
+            # Over-fetch by 1 to detect if there are more
+            results = db.search_plans(query, limit=limit + 1, project=project)
+        if offset:
+            results = results[offset:]
+        has_more = len(results) > limit
+        results = results[:limit]
         if not results:
             return "No matching plans."
         lines: list[str] = []
@@ -1180,6 +1205,9 @@ def plan_search(query: str, project: str = "", limit: int = 5) -> str:
                 f"  outcome={r['outcome']}  tags={r['tags']}\n"
                 f"  plan: {plan_preview}..."
             )
+        shown_end = offset + len(results)
+        if has_more:
+            lines.append(f"\n--- showing {offset + 1}-{shown_end}. may have more: offset={shown_end}")
         return "\n\n".join(lines)
     except Exception as e:
         return f"Error: {e}"
@@ -1205,11 +1233,15 @@ def catalog_search(
     owner: str = "",
     file_path: str = "",
     limit: int = 20,
+    offset: int = 0,
 ) -> list[dict]:
     """Find documents by metadata (title, author, corpus, file path).
 
+    Results are paged. When results are truncated, a ``_pagination`` entry appears
+    at the end with ``next_offset``. Pass that value as ``offset`` to get the next page.
+
     Returns catalog entries with tumbler, physical_collection, and metadata — NOT document
-    content. Use the `search` tool for semantic content search within collections.
+    content. Use the ``search`` tool for semantic content search within collections.
     Use catalog_search first to discover WHICH collections to search, then search for content.
 
     Filters: query (free-text), author, corpus, owner, file_path, content_type (exact match).
@@ -1246,9 +1278,9 @@ def catalog_search(
             sql = (
                 "SELECT tumbler, title, author, year, content_type, file_path, "
                 "corpus, physical_collection, chunk_count, head_hash, indexed_at, metadata "
-                f"FROM documents WHERE {' AND '.join(conditions)} LIMIT ?"
+                f"FROM documents WHERE {' AND '.join(conditions)} LIMIT ? OFFSET ?"
             )
-            params.append(limit)
+            params.extend([limit + 1, offset])
             rows = cat._db.execute(sql, params).fetchall()
             from nexus.catalog.catalog import CatalogEntry
             entries = [
@@ -1260,7 +1292,12 @@ def catalog_search(
                 )
                 for r in rows
             ]
-            return [_entry_to_dict(e) for e in entries]
+            has_more = len(entries) > limit
+            entries = entries[:limit]
+            result = [_entry_to_dict(e) for e in entries]
+            if has_more:
+                result.append({"_pagination": {"next_offset": offset + limit, "limit": limit}})
+            return result
 
         # FTS5 free-text search (append author to query if both provided)
         fts_query = query
@@ -1268,8 +1305,12 @@ def catalog_search(
             fts_query = f"{query} {author}"
         if not fts_query.strip():
             return [{"error": "query or at least one filter required"}]
-        results = cat.find(fts_query, content_type=content_type or None)[:limit]
-        return [_entry_to_dict(e) for e in results]
+        all_results = cat.find(fts_query, content_type=content_type or None)
+        page = all_results[offset:offset + limit]
+        result = [_entry_to_dict(e) for e in page]
+        if offset + limit < len(all_results):
+            result.append({"_pagination": {"next_offset": offset + limit, "limit": limit}})
+        return result
     except Exception as e:
         return [{"error": str(e)}]
 
@@ -1315,7 +1356,10 @@ def catalog_list(
     limit: int = 50,
     offset: int = 0,
 ) -> list[dict]:
-    """List catalog entries with optional filters."""
+    """List catalog entries with optional filters.
+
+    Results are paged. When truncated, a ``_pagination`` entry appears at the end
+    with ``next_offset``. Pass that as ``offset`` to get the next page."""
     cat, err = _require_catalog()
     if err:
         return [{"error": err}]
@@ -1347,7 +1391,11 @@ def catalog_list(
             ]
         if content_type:
             entries = [e for e in entries if e.content_type == content_type]
-        return [_entry_to_dict(e) for e in entries[:limit]]
+        page = entries[:limit]
+        result = [_entry_to_dict(e) for e in page]
+        if len(entries) > limit:
+            result.append({"_pagination": {"next_offset": offset + limit, "limit": limit}})
+        return result
     except Exception as e:
         return [{"error": str(e)}]
 
@@ -1592,6 +1640,9 @@ def catalog_link_query(
 ) -> list[dict]:
     """Query links by any combination of filters. For admin/audit use.
 
+    Results are paged. When truncated, a ``_pagination`` entry appears at the end
+    with ``next_offset``. Pass that as ``offset`` to get the next page.
+
     NOT a query-planner step — use catalog_links for graph traversal.
     Use for: audit by creator, find all links of a type, count generator output.
     created_at_before: ISO timestamp — only links created before this time.
@@ -1606,9 +1657,14 @@ def catalog_link_query(
             from_t=from_tumbler, to_t=to_tumbler, link_type=link_type,
             created_by=created_by, direction=direction, tumbler=tumbler,
             created_at_before=created_at_before,
-            limit=limit, offset=offset,
+            limit=limit + 1, offset=offset,
         )
-        return [_link_to_dict(l) for l in links]
+        has_more = len(links) > limit
+        links = links[:limit]
+        result = [_link_to_dict(l) for l in links]
+        if has_more:
+            result.append({"_pagination": {"next_offset": offset + limit, "limit": limit}})
+        return result
     except Exception as e:
         return [{"error": str(e)}]
 

{conexus-3.2.2 → conexus-3.2.4}/src/nexus/md_chunker.py

@@ -284,10 +284,18 @@ class SemanticMarkdownChunker:
                 current_tokens += part_tokens
                 current_end_char = part_end_char
             else:
+                is_code = part.get("is_code_block", False)
+                if self.preserve_code_blocks and is_code:
+                    # Emit code blocks atomically — never truncate, even if oversized.
+                    pass
+                elif len(part_text) > self.max_chars:
+                    # Truncate oversized non-code parts to prevent unbounded chunk sizes.
+                    part_text = part_text[: self.max_chars]
                 if current_parts:
+                    emitted_text = "\n\n".join(current_parts)
                     chunks.append(
                         self._make_chunk(
-                            "\n\n".join(current_parts),
+                            emitted_text,
                             chunk_index,
                             base_metadata,
                             section["header_path"],
@@ -297,16 +305,21 @@ class SemanticMarkdownChunker:
                     )
                     chunk_index += 1
                     section_start_char = current_end_char
-                is_code = part.get("is_code_block", False)
-                if self.preserve_code_blocks and is_code:
-                    # Emit code blocks atomically — never truncate, even if oversized.
-                    pass
-                elif len(part_text) > self.max_chars:
-                    # Truncate oversized non-code parts to prevent unbounded chunk sizes.
-                    part_text = part_text[: self.max_chars]
-                header_tokens = len(header_text) / _CHARS_PER_TOKEN if header_text else 0.0
-                current_parts = [header_text, part_text] if header_text else [part_text]
-                current_tokens = header_tokens + len(part_text) / _CHARS_PER_TOKEN
+                    if self.overlap_chars > 0:
+                        # Compute overlap from content only (skip header) to
+                        # avoid duplicating the header when emitted_text is
+                        # shorter than overlap_chars.
+                        content_text = "\n\n".join(current_parts[1:]) if header_text else emitted_text
+                        overlap_tail = content_text[-self.overlap_chars:] if content_text else ""
+                        if header_text:
+                            current_parts = [header_text, overlap_tail, part_text]
+                        else:
+                            current_parts = [overlap_tail, part_text]
+                    else:
+                        current_parts = [header_text, part_text] if header_text else [part_text]
+                else:
+                    current_parts = [header_text, part_text] if header_text else [part_text]
+                current_tokens = sum(len(p) for p in current_parts) / _CHARS_PER_TOKEN
                 current_end_char = part_end_char
 
         if current_parts: