conexus 4.1.2__tar.gz → 4.2.2__tar.gz

{conexus-4.1.2 → conexus-4.2.2}/.claude-plugin/marketplace.json

@@ -9,13 +9,13 @@
       "name": "nx",
       "source": "./nx",
       "description": "Self-hosted three-tier knowledge management with 16 specialized agents, analytical query pipelines, semantic search, and RDR decision tracking for Claude Code.",
-      "version": "4.1.2"
+      "version": "4.2.2"
     },
     {
       "name": "sn",
       "source": "./sn",
       "description": "Injects Serena and Context7 MCP tool usage guidance into subagents via SubagentStart hook.",
-      "version": "4.1.2"
+      "version": "4.2.2"
     }
   ]
 }

{conexus-4.1.2 → conexus-4.2.2}/.github/workflows/ci.yml

@@ -16,7 +16,7 @@ jobs:
   test:
     name: pytest (Python ${{ matrix.python-version }})
     runs-on: ubuntu-latest
-    timeout-minutes: 10
+    timeout-minutes: 15
     strategy:
       fail-fast: false
       matrix:

{conexus-4.1.2 → conexus-4.2.2}/CHANGELOG.md

@@ -6,6 +6,69 @@ Versioning follows [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
 ## [Unreleased]
 
+## [4.2.2] - 2026-04-14
+
+(Note: v4.2.1 was tagged but never published due to a test failure. v4.2.2 supersedes it and includes all 4.2.1 changes plus the ChromaDB Cloud quota audit and observability improvements found during a live shakeout.)
+
+### Added
+
+- **`nx doctor` PyPI version check**: `_check_cli_version` queries https://pypi.org/pypi/conexus/json (3-second timeout) and reports current vs latest. When behind, suggests `uv tool upgrade conexus`. Network failures are silent (offline-tolerant).
+- **`nx upgrade --skip-t3` flag**: skip T3 upgrade steps (e.g., heavy cross-collection projection backfill) for fast T2-only migrations.
+- **`backfill_projection` per-collection progress**: prints `[i/N] collection: chunks, matches, attempted (elapsed)` to stderr during the T3 backfill, plus a final summary with total time and actual rows stored. Previously the backfill was silent for many minutes on large repos.
+- **`CatalogTaxonomy._paginated_get`** + **`_batched_upsert`** helpers: wrap ChromaDB calls with the 300-record per-call cap (`MAX_QUERY_RESULTS` / `MAX_RECORDS_PER_WRITE`).
+- **CLAUDE.md "External Service Limits" section**: documents ChromaDB Cloud + Voyage AI quotas with a reference table. Mandatory consult before any new ChromaDB call.
+
+### Fixed
+
+- **`project_against` paginated `coll.get()`**: `_PAGE = 2000` exceeded the ChromaDB Cloud Get quota of 300, causing `nx taxonomy project` to fail on real cloud collections. Now `_PAGE = 300` with paginated source-collection fetch via `_paginated_get`.
+- **4 unbounded `coll.get()` calls** in `catalog_taxonomy.py` (`_discover_cross_links`, `project_against` centroid filter, `rebuild_taxonomy` rebuild + cleanup paths) wrapped in `_paginated_get` to avoid OOM and quota errors at scale.
+- **3 `centroid_coll.upsert()` sites** wrapped in `_batched_upsert` (defensive against `MAX_RECORDS_PER_WRITE = 300`).
+- **`rebuild_taxonomy` cleanup**: paginated GET + batched DELETE so collections with >300 centroids don't fail rebuild.
+- **`nx taxonomy links` invisible cross-collection links**: command queried `compute_topic_links` (catalog-derived only) and ignored the `topic_links` table. Cross-collection projection links written by `_discover_cross_links` and `generate_cooccurrence_links` were invisible. Now displays all rows in `topic_links` with `[collection]` prefix on each topic. New `--refresh` flag re-runs catalog-derived computation explicitly.
+- **`backfill_projection` misleading count**: reported "X assignments" using the per-call attempt count, but `INSERT OR IGNORE` deduplicates. Now reports "X stored (Y attempted)" using `COUNT(*) FROM topic_assignments WHERE assigned_by = 'projection'`.
+- **Plugin/CLI version mismatch UX**: when the nx plugin is upgraded but the conexus CLI is not, the `nx upgrade --auto` SessionStart hook would print a cryptic Click error. Now prints a helpful message: `nx plugin requires conexus >= 4.2.0 — run: uv tool upgrade conexus`.
+
+## [4.2.0] - 2026-04-14
+
+### Added
+
+- **Idempotent upgrade mechanism** (RDR-076): centralised T2 schema migration registry in `src/nexus/db/migrations.py` with version-gated `Migration(introduced, name, fn)` entries. `apply_pending(conn, current_version)` runs migrations between last-seen version (stored in `_nexus_version` table) and current CLI version. Each migration is idempotent via `PRAGMA table_info()` / `sqlite_master` guards.
+- **`nx upgrade` CLI command** with `--dry-run`, `--force`, `--auto` flags for applying pending T2 migrations and T3 upgrade steps.
+- **Auto-upgrade on SessionStart**: `nx upgrade --auto` runs as the first SessionStart hook — T2 migrations apply silently on every session start.
+- **`T3UpgradeStep` typed interface** for ChromaDB operations (backfills, re-indexing) that require a `T3Database` client.
+- **`nx doctor --check-schema`** validates T2 database schema and reports pending migrations.
+- **MCP version compatibility check**: synchronous check in MCP `main()` that warns on major/minor version divergence between CLI and stored version.
+- **Cross-collection topic projection** (RDR-075): `nx taxonomy project SOURCE` command computes cosine similarity between source chunk embeddings and target collection centroids via normalized matrix multiply. Flags: `--against TARGETS`, `--threshold N` (default 0.85), `--top-k N`, `--persist`, `--backfill`.
+- **Automatic cross-collection projection** in `taxonomy_assign_hook`: every `store_put` now projects against foreign collection centroids in addition to same-collection assignment. New rows use `assigned_by='projection'`.
+- **Cross-collection topic links**: `_discover_cross_links` (centroid-level similarity at discover time) and `generate_cooccurrence_links` (SQL self-join on shared doc co-assignments) populate the `topic_links` table with `link_types=["projection"]` or `["cooccurrence"]`.
+- **`list_sibling_collections()`** in `registry.py` auto-detects related collections from the `{prefix}{name}-{hash8}` naming scheme. Used as the default `--against` target for `nx taxonomy project`.
+- **T3 projection backfill**: `T3UpgradeStep("4.2.0", "Backfill cross-collection projection", ...)` runs via `nx upgrade` (not `--auto`) to populate cross-collection assignments and links for existing installs.
+- **`cross_collection` parameter** on `assign_single` and `assign_batch` — when True, queries only foreign centroids (`$ne collection_name` filter) for cross-collection projection.
+- **Incremental taxonomy assignment during indexing** (RDR-070): `taxonomy_assign_batch` wired into `code_indexer`, `prose_indexer`, `pipeline_stages` uploader, and `doc_indexer`. Chunks assigned to nearest topics immediately after upsert.
+- **`indexer_utils`** gitignore/repo-root helpers: `find_repo_root()`, `should_ignore()`, `load_ignore_patterns()`, `is_gitignored()`. PDF batch mode now respects `.nexusignore`.
+
+### Fixed
+
+- RDR close gate heading normalization: `_extract_section` now accepts both `## Problem` and `## Problem Statement` heading variants; gap regex broadened from `^#### Gap \d+:` to `^#{3,5} Gap \d+:` (accepts h3–h5).
+- `doctor --check-schema` uses `PRAGMA busy_timeout=2000` to prevent `database is locked` during concurrent upgrades.
+- `upsert_topic_links` no longer deletes all rows before inserting — preserves projection links from `_discover_cross_links`.
+- `_parse_version` normalizes to 3-component tuples (`(3, 7)` → `(3, 7, 0)`) to avoid unexpected ordering.
+
+### Changed
+
+- `assign_batch` batches all embeddings into a single ChromaDB query (was N individual queries).
+- `project_against` paginates source collection fetch (2000-chunk pages) to prevent OOM on large collections.
+- `generate_cooccurrence_links` uses a SQL self-join on `topic_assignments` instead of loading the full table into Python memory.
+- Domain store `_migrate_*_if_needed()` methods now delegate to the centralised migration registry.
+
+### Docs
+
+- Full `nx upgrade` section added to `docs/cli-reference.md`
+- `nx taxonomy project` subcommand documented in `docs/cli-reference.md`
+- Migration Registry section added to `docs/architecture.md` replacing old ad-hoc migration paragraph
+- Source Layout in `CLAUDE.md` updated with `migrations.py`, `upgrade.py`, and updated descriptions
+- Release checklist in `docs/contributing.md` now includes `migrations.py` verification
+
 ## [4.1.2] - 2026-04-13
 
 ### Fixed

{conexus-4.1.2 → conexus-4.2.2}/CLAUDE.md

@@ -105,12 +105,13 @@ src/nexus/           # Core package
     config_cmd.py    # nx config
     hooks.py         # nx hooks (user-facing hook management)
     hook.py          # nx hook (hidden; git hook stanza management)
-    doctor.py        # nx doctor
+    doctor.py        # nx doctor (includes --check-schema for T2 validation, RDR-076)
+    upgrade.py       # nx upgrade (--dry-run, --force, --auto; T2 migrations + T3 upgrade steps, RDR-076)
     enrich.py        # nx enrich
     catalog.py       # nx catalog
     mineru.py        # nx mineru
     console.py       # nx console (embedded web UI server)
-    taxonomy_cmd.py  # nx taxonomy (topic browsing and discovery)
+    taxonomy_cmd.py  # nx taxonomy (topic browsing, discovery, cross-collection projection via project subcommand)
     _helpers.py      # Shared CLI helpers (default_db_path)
     _provision.py    # ChromaDB Cloud provisioning helpers
   catalog/           # Xanadu-inspired document catalog (JSONL truth + SQLite cache)
@@ -126,6 +127,7 @@ src/nexus/           # Core package
     watchers.py      # File/event watchers for live UI updates
     routes/          # FastAPI routers (activity, campaigns, health, partials)
   db/                # Storage tier implementations
+    migrations.py    # Centralised T2 migration registry: Migration dataclass, apply_pending(), T3UpgradeStep, version tracking (RDR-076)
     t1.py            # T1 ChromaDB client (ephemeral or HTTP)
     t2/              # T2 SQLite package: MemoryStore, PlanLibrary, CatalogTaxonomy, Telemetry domain stores + T2Database facade
     t3.py            # T3 ChromaDB client (persistent local or cloud)
@@ -138,7 +140,7 @@ src/nexus/           # Core package
   code_indexer.py    # Code file indexing: AST chunking, context extraction, Voyage AI embedding (extracted from indexer.py, RDR-032)
   prose_indexer.py   # Prose file indexing: semantic markdown chunking, CCE embedding (extracted from indexer.py, RDR-032)
   index_context.py   # IndexContext dataclass: shared indexing parameters replacing 12-parameter signatures
-  indexer_utils.py   # Shared indexing utilities: staleness checks, context-prefix building
+  indexer_utils.py   # Shared indexing utilities: staleness checks, context-prefix building, gitignore/repo-root helpers
   classifier.py      # File classification: CODE / PROSE / PDF / SKIP
   chunker.py         # Tree-sitter AST chunking (31 languages)
   languages.py       # LANGUAGE_REGISTRY: unified extension→language map (single source of truth)
@@ -160,7 +162,7 @@ src/nexus/           # Core package
   ripgrep_cache.py   # ripgrep integration for hybrid search
   session.py         # Session lifecycle (T1 server start/connect)
   config.py          # Config hierarchy + .nexus.yml
-  registry.py        # Collection → database routing
+  registry.py        # Collection → database routing + list_sibling_collections() for cross-collection scoping (RDR-075)
   corpus.py          # Corpus naming utilities
   formatters.py      # Result display formatting
   types.py           # Shared type definitions
@@ -171,12 +173,34 @@ src/nexus/           # Core package
   logging_setup.py   # Structured logging configuration (cli/console/mcp/hook modes, rotating file handler)
   taxonomy.py        # Deprecation shim — imports forwarded to db.t2.catalog_taxonomy (RDR-070)
   mcp_server.py      # Backward-compat shim — re-exports all MCP tools from nexus.mcp package
-  mcp_infra.py       # MCP server infrastructure: singletons, caching, test injection, taxonomy_assign_hook (post-store topic assignment)
+  mcp_infra.py       # MCP server infrastructure: singletons, caching, test injection, taxonomy_assign_hook (post-store topic assignment + cross-collection projection), check_version_compatibility (RDR-076)
 nx/                  # Claude Code plugin (skills, agents, hooks, slash commands)
 tests/               # pytest suite (unit + integration + e2e/)
 docs/                # Documentation (architecture.md is the module map)
 ```
 
+## External Service Limits — CHECK BEFORE EVERY CALL
+
+**ALWAYS** consult `src/nexus/db/chroma_quotas.py` (the `QUOTAS` dataclass and `QuotaValidator`) before writing any ChromaDB call. Violating these at runtime produces `ChromaError: Quota exceeded` — costly to debug, embarrassing in release.
+
+**ChromaDB Cloud free-tier caps** (single source of truth: `chroma_quotas.py`):
+
+| Operation | Limit | Notes |
+|-----------|-------|-------|
+| `coll.get(limit=N)` | N ≤ 300 | Same cap as queries. `_PAGE = 300` max for pagination |
+| `coll.query(n_results=N)` | N ≤ 300 | `MAX_QUERY_RESULTS` |
+| `coll.upsert/add(ids=[...])` | ≤ 300 records | `MAX_RECORDS_PER_WRITE` |
+| Concurrent reads per coll | ≤ 10 | `MAX_CONCURRENT_READS` |
+| Concurrent writes per coll | ≤ 10 | `MAX_CONCURRENT_WRITES` |
+| Document size | ≤ 16384 bytes | `MAX_DOCUMENT_BYTES` (use `SAFE_CHUNK_BYTES = 12288`) |
+| Query string | ≤ 256 chars | `MAX_QUERY_STRING_CHARS` |
+| `where` predicates | ≤ 8 top-level | `MAX_WHERE_PREDICATES` |
+| Embedding dims | ≤ 4096 | `MAX_EMBEDDING_DIMENSIONS` |
+
+**Voyage AI**: `voyage-3` / `voyage-code-3` / `voyage-context-3` = 1024 dims, 32k tokens/request. Batch requests up to 128 inputs. Use `nexus.retry._voyage_with_retry` for transient failure handling.
+
+**Rule of thumb**: paginating through a large ChromaDB collection requires `limit ≤ 300` per call. When fetching N documents, use `offset += 300` in a loop. `MAX_RECORDS_PER_WRITE = 300` means upsert batches must also be capped.
+
 ## Development Conventions
 
 - **Python 3.12+**: use `match/case`, `tomllib`, `typing.Protocol`, walrus operator freely

{conexus-4.1.2 → conexus-4.2.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: conexus
-Version: 4.1.2
+Version: 4.2.2
 Summary: Self-hosted semantic search and knowledge management for LLM-driven development
 Project-URL: Homepage, https://github.com/Hellblazer/nexus
 Project-URL: Repository, https://github.com/Hellblazer/nexus
@@ -183,10 +183,11 @@ The `nx` command provides direct access to all storage tiers, indexing, and sear
 | `nx scratch` | Inter-agent session context (in-memory, no API keys) |
 | `nx collection` | Inspect and manage cloud collections |
 | `nx config` | Credentials and settings |
-| `nx doctor` | Health check — verifies dependencies, credentials, connectivity |
+| `nx upgrade` | Run pending database migrations and T3 upgrade steps |
+| `nx doctor` | Health check — verifies dependencies, credentials, connectivity, schema |
 | `nx hooks` | Install git hooks for automatic re-indexing on commit |
 | `nx catalog` | Document registry — search, link, audit the catalog metadata layer |
-| `nx taxonomy` | Topic taxonomy — review, rename, merge, and split auto-discovered topic clusters |
+| `nx taxonomy` | Topic taxonomy — discover, project across collections, review, merge, split |
 | `nx context` | Project context cache — topic map for agent cold-start acceleration |
 | `nx mineru` | MinerU server lifecycle (start/stop/status) for PDF extraction |
 

{conexus-4.1.2 → conexus-4.2.2}/README.md

@@ -135,10 +135,11 @@ The `nx` command provides direct access to all storage tiers, indexing, and sear
 | `nx scratch` | Inter-agent session context (in-memory, no API keys) |
 | `nx collection` | Inspect and manage cloud collections |
 | `nx config` | Credentials and settings |
-| `nx doctor` | Health check — verifies dependencies, credentials, connectivity |
+| `nx upgrade` | Run pending database migrations and T3 upgrade steps |
+| `nx doctor` | Health check — verifies dependencies, credentials, connectivity, schema |
 | `nx hooks` | Install git hooks for automatic re-indexing on commit |
 | `nx catalog` | Document registry — search, link, audit the catalog metadata layer |
-| `nx taxonomy` | Topic taxonomy — review, rename, merge, and split auto-discovered topic clusters |
+| `nx taxonomy` | Topic taxonomy — discover, project across collections, review, merge, split |
 | `nx context` | Project context cache — topic map for agent cold-start acceleration |
 | `nx mineru` | MinerU server lifecycle (start/stop/status) for PDF extraction |
 

{conexus-4.1.2 → conexus-4.2.2}/docs/architecture.md

@@ -242,14 +242,25 @@ Phase 2 consequences:
   own `threading.Lock` plus the SQLite file-level write lock -- callers
   never see `OperationalError: database is locked`.
 
-**Migrations**: Each store owns its schema-migration guards and runs
-them the first time it opens a given database path. The guards are
-per-domain, so concurrent `T2Database` constructors on the same path
-can each reach their own `_init_schema` without coordinating through a
-single global migration lock. (`T2Database.__init__` itself constructs
-the four stores sequentially in dependency order -- the per-domain
-guard matters for multi-process / multi-constructor races, not for the
-sequential initialization inside a single `T2Database`.)
+**Migration Registry** (RDR-076): All T2 schema migrations are centralised in
+`src/nexus/db/migrations.py`. The `MIGRATIONS` list contains version-tagged
+`Migration(introduced, name, fn)` entries. `apply_pending(conn, current_version)`
+runs migrations between the last-seen version (stored in `_nexus_version` table)
+and the current CLI version. Each migration function is idempotent via
+`PRAGMA table_info()` or `sqlite_master` guards.
+
+`T2Database.__init__()` opens a transient connection, calls `apply_pending()`,
+closes it, then constructs the four domain stores. The `_upgrade_done` set
+(guarded by `_upgrade_lock`) provides a process-level fast path — subsequent
+constructions skip all DB access. Domain stores retain their own
+`_migrated_paths` guards for standalone construction outside `T2Database`.
+
+**T3 Upgrade Steps**: `T3UpgradeStep(introduced, name, fn)` entries in the
+`T3_UPGRADES` list handle ChromaDB operations (backfills, re-indexing) that
+require a `T3Database` client. These run via `nx upgrade` (not `--auto` mode).
+
+**Auto-upgrade**: `nx upgrade --auto` runs as the first SessionStart hook,
+applying T2 migrations silently. T3 steps are skipped in auto mode.
 
 **In-memory SQLite**: Tests that want an ephemeral database should use
 a temp file path, not `":memory:"` -- `:memory:` databases are

{conexus-4.1.2 → conexus-4.2.2}/docs/cli-reference.md

@@ -325,6 +325,9 @@ nx taxonomy merge "source" "target"             # merge topics
 nx taxonomy split "label" --k 3                 # split into sub-topics
 nx taxonomy links                               # show inter-topic relationships
 nx taxonomy rebuild -c docs__nexus              # full rebuild
+nx taxonomy project code__nexus                 # project against sibling collections
+nx taxonomy project code__nexus --against knowledge__art  # explicit targets
+nx taxonomy project --backfill --persist        # project all collections
 ```
 
 | Subcommand | Description |
@@ -341,6 +344,7 @@ nx taxonomy rebuild -c docs__nexus              # full rebuild
 | `split LABEL --k N` | Split into N sub-topics via KMeans. `-c NAME` scopes label lookup |
 | `links` | Inter-topic link counts from catalog graph. `-c NAME` filters by collection |
 | `rebuild` | Full re-cluster (alias for `discover --force`). `-c NAME` required |
+| `project SOURCE` | Cross-collection projection: match chunks against other collections' centroids. `--against TARGETS` for explicit targets (default: sibling collections). `--threshold N` (default 0.85). `--persist` to write assignments. `--backfill` to project all collections against each other |
 
 **Configuration** (in `.nexus.yml`):
 
@@ -350,7 +354,7 @@ taxonomy:
   local_exclude_collections: []       # default: ["code__*"] — MiniLM clusters poorly on code
 ```
 
-**Upgrade path**: Run `nx taxonomy discover --all` once after upgrading to populate topics for existing collections.
+**Upgrade path**: Run `nx upgrade` after upgrading to apply pending migrations and T3 upgrade steps (including cross-collection projection backfill). Run `nx taxonomy discover --all` to populate topics for new collections.
 
 ---
 
@@ -621,6 +625,35 @@ nx doctor --fix-paths --dry-run # Preview migration without applying
 
 The `--fix` flag retroactively applies HNSW `search_ef` tuning to all existing local-mode collections. New collections get this automatically. In cloud mode (SPANN), prints a skip message — SPANN defaults are adequate.
 
+```
+nx doctor --check-schema          # Validate T2 database schema and report pending migrations
+```
+
+---
+
+## nx upgrade
+
+Run pending database migrations and T3 upgrade steps.
+
+```
+nx upgrade                        # Apply all pending T2 + T3 migrations
+nx upgrade --dry-run              # List pending migrations without executing
+nx upgrade --force                # Reset version gate and re-run all migrations
+nx upgrade --auto                 # Quiet mode for hook invocation (T2 only, exit 0 always)
+```
+
+| Flag | Description |
+|------|-------------|
+| `--dry-run` | List pending migrations without executing (creates base tables if absent) |
+| `--force` | Reset version gate to 0.0.0 and re-run all migrations. Per-migration idempotency guards still apply |
+| `--auto` | Quiet mode for SessionStart hook. T2 migrations only (T3 skipped — may exceed hook timeout). Exit 0 always |
+
+**How it works**: The CLI version (`importlib.metadata.version("conexus")`) is compared against the last-seen version stored in T2 (`_nexus_version` table). Migrations tagged with versions between last-seen and current are executed. Each migration is idempotent via `PRAGMA table_info()` / `sqlite_master` guards.
+
+**Auto-upgrade**: `nx upgrade --auto` runs as the first SessionStart hook in the Claude Code plugin. T2 migrations apply silently on every session start. T3 upgrade steps (e.g., cross-collection projection backfill) run only via explicit `nx upgrade`.
+
+**Adding new migrations**: Append a `Migration("x.y.z", "description", fn)` entry to the `MIGRATIONS` list in `src/nexus/db/migrations.py`. For T3 operations, use `T3UpgradeStep`.
+
 ---
 
 ## nx console

{conexus-4.1.2 → conexus-4.2.2}/docs/contributing.md

@@ -254,6 +254,7 @@ Every step below is **required**. Missing any one of them has caused problems in
 | `docs/configuration.md` | new config keys or tuning parameters |
 | `docs/storage-tiers.md` | new storage capabilities |
 | `README.md` | high-level feature descriptions |
+| `src/nexus/db/migrations.py` | verify `PRE_REGISTRY_VERSION` matches previous release; new T2 migrations in `MIGRATIONS` list; new T3 steps in `T3_UPGRADES` |
 
 ### Pre-push release checklist
 

{conexus-4.1.2 → conexus-4.2.2}/docs/rdr/README.md

@@ -96,6 +96,8 @@ An RDR (Research-Design-Review) is a short document that records a technical dec
 | [RDR-072](rdr-072-progressive-context-loading.md) | Progressive Context Loading | Feature | Closed | 2026-04-13 |
 | [RDR-073](rdr-073-temporal-entity-knowledge-graph.md) | Temporal Entity Knowledge Graph | Feature | Deferred | 2026-04-13 |
 | [RDR-074](rdr-074-permanence-mode.md) | Permanence Mode | Feature | Deferred | 2026-04-13 |
+| [RDR-075](rdr-075-cross-collection-topic-projection.md) | Cross-Collection Topic Projection | Feature | Closed (implemented) | 2026-04-13 |
+| [RDR-076](rdr-076-idempotent-upgrade-mechanism.md) | Idempotent Upgrade Mechanism | Architecture | Closed (implemented) | 2026-04-13 |
 
 ---