gitnexus 1.6.9-rc.35 → 1.6.9-rc.36

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -357,13 +357,14 @@ npm install -g gitnexus
357
357
 
358
358
  GitNexus uses optional DuckDB extensions for BM25 and vector search. The `gitnexus serve` and MCP read paths only ever try to `LOAD` the extensions — they never block on a network install. The `analyze` command, by default, attempts one bounded out-of-process `INSTALL` if `LOAD` fails and proceeds even when that install times out, so the index is always written to disk; BM25/vector search degrade gracefully until the extensions become available.
359
359
 
360
- Configure the behavior with two environment variables:
360
+ Configure the behavior with these environment variables:
361
361
 
362
362
  | Variable | Values | Default | Effect |
363
363
  | -------------------------------------------- | ---------------------------- | ------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
364
364
  | `GITNEXUS_LBUG_EXTENSION_INSTALL` | `auto`, `load-only`, `never` | `auto` | `auto` runs one bounded INSTALL if LOAD fails. `load-only` only uses already-installed extensions (recommended for offline / firewalled environments). `never` skips optional extensions entirely. |
365
365
  | `GITNEXUS_LBUG_EXTENSION_INSTALL_TIMEOUT_MS` | positive integer | `15000` | Wall-clock budget for the out-of-process `INSTALL` child before it is killed. |
366
366
  | `GITNEXUS_FTS_STEMMER` | supported LadybugDB stemmer | `porter` | Stemmer used when rebuilding BM25/FTS indexes. Use `none` for CJK-heavy repositories, or a language stemmer such as `german`, `french`, or `spanish` when that better matches repository comments and identifiers. Re-run `gitnexus analyze --repair-fts` after changing it. |
367
+ | `GITNEXUS_FTS_CJK_SEGMENTATION` | `none`, `bigram` | `none` | `bigram` inserts overlapping character-bigram boundaries into Chinese/Japanese Han-ideograph spans in `content`/`description` before FTS indexing, so LadybugDB's space-only tokenizer can see sub-phrase word boundaries. Scoped to CJK Unified Ideographs only — Japanese Hiragana/Katakana and Korean Hangul are not currently segmented. Unlike `GITNEXUS_FTS_STEMMER`, this rewrites stored text — enabling it on an already-indexed repo requires a full `gitnexus analyze --force`; neither `--repair-fts` nor a plain incremental `analyze` applies it to previously-indexed files. Set the same value wherever `analyze` and search-serving processes (CLI query, MCP server, web server) run. |
367
368
  | `GITNEXUS_WAL_CHECKPOINT_THRESHOLD` | integer `>= -1` | `67108864` (64 MiB) | LadybugDB WAL auto-checkpoint threshold during analyze (bytes). Auto-checkpoint remains enabled; `-1` keeps Ladybug's stock ~16 MiB. Larger thresholds reduce checkpoint frequency but increase the WAL size at rotation time — choose a smaller value on disk-constrained environments. |
368
369
 
369
370
  ```bash
@@ -375,6 +376,11 @@ GITNEXUS_LBUG_EXTENSION_INSTALL_TIMEOUT_MS=30000 npx gitnexus analyze
375
376
 
376
377
  # CJK-heavy codebase: rebuild keyword indexes without English stemming
377
378
  GITNEXUS_FTS_STEMMER=none npx gitnexus analyze --repair-fts
379
+
380
+ # CJK-heavy codebase: enable sub-phrase search over Chinese/Japanese Han text.
381
+ # On an already-indexed repo, the first run after enabling this MUST be --force —
382
+ # --repair-fts and plain incremental `analyze` both leave old files un-segmented.
383
+ GITNEXUS_FTS_CJK_SEGMENTATION=bigram npx gitnexus analyze --force
378
384
  ```
379
385
 
380
386
  ### Analysis runs out of memory
@@ -25,18 +25,24 @@ import { NodeTableName } from './schema.js';
25
25
  * case that matters: one more oversized row lands right after the buffer was
26
26
  * just under this threshold, before the flush fires. That row is capped at
27
27
  * TREE_SITTER_MAX_BUFFER (32MB, hard-clamped — GITNEXUS_MAX_FILE_SIZE cannot
28
- * raise it), and escapeCSVField's quote-doubling can at most double it. So the
29
- * peak joined-string size is bounded by
30
- * FLUSH_BYTES + 2 * TREE_SITTER_MAX_BUFFER 8MB + 64MB = 72MB,
31
- * versus Node's `buffer.constants.MAX_STRING_LENGTH` (~512MB) that throws
32
- * `RangeError: Invalid string length` past it a >7x margin (see the
28
+ * raise it). Two transforms can each grow it before it reaches the buffer:
29
+ * `applyCjkSegmentationIfEnabled` (#2331, `CJK_BIGRAM_WORST_CASE_GROWTH_FACTOR`
30
+ * on an all-CJK row when `GITNEXUS_FTS_CJK_SEGMENTATION=bigram` the single
31
+ * source of truth for that ratio, imported by the paired test) and
32
+ * `escapeCSVField`'s worst-case quote-doubling (2x). So the peak joined-string
33
+ * size is bounded by
34
+ * FLUSH_BYTES + 2 * CJK_BIGRAM_WORST_CASE_GROWTH_FACTOR * TREE_SITTER_MAX_BUFFER
35
+ * ≈ 8MB + 149MB ≈ 157MB,
36
+ * versus Node's `buffer.constants.MAX_STRING_LENGTH` (~512MB) — the test
37
+ * actually enforces half of that (~256MB), for a ~1.63x margin (see the
33
38
  * `shouldFlushCSVBuffer stays within the V8 string-length ceiling` test,
34
- * which fails loudly if either constant ever moves this margin the wrong
35
- * way). Raising FLUSH_BYTES trades fewer/larger flushes for less margin;
36
- * lowering it trades the reverse for lower peak transient memory. Change the
37
- * constant directly if a real workload needs a different point on that
38
- * curve a per-host env var would let the margin get silently reintroduced
39
- * by an operator with no way to know why 512MB is dangerous.
39
+ * which fails loudly if any of these constants ever moves this margin the
40
+ * wrong way). With segmentation disabled (default), the old ~3.56x margin
41
+ * still applies. Raising FLUSH_BYTES trades fewer/larger flushes for less
42
+ * margin; lowering it trades the reverse for lower peak transient memory.
43
+ * Change the constant directly if a real workload needs a different point on
44
+ * that curve a per-host env var would let the margin get silently
45
+ * reintroduced by an operator with no way to know why 512MB is dangerous.
40
46
  */
41
47
  export declare const FLUSH_BYTES: number;
42
48
  export declare const shouldFlushCSVBuffer: (byteCount: number) => boolean;
@@ -44,6 +50,33 @@ export declare const sanitizeUTF8: (str: string) => string;
44
50
  export declare const escapeCSVField: (value: string | number | undefined | null) => string;
45
51
  export declare const escapeCSVNumber: (value: number | undefined | null, defaultValue?: number) => string;
46
52
  export declare const isBinaryContent: (content: string) => boolean;
53
+ /**
54
+ * Flatten newlines and tabs to single spaces for FTS-indexed text columns
55
+ * (`content`, `description`) — the real fix for #2317.
56
+ *
57
+ * Ladybug's full-text-search tokenizer splits ONLY on the space character —
58
+ * `\n`, `\r`, and `\t` are NOT token delimiters. So multiline text indexes as
59
+ * a handful of giant tokens (each whole line, joined across lines), and a
60
+ * word query matches none of them: `searchFTSFromLbug('foo')` misses a file
61
+ * whose content is `... \nfoo\n ...`. Removing the 10KB cap (#2333/#2317)
62
+ * stores the full body but leaves it unsearchable; collapsing intra-text
63
+ * whitespace to spaces is what actually makes every word searchable.
64
+ *
65
+ * This rewrites the STORED column too (the same value is COPYed in), so File
66
+ * content returned via the graph API is space-flattened — an accepted trade
67
+ * for making file/symbol text searchable. Leading/trailing/empty are no-ops.
68
+ *
69
+ * Callers apply `applyCjkSegmentationIfEnabled` (#2331) to the text *before*
70
+ * this flatten, so a CJK phrase split across a line-wrap loses its boundary
71
+ * bigram (run detection resets at whitespace) — an accepted limitation, see
72
+ * the plan's Scope Boundaries.
73
+ *
74
+ * Exported (#2339) so `bm25-index.ts`'s query path can compose it in the
75
+ * same order on incoming search queries, keeping index-time and query-time
76
+ * text transforms symmetric — a literal tab/newline in a query would
77
+ * otherwise fail to match whitespace-normalized indexed content.
78
+ */
79
+ export declare const normalizeFtsText: (text: string) => string;
47
80
  /** Canonical relationship CSV header — shared by the emit pass and the
48
81
  * `splitRelCsvByLabelPair` differential oracle. */
49
82
  export declare const REL_CSV_HEADER = "from,to,type,confidence,reason,step";
@@ -17,6 +17,7 @@ import path from 'path';
17
17
  import { NODE_TABLES } from './schema.js';
18
18
  import { RelPairRouter } from './rel-pair-routing.js';
19
19
  import { parseTruthyEnv } from '../ingestion/utils/env.js';
20
+ import { applyCjkSegmentationIfEnabled } from '../search/cjk-segmentation.js';
20
21
  /**
21
22
  * Deterministic output ordering — optional (out-of-core / windowed-resolve
22
23
  * enabler). When `GITNEXUS_SORT_GRAPH_OUTPUT` is set, nodes and relationships
@@ -42,18 +43,24 @@ const orderedRelationships = (graph, sorted) => sorted ? [...graph.iterRelations
42
43
  * case that matters: one more oversized row lands right after the buffer was
43
44
  * just under this threshold, before the flush fires. That row is capped at
44
45
  * TREE_SITTER_MAX_BUFFER (32MB, hard-clamped — GITNEXUS_MAX_FILE_SIZE cannot
45
- * raise it), and escapeCSVField's quote-doubling can at most double it. So the
46
- * peak joined-string size is bounded by
47
- * FLUSH_BYTES + 2 * TREE_SITTER_MAX_BUFFER 8MB + 64MB = 72MB,
48
- * versus Node's `buffer.constants.MAX_STRING_LENGTH` (~512MB) that throws
49
- * `RangeError: Invalid string length` past it a >7x margin (see the
46
+ * raise it). Two transforms can each grow it before it reaches the buffer:
47
+ * `applyCjkSegmentationIfEnabled` (#2331, `CJK_BIGRAM_WORST_CASE_GROWTH_FACTOR`
48
+ * on an all-CJK row when `GITNEXUS_FTS_CJK_SEGMENTATION=bigram` the single
49
+ * source of truth for that ratio, imported by the paired test) and
50
+ * `escapeCSVField`'s worst-case quote-doubling (2x). So the peak joined-string
51
+ * size is bounded by
52
+ * FLUSH_BYTES + 2 * CJK_BIGRAM_WORST_CASE_GROWTH_FACTOR * TREE_SITTER_MAX_BUFFER
53
+ * ≈ 8MB + 149MB ≈ 157MB,
54
+ * versus Node's `buffer.constants.MAX_STRING_LENGTH` (~512MB) — the test
55
+ * actually enforces half of that (~256MB), for a ~1.63x margin (see the
50
56
  * `shouldFlushCSVBuffer stays within the V8 string-length ceiling` test,
51
- * which fails loudly if either constant ever moves this margin the wrong
52
- * way). Raising FLUSH_BYTES trades fewer/larger flushes for less margin;
53
- * lowering it trades the reverse for lower peak transient memory. Change the
54
- * constant directly if a real workload needs a different point on that
55
- * curve a per-host env var would let the margin get silently reintroduced
56
- * by an operator with no way to know why 512MB is dangerous.
57
+ * which fails loudly if any of these constants ever moves this margin the
58
+ * wrong way). With segmentation disabled (default), the old ~3.56x margin
59
+ * still applies. Raising FLUSH_BYTES trades fewer/larger flushes for less
60
+ * margin; lowering it trades the reverse for lower peak transient memory.
61
+ * Change the constant directly if a real workload needs a different point on
62
+ * that curve a per-host env var would let the margin get silently
63
+ * reintroduced by an operator with no way to know why 512MB is dangerous.
57
64
  */
58
65
  export const FLUSH_BYTES = 8 * 1024 * 1024;
59
66
  export const shouldFlushCSVBuffer = (byteCount) => byteCount >= FLUSH_BYTES;
@@ -166,8 +173,20 @@ class FileContentCache {
166
173
  * This rewrites the STORED column too (the same value is COPYed in), so File
167
174
  * content returned via the graph API is space-flattened — an accepted trade
168
175
  * for making file/symbol text searchable. Leading/trailing/empty are no-ops.
176
+ *
177
+ * Callers apply `applyCjkSegmentationIfEnabled` (#2331) to the text *before*
178
+ * this flatten, so a CJK phrase split across a line-wrap loses its boundary
179
+ * bigram (run detection resets at whitespace) — an accepted limitation, see
180
+ * the plan's Scope Boundaries.
181
+ *
182
+ * Exported (#2339) so `bm25-index.ts`'s query path can compose it in the
183
+ * same order on incoming search queries, keeping index-time and query-time
184
+ * text transforms symmetric — a literal tab/newline in a query would
185
+ * otherwise fail to match whitespace-normalized indexed content.
169
186
  */
170
- const normalizeFtsText = (text) => text.replace(/[\r\n\t]+/g, ' ');
187
+ export const normalizeFtsText = (text) => text.replace(/[\r\n\t]+/g, ' ');
188
+ /** Composes both FTS-text transforms for the `description` column — one place for the six emission sites below to call, instead of repeating the composition. */
189
+ const formatFtsDescription = (description) => normalizeFtsText(applyCjkSegmentationIfEnabled(description));
171
190
  const extractContent = async (node, contentCache) => {
172
191
  const filePath = node.properties.filePath;
173
192
  const content = await contentCache.get(filePath);
@@ -183,7 +202,7 @@ const extractContent = async (node, contentCache) => {
183
202
  // and only whitespace-normalized for the tokenizer. The symbol snippet path
184
203
  // below, by contrast, deliberately stays capped at MAX_SNIPPET.
185
204
  if (node.label === 'File') {
186
- return normalizeFtsText(content);
205
+ return normalizeFtsText(applyCjkSegmentationIfEnabled(content));
187
206
  }
188
207
  const startLine = node.properties.startLine;
189
208
  const endLine = node.properties.endLine;
@@ -195,7 +214,7 @@ const extractContent = async (node, contentCache) => {
195
214
  const snippet = lines.slice(start, end + 1).join('\n');
196
215
  const MAX_SNIPPET = 5000;
197
216
  const capped = snippet.length > MAX_SNIPPET ? snippet.slice(0, MAX_SNIPPET) + '\n... [truncated]' : snippet;
198
- return normalizeFtsText(capped);
217
+ return normalizeFtsText(applyCjkSegmentationIfEnabled(capped));
199
218
  };
200
219
  // ============================================================================
201
220
  // BUFFERED CSV WRITER
@@ -429,7 +448,7 @@ export const streamAllCSVsToDisk = async (graph, repoPath, csvDir, onNodePhaseCo
429
448
  escapeCSVField(node.properties.name || ''),
430
449
  escapeCSVField(node.properties.heuristicLabel || ''),
431
450
  keywordsStr,
432
- escapeCSVField(normalizeFtsText(node.properties.description || '')),
451
+ escapeCSVField(formatFtsDescription(node.properties.description || '')),
433
452
  escapeCSVField(node.properties.enrichedBy || 'heuristic'),
434
453
  escapeCSVNumber(node.properties.cohesion, 0),
435
454
  escapeCSVNumber(node.properties.symbolCount, 0),
@@ -461,7 +480,7 @@ export const streamAllCSVsToDisk = async (graph, repoPath, csvDir, onNodePhaseCo
461
480
  escapeCSVNumber(node.properties.endLine, -1),
462
481
  node.properties.isExported ? 'true' : 'false',
463
482
  escapeCSVField(content),
464
- escapeCSVField(normalizeFtsText(node.properties.description || '')),
483
+ escapeCSVField(formatFtsDescription(node.properties.description || '')),
465
484
  escapeCSVNumber(node.properties.parameterCount, 0),
466
485
  escapeCSVField(node.properties.returnType || ''),
467
486
  ].join(','));
@@ -477,7 +496,7 @@ export const streamAllCSVsToDisk = async (graph, repoPath, csvDir, onNodePhaseCo
477
496
  escapeCSVNumber(node.properties.endLine, -1),
478
497
  escapeCSVNumber(node.properties.level, 1),
479
498
  escapeCSVField(content),
480
- escapeCSVField(normalizeFtsText(node.properties.description || '')),
499
+ escapeCSVField(formatFtsDescription(node.properties.description || '')),
481
500
  ].join(','));
482
501
  break;
483
502
  }
@@ -507,7 +526,7 @@ export const streamAllCSVsToDisk = async (graph, repoPath, csvDir, onNodePhaseCo
507
526
  escapeCSVField(node.id),
508
527
  escapeCSVField(node.properties.name || ''),
509
528
  escapeCSVField(node.properties.filePath || ''),
510
- escapeCSVField(normalizeFtsText(node.properties.description || '')),
529
+ escapeCSVField(formatFtsDescription(node.properties.description || '')),
511
530
  ].join(','));
512
531
  break;
513
532
  case 'BasicBlock':
@@ -526,7 +545,7 @@ export const streamAllCSVsToDisk = async (graph, repoPath, csvDir, onNodePhaseCo
526
545
  escapeCSVNumber(node.properties.endLine, -1),
527
546
  node.properties.isExported ? 'true' : 'false',
528
547
  escapeCSVField(content),
529
- escapeCSVField(normalizeFtsText(node.properties.description || '')),
548
+ escapeCSVField(formatFtsDescription(node.properties.description || '')),
530
549
  ].join(','));
531
550
  }
532
551
  else {
@@ -541,7 +560,7 @@ export const streamAllCSVsToDisk = async (graph, repoPath, csvDir, onNodePhaseCo
541
560
  escapeCSVNumber(node.properties.startLine, -1),
542
561
  escapeCSVNumber(node.properties.endLine, -1),
543
562
  escapeCSVField(content),
544
- escapeCSVField(normalizeFtsText(node.properties.description || '')),
563
+ escapeCSVField(formatFtsDescription(node.properties.description || '')),
545
564
  ...(node.label === 'Property'
546
565
  ? [escapeCSVField(node.properties.declaredType || '')]
547
566
  : []),
@@ -15,6 +15,7 @@ import { runPipelineFromRepo } from './ingestion/pipeline.js';
15
15
  import { resetDegradedParseCounter } from './tree-sitter/safe-parse.js';
16
16
  import { initLbug, loadGraphToLbug, getLbugStats, executeQuery, executeWithReusedStatement, closeLbug, closeLbugBeforeExit, loadCachedEmbeddings, deleteNodesForFile, deleteAllCommunitiesAndProcesses, deleteAllInterprocTaintPaths, deleteAllCallSummaries, queryImporters, loadFTSExtension, } from './lbug/lbug-adapter.js';
17
17
  import { createSearchFTSIndexes, initialiseSearchFTSStemmer, verifySearchFTSIndexes, } from './search/fts-indexes.js';
18
+ import { cjkSegmentationModeMismatch, getSearchFTSCjkSegmentation, initialiseSearchFTSCjkSegmentation, } from './search/cjk-segmentation.js';
18
19
  import { resolveAnalyzeInstallPolicy } from './lbug/extension-loader.js';
19
20
  import { startWalCheckpointDriver, } from './lbug/wal-checkpoint-driver.js';
20
21
  import { getStoragePaths, resolveBranchPlacement, saveMeta, loadMeta, ensureGitNexusIgnored, registerRepo, isRepoRegistered, cleanupOldKuzuFiles, INCREMENTAL_SCHEMA_VERSION, } from '../storage/repo-manager.js';
@@ -259,6 +260,7 @@ export async function runFullAnalysis(repoPath, options, callbacks) {
259
260
  // parse/load phases. A typo fails here in ms; createSearchFTSIndexes reuses
260
261
  // the cached value via getSearchFTSStemmer.
261
262
  initialiseSearchFTSStemmer();
263
+ initialiseSearchFTSCjkSegmentation();
262
264
  // Scope the degraded-parse log throttle to this run. On a reused process
263
265
  // (e.g. tests, or any host that calls runFullAnalysis more than once) the
264
266
  // module-level counter would otherwise stay saturated and suppress every
@@ -459,6 +461,13 @@ export async function runFullAnalysis(repoPath, options, callbacks) {
459
461
  `forcing a full rebuild so persisted rows match the current schema.`);
460
462
  options = { ...options, force: true };
461
463
  }
464
+ if (existingMeta &&
465
+ cjkSegmentationModeMismatch(existingMeta.cjkSegmentation, getSearchFTSCjkSegmentation())) {
466
+ log(`CJK segmentation mode changed (index built with '${existingMeta.cjkSegmentation ?? 'none'}', ` +
467
+ `this run resolves '${getSearchFTSCjkSegmentation()}'); forcing a full rebuild so indexed ` +
468
+ `text and query-time segmentation stay in sync.`);
469
+ options = { ...options, force: true };
470
+ }
462
471
  // ── Early-return: already up to date ──────────────────────────────
463
472
  if (existingMeta && !options.force && existingMeta.lastCommit === currentCommit) {
464
473
  // Non-git folders have currentCommit = '' — always rebuild since we can't detect changes
@@ -1074,6 +1083,10 @@ export async function runFullAnalysis(repoPath, options, callbacks) {
1074
1083
  // incrementalInProgress to undefined explicitly clears any prior
1075
1084
  // dirty flag (full and incremental success paths converge here).
1076
1085
  schemaVersion: hasGitDir(repoPath) ? INCREMENTAL_SCHEMA_VERSION : undefined,
1086
+ // Always stamped with the live resolved mode (#2331/#2339) — unlike
1087
+ // `pdg` below, 'none' is a meaningful value to compare, not an
1088
+ // absence, so this is never conditionally omitted.
1089
+ cjkSegmentation: getSearchFTSCjkSegmentation(),
1077
1090
  fileHashes: hasGitDir(repoPath) ? newFileHashesRecord : undefined,
1078
1091
  // This branch's full live chunk-key set (#2106 R6). `usedKeys` is every
1079
1092
  // chunk hash touched in this scan — cache HITS included (see parse-impl
@@ -5,7 +5,9 @@
5
5
  * Always reads from the database (no cached state to drift).
6
6
  */
7
7
  import { queryFTS } from '../lbug/lbug-adapter.js';
8
+ import { normalizeFtsText } from '../lbug/csv-generator.js';
8
9
  import { FTS_INDEXES } from './fts-schema.js';
10
+ import { applyCjkSegmentationIfEnabled, MAX_CJK_SEGMENTATION_QUERY_LENGTH, } from './cjk-segmentation.js';
9
11
  /**
10
12
  * Execute a single FTS query via a custom executor (for MCP connection pool).
11
13
  * Returns `null` when the query fails (e.g. FTS index does not exist) so the
@@ -46,6 +48,21 @@ async function queryFTSViaExecutor(executor, tableName, indexName, query, limit)
46
48
  * @returns Ranked search results from FTS indexes
47
49
  */
48
50
  export const searchFTSFromLbug = async (query, limit = 20, repoId) => {
51
+ // Applied once, up front, so every downstream branch searches with the
52
+ // same text the index was built from (#2331/#2339) — index-time and
53
+ // query-time text transforms must never diverge, since QUERY_FTS_INDEX
54
+ // cannot derive a tokenizer from the index it queries. Composed in the
55
+ // SAME order as the write path (csv-generator.ts's formatFtsDescription /
56
+ // extractContent: normalizeFtsText(applyCjkSegmentationIfEnabled(text))):
57
+ // CJK segmentation is no-op when disabled (default); normalizeFtsText
58
+ // (collapsing \r\n\t to a space) applies unconditionally — it has no
59
+ // per-character cost concern, unlike CJK segmentation, so it's not gated
60
+ // by the length cap. Segmentation itself is skipped for pathologically
61
+ // long queries (see MAX_CJK_SEGMENTATION_QUERY_LENGTH) — the query still
62
+ // searches correctly, just without CJK sub-phrase segmentation.
63
+ const searchQuery = normalizeFtsText(query.length <= MAX_CJK_SEGMENTATION_QUERY_LENGTH
64
+ ? applyCjkSegmentationIfEnabled(query)
65
+ : query);
49
66
  const resultsByIndex = [];
50
67
  let queriesSucceeded = 0;
51
68
  if (repoId) {
@@ -56,7 +73,7 @@ export const searchFTSFromLbug = async (query, limit = 20, repoId) => {
56
73
  const { executeParameterized } = poolMod;
57
74
  const executor = (cypher, params) => executeParameterized(repoId, cypher, params);
58
75
  for (const { table, indexName } of FTS_INDEXES) {
59
- const result = await queryFTSViaExecutor(executor, table, indexName, query, limit);
76
+ const result = await queryFTSViaExecutor(executor, table, indexName, searchQuery, limit);
60
77
  if (result !== null) {
61
78
  queriesSucceeded++;
62
79
  resultsByIndex.push(result);
@@ -67,7 +84,7 @@ export const searchFTSFromLbug = async (query, limit = 20, repoId) => {
67
84
  // Use core lbug adapter (CLI / pipeline context) — also sequential for safety.
68
85
  for (const { table, indexName } of FTS_INDEXES) {
69
86
  try {
70
- const result = await queryFTS(table, indexName, query, limit, false);
87
+ const result = await queryFTS(table, indexName, searchQuery, limit, false);
71
88
  queriesSucceeded++;
72
89
  resultsByIndex.push(result);
73
90
  }
@@ -0,0 +1,126 @@
1
+ /**
2
+ * CJK bigram segmentation for FTS search (#2331)
3
+ *
4
+ * LadybugDB's bundled FTS tokenizer splits only on the space character, so a
5
+ * contiguous CJK (Chinese/Japanese/Korean) span indexes as one giant token and
6
+ * sub-phrase queries never match. `segmentCjkSpans` addresses the Han-ideograph
7
+ * case (Chinese text and Japanese Kanji — see scope note below) by rewriting
8
+ * each contiguous run of CJK Unified Ideographs into space-separated overlapping character
9
+ * bigrams (`采购订单` -> `采购 购订 订单`), the same technique MySQL's `ngram`
10
+ * fulltext parser, Elasticsearch's `cjk` analyzer, and Lucene's
11
+ * `CJKBigramFilter` use by default. For any exact contiguous substring query
12
+ * of length >= 2, its bigram decomposition is a subset of the source text's
13
+ * bigram decomposition, so sub-phrase matching works without needing a
14
+ * dictionary or boundary-alignment luck.
15
+ *
16
+ * Scoped to the core CJK Unified Ideographs block (U+4E00-U+9FFF) only —
17
+ * covers Chinese text and Japanese Kanji. Hiragana, Katakana, and Hangul
18
+ * Syllables are deliberately excluded for now (see plan Scope Boundaries);
19
+ * extend `CJK_UNIFIED_IDEOGRAPHS` below if that need arises.
20
+ */
21
+ /**
22
+ * Worst-case output/input byte ratio for `segmentCjkSpans` on an all-CJK run:
23
+ * each adjacent character pair becomes a 2-character bigram plus a 1-byte
24
+ * separator, i.e. ~7 output bytes per 3 input bytes of UTF-8 CJK text (each
25
+ * CJK character is 3 bytes). Single source of truth — imported by both the
26
+ * CSV-flush safety-margin test (`csv-pipeline.test.ts`) and the growth-factor
27
+ * regression guard (`cjk-segmentation.test.ts`), and referenced by name in
28
+ * `csv-generator.ts`'s `FLUSH_BYTES` margin comment, so all three stay in
29
+ * sync if the algorithm's expansion ratio ever changes.
30
+ */
31
+ export declare const CJK_BIGRAM_WORST_CASE_GROWTH_FACTOR: number;
32
+ /**
33
+ * A real search query is always a short phrase — unlike indexed File content
34
+ * (deliberately uncapped, #2317/#2323), nothing else bounds a query's length
35
+ * before it reaches `segmentCjkSpans`. Without a cap, a pathologically long
36
+ * query string (accidental or adversarial) would pay `segmentCjkSpans`'s
37
+ * per-character allocation cost on every search request. 2000 characters
38
+ * comfortably covers any real natural-language query.
39
+ *
40
+ * Lives here rather than in `bm25-index.ts` (its only other consumer) so
41
+ * `local-backend.ts` can import it statically alongside this module's other
42
+ * symbols — `bm25-index.ts` transitively imports `@ladybugdb/core` (a native
43
+ * binding, via `lbug-adapter.js`), which is exactly the kind of module
44
+ * `local-backend.ts`'s `bm25Search` deliberately dynamic-imports instead of
45
+ * statically (#1489: can fail in sandboxed MCP contexts). A static import of
46
+ * even one constant from `bm25-index.ts` would force that native binding to
47
+ * load at MCP-server startup instead of at first query.
48
+ */
49
+ export declare const MAX_CJK_SEGMENTATION_QUERY_LENGTH = 2000;
50
+ /**
51
+ * True if `text` contains at least one CJK Unified Ideograph, including a
52
+ * single character. Generic presence check — for gating the "enable bigram
53
+ * mode" query warning specifically, use {@link containsSegmentableCjkRun}
54
+ * instead (#2339): a lone CJK character can never be bigram-segmented, so
55
+ * this broader check would misleadingly flag queries bigram mode can't help.
56
+ */
57
+ export declare const containsCjkIdeograph: (text: string) => boolean;
58
+ /**
59
+ * True if `text` contains a CJK run of 2+ contiguous ideographs —
60
+ * i.e. a span `segmentCjkSpans` can actually bigram-segment. A lone CJK
61
+ * character can never be segmented (no possible pairing), so callers
62
+ * warning "enable bigram mode" for a query should gate on this, not on
63
+ * `containsCjkIdeograph` (#2339). Uses its own non-global RegExp instance
64
+ * (see `CJK_SEGMENTABLE_RUN_RE` above) — never call `.test()` on the
65
+ * shared, global-flagged `CJK_RUN_RE` directly.
66
+ */
67
+ export declare const containsSegmentableCjkRun: (text: string) => boolean;
68
+ /**
69
+ * Rewrite contiguous CJK spans in `text` into space-separated overlapping
70
+ * bigrams (a run of exactly 2 chars becomes a single bigram; a lone CJK
71
+ * char has no possible pairing and passes through unchanged). Non-CJK text
72
+ * is never touched by `replace` in the first place, so a run's boundary
73
+ * spacing is decided by peeking at the *original* string's neighboring
74
+ * character (via the callback's `offset`/`full` args) rather than tracking
75
+ * state across matches — each match stays independent even when two CJK
76
+ * runs sit close together, and a space is added only when the neighbor
77
+ * isn't already whitespace, so the whitespace-splitting FTS tokenizer
78
+ * treats runs as separate tokens (`ERP审批流程` -> `ERP 审批 批流 流程`,
79
+ * not `ERP审批 批流 流程`).
80
+ */
81
+ export declare const segmentCjkSpans: (text: string) => string;
82
+ export declare const DEFAULT_FTS_CJK_SEGMENTATION = "none";
83
+ /**
84
+ * True if `value` is one of the recognized segmentation modes. Callers that
85
+ * interpolate a persisted `RepoMeta.cjkSegmentation` value into agent-visible
86
+ * text (e.g. the MCP query-tool's mode-drift warning, #2339) must validate
87
+ * with this first — that field comes from `meta.json`, a schema-less
88
+ * `JSON.parse` of on-disk state inside the analyzed repo, not a trusted
89
+ * input, so an unvalidated value could otherwise be echoed verbatim into
90
+ * tool output an agent is expected to trust and act on.
91
+ */
92
+ export declare const isSupportedCjkSegmentationMode: (value: unknown) => value is string;
93
+ /**
94
+ * Resolve + validate `GITNEXUS_FTS_CJK_SEGMENTATION` once, up front at analyze
95
+ * startup, and cache it — mirrors `initialiseSearchFTSStemmer` so an invalid
96
+ * value fails in milliseconds instead of partway through a run. The cached
97
+ * value is what {@link getSearchFTSCjkSegmentation} returns for the rest of
98
+ * the run, so config is read and validated in exactly one place.
99
+ */
100
+ export declare function initialiseSearchFTSCjkSegmentation(): string;
101
+ /**
102
+ * Return the mode resolved by {@link initialiseSearchFTSCjkSegmentation}.
103
+ * Falls back to resolving on demand when init was never called (read-only
104
+ * hosts, unit tests) so validation always applies.
105
+ */
106
+ export declare function getSearchFTSCjkSegmentation(): string;
107
+ /**
108
+ * Whether the CJK segmentation mode an index was built under (as persisted in
109
+ * `RepoMeta.cjkSegmentation`) differs from the mode the live process resolves
110
+ * (#2331/#2339) — used by `run-analyze.ts` to force a full rebuild on drift,
111
+ * and by the MCP query path to warn when a repo's index and the serving
112
+ * process disagree. A single scalar, so a plain equality check suffices —
113
+ * unlike `pdgModeMismatch` in `run-analyze.ts`, no key-union comparator is
114
+ * needed. An absent recorded stamp defaults to 'none' (this feature's own
115
+ * default), so a repo that never touched this feature never mismatches.
116
+ * Pure + exported for testing. Lives here (not `run-analyze.ts`) so callers
117
+ * that only need this comparator — e.g. the MCP query path — don't have to
118
+ * pull in the full analyze-pipeline module.
119
+ */
120
+ export declare const cjkSegmentationModeMismatch: (recorded: string | undefined, resolved: string) => boolean;
121
+ /**
122
+ * The single entry point the write path (`csv-generator.ts`) and read path
123
+ * (`bm25-index.ts`) both call, so indexed text and query text are always
124
+ * segmented identically. No-ops when the resolved mode is `none` (default).
125
+ */
126
+ export declare const applyCjkSegmentationIfEnabled: (text: string) => string;
@@ -0,0 +1,176 @@
1
+ /**
2
+ * CJK bigram segmentation for FTS search (#2331)
3
+ *
4
+ * LadybugDB's bundled FTS tokenizer splits only on the space character, so a
5
+ * contiguous CJK (Chinese/Japanese/Korean) span indexes as one giant token and
6
+ * sub-phrase queries never match. `segmentCjkSpans` addresses the Han-ideograph
7
+ * case (Chinese text and Japanese Kanji — see scope note below) by rewriting
8
+ * each contiguous run of CJK Unified Ideographs into space-separated overlapping character
9
+ * bigrams (`采购订单` -> `采购 购订 订单`), the same technique MySQL's `ngram`
10
+ * fulltext parser, Elasticsearch's `cjk` analyzer, and Lucene's
11
+ * `CJKBigramFilter` use by default. For any exact contiguous substring query
12
+ * of length >= 2, its bigram decomposition is a subset of the source text's
13
+ * bigram decomposition, so sub-phrase matching works without needing a
14
+ * dictionary or boundary-alignment luck.
15
+ *
16
+ * Scoped to the core CJK Unified Ideographs block (U+4E00-U+9FFF) only —
17
+ * covers Chinese text and Japanese Kanji. Hiragana, Katakana, and Hangul
18
+ * Syllables are deliberately excluded for now (see plan Scope Boundaries);
19
+ * extend `CJK_UNIFIED_IDEOGRAPHS` below if that need arises.
20
+ */
21
+ /** The core CJK Unified Ideographs block — single source of truth for both regexes below. */
22
+ const CJK_UNIFIED_IDEOGRAPHS = '[\\u4e00-\\u9fff]';
23
+ const CJK_CHAR_RE = new RegExp(CJK_UNIFIED_IDEOGRAPHS);
24
+ const CJK_RUN_RE = new RegExp(`${CJK_UNIFIED_IDEOGRAPHS}{2,}`, 'g');
25
+ // Same pattern as CJK_RUN_RE, but WITHOUT the 'g' flag — kept as a separate
26
+ // instance deliberately. RegExp.prototype.test() on a global-flagged regex
27
+ // is stateful (mutates lastIndex between calls); CJK_RUN_RE only stays safe
28
+ // today because its one consumer (segmentCjkSpans) drives it exclusively via
29
+ // String.prototype.replace, which always resets matching from index 0. A
30
+ // second consumer calling .test() on that same shared instance would leak
31
+ // state across calls (and across requests, in a long-lived process).
32
+ const CJK_SEGMENTABLE_RUN_RE = new RegExp(`${CJK_UNIFIED_IDEOGRAPHS}{2,}`);
33
+ const WHITESPACE_RE = /\s/;
34
+ /**
35
+ * Worst-case output/input byte ratio for `segmentCjkSpans` on an all-CJK run:
36
+ * each adjacent character pair becomes a 2-character bigram plus a 1-byte
37
+ * separator, i.e. ~7 output bytes per 3 input bytes of UTF-8 CJK text (each
38
+ * CJK character is 3 bytes). Single source of truth — imported by both the
39
+ * CSV-flush safety-margin test (`csv-pipeline.test.ts`) and the growth-factor
40
+ * regression guard (`cjk-segmentation.test.ts`), and referenced by name in
41
+ * `csv-generator.ts`'s `FLUSH_BYTES` margin comment, so all three stay in
42
+ * sync if the algorithm's expansion ratio ever changes.
43
+ */
44
+ export const CJK_BIGRAM_WORST_CASE_GROWTH_FACTOR = 7 / 3;
45
+ /**
46
+ * A real search query is always a short phrase — unlike indexed File content
47
+ * (deliberately uncapped, #2317/#2323), nothing else bounds a query's length
48
+ * before it reaches `segmentCjkSpans`. Without a cap, a pathologically long
49
+ * query string (accidental or adversarial) would pay `segmentCjkSpans`'s
50
+ * per-character allocation cost on every search request. 2000 characters
51
+ * comfortably covers any real natural-language query.
52
+ *
53
+ * Lives here rather than in `bm25-index.ts` (its only other consumer) so
54
+ * `local-backend.ts` can import it statically alongside this module's other
55
+ * symbols — `bm25-index.ts` transitively imports `@ladybugdb/core` (a native
56
+ * binding, via `lbug-adapter.js`), which is exactly the kind of module
57
+ * `local-backend.ts`'s `bm25Search` deliberately dynamic-imports instead of
58
+ * statically (#1489: can fail in sandboxed MCP contexts). A static import of
59
+ * even one constant from `bm25-index.ts` would force that native binding to
60
+ * load at MCP-server startup instead of at first query.
61
+ */
62
+ export const MAX_CJK_SEGMENTATION_QUERY_LENGTH = 2000;
63
+ /**
64
+ * True if `text` contains at least one CJK Unified Ideograph, including a
65
+ * single character. Generic presence check — for gating the "enable bigram
66
+ * mode" query warning specifically, use {@link containsSegmentableCjkRun}
67
+ * instead (#2339): a lone CJK character can never be bigram-segmented, so
68
+ * this broader check would misleadingly flag queries bigram mode can't help.
69
+ */
70
+ export const containsCjkIdeograph = (text) => CJK_CHAR_RE.test(text);
71
+ /**
72
+ * True if `text` contains a CJK run of 2+ contiguous ideographs —
73
+ * i.e. a span `segmentCjkSpans` can actually bigram-segment. A lone CJK
74
+ * character can never be segmented (no possible pairing), so callers
75
+ * warning "enable bigram mode" for a query should gate on this, not on
76
+ * `containsCjkIdeograph` (#2339). Uses its own non-global RegExp instance
77
+ * (see `CJK_SEGMENTABLE_RUN_RE` above) — never call `.test()` on the
78
+ * shared, global-flagged `CJK_RUN_RE` directly.
79
+ */
80
+ export const containsSegmentableCjkRun = (text) => CJK_SEGMENTABLE_RUN_RE.test(text);
81
+ /**
82
+ * Rewrite contiguous CJK spans in `text` into space-separated overlapping
83
+ * bigrams (a run of exactly 2 chars becomes a single bigram; a lone CJK
84
+ * char has no possible pairing and passes through unchanged). Non-CJK text
85
+ * is never touched by `replace` in the first place, so a run's boundary
86
+ * spacing is decided by peeking at the *original* string's neighboring
87
+ * character (via the callback's `offset`/`full` args) rather than tracking
88
+ * state across matches — each match stays independent even when two CJK
89
+ * runs sit close together, and a space is added only when the neighbor
90
+ * isn't already whitespace, so the whitespace-splitting FTS tokenizer
91
+ * treats runs as separate tokens (`ERP审批流程` -> `ERP 审批 批流 流程`,
92
+ * not `ERP审批 批流 流程`).
93
+ */
94
+ export const segmentCjkSpans = (text) => text.replace(CJK_RUN_RE, (run, offset, full) => {
95
+ const bigrams = [];
96
+ for (let i = 0; i < run.length - 1; i++)
97
+ bigrams.push(run.slice(i, i + 2));
98
+ const before = full[offset - 1];
99
+ const after = full[offset + run.length];
100
+ const leadingSpace = before !== undefined && !WHITESPACE_RE.test(before) ? ' ' : '';
101
+ const trailingSpace = after !== undefined && !WHITESPACE_RE.test(after) ? ' ' : '';
102
+ return leadingSpace + bigrams.join(' ') + trailingSpace;
103
+ });
104
+ // ============================================================================
105
+ // GITNEXUS_FTS_CJK_SEGMENTATION — env var validation and the segmentation gate
106
+ // ============================================================================
107
+ /**
108
+ * Modes shipped by this plan. Deliberately does not include a `'jieba'`
109
+ * value: LadybugDB's native `tokenizer := 'jieba'` parameter FATAL-crashes
110
+ * the process without a bundled dictionary (no such dictionary ships with
111
+ * `@ladybugdb/core`), and `QUERY_FTS_INDEX` has no way to apply it to a query
112
+ * string anyway — see the plan's Key Technical Decision 1. Stubbing an
113
+ * unimplemented option here would misrepresent it as available.
114
+ */
115
+ const SUPPORTED_FTS_CJK_SEGMENTATION_MODES = new Set(['none', 'bigram']);
116
+ export const DEFAULT_FTS_CJK_SEGMENTATION = 'none';
117
+ /**
118
+ * True if `value` is one of the recognized segmentation modes. Callers that
119
+ * interpolate a persisted `RepoMeta.cjkSegmentation` value into agent-visible
120
+ * text (e.g. the MCP query-tool's mode-drift warning, #2339) must validate
121
+ * with this first — that field comes from `meta.json`, a schema-less
122
+ * `JSON.parse` of on-disk state inside the analyzed repo, not a trusted
123
+ * input, so an unvalidated value could otherwise be echoed verbatim into
124
+ * tool output an agent is expected to trust and act on.
125
+ */
126
+ export const isSupportedCjkSegmentationMode = (value) => typeof value === 'string' && SUPPORTED_FTS_CJK_SEGMENTATION_MODES.has(value);
127
+ let resolvedCjkSegmentation;
128
+ /** Read + validate `GITNEXUS_FTS_CJK_SEGMENTATION`. Throws on an unsupported value. */
129
+ function resolveFTSCjkSegmentation() {
130
+ const raw = process.env.GITNEXUS_FTS_CJK_SEGMENTATION?.trim().toLowerCase();
131
+ if (!raw)
132
+ return DEFAULT_FTS_CJK_SEGMENTATION;
133
+ if (SUPPORTED_FTS_CJK_SEGMENTATION_MODES.has(raw))
134
+ return raw;
135
+ throw new Error(`Invalid GITNEXUS_FTS_CJK_SEGMENTATION "${process.env.GITNEXUS_FTS_CJK_SEGMENTATION}". ` +
136
+ `Expected one of: ${[...SUPPORTED_FTS_CJK_SEGMENTATION_MODES].sort().join(', ')}.`);
137
+ }
138
+ /**
139
+ * Resolve + validate `GITNEXUS_FTS_CJK_SEGMENTATION` once, up front at analyze
140
+ * startup, and cache it — mirrors `initialiseSearchFTSStemmer` so an invalid
141
+ * value fails in milliseconds instead of partway through a run. The cached
142
+ * value is what {@link getSearchFTSCjkSegmentation} returns for the rest of
143
+ * the run, so config is read and validated in exactly one place.
144
+ */
145
+ export function initialiseSearchFTSCjkSegmentation() {
146
+ resolvedCjkSegmentation = resolveFTSCjkSegmentation();
147
+ return resolvedCjkSegmentation;
148
+ }
149
+ /**
150
+ * Return the mode resolved by {@link initialiseSearchFTSCjkSegmentation}.
151
+ * Falls back to resolving on demand when init was never called (read-only
152
+ * hosts, unit tests) so validation always applies.
153
+ */
154
+ export function getSearchFTSCjkSegmentation() {
155
+ return resolvedCjkSegmentation ?? resolveFTSCjkSegmentation();
156
+ }
157
+ /**
158
+ * Whether the CJK segmentation mode an index was built under (as persisted in
159
+ * `RepoMeta.cjkSegmentation`) differs from the mode the live process resolves
160
+ * (#2331/#2339) — used by `run-analyze.ts` to force a full rebuild on drift,
161
+ * and by the MCP query path to warn when a repo's index and the serving
162
+ * process disagree. A single scalar, so a plain equality check suffices —
163
+ * unlike `pdgModeMismatch` in `run-analyze.ts`, no key-union comparator is
164
+ * needed. An absent recorded stamp defaults to 'none' (this feature's own
165
+ * default), so a repo that never touched this feature never mismatches.
166
+ * Pure + exported for testing. Lives here (not `run-analyze.ts`) so callers
167
+ * that only need this comparator — e.g. the MCP query path — don't have to
168
+ * pull in the full analyze-pipeline module.
169
+ */
170
+ export const cjkSegmentationModeMismatch = (recorded, resolved) => (recorded ?? 'none') !== resolved;
171
+ /**
172
+ * The single entry point the write path (`csv-generator.ts`) and read path
173
+ * (`bm25-index.ts`) both call, so indexed text and query text are always
174
+ * segmented identically. No-ops when the resolved mode is `none` (default).
175
+ */
176
+ export const applyCjkSegmentationIfEnabled = (text) => getSearchFTSCjkSegmentation() === 'bigram' ? segmentCjkSpans(text) : text;
@@ -26,6 +26,7 @@ import { rankExactEmbeddingRows, } from '../../core/embeddings/exact-search.js';
26
26
  import { EMBEDDING_TABLE_NAME, EMBEDDING_INDEX_NAME } from '../../core/lbug/schema.js';
27
27
  import { getExactScanLimit, isVectorExtensionSupportedByPlatform, } from '../../core/platform/capabilities.js';
28
28
  import { PhaseTimer } from '../../core/search/phase-timer.js';
29
+ import { cjkSegmentationModeMismatch, containsSegmentableCjkRun, getSearchFTSCjkSegmentation, isSupportedCjkSegmentationMode, MAX_CJK_SEGMENTATION_QUERY_LENGTH, } from '../../core/search/cjk-segmentation.js';
29
30
  import { checkStalenessAsync, checkCwdMatch } from '../../core/git-staleness.js';
30
31
  import { logger } from '../../core/logger.js';
31
32
  import { LIST_REPOS_DEFAULT_LIMIT, LIST_REPOS_MAX_LIMIT, EXPLAIN_DEFAULT_LIMIT, EXPLAIN_MAX_LIMIT, PDG_QUERY_DEFAULT_LIMIT, PDG_QUERY_MAX_LIMIT, } from '../tools.js';
@@ -1579,6 +1580,68 @@ export class LocalBackend {
1579
1580
  if (!ftsUsed) {
1580
1581
  warnings.push('FTS indexes missing — keyword search degraded. Run: gitnexus analyze --repair-fts (or gitnexus analyze --force) to rebuild indexes.');
1581
1582
  }
1583
+ // #2331: a CJK query against a server process resolving
1584
+ // GITNEXUS_FTS_CJK_SEGMENTATION to 'none' silently misses sub-phrase
1585
+ // matches with no other signal — this is the only place an agent driving
1586
+ // GitNexus through the query tool can learn the capability exists.
1587
+ try {
1588
+ const cjkMode = getSearchFTSCjkSegmentation();
1589
+ if (containsSegmentableCjkRun(searchQuery) && cjkMode !== 'bigram') {
1590
+ warnings.push('Query contains CJK characters — sub-phrase matches require GITNEXUS_FTS_CJK_SEGMENTATION=bigram set for both `analyze` and this server process, then `gitnexus analyze --force`.');
1591
+ }
1592
+ else if (cjkMode === 'bigram' &&
1593
+ searchQuery.length > MAX_CJK_SEGMENTATION_QUERY_LENGTH &&
1594
+ containsSegmentableCjkRun(searchQuery)) {
1595
+ // #2339: bigram mode is enabled, but the query exceeds the length
1596
+ // cap that guards segmentCjkSpans's per-character allocation cost —
1597
+ // applyCjkSegmentationIfEnabled silently skips segmentation above
1598
+ // this length, so an over-cap CJK query returns zero results for
1599
+ // text that IS indexed and present verbatim, with no other signal.
1600
+ warnings.push(`Query exceeds the ${MAX_CJK_SEGMENTATION_QUERY_LENGTH}-character CJK segmentation cap — ` +
1601
+ 'sub-phrase matches are skipped for this query even though GITNEXUS_FTS_CJK_SEGMENTATION=bigram is enabled. Shorten the query to search within the cap.');
1602
+ }
1603
+ }
1604
+ catch (err) {
1605
+ // Best-effort diagnostic only — never fail the query over it.
1606
+ logQueryError('query:cjk-warning', err);
1607
+ }
1608
+ // #2339: the checks above only compare the QUERY's own content against
1609
+ // the live process's mode — they can't detect "server mode is 'bigram'
1610
+ // but the on-disk index was actually built under 'none'/legacy" (env var
1611
+ // changed without a full --force re-analyze, or a plain/--repair-fts
1612
+ // analyze ran instead). That mismatch affects every CJK query against
1613
+ // this repo, not just one whose own text happens to contain CJK, so it's
1614
+ // a separate, unconditional check — not folded into the branches above.
1615
+ try {
1616
+ const meta = await loadMeta(path.dirname(repo.lbugPath));
1617
+ // meta.json is on-disk state inside the analyzed repo, read via a
1618
+ // schema-less JSON.parse — not trusted input. Validate before
1619
+ // interpolating it into agent-visible tool output (#2339): an
1620
+ // unrecognized value is itself evidence of a corrupt/foreign index,
1621
+ // reported generically rather than echoed verbatim.
1622
+ const persistedMode = meta?.cjkSegmentation;
1623
+ if (meta && persistedMode !== undefined && !isSupportedCjkSegmentationMode(persistedMode)) {
1624
+ warnings.push("This repo's index metadata has an unrecognized CJK segmentation mode stamp — the index " +
1625
+ 'may be corrupt or from an incompatible GitNexus version. Run `gitnexus analyze --force` to rebuild it.');
1626
+ }
1627
+ else if (meta &&
1628
+ cjkSegmentationModeMismatch(meta.cjkSegmentation, getSearchFTSCjkSegmentation())) {
1629
+ warnings.push(`Index was built with CJK segmentation mode '${meta.cjkSegmentation ?? 'none'}', but this ` +
1630
+ `server is resolving '${getSearchFTSCjkSegmentation()}' — sub-phrase CJK search results ` +
1631
+ 'may be incomplete. Set GITNEXUS_FTS_CJK_SEGMENTATION to the same value for both the ' +
1632
+ '`analyze` process and this server, then run `gitnexus analyze --force` to rebuild under ' +
1633
+ "the agreed mode (do not assume the live server's mode is the one to keep — re-analyzing " +
1634
+ 'under the wrong mode can strip an already-working bigram-segmented index back to `none`).');
1635
+ }
1636
+ }
1637
+ catch (err) {
1638
+ // loadMeta() itself never throws (it returns null on any read/parse
1639
+ // failure) — the actual throw source here is getSearchFTSCjkSegmentation()
1640
+ // on an invalid env value, same root cause as the catch above. This is
1641
+ // a separate, independently-guarded diagnostic though, so it gets its
1642
+ // own log context rather than sharing 'query:cjk-warning'.
1643
+ logQueryError('query:cjk-mode-drift', err);
1644
+ }
1582
1645
  if (enrichmentDegraded) {
1583
1646
  warnings.push('Symbol enrichment partially failed — some process/cohesion/content data may be missing from these results (see server logs).');
1584
1647
  }
@@ -73,6 +73,16 @@ export interface RepoMeta {
73
73
  * full rebuild rather than risk an inconsistent incremental update.
74
74
  */
75
75
  schemaVersion?: number;
76
+ /**
77
+ * The resolved GITNEXUS_FTS_CJK_SEGMENTATION mode ('none' | 'bigram') the
78
+ * existing index's content/description columns were last written under
79
+ * (#2331/#2339). On mismatch with the live process's resolved mode,
80
+ * runFullAnalysis forces a full rebuild so indexed text and query-time
81
+ * segmentation never diverge. Always stamped (never omitted), unlike
82
+ * `pdg` below — the default 'none' is itself a meaningful value to
83
+ * compare, not an absence.
84
+ */
85
+ cjkSegmentation?: string;
76
86
  /**
77
87
  * SHA-256 of every file's content at the time of the last successful
78
88
  * indexing run. The next run computes current hashes and diffs against
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "gitnexus",
3
- "version": "1.6.9-rc.35",
3
+ "version": "1.6.9-rc.36",
4
4
  "description": "Graph-powered code intelligence for AI agents. Index any codebase, query via MCP or CLI.",
5
5
  "author": "Abhigyan Patwari",
6
6
  "license": "PolyForm-Noncommercial-1.0.0",