gitnexus 1.6.9-rc.35 → 1.6.9-rc.36
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +7 -1
- package/dist/core/lbug/csv-generator.d.ts +44 -11
- package/dist/core/lbug/csv-generator.js +39 -20
- package/dist/core/run-analyze.js +13 -0
- package/dist/core/search/bm25-index.js +19 -2
- package/dist/core/search/cjk-segmentation.d.ts +126 -0
- package/dist/core/search/cjk-segmentation.js +176 -0
- package/dist/mcp/local/local-backend.js +63 -0
- package/dist/storage/repo-manager.d.ts +10 -0
- package/package.json +1 -1
package/README.md
CHANGED
|
@@ -357,13 +357,14 @@ npm install -g gitnexus
|
|
|
357
357
|
|
|
358
358
|
GitNexus uses optional DuckDB extensions for BM25 and vector search. The `gitnexus serve` and MCP read paths only ever try to `LOAD` the extensions — they never block on a network install. The `analyze` command, by default, attempts one bounded out-of-process `INSTALL` if `LOAD` fails and proceeds even when that install times out, so the index is always written to disk; BM25/vector search degrade gracefully until the extensions become available.
|
|
359
359
|
|
|
360
|
-
Configure the behavior with
|
|
360
|
+
Configure the behavior with these environment variables:
|
|
361
361
|
|
|
362
362
|
| Variable | Values | Default | Effect |
|
|
363
363
|
| -------------------------------------------- | ---------------------------- | ------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
364
364
|
| `GITNEXUS_LBUG_EXTENSION_INSTALL` | `auto`, `load-only`, `never` | `auto` | `auto` runs one bounded INSTALL if LOAD fails. `load-only` only uses already-installed extensions (recommended for offline / firewalled environments). `never` skips optional extensions entirely. |
|
|
365
365
|
| `GITNEXUS_LBUG_EXTENSION_INSTALL_TIMEOUT_MS` | positive integer | `15000` | Wall-clock budget for the out-of-process `INSTALL` child before it is killed. |
|
|
366
366
|
| `GITNEXUS_FTS_STEMMER` | supported LadybugDB stemmer | `porter` | Stemmer used when rebuilding BM25/FTS indexes. Use `none` for CJK-heavy repositories, or a language stemmer such as `german`, `french`, or `spanish` when that better matches repository comments and identifiers. Re-run `gitnexus analyze --repair-fts` after changing it. |
|
|
367
|
+
| `GITNEXUS_FTS_CJK_SEGMENTATION` | `none`, `bigram` | `none` | `bigram` inserts overlapping character-bigram boundaries into Chinese/Japanese Han-ideograph spans in `content`/`description` before FTS indexing, so LadybugDB's space-only tokenizer can see sub-phrase word boundaries. Scoped to CJK Unified Ideographs only — Japanese Hiragana/Katakana and Korean Hangul are not currently segmented. Unlike `GITNEXUS_FTS_STEMMER`, this rewrites stored text — enabling it on an already-indexed repo requires a full `gitnexus analyze --force`; neither `--repair-fts` nor a plain incremental `analyze` applies it to previously-indexed files. Set the same value wherever `analyze` and search-serving processes (CLI query, MCP server, web server) run. |
|
|
367
368
|
| `GITNEXUS_WAL_CHECKPOINT_THRESHOLD` | integer `>= -1` | `67108864` (64 MiB) | LadybugDB WAL auto-checkpoint threshold during analyze (bytes). Auto-checkpoint remains enabled; `-1` keeps Ladybug's stock ~16 MiB. Larger thresholds reduce checkpoint frequency but increase the WAL size at rotation time — choose a smaller value on disk-constrained environments. |
|
|
368
369
|
|
|
369
370
|
```bash
|
|
@@ -375,6 +376,11 @@ GITNEXUS_LBUG_EXTENSION_INSTALL_TIMEOUT_MS=30000 npx gitnexus analyze
|
|
|
375
376
|
|
|
376
377
|
# CJK-heavy codebase: rebuild keyword indexes without English stemming
|
|
377
378
|
GITNEXUS_FTS_STEMMER=none npx gitnexus analyze --repair-fts
|
|
379
|
+
|
|
380
|
+
# CJK-heavy codebase: enable sub-phrase search over Chinese/Japanese Han text.
|
|
381
|
+
# On an already-indexed repo, the first run after enabling this MUST be --force —
|
|
382
|
+
# --repair-fts and plain incremental `analyze` both leave old files un-segmented.
|
|
383
|
+
GITNEXUS_FTS_CJK_SEGMENTATION=bigram npx gitnexus analyze --force
|
|
378
384
|
```
|
|
379
385
|
|
|
380
386
|
### Analysis runs out of memory
|
|
@@ -25,18 +25,24 @@ import { NodeTableName } from './schema.js';
|
|
|
25
25
|
* case that matters: one more oversized row lands right after the buffer was
|
|
26
26
|
* just under this threshold, before the flush fires. That row is capped at
|
|
27
27
|
* TREE_SITTER_MAX_BUFFER (32MB, hard-clamped — GITNEXUS_MAX_FILE_SIZE cannot
|
|
28
|
-
* raise it)
|
|
29
|
-
*
|
|
30
|
-
*
|
|
31
|
-
*
|
|
32
|
-
* `
|
|
28
|
+
* raise it). Two transforms can each grow it before it reaches the buffer:
|
|
29
|
+
* `applyCjkSegmentationIfEnabled` (#2331, `CJK_BIGRAM_WORST_CASE_GROWTH_FACTOR`
|
|
30
|
+
* on an all-CJK row when `GITNEXUS_FTS_CJK_SEGMENTATION=bigram` — the single
|
|
31
|
+
* source of truth for that ratio, imported by the paired test) and
|
|
32
|
+
* `escapeCSVField`'s worst-case quote-doubling (2x). So the peak joined-string
|
|
33
|
+
* size is bounded by
|
|
34
|
+
* FLUSH_BYTES + 2 * CJK_BIGRAM_WORST_CASE_GROWTH_FACTOR * TREE_SITTER_MAX_BUFFER
|
|
35
|
+
* ≈ 8MB + 149MB ≈ 157MB,
|
|
36
|
+
* versus Node's `buffer.constants.MAX_STRING_LENGTH` (~512MB) — the test
|
|
37
|
+
* actually enforces half of that (~256MB), for a ~1.63x margin (see the
|
|
33
38
|
* `shouldFlushCSVBuffer stays within the V8 string-length ceiling` test,
|
|
34
|
-
* which fails loudly if
|
|
35
|
-
* way).
|
|
36
|
-
*
|
|
37
|
-
*
|
|
38
|
-
*
|
|
39
|
-
*
|
|
39
|
+
* which fails loudly if any of these constants ever moves this margin the
|
|
40
|
+
* wrong way). With segmentation disabled (default), the old ~3.56x margin
|
|
41
|
+
* still applies. Raising FLUSH_BYTES trades fewer/larger flushes for less
|
|
42
|
+
* margin; lowering it trades the reverse for lower peak transient memory.
|
|
43
|
+
* Change the constant directly if a real workload needs a different point on
|
|
44
|
+
* that curve — a per-host env var would let the margin get silently
|
|
45
|
+
* reintroduced by an operator with no way to know why 512MB is dangerous.
|
|
40
46
|
*/
|
|
41
47
|
export declare const FLUSH_BYTES: number;
|
|
42
48
|
export declare const shouldFlushCSVBuffer: (byteCount: number) => boolean;
|
|
@@ -44,6 +50,33 @@ export declare const sanitizeUTF8: (str: string) => string;
|
|
|
44
50
|
export declare const escapeCSVField: (value: string | number | undefined | null) => string;
|
|
45
51
|
export declare const escapeCSVNumber: (value: number | undefined | null, defaultValue?: number) => string;
|
|
46
52
|
export declare const isBinaryContent: (content: string) => boolean;
|
|
53
|
+
/**
|
|
54
|
+
* Flatten newlines and tabs to single spaces for FTS-indexed text columns
|
|
55
|
+
* (`content`, `description`) — the real fix for #2317.
|
|
56
|
+
*
|
|
57
|
+
* Ladybug's full-text-search tokenizer splits ONLY on the space character —
|
|
58
|
+
* `\n`, `\r`, and `\t` are NOT token delimiters. So multiline text indexes as
|
|
59
|
+
* a handful of giant tokens (each whole line, joined across lines), and a
|
|
60
|
+
* word query matches none of them: `searchFTSFromLbug('foo')` misses a file
|
|
61
|
+
* whose content is `... \nfoo\n ...`. Removing the 10KB cap (#2333/#2317)
|
|
62
|
+
* stores the full body but leaves it unsearchable; collapsing intra-text
|
|
63
|
+
* whitespace to spaces is what actually makes every word searchable.
|
|
64
|
+
*
|
|
65
|
+
* This rewrites the STORED column too (the same value is COPYed in), so File
|
|
66
|
+
* content returned via the graph API is space-flattened — an accepted trade
|
|
67
|
+
* for making file/symbol text searchable. Leading/trailing/empty are no-ops.
|
|
68
|
+
*
|
|
69
|
+
* Callers apply `applyCjkSegmentationIfEnabled` (#2331) to the text *before*
|
|
70
|
+
* this flatten, so a CJK phrase split across a line-wrap loses its boundary
|
|
71
|
+
* bigram (run detection resets at whitespace) — an accepted limitation, see
|
|
72
|
+
* the plan's Scope Boundaries.
|
|
73
|
+
*
|
|
74
|
+
* Exported (#2339) so `bm25-index.ts`'s query path can compose it in the
|
|
75
|
+
* same order on incoming search queries, keeping index-time and query-time
|
|
76
|
+
* text transforms symmetric — a literal tab/newline in a query would
|
|
77
|
+
* otherwise fail to match whitespace-normalized indexed content.
|
|
78
|
+
*/
|
|
79
|
+
export declare const normalizeFtsText: (text: string) => string;
|
|
47
80
|
/** Canonical relationship CSV header — shared by the emit pass and the
|
|
48
81
|
* `splitRelCsvByLabelPair` differential oracle. */
|
|
49
82
|
export declare const REL_CSV_HEADER = "from,to,type,confidence,reason,step";
|
|
@@ -17,6 +17,7 @@ import path from 'path';
|
|
|
17
17
|
import { NODE_TABLES } from './schema.js';
|
|
18
18
|
import { RelPairRouter } from './rel-pair-routing.js';
|
|
19
19
|
import { parseTruthyEnv } from '../ingestion/utils/env.js';
|
|
20
|
+
import { applyCjkSegmentationIfEnabled } from '../search/cjk-segmentation.js';
|
|
20
21
|
/**
|
|
21
22
|
* Deterministic output ordering — optional (out-of-core / windowed-resolve
|
|
22
23
|
* enabler). When `GITNEXUS_SORT_GRAPH_OUTPUT` is set, nodes and relationships
|
|
@@ -42,18 +43,24 @@ const orderedRelationships = (graph, sorted) => sorted ? [...graph.iterRelations
|
|
|
42
43
|
* case that matters: one more oversized row lands right after the buffer was
|
|
43
44
|
* just under this threshold, before the flush fires. That row is capped at
|
|
44
45
|
* TREE_SITTER_MAX_BUFFER (32MB, hard-clamped — GITNEXUS_MAX_FILE_SIZE cannot
|
|
45
|
-
* raise it)
|
|
46
|
-
*
|
|
47
|
-
*
|
|
48
|
-
*
|
|
49
|
-
* `
|
|
46
|
+
* raise it). Two transforms can each grow it before it reaches the buffer:
|
|
47
|
+
* `applyCjkSegmentationIfEnabled` (#2331, `CJK_BIGRAM_WORST_CASE_GROWTH_FACTOR`
|
|
48
|
+
* on an all-CJK row when `GITNEXUS_FTS_CJK_SEGMENTATION=bigram` — the single
|
|
49
|
+
* source of truth for that ratio, imported by the paired test) and
|
|
50
|
+
* `escapeCSVField`'s worst-case quote-doubling (2x). So the peak joined-string
|
|
51
|
+
* size is bounded by
|
|
52
|
+
* FLUSH_BYTES + 2 * CJK_BIGRAM_WORST_CASE_GROWTH_FACTOR * TREE_SITTER_MAX_BUFFER
|
|
53
|
+
* ≈ 8MB + 149MB ≈ 157MB,
|
|
54
|
+
* versus Node's `buffer.constants.MAX_STRING_LENGTH` (~512MB) — the test
|
|
55
|
+
* actually enforces half of that (~256MB), for a ~1.63x margin (see the
|
|
50
56
|
* `shouldFlushCSVBuffer stays within the V8 string-length ceiling` test,
|
|
51
|
-
* which fails loudly if
|
|
52
|
-
* way).
|
|
53
|
-
*
|
|
54
|
-
*
|
|
55
|
-
*
|
|
56
|
-
*
|
|
57
|
+
* which fails loudly if any of these constants ever moves this margin the
|
|
58
|
+
* wrong way). With segmentation disabled (default), the old ~3.56x margin
|
|
59
|
+
* still applies. Raising FLUSH_BYTES trades fewer/larger flushes for less
|
|
60
|
+
* margin; lowering it trades the reverse for lower peak transient memory.
|
|
61
|
+
* Change the constant directly if a real workload needs a different point on
|
|
62
|
+
* that curve — a per-host env var would let the margin get silently
|
|
63
|
+
* reintroduced by an operator with no way to know why 512MB is dangerous.
|
|
57
64
|
*/
|
|
58
65
|
export const FLUSH_BYTES = 8 * 1024 * 1024;
|
|
59
66
|
export const shouldFlushCSVBuffer = (byteCount) => byteCount >= FLUSH_BYTES;
|
|
@@ -166,8 +173,20 @@ class FileContentCache {
|
|
|
166
173
|
* This rewrites the STORED column too (the same value is COPYed in), so File
|
|
167
174
|
* content returned via the graph API is space-flattened — an accepted trade
|
|
168
175
|
* for making file/symbol text searchable. Leading/trailing/empty are no-ops.
|
|
176
|
+
*
|
|
177
|
+
* Callers apply `applyCjkSegmentationIfEnabled` (#2331) to the text *before*
|
|
178
|
+
* this flatten, so a CJK phrase split across a line-wrap loses its boundary
|
|
179
|
+
* bigram (run detection resets at whitespace) — an accepted limitation, see
|
|
180
|
+
* the plan's Scope Boundaries.
|
|
181
|
+
*
|
|
182
|
+
* Exported (#2339) so `bm25-index.ts`'s query path can compose it in the
|
|
183
|
+
* same order on incoming search queries, keeping index-time and query-time
|
|
184
|
+
* text transforms symmetric — a literal tab/newline in a query would
|
|
185
|
+
* otherwise fail to match whitespace-normalized indexed content.
|
|
169
186
|
*/
|
|
170
|
-
const normalizeFtsText = (text) => text.replace(/[\r\n\t]+/g, ' ');
|
|
187
|
+
export const normalizeFtsText = (text) => text.replace(/[\r\n\t]+/g, ' ');
|
|
188
|
+
/** Composes both FTS-text transforms for the `description` column — one place for the six emission sites below to call, instead of repeating the composition. */
|
|
189
|
+
const formatFtsDescription = (description) => normalizeFtsText(applyCjkSegmentationIfEnabled(description));
|
|
171
190
|
const extractContent = async (node, contentCache) => {
|
|
172
191
|
const filePath = node.properties.filePath;
|
|
173
192
|
const content = await contentCache.get(filePath);
|
|
@@ -183,7 +202,7 @@ const extractContent = async (node, contentCache) => {
|
|
|
183
202
|
// and only whitespace-normalized for the tokenizer. The symbol snippet path
|
|
184
203
|
// below, by contrast, deliberately stays capped at MAX_SNIPPET.
|
|
185
204
|
if (node.label === 'File') {
|
|
186
|
-
return normalizeFtsText(content);
|
|
205
|
+
return normalizeFtsText(applyCjkSegmentationIfEnabled(content));
|
|
187
206
|
}
|
|
188
207
|
const startLine = node.properties.startLine;
|
|
189
208
|
const endLine = node.properties.endLine;
|
|
@@ -195,7 +214,7 @@ const extractContent = async (node, contentCache) => {
|
|
|
195
214
|
const snippet = lines.slice(start, end + 1).join('\n');
|
|
196
215
|
const MAX_SNIPPET = 5000;
|
|
197
216
|
const capped = snippet.length > MAX_SNIPPET ? snippet.slice(0, MAX_SNIPPET) + '\n... [truncated]' : snippet;
|
|
198
|
-
return normalizeFtsText(capped);
|
|
217
|
+
return normalizeFtsText(applyCjkSegmentationIfEnabled(capped));
|
|
199
218
|
};
|
|
200
219
|
// ============================================================================
|
|
201
220
|
// BUFFERED CSV WRITER
|
|
@@ -429,7 +448,7 @@ export const streamAllCSVsToDisk = async (graph, repoPath, csvDir, onNodePhaseCo
|
|
|
429
448
|
escapeCSVField(node.properties.name || ''),
|
|
430
449
|
escapeCSVField(node.properties.heuristicLabel || ''),
|
|
431
450
|
keywordsStr,
|
|
432
|
-
escapeCSVField(
|
|
451
|
+
escapeCSVField(formatFtsDescription(node.properties.description || '')),
|
|
433
452
|
escapeCSVField(node.properties.enrichedBy || 'heuristic'),
|
|
434
453
|
escapeCSVNumber(node.properties.cohesion, 0),
|
|
435
454
|
escapeCSVNumber(node.properties.symbolCount, 0),
|
|
@@ -461,7 +480,7 @@ export const streamAllCSVsToDisk = async (graph, repoPath, csvDir, onNodePhaseCo
|
|
|
461
480
|
escapeCSVNumber(node.properties.endLine, -1),
|
|
462
481
|
node.properties.isExported ? 'true' : 'false',
|
|
463
482
|
escapeCSVField(content),
|
|
464
|
-
escapeCSVField(
|
|
483
|
+
escapeCSVField(formatFtsDescription(node.properties.description || '')),
|
|
465
484
|
escapeCSVNumber(node.properties.parameterCount, 0),
|
|
466
485
|
escapeCSVField(node.properties.returnType || ''),
|
|
467
486
|
].join(','));
|
|
@@ -477,7 +496,7 @@ export const streamAllCSVsToDisk = async (graph, repoPath, csvDir, onNodePhaseCo
|
|
|
477
496
|
escapeCSVNumber(node.properties.endLine, -1),
|
|
478
497
|
escapeCSVNumber(node.properties.level, 1),
|
|
479
498
|
escapeCSVField(content),
|
|
480
|
-
escapeCSVField(
|
|
499
|
+
escapeCSVField(formatFtsDescription(node.properties.description || '')),
|
|
481
500
|
].join(','));
|
|
482
501
|
break;
|
|
483
502
|
}
|
|
@@ -507,7 +526,7 @@ export const streamAllCSVsToDisk = async (graph, repoPath, csvDir, onNodePhaseCo
|
|
|
507
526
|
escapeCSVField(node.id),
|
|
508
527
|
escapeCSVField(node.properties.name || ''),
|
|
509
528
|
escapeCSVField(node.properties.filePath || ''),
|
|
510
|
-
escapeCSVField(
|
|
529
|
+
escapeCSVField(formatFtsDescription(node.properties.description || '')),
|
|
511
530
|
].join(','));
|
|
512
531
|
break;
|
|
513
532
|
case 'BasicBlock':
|
|
@@ -526,7 +545,7 @@ export const streamAllCSVsToDisk = async (graph, repoPath, csvDir, onNodePhaseCo
|
|
|
526
545
|
escapeCSVNumber(node.properties.endLine, -1),
|
|
527
546
|
node.properties.isExported ? 'true' : 'false',
|
|
528
547
|
escapeCSVField(content),
|
|
529
|
-
escapeCSVField(
|
|
548
|
+
escapeCSVField(formatFtsDescription(node.properties.description || '')),
|
|
530
549
|
].join(','));
|
|
531
550
|
}
|
|
532
551
|
else {
|
|
@@ -541,7 +560,7 @@ export const streamAllCSVsToDisk = async (graph, repoPath, csvDir, onNodePhaseCo
|
|
|
541
560
|
escapeCSVNumber(node.properties.startLine, -1),
|
|
542
561
|
escapeCSVNumber(node.properties.endLine, -1),
|
|
543
562
|
escapeCSVField(content),
|
|
544
|
-
escapeCSVField(
|
|
563
|
+
escapeCSVField(formatFtsDescription(node.properties.description || '')),
|
|
545
564
|
...(node.label === 'Property'
|
|
546
565
|
? [escapeCSVField(node.properties.declaredType || '')]
|
|
547
566
|
: []),
|
package/dist/core/run-analyze.js
CHANGED
|
@@ -15,6 +15,7 @@ import { runPipelineFromRepo } from './ingestion/pipeline.js';
|
|
|
15
15
|
import { resetDegradedParseCounter } from './tree-sitter/safe-parse.js';
|
|
16
16
|
import { initLbug, loadGraphToLbug, getLbugStats, executeQuery, executeWithReusedStatement, closeLbug, closeLbugBeforeExit, loadCachedEmbeddings, deleteNodesForFile, deleteAllCommunitiesAndProcesses, deleteAllInterprocTaintPaths, deleteAllCallSummaries, queryImporters, loadFTSExtension, } from './lbug/lbug-adapter.js';
|
|
17
17
|
import { createSearchFTSIndexes, initialiseSearchFTSStemmer, verifySearchFTSIndexes, } from './search/fts-indexes.js';
|
|
18
|
+
import { cjkSegmentationModeMismatch, getSearchFTSCjkSegmentation, initialiseSearchFTSCjkSegmentation, } from './search/cjk-segmentation.js';
|
|
18
19
|
import { resolveAnalyzeInstallPolicy } from './lbug/extension-loader.js';
|
|
19
20
|
import { startWalCheckpointDriver, } from './lbug/wal-checkpoint-driver.js';
|
|
20
21
|
import { getStoragePaths, resolveBranchPlacement, saveMeta, loadMeta, ensureGitNexusIgnored, registerRepo, isRepoRegistered, cleanupOldKuzuFiles, INCREMENTAL_SCHEMA_VERSION, } from '../storage/repo-manager.js';
|
|
@@ -259,6 +260,7 @@ export async function runFullAnalysis(repoPath, options, callbacks) {
|
|
|
259
260
|
// parse/load phases. A typo fails here in ms; createSearchFTSIndexes reuses
|
|
260
261
|
// the cached value via getSearchFTSStemmer.
|
|
261
262
|
initialiseSearchFTSStemmer();
|
|
263
|
+
initialiseSearchFTSCjkSegmentation();
|
|
262
264
|
// Scope the degraded-parse log throttle to this run. On a reused process
|
|
263
265
|
// (e.g. tests, or any host that calls runFullAnalysis more than once) the
|
|
264
266
|
// module-level counter would otherwise stay saturated and suppress every
|
|
@@ -459,6 +461,13 @@ export async function runFullAnalysis(repoPath, options, callbacks) {
|
|
|
459
461
|
`forcing a full rebuild so persisted rows match the current schema.`);
|
|
460
462
|
options = { ...options, force: true };
|
|
461
463
|
}
|
|
464
|
+
if (existingMeta &&
|
|
465
|
+
cjkSegmentationModeMismatch(existingMeta.cjkSegmentation, getSearchFTSCjkSegmentation())) {
|
|
466
|
+
log(`CJK segmentation mode changed (index built with '${existingMeta.cjkSegmentation ?? 'none'}', ` +
|
|
467
|
+
`this run resolves '${getSearchFTSCjkSegmentation()}'); forcing a full rebuild so indexed ` +
|
|
468
|
+
`text and query-time segmentation stay in sync.`);
|
|
469
|
+
options = { ...options, force: true };
|
|
470
|
+
}
|
|
462
471
|
// ── Early-return: already up to date ──────────────────────────────
|
|
463
472
|
if (existingMeta && !options.force && existingMeta.lastCommit === currentCommit) {
|
|
464
473
|
// Non-git folders have currentCommit = '' — always rebuild since we can't detect changes
|
|
@@ -1074,6 +1083,10 @@ export async function runFullAnalysis(repoPath, options, callbacks) {
|
|
|
1074
1083
|
// incrementalInProgress to undefined explicitly clears any prior
|
|
1075
1084
|
// dirty flag (full and incremental success paths converge here).
|
|
1076
1085
|
schemaVersion: hasGitDir(repoPath) ? INCREMENTAL_SCHEMA_VERSION : undefined,
|
|
1086
|
+
// Always stamped with the live resolved mode (#2331/#2339) — unlike
|
|
1087
|
+
// `pdg` below, 'none' is a meaningful value to compare, not an
|
|
1088
|
+
// absence, so this is never conditionally omitted.
|
|
1089
|
+
cjkSegmentation: getSearchFTSCjkSegmentation(),
|
|
1077
1090
|
fileHashes: hasGitDir(repoPath) ? newFileHashesRecord : undefined,
|
|
1078
1091
|
// This branch's full live chunk-key set (#2106 R6). `usedKeys` is every
|
|
1079
1092
|
// chunk hash touched in this scan — cache HITS included (see parse-impl
|
|
@@ -5,7 +5,9 @@
|
|
|
5
5
|
* Always reads from the database (no cached state to drift).
|
|
6
6
|
*/
|
|
7
7
|
import { queryFTS } from '../lbug/lbug-adapter.js';
|
|
8
|
+
import { normalizeFtsText } from '../lbug/csv-generator.js';
|
|
8
9
|
import { FTS_INDEXES } from './fts-schema.js';
|
|
10
|
+
import { applyCjkSegmentationIfEnabled, MAX_CJK_SEGMENTATION_QUERY_LENGTH, } from './cjk-segmentation.js';
|
|
9
11
|
/**
|
|
10
12
|
* Execute a single FTS query via a custom executor (for MCP connection pool).
|
|
11
13
|
* Returns `null` when the query fails (e.g. FTS index does not exist) so the
|
|
@@ -46,6 +48,21 @@ async function queryFTSViaExecutor(executor, tableName, indexName, query, limit)
|
|
|
46
48
|
* @returns Ranked search results from FTS indexes
|
|
47
49
|
*/
|
|
48
50
|
export const searchFTSFromLbug = async (query, limit = 20, repoId) => {
|
|
51
|
+
// Applied once, up front, so every downstream branch searches with the
|
|
52
|
+
// same text the index was built from (#2331/#2339) — index-time and
|
|
53
|
+
// query-time text transforms must never diverge, since QUERY_FTS_INDEX
|
|
54
|
+
// cannot derive a tokenizer from the index it queries. Composed in the
|
|
55
|
+
// SAME order as the write path (csv-generator.ts's formatFtsDescription /
|
|
56
|
+
// extractContent: normalizeFtsText(applyCjkSegmentationIfEnabled(text))):
|
|
57
|
+
// CJK segmentation is no-op when disabled (default); normalizeFtsText
|
|
58
|
+
// (collapsing \r\n\t to a space) applies unconditionally — it has no
|
|
59
|
+
// per-character cost concern, unlike CJK segmentation, so it's not gated
|
|
60
|
+
// by the length cap. Segmentation itself is skipped for pathologically
|
|
61
|
+
// long queries (see MAX_CJK_SEGMENTATION_QUERY_LENGTH) — the query still
|
|
62
|
+
// searches correctly, just without CJK sub-phrase segmentation.
|
|
63
|
+
const searchQuery = normalizeFtsText(query.length <= MAX_CJK_SEGMENTATION_QUERY_LENGTH
|
|
64
|
+
? applyCjkSegmentationIfEnabled(query)
|
|
65
|
+
: query);
|
|
49
66
|
const resultsByIndex = [];
|
|
50
67
|
let queriesSucceeded = 0;
|
|
51
68
|
if (repoId) {
|
|
@@ -56,7 +73,7 @@ export const searchFTSFromLbug = async (query, limit = 20, repoId) => {
|
|
|
56
73
|
const { executeParameterized } = poolMod;
|
|
57
74
|
const executor = (cypher, params) => executeParameterized(repoId, cypher, params);
|
|
58
75
|
for (const { table, indexName } of FTS_INDEXES) {
|
|
59
|
-
const result = await queryFTSViaExecutor(executor, table, indexName,
|
|
76
|
+
const result = await queryFTSViaExecutor(executor, table, indexName, searchQuery, limit);
|
|
60
77
|
if (result !== null) {
|
|
61
78
|
queriesSucceeded++;
|
|
62
79
|
resultsByIndex.push(result);
|
|
@@ -67,7 +84,7 @@ export const searchFTSFromLbug = async (query, limit = 20, repoId) => {
|
|
|
67
84
|
// Use core lbug adapter (CLI / pipeline context) — also sequential for safety.
|
|
68
85
|
for (const { table, indexName } of FTS_INDEXES) {
|
|
69
86
|
try {
|
|
70
|
-
const result = await queryFTS(table, indexName,
|
|
87
|
+
const result = await queryFTS(table, indexName, searchQuery, limit, false);
|
|
71
88
|
queriesSucceeded++;
|
|
72
89
|
resultsByIndex.push(result);
|
|
73
90
|
}
|
|
@@ -0,0 +1,126 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* CJK bigram segmentation for FTS search (#2331)
|
|
3
|
+
*
|
|
4
|
+
* LadybugDB's bundled FTS tokenizer splits only on the space character, so a
|
|
5
|
+
* contiguous CJK (Chinese/Japanese/Korean) span indexes as one giant token and
|
|
6
|
+
* sub-phrase queries never match. `segmentCjkSpans` addresses the Han-ideograph
|
|
7
|
+
* case (Chinese text and Japanese Kanji — see scope note below) by rewriting
|
|
8
|
+
* each contiguous run of CJK Unified Ideographs into space-separated overlapping character
|
|
9
|
+
* bigrams (`采购订单` -> `采购 购订 订单`), the same technique MySQL's `ngram`
|
|
10
|
+
* fulltext parser, Elasticsearch's `cjk` analyzer, and Lucene's
|
|
11
|
+
* `CJKBigramFilter` use by default. For any exact contiguous substring query
|
|
12
|
+
* of length >= 2, its bigram decomposition is a subset of the source text's
|
|
13
|
+
* bigram decomposition, so sub-phrase matching works without needing a
|
|
14
|
+
* dictionary or boundary-alignment luck.
|
|
15
|
+
*
|
|
16
|
+
* Scoped to the core CJK Unified Ideographs block (U+4E00-U+9FFF) only —
|
|
17
|
+
* covers Chinese text and Japanese Kanji. Hiragana, Katakana, and Hangul
|
|
18
|
+
* Syllables are deliberately excluded for now (see plan Scope Boundaries);
|
|
19
|
+
* extend `CJK_UNIFIED_IDEOGRAPHS` below if that need arises.
|
|
20
|
+
*/
|
|
21
|
+
/**
|
|
22
|
+
* Worst-case output/input byte ratio for `segmentCjkSpans` on an all-CJK run:
|
|
23
|
+
* each adjacent character pair becomes a 2-character bigram plus a 1-byte
|
|
24
|
+
* separator, i.e. ~7 output bytes per 3 input bytes of UTF-8 CJK text (each
|
|
25
|
+
* CJK character is 3 bytes). Single source of truth — imported by both the
|
|
26
|
+
* CSV-flush safety-margin test (`csv-pipeline.test.ts`) and the growth-factor
|
|
27
|
+
* regression guard (`cjk-segmentation.test.ts`), and referenced by name in
|
|
28
|
+
* `csv-generator.ts`'s `FLUSH_BYTES` margin comment, so all three stay in
|
|
29
|
+
* sync if the algorithm's expansion ratio ever changes.
|
|
30
|
+
*/
|
|
31
|
+
export declare const CJK_BIGRAM_WORST_CASE_GROWTH_FACTOR: number;
|
|
32
|
+
/**
|
|
33
|
+
* A real search query is always a short phrase — unlike indexed File content
|
|
34
|
+
* (deliberately uncapped, #2317/#2323), nothing else bounds a query's length
|
|
35
|
+
* before it reaches `segmentCjkSpans`. Without a cap, a pathologically long
|
|
36
|
+
* query string (accidental or adversarial) would pay `segmentCjkSpans`'s
|
|
37
|
+
* per-character allocation cost on every search request. 2000 characters
|
|
38
|
+
* comfortably covers any real natural-language query.
|
|
39
|
+
*
|
|
40
|
+
* Lives here rather than in `bm25-index.ts` (its only other consumer) so
|
|
41
|
+
* `local-backend.ts` can import it statically alongside this module's other
|
|
42
|
+
* symbols — `bm25-index.ts` transitively imports `@ladybugdb/core` (a native
|
|
43
|
+
* binding, via `lbug-adapter.js`), which is exactly the kind of module
|
|
44
|
+
* `local-backend.ts`'s `bm25Search` deliberately dynamic-imports instead of
|
|
45
|
+
* statically (#1489: can fail in sandboxed MCP contexts). A static import of
|
|
46
|
+
* even one constant from `bm25-index.ts` would force that native binding to
|
|
47
|
+
* load at MCP-server startup instead of at first query.
|
|
48
|
+
*/
|
|
49
|
+
export declare const MAX_CJK_SEGMENTATION_QUERY_LENGTH = 2000;
|
|
50
|
+
/**
|
|
51
|
+
* True if `text` contains at least one CJK Unified Ideograph, including a
|
|
52
|
+
* single character. Generic presence check — for gating the "enable bigram
|
|
53
|
+
* mode" query warning specifically, use {@link containsSegmentableCjkRun}
|
|
54
|
+
* instead (#2339): a lone CJK character can never be bigram-segmented, so
|
|
55
|
+
* this broader check would misleadingly flag queries bigram mode can't help.
|
|
56
|
+
*/
|
|
57
|
+
export declare const containsCjkIdeograph: (text: string) => boolean;
|
|
58
|
+
/**
|
|
59
|
+
* True if `text` contains a CJK run of 2+ contiguous ideographs —
|
|
60
|
+
* i.e. a span `segmentCjkSpans` can actually bigram-segment. A lone CJK
|
|
61
|
+
* character can never be segmented (no possible pairing), so callers
|
|
62
|
+
* warning "enable bigram mode" for a query should gate on this, not on
|
|
63
|
+
* `containsCjkIdeograph` (#2339). Uses its own non-global RegExp instance
|
|
64
|
+
* (see `CJK_SEGMENTABLE_RUN_RE` above) — never call `.test()` on the
|
|
65
|
+
* shared, global-flagged `CJK_RUN_RE` directly.
|
|
66
|
+
*/
|
|
67
|
+
export declare const containsSegmentableCjkRun: (text: string) => boolean;
|
|
68
|
+
/**
|
|
69
|
+
* Rewrite contiguous CJK spans in `text` into space-separated overlapping
|
|
70
|
+
* bigrams (a run of exactly 2 chars becomes a single bigram; a lone CJK
|
|
71
|
+
* char has no possible pairing and passes through unchanged). Non-CJK text
|
|
72
|
+
* is never touched by `replace` in the first place, so a run's boundary
|
|
73
|
+
* spacing is decided by peeking at the *original* string's neighboring
|
|
74
|
+
* character (via the callback's `offset`/`full` args) rather than tracking
|
|
75
|
+
* state across matches — each match stays independent even when two CJK
|
|
76
|
+
* runs sit close together, and a space is added only when the neighbor
|
|
77
|
+
* isn't already whitespace, so the whitespace-splitting FTS tokenizer
|
|
78
|
+
* treats runs as separate tokens (`ERP审批流程` -> `ERP 审批 批流 流程`,
|
|
79
|
+
* not `ERP审批 批流 流程`).
|
|
80
|
+
*/
|
|
81
|
+
export declare const segmentCjkSpans: (text: string) => string;
|
|
82
|
+
export declare const DEFAULT_FTS_CJK_SEGMENTATION = "none";
|
|
83
|
+
/**
|
|
84
|
+
* True if `value` is one of the recognized segmentation modes. Callers that
|
|
85
|
+
* interpolate a persisted `RepoMeta.cjkSegmentation` value into agent-visible
|
|
86
|
+
* text (e.g. the MCP query-tool's mode-drift warning, #2339) must validate
|
|
87
|
+
* with this first — that field comes from `meta.json`, a schema-less
|
|
88
|
+
* `JSON.parse` of on-disk state inside the analyzed repo, not a trusted
|
|
89
|
+
* input, so an unvalidated value could otherwise be echoed verbatim into
|
|
90
|
+
* tool output an agent is expected to trust and act on.
|
|
91
|
+
*/
|
|
92
|
+
export declare const isSupportedCjkSegmentationMode: (value: unknown) => value is string;
|
|
93
|
+
/**
|
|
94
|
+
* Resolve + validate `GITNEXUS_FTS_CJK_SEGMENTATION` once, up front at analyze
|
|
95
|
+
* startup, and cache it — mirrors `initialiseSearchFTSStemmer` so an invalid
|
|
96
|
+
* value fails in milliseconds instead of partway through a run. The cached
|
|
97
|
+
* value is what {@link getSearchFTSCjkSegmentation} returns for the rest of
|
|
98
|
+
* the run, so config is read and validated in exactly one place.
|
|
99
|
+
*/
|
|
100
|
+
export declare function initialiseSearchFTSCjkSegmentation(): string;
|
|
101
|
+
/**
|
|
102
|
+
* Return the mode resolved by {@link initialiseSearchFTSCjkSegmentation}.
|
|
103
|
+
* Falls back to resolving on demand when init was never called (read-only
|
|
104
|
+
* hosts, unit tests) so validation always applies.
|
|
105
|
+
*/
|
|
106
|
+
export declare function getSearchFTSCjkSegmentation(): string;
|
|
107
|
+
/**
|
|
108
|
+
* Whether the CJK segmentation mode an index was built under (as persisted in
|
|
109
|
+
* `RepoMeta.cjkSegmentation`) differs from the mode the live process resolves
|
|
110
|
+
* (#2331/#2339) — used by `run-analyze.ts` to force a full rebuild on drift,
|
|
111
|
+
* and by the MCP query path to warn when a repo's index and the serving
|
|
112
|
+
* process disagree. A single scalar, so a plain equality check suffices —
|
|
113
|
+
* unlike `pdgModeMismatch` in `run-analyze.ts`, no key-union comparator is
|
|
114
|
+
* needed. An absent recorded stamp defaults to 'none' (this feature's own
|
|
115
|
+
* default), so a repo that never touched this feature never mismatches.
|
|
116
|
+
* Pure + exported for testing. Lives here (not `run-analyze.ts`) so callers
|
|
117
|
+
* that only need this comparator — e.g. the MCP query path — don't have to
|
|
118
|
+
* pull in the full analyze-pipeline module.
|
|
119
|
+
*/
|
|
120
|
+
export declare const cjkSegmentationModeMismatch: (recorded: string | undefined, resolved: string) => boolean;
|
|
121
|
+
/**
|
|
122
|
+
* The single entry point the write path (`csv-generator.ts`) and read path
|
|
123
|
+
* (`bm25-index.ts`) both call, so indexed text and query text are always
|
|
124
|
+
* segmented identically. No-ops when the resolved mode is `none` (default).
|
|
125
|
+
*/
|
|
126
|
+
export declare const applyCjkSegmentationIfEnabled: (text: string) => string;
|
|
@@ -0,0 +1,176 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* CJK bigram segmentation for FTS search (#2331)
|
|
3
|
+
*
|
|
4
|
+
* LadybugDB's bundled FTS tokenizer splits only on the space character, so a
|
|
5
|
+
* contiguous CJK (Chinese/Japanese/Korean) span indexes as one giant token and
|
|
6
|
+
* sub-phrase queries never match. `segmentCjkSpans` addresses the Han-ideograph
|
|
7
|
+
* case (Chinese text and Japanese Kanji — see scope note below) by rewriting
|
|
8
|
+
* each contiguous run of CJK Unified Ideographs into space-separated overlapping character
|
|
9
|
+
* bigrams (`采购订单` -> `采购 购订 订单`), the same technique MySQL's `ngram`
|
|
10
|
+
* fulltext parser, Elasticsearch's `cjk` analyzer, and Lucene's
|
|
11
|
+
* `CJKBigramFilter` use by default. For any exact contiguous substring query
|
|
12
|
+
* of length >= 2, its bigram decomposition is a subset of the source text's
|
|
13
|
+
* bigram decomposition, so sub-phrase matching works without needing a
|
|
14
|
+
* dictionary or boundary-alignment luck.
|
|
15
|
+
*
|
|
16
|
+
* Scoped to the core CJK Unified Ideographs block (U+4E00-U+9FFF) only —
|
|
17
|
+
* covers Chinese text and Japanese Kanji. Hiragana, Katakana, and Hangul
|
|
18
|
+
* Syllables are deliberately excluded for now (see plan Scope Boundaries);
|
|
19
|
+
* extend `CJK_UNIFIED_IDEOGRAPHS` below if that need arises.
|
|
20
|
+
*/
|
|
21
|
+
/** The core CJK Unified Ideographs block — single source of truth for both regexes below. */
|
|
22
|
+
const CJK_UNIFIED_IDEOGRAPHS = '[\\u4e00-\\u9fff]';
|
|
23
|
+
const CJK_CHAR_RE = new RegExp(CJK_UNIFIED_IDEOGRAPHS);
|
|
24
|
+
const CJK_RUN_RE = new RegExp(`${CJK_UNIFIED_IDEOGRAPHS}{2,}`, 'g');
|
|
25
|
+
// Same pattern as CJK_RUN_RE, but WITHOUT the 'g' flag — kept as a separate
|
|
26
|
+
// instance deliberately. RegExp.prototype.test() on a global-flagged regex
|
|
27
|
+
// is stateful (mutates lastIndex between calls); CJK_RUN_RE only stays safe
|
|
28
|
+
// today because its one consumer (segmentCjkSpans) drives it exclusively via
|
|
29
|
+
// String.prototype.replace, which always resets matching from index 0. A
|
|
30
|
+
// second consumer calling .test() on that same shared instance would leak
|
|
31
|
+
// state across calls (and across requests, in a long-lived process).
|
|
32
|
+
const CJK_SEGMENTABLE_RUN_RE = new RegExp(`${CJK_UNIFIED_IDEOGRAPHS}{2,}`);
|
|
33
|
+
const WHITESPACE_RE = /\s/;
|
|
34
|
+
/**
|
|
35
|
+
* Worst-case output/input byte ratio for `segmentCjkSpans` on an all-CJK run:
|
|
36
|
+
* each adjacent character pair becomes a 2-character bigram plus a 1-byte
|
|
37
|
+
* separator, i.e. ~7 output bytes per 3 input bytes of UTF-8 CJK text (each
|
|
38
|
+
* CJK character is 3 bytes). Single source of truth — imported by both the
|
|
39
|
+
* CSV-flush safety-margin test (`csv-pipeline.test.ts`) and the growth-factor
|
|
40
|
+
* regression guard (`cjk-segmentation.test.ts`), and referenced by name in
|
|
41
|
+
* `csv-generator.ts`'s `FLUSH_BYTES` margin comment, so all three stay in
|
|
42
|
+
* sync if the algorithm's expansion ratio ever changes.
|
|
43
|
+
*/
|
|
44
|
+
export const CJK_BIGRAM_WORST_CASE_GROWTH_FACTOR = 7 / 3;
|
|
45
|
+
/**
|
|
46
|
+
* A real search query is always a short phrase — unlike indexed File content
|
|
47
|
+
* (deliberately uncapped, #2317/#2323), nothing else bounds a query's length
|
|
48
|
+
* before it reaches `segmentCjkSpans`. Without a cap, a pathologically long
|
|
49
|
+
* query string (accidental or adversarial) would pay `segmentCjkSpans`'s
|
|
50
|
+
* per-character allocation cost on every search request. 2000 characters
|
|
51
|
+
* comfortably covers any real natural-language query.
|
|
52
|
+
*
|
|
53
|
+
* Lives here rather than in `bm25-index.ts` (its only other consumer) so
|
|
54
|
+
* `local-backend.ts` can import it statically alongside this module's other
|
|
55
|
+
* symbols — `bm25-index.ts` transitively imports `@ladybugdb/core` (a native
|
|
56
|
+
* binding, via `lbug-adapter.js`), which is exactly the kind of module
|
|
57
|
+
* `local-backend.ts`'s `bm25Search` deliberately dynamic-imports instead of
|
|
58
|
+
* statically (#1489: can fail in sandboxed MCP contexts). A static import of
|
|
59
|
+
* even one constant from `bm25-index.ts` would force that native binding to
|
|
60
|
+
* load at MCP-server startup instead of at first query.
|
|
61
|
+
*/
|
|
62
|
+
export const MAX_CJK_SEGMENTATION_QUERY_LENGTH = 2000;
|
|
63
|
+
/**
|
|
64
|
+
* True if `text` contains at least one CJK Unified Ideograph, including a
|
|
65
|
+
* single character. Generic presence check — for gating the "enable bigram
|
|
66
|
+
* mode" query warning specifically, use {@link containsSegmentableCjkRun}
|
|
67
|
+
* instead (#2339): a lone CJK character can never be bigram-segmented, so
|
|
68
|
+
* this broader check would misleadingly flag queries bigram mode can't help.
|
|
69
|
+
*/
|
|
70
|
+
export const containsCjkIdeograph = (text) => CJK_CHAR_RE.test(text);
|
|
71
|
+
/**
|
|
72
|
+
* True if `text` contains a CJK run of 2+ contiguous ideographs —
|
|
73
|
+
* i.e. a span `segmentCjkSpans` can actually bigram-segment. A lone CJK
|
|
74
|
+
* character can never be segmented (no possible pairing), so callers
|
|
75
|
+
* warning "enable bigram mode" for a query should gate on this, not on
|
|
76
|
+
* `containsCjkIdeograph` (#2339). Uses its own non-global RegExp instance
|
|
77
|
+
* (see `CJK_SEGMENTABLE_RUN_RE` above) — never call `.test()` on the
|
|
78
|
+
* shared, global-flagged `CJK_RUN_RE` directly.
|
|
79
|
+
*/
|
|
80
|
+
export const containsSegmentableCjkRun = (text) => CJK_SEGMENTABLE_RUN_RE.test(text);
|
|
81
|
+
/**
|
|
82
|
+
* Rewrite contiguous CJK spans in `text` into space-separated overlapping
|
|
83
|
+
* bigrams (a run of exactly 2 chars becomes a single bigram; a lone CJK
|
|
84
|
+
* char has no possible pairing and passes through unchanged). Non-CJK text
|
|
85
|
+
* is never touched by `replace` in the first place, so a run's boundary
|
|
86
|
+
* spacing is decided by peeking at the *original* string's neighboring
|
|
87
|
+
* character (via the callback's `offset`/`full` args) rather than tracking
|
|
88
|
+
* state across matches — each match stays independent even when two CJK
|
|
89
|
+
* runs sit close together, and a space is added only when the neighbor
|
|
90
|
+
* isn't already whitespace, so the whitespace-splitting FTS tokenizer
|
|
91
|
+
* treats runs as separate tokens (`ERP审批流程` -> `ERP 审批 批流 流程`,
|
|
92
|
+
* not `ERP审批 批流 流程`).
|
|
93
|
+
*/
|
|
94
|
+
export const segmentCjkSpans = (text) => text.replace(CJK_RUN_RE, (run, offset, full) => {
|
|
95
|
+
const bigrams = [];
|
|
96
|
+
for (let i = 0; i < run.length - 1; i++)
|
|
97
|
+
bigrams.push(run.slice(i, i + 2));
|
|
98
|
+
const before = full[offset - 1];
|
|
99
|
+
const after = full[offset + run.length];
|
|
100
|
+
const leadingSpace = before !== undefined && !WHITESPACE_RE.test(before) ? ' ' : '';
|
|
101
|
+
const trailingSpace = after !== undefined && !WHITESPACE_RE.test(after) ? ' ' : '';
|
|
102
|
+
return leadingSpace + bigrams.join(' ') + trailingSpace;
|
|
103
|
+
});
|
|
104
|
+
// ============================================================================
|
|
105
|
+
// GITNEXUS_FTS_CJK_SEGMENTATION — env var validation and the segmentation gate
|
|
106
|
+
// ============================================================================
|
|
107
|
+
/**
|
|
108
|
+
* Modes shipped by this plan. Deliberately does not include a `'jieba'`
|
|
109
|
+
* value: LadybugDB's native `tokenizer := 'jieba'` parameter FATAL-crashes
|
|
110
|
+
* the process without a bundled dictionary (no such dictionary ships with
|
|
111
|
+
* `@ladybugdb/core`), and `QUERY_FTS_INDEX` has no way to apply it to a query
|
|
112
|
+
* string anyway — see the plan's Key Technical Decision 1. Stubbing an
|
|
113
|
+
* unimplemented option here would misrepresent it as available.
|
|
114
|
+
*/
|
|
115
|
+
const SUPPORTED_FTS_CJK_SEGMENTATION_MODES = new Set(['none', 'bigram']);
|
|
116
|
+
export const DEFAULT_FTS_CJK_SEGMENTATION = 'none';
|
|
117
|
+
/**
|
|
118
|
+
* True if `value` is one of the recognized segmentation modes. Callers that
|
|
119
|
+
* interpolate a persisted `RepoMeta.cjkSegmentation` value into agent-visible
|
|
120
|
+
* text (e.g. the MCP query-tool's mode-drift warning, #2339) must validate
|
|
121
|
+
* with this first — that field comes from `meta.json`, a schema-less
|
|
122
|
+
* `JSON.parse` of on-disk state inside the analyzed repo, not a trusted
|
|
123
|
+
* input, so an unvalidated value could otherwise be echoed verbatim into
|
|
124
|
+
* tool output an agent is expected to trust and act on.
|
|
125
|
+
*/
|
|
126
|
+
export const isSupportedCjkSegmentationMode = (value) => typeof value === 'string' && SUPPORTED_FTS_CJK_SEGMENTATION_MODES.has(value);
|
|
127
|
+
let resolvedCjkSegmentation;
|
|
128
|
+
/** Read + validate `GITNEXUS_FTS_CJK_SEGMENTATION`. Throws on an unsupported value. */
|
|
129
|
+
function resolveFTSCjkSegmentation() {
|
|
130
|
+
const raw = process.env.GITNEXUS_FTS_CJK_SEGMENTATION?.trim().toLowerCase();
|
|
131
|
+
if (!raw)
|
|
132
|
+
return DEFAULT_FTS_CJK_SEGMENTATION;
|
|
133
|
+
if (SUPPORTED_FTS_CJK_SEGMENTATION_MODES.has(raw))
|
|
134
|
+
return raw;
|
|
135
|
+
throw new Error(`Invalid GITNEXUS_FTS_CJK_SEGMENTATION "${process.env.GITNEXUS_FTS_CJK_SEGMENTATION}". ` +
|
|
136
|
+
`Expected one of: ${[...SUPPORTED_FTS_CJK_SEGMENTATION_MODES].sort().join(', ')}.`);
|
|
137
|
+
}
|
|
138
|
+
/**
|
|
139
|
+
* Resolve + validate `GITNEXUS_FTS_CJK_SEGMENTATION` once, up front at analyze
|
|
140
|
+
* startup, and cache it — mirrors `initialiseSearchFTSStemmer` so an invalid
|
|
141
|
+
* value fails in milliseconds instead of partway through a run. The cached
|
|
142
|
+
* value is what {@link getSearchFTSCjkSegmentation} returns for the rest of
|
|
143
|
+
* the run, so config is read and validated in exactly one place.
|
|
144
|
+
*/
|
|
145
|
+
export function initialiseSearchFTSCjkSegmentation() {
|
|
146
|
+
resolvedCjkSegmentation = resolveFTSCjkSegmentation();
|
|
147
|
+
return resolvedCjkSegmentation;
|
|
148
|
+
}
|
|
149
|
+
/**
|
|
150
|
+
* Return the mode resolved by {@link initialiseSearchFTSCjkSegmentation}.
|
|
151
|
+
* Falls back to resolving on demand when init was never called (read-only
|
|
152
|
+
* hosts, unit tests) so validation always applies.
|
|
153
|
+
*/
|
|
154
|
+
export function getSearchFTSCjkSegmentation() {
|
|
155
|
+
return resolvedCjkSegmentation ?? resolveFTSCjkSegmentation();
|
|
156
|
+
}
|
|
157
|
+
/**
|
|
158
|
+
* Whether the CJK segmentation mode an index was built under (as persisted in
|
|
159
|
+
* `RepoMeta.cjkSegmentation`) differs from the mode the live process resolves
|
|
160
|
+
* (#2331/#2339) — used by `run-analyze.ts` to force a full rebuild on drift,
|
|
161
|
+
* and by the MCP query path to warn when a repo's index and the serving
|
|
162
|
+
* process disagree. A single scalar, so a plain equality check suffices —
|
|
163
|
+
* unlike `pdgModeMismatch` in `run-analyze.ts`, no key-union comparator is
|
|
164
|
+
* needed. An absent recorded stamp defaults to 'none' (this feature's own
|
|
165
|
+
* default), so a repo that never touched this feature never mismatches.
|
|
166
|
+
* Pure + exported for testing. Lives here (not `run-analyze.ts`) so callers
|
|
167
|
+
* that only need this comparator — e.g. the MCP query path — don't have to
|
|
168
|
+
* pull in the full analyze-pipeline module.
|
|
169
|
+
*/
|
|
170
|
+
export const cjkSegmentationModeMismatch = (recorded, resolved) => (recorded ?? 'none') !== resolved;
|
|
171
|
+
/**
|
|
172
|
+
* The single entry point the write path (`csv-generator.ts`) and read path
|
|
173
|
+
* (`bm25-index.ts`) both call, so indexed text and query text are always
|
|
174
|
+
* segmented identically. No-ops when the resolved mode is `none` (default).
|
|
175
|
+
*/
|
|
176
|
+
export const applyCjkSegmentationIfEnabled = (text) => getSearchFTSCjkSegmentation() === 'bigram' ? segmentCjkSpans(text) : text;
|
|
@@ -26,6 +26,7 @@ import { rankExactEmbeddingRows, } from '../../core/embeddings/exact-search.js';
|
|
|
26
26
|
import { EMBEDDING_TABLE_NAME, EMBEDDING_INDEX_NAME } from '../../core/lbug/schema.js';
|
|
27
27
|
import { getExactScanLimit, isVectorExtensionSupportedByPlatform, } from '../../core/platform/capabilities.js';
|
|
28
28
|
import { PhaseTimer } from '../../core/search/phase-timer.js';
|
|
29
|
+
import { cjkSegmentationModeMismatch, containsSegmentableCjkRun, getSearchFTSCjkSegmentation, isSupportedCjkSegmentationMode, MAX_CJK_SEGMENTATION_QUERY_LENGTH, } from '../../core/search/cjk-segmentation.js';
|
|
29
30
|
import { checkStalenessAsync, checkCwdMatch } from '../../core/git-staleness.js';
|
|
30
31
|
import { logger } from '../../core/logger.js';
|
|
31
32
|
import { LIST_REPOS_DEFAULT_LIMIT, LIST_REPOS_MAX_LIMIT, EXPLAIN_DEFAULT_LIMIT, EXPLAIN_MAX_LIMIT, PDG_QUERY_DEFAULT_LIMIT, PDG_QUERY_MAX_LIMIT, } from '../tools.js';
|
|
@@ -1579,6 +1580,68 @@ export class LocalBackend {
|
|
|
1579
1580
|
if (!ftsUsed) {
|
|
1580
1581
|
warnings.push('FTS indexes missing — keyword search degraded. Run: gitnexus analyze --repair-fts (or gitnexus analyze --force) to rebuild indexes.');
|
|
1581
1582
|
}
|
|
1583
|
+
// #2331: a CJK query against a server process resolving
|
|
1584
|
+
// GITNEXUS_FTS_CJK_SEGMENTATION to 'none' silently misses sub-phrase
|
|
1585
|
+
// matches with no other signal — this is the only place an agent driving
|
|
1586
|
+
// GitNexus through the query tool can learn the capability exists.
|
|
1587
|
+
try {
|
|
1588
|
+
const cjkMode = getSearchFTSCjkSegmentation();
|
|
1589
|
+
if (containsSegmentableCjkRun(searchQuery) && cjkMode !== 'bigram') {
|
|
1590
|
+
warnings.push('Query contains CJK characters — sub-phrase matches require GITNEXUS_FTS_CJK_SEGMENTATION=bigram set for both `analyze` and this server process, then `gitnexus analyze --force`.');
|
|
1591
|
+
}
|
|
1592
|
+
else if (cjkMode === 'bigram' &&
|
|
1593
|
+
searchQuery.length > MAX_CJK_SEGMENTATION_QUERY_LENGTH &&
|
|
1594
|
+
containsSegmentableCjkRun(searchQuery)) {
|
|
1595
|
+
// #2339: bigram mode is enabled, but the query exceeds the length
|
|
1596
|
+
// cap that guards segmentCjkSpans's per-character allocation cost —
|
|
1597
|
+
// applyCjkSegmentationIfEnabled silently skips segmentation above
|
|
1598
|
+
// this length, so an over-cap CJK query returns zero results for
|
|
1599
|
+
// text that IS indexed and present verbatim, with no other signal.
|
|
1600
|
+
warnings.push(`Query exceeds the ${MAX_CJK_SEGMENTATION_QUERY_LENGTH}-character CJK segmentation cap — ` +
|
|
1601
|
+
'sub-phrase matches are skipped for this query even though GITNEXUS_FTS_CJK_SEGMENTATION=bigram is enabled. Shorten the query to search within the cap.');
|
|
1602
|
+
}
|
|
1603
|
+
}
|
|
1604
|
+
catch (err) {
|
|
1605
|
+
// Best-effort diagnostic only — never fail the query over it.
|
|
1606
|
+
logQueryError('query:cjk-warning', err);
|
|
1607
|
+
}
|
|
1608
|
+
// #2339: the checks above only compare the QUERY's own content against
|
|
1609
|
+
// the live process's mode — they can't detect "server mode is 'bigram'
|
|
1610
|
+
// but the on-disk index was actually built under 'none'/legacy" (env var
|
|
1611
|
+
// changed without a full --force re-analyze, or a plain/--repair-fts
|
|
1612
|
+
// analyze ran instead). That mismatch affects every CJK query against
|
|
1613
|
+
// this repo, not just one whose own text happens to contain CJK, so it's
|
|
1614
|
+
// a separate, unconditional check — not folded into the branches above.
|
|
1615
|
+
try {
|
|
1616
|
+
const meta = await loadMeta(path.dirname(repo.lbugPath));
|
|
1617
|
+
// meta.json is on-disk state inside the analyzed repo, read via a
|
|
1618
|
+
// schema-less JSON.parse — not trusted input. Validate before
|
|
1619
|
+
// interpolating it into agent-visible tool output (#2339): an
|
|
1620
|
+
// unrecognized value is itself evidence of a corrupt/foreign index,
|
|
1621
|
+
// reported generically rather than echoed verbatim.
|
|
1622
|
+
const persistedMode = meta?.cjkSegmentation;
|
|
1623
|
+
if (meta && persistedMode !== undefined && !isSupportedCjkSegmentationMode(persistedMode)) {
|
|
1624
|
+
warnings.push("This repo's index metadata has an unrecognized CJK segmentation mode stamp — the index " +
|
|
1625
|
+
'may be corrupt or from an incompatible GitNexus version. Run `gitnexus analyze --force` to rebuild it.');
|
|
1626
|
+
}
|
|
1627
|
+
else if (meta &&
|
|
1628
|
+
cjkSegmentationModeMismatch(meta.cjkSegmentation, getSearchFTSCjkSegmentation())) {
|
|
1629
|
+
warnings.push(`Index was built with CJK segmentation mode '${meta.cjkSegmentation ?? 'none'}', but this ` +
|
|
1630
|
+
`server is resolving '${getSearchFTSCjkSegmentation()}' — sub-phrase CJK search results ` +
|
|
1631
|
+
'may be incomplete. Set GITNEXUS_FTS_CJK_SEGMENTATION to the same value for both the ' +
|
|
1632
|
+
'`analyze` process and this server, then run `gitnexus analyze --force` to rebuild under ' +
|
|
1633
|
+
"the agreed mode (do not assume the live server's mode is the one to keep — re-analyzing " +
|
|
1634
|
+
'under the wrong mode can strip an already-working bigram-segmented index back to `none`).');
|
|
1635
|
+
}
|
|
1636
|
+
}
|
|
1637
|
+
catch (err) {
|
|
1638
|
+
// loadMeta() itself never throws (it returns null on any read/parse
|
|
1639
|
+
// failure) — the actual throw source here is getSearchFTSCjkSegmentation()
|
|
1640
|
+
// on an invalid env value, same root cause as the catch above. This is
|
|
1641
|
+
// a separate, independently-guarded diagnostic though, so it gets its
|
|
1642
|
+
// own log context rather than sharing 'query:cjk-warning'.
|
|
1643
|
+
logQueryError('query:cjk-mode-drift', err);
|
|
1644
|
+
}
|
|
1582
1645
|
if (enrichmentDegraded) {
|
|
1583
1646
|
warnings.push('Symbol enrichment partially failed — some process/cohesion/content data may be missing from these results (see server logs).');
|
|
1584
1647
|
}
|
|
@@ -73,6 +73,16 @@ export interface RepoMeta {
|
|
|
73
73
|
* full rebuild rather than risk an inconsistent incremental update.
|
|
74
74
|
*/
|
|
75
75
|
schemaVersion?: number;
|
|
76
|
+
/**
|
|
77
|
+
* The resolved GITNEXUS_FTS_CJK_SEGMENTATION mode ('none' | 'bigram') the
|
|
78
|
+
* existing index's content/description columns were last written under
|
|
79
|
+
* (#2331/#2339). On mismatch with the live process's resolved mode,
|
|
80
|
+
* runFullAnalysis forces a full rebuild so indexed text and query-time
|
|
81
|
+
* segmentation never diverge. Always stamped (never omitted), unlike
|
|
82
|
+
* `pdg` below — the default 'none' is itself a meaningful value to
|
|
83
|
+
* compare, not an absence.
|
|
84
|
+
*/
|
|
85
|
+
cjkSegmentation?: string;
|
|
76
86
|
/**
|
|
77
87
|
* SHA-256 of every file's content at the time of the last successful
|
|
78
88
|
* indexing run. The next run computes current hashes and diffs against
|
package/package.json
CHANGED