@dikolab/kbdb 0.4.3 → 0.6.0

package/docs/goals/semantic-dedup.md

@@ -1,233 +0,0 @@
-# Semantic Near-Duplicate Detection
-
-*Cosine similarity on embeddings as a complementary
-detection layer alongside Jaccard -- closes the
-vocabulary-gap ceiling identified in gap #6.*
-
-## Goal
-
-The existing near-duplicate detection computes a Jaccard
-coefficient on stemmed term sets of the incoming section
-versus all indexed sections. Sections that exceed a
-similarity threshold of 0.85 are reported as
-near-duplicates in `AddSectionResult.near_duplicates`.
-
-This mechanism catches lexically similar content
-reliably. It does not catch semantically equivalent
-sections that use different vocabulary. Example: a
-section stored as "We selected PostgreSQL for its JSONB
-support" and a later section written as "Chose Postgres
-due to native JSON column type" express the same
-decision. Their stemmed term sets share few tokens --
-Jaccard produces a low score, no duplicate is flagged,
-and the knowledge base silently accumulates redundant
-content.
-
-This gap is identified as the semantic dedup ceiling in
-[Opus 4.8 Second Brain Review](../issues/2026-06-23-opus-48-second-brain-review.md)
-(gap #6). Semantic dedup closes it by adding a cosine
-similarity pass on embedding vectors as a complementary
-detection method when an embedding provider is
-configured. The feature is conditional on
-[Hybrid Search](hybrid-search.md) -- if no
-`EmbeddingProvider` is configured, only the Jaccard pass
-runs and behaviour is identical to today.
-
-## Two-Pass Detection Algorithm
-
-Detection proceeds in two sequential passes on every
-`add_section` call.
-
-### Pass 1: Jaccard (always runs)
-
-Compute the Jaccard coefficient between the incoming
-section's stemmed term set and each existing section's
-term set. The term sets are derived from the inverted
-index -- no section files are read.
-
-Sections whose Jaccard score meets or exceeds 0.85 are
-flagged as lexical near-duplicates and appended to
-`near_duplicates` with `method: "jaccard"`.
-
-Sections whose Jaccard score falls in the range
-[0.40, 0.85) -- the gray zone -- are retained as
-candidates for Pass 2.
-
-Sections with a Jaccard score below 0.40 are discarded
-with no further processing.
-
-### Pass 2: Cosine (when embeddings available)
-
-Pass 2 runs when the active `EmbeddingProvider` has
-non-zero dimensions and `vectors.idx` is present and
-not stale. With the built-in `TfidfEmbeddingProvider`
-as the default (256 dimensions), Pass 2 runs
-automatically without configuration. It is skipped
-only when embeddings are explicitly disabled
-(`provider = "none"` in `worker.toml`).
-
-For each gray-zone candidate from Pass 1, retrieve the
-candidate's embedding from `vectors.idx`. If the
-candidate's embedding is missing (e.g. the embedding
-provider failed when that section was learned), skip
-that candidate silently. Compute cosine similarity
-between the incoming section's embedding and the
-candidate's embedding.
-
-The incoming section's embedding is computed by the
-daemon as part of the same learn operation that triggers
-the add (see the embedding lifecycle in
-[Hybrid Search](hybrid-search.md)). If the embedding
-computation for the incoming section itself failed, Pass
-2 is skipped entirely for that call.
-
-Candidates whose cosine similarity meets or exceeds 0.92
-are flagged as semantic near-duplicates and appended to
-`near_duplicates` with `method: "cosine"`.
-
-### Efficiency Rationale
-
-Jaccard operates on integer set intersections over the
-inverted index -- it is cheap. Cosine operates on
-floating-point vector dot products and requires reading
-from `vectors.idx` -- it is more expensive. Restricting
-cosine to the gray zone (Jaccard >= 0.40) means cosine
-is invoked only for sections that already share moderate
-lexical overlap with the candidate. Sections with very
-low Jaccard scores are unlikely to be semantic
-duplicates, so skipping them avoids unnecessary work
-without meaningful loss in detection coverage.
-
-## Detection Thresholds
-
-| Method           | Threshold | Meaning                      |
-|------------------|-----------|------------------------------|
-| Jaccard          | >= 0.85   | Lexical near-duplicate       |
-| Jaccard pre-filter | >= 0.40 | Candidate for cosine pass    |
-| Cosine           | >= 0.92   | Semantic near-duplicate      |
-
-Thresholds are not configurable in v1. They are
-hardcoded constants. If operational experience shows
-that the defaults produce excessive false positives or
-false negatives, they can be exposed as `worker.toml`
-configuration keys in a later version.
-
-## Response Format
-
-The `near_duplicates` array in `AddSectionResult` gains
-a `method` field on every entry. The field identifies
-which detection pass produced the entry.
-
-```json
-{
-   "kbid": "<26-char hash>",
-   "near_duplicates": [
-      {
-         "kbid": "abc123...",
-         "similarity": 0.89,
-         "method": "jaccard"
-      },
-      {
-         "kbid": "def456...",
-         "similarity": 0.95,
-         "method": "cosine"
-      }
-   ]
-}
-```
-
-The `method` field is always present on every
-near-duplicate entry, including in responses from
-deployments where only Jaccard runs. Callers can rely
-on the field being present unconditionally.
-
-Entries produced by Pass 1 always carry `method:
-"jaccard"`. Entries produced by Pass 2 always carry
-`method: "cosine"`. No entry can carry both methods --
-a section that crosses the Jaccard threshold of 0.85
-is already flagged in Pass 1 and is not re-evaluated
-in Pass 2.
-
-## Dependency on Hybrid Search
-
-Semantic dedup reuses the infrastructure introduced by
-[Hybrid Search](hybrid-search.md) without adding any
-new infrastructure of its own.
-
-- The `EmbeddingProvider` interface defined in Hybrid
-  Search provides the embedding computation at learn
-  time.
-- `vectors.idx` provides the embedding lookup at
-  duplicate-detection time. No second embedding call is
-  made -- the stored vectors are read directly.
-- When `NullEmbeddingProvider` is active (no `[embedding]`
-  section in `worker.toml`), Pass 2 never runs. The
-  `near_duplicates` array may still contain Jaccard
-  entries; it will never contain cosine entries.
-- Stale vector index conditions (model name mismatch
-  between `vectors-meta.toml` and the configured
-  provider) are treated the same as a missing
-  `vectors.idx`: Pass 2 is skipped for that call.
-
-## Contradiction Detection Enhancement
-
-The existing `divergent_content` integrity check
-identifies sections that share the same `source_key`
-but have different content -- contradictory knowledge
-from the same origin that supersession should have
-resolved.
-
-With embeddings available, integrity check can extend
-this check to surface potential semantic contradictions
-that lack a shared source key: sections with high cosine
-similarity (>= 0.92) and conflicting content signals.
-This is a secondary, advisory output in the integrity
-check response -- it does not block any operation.
-
-The enhancement runs only when the integrity check is
-invoked and `vectors.idx` is present and not stale. It
-adds a new `semantic_contradictions` array to the
-integrity check response, alongside the existing
-`divergent_content` error entries:
-
-```json
-{
-   "semantic_contradictions": [
-      {
-         "kbid_a": "abc123...",
-         "kbid_b": "def456...",
-         "cosine_similarity": 0.94
-      }
-   ]
-}
-```
-
-When no embedding provider is configured or
-`vectors.idx` is absent, `semantic_contradictions` is
-an empty array.
-
-## Design Constraints
-
-- Semantic dedup is additive. It never replaces the
-  Jaccard pass. Both passes coexist, and the Jaccard
-  pass always runs regardless of embedding configuration.
-- The `method` field is always present on
-  `near_duplicates` entries, even in responses from
-  deployments where only Jaccard runs.
-- Cosine similarity computation reuses existing
-  embedding vectors stored in `vectors.idx`. No
-  additional embedding API call is made during
-  duplicate detection.
-- If Pass 2 is skipped (no provider, stale index, or
-  embedding failure for the incoming section), the
-  response omits cosine entries silently. No warning
-  field is added for a skipped cosine pass -- the
-  absence of cosine entries is not an error condition
-  and is expected in lexical-only deployments.
-- Thresholds are hardcoded in v1. Exposure as
-  configuration keys is deferred to a later version
-  after operational validation.
-- The `semantic_contradictions` enhancement to integrity
-  check is advisory. It does not change `ok` to `false`
-  on its own -- it provides supplementary information
-  for human review.