@c4a/extract 0.5.41-beta.7 → 0.6.0-alpha.1

package/README.md

@@ -1,15 +1,16 @@
 # @c4a/extract
 
-Code extraction framework for C4A. It owns the language-plugin protocol, repository runner, raw code snapshot contract, digest generation, and shared Tree-sitter parsing utilities.
+Code extraction framework for Context. It owns the language-plugin protocol, repository runner, raw code snapshot contract, digest generation, and shared Tree-sitter parsing utilities.
 
 ## Role in the Monorepo
 
-`@c4a/extract` is the protocol and runner layer under `context capture --code`.
+`@c4a/extract` is the protocol and runner layer used by declared Context
+project extraction phases such as `extractTs({ source, collection: "wikis" })`.
 
 - Language plugins implement `ExtractionPlugin` and return `ExtractionResult` v2.
-- The runner loads one or more plugins, scans repository modules, emits progress/module-error/summary events, and can build a raw code snapshot payload.
-- `@c4a/context-cli` writes the snapshot under `.context/raw/aspect/code/<source-slug>/<snapshot-id>/`.
-- `context compile --code <source-slug>` reads that snapshot and materializes package/category/symbol knowledge Nodes.
+- The runner loads one or more plugins, scans repository modules, emits progress/module-error/summary events, and can build an optional low-level code snapshot payload for tooling/tests.
+- `@c4a/context-cli` invokes the runner from `context run extract:<source>:wikis` after the workspace declares an `extractTs` phase.
+- The project flow stages review candidates under `unapproved/entities.jsonl` and keeps runtime candidate artifacts under `.tmp/context-runtime/extract/`; approved Markdown is written only by review/apply.
 
 **Depends on:** `@c4a/core`, `web-tree-sitter`, `zod`
 
@@ -67,7 +68,10 @@ Every plugin returns:
 
 ### 3. Repository Runner Protocol
 
-The package exposes `c4a-extract-code`, a NDJSON runner used by `context capture --code`.
+The package exposes `c4a-extract-code`, a low-level NDJSON runner. The current
+Context project flow normally reaches it through a declared `extractTs` phase
+and `context run extract:<source>:wikis`; agents should not hand-build runner
+input for normal workspace operation.
 
 Input is JSON on stdin:
 
@@ -129,7 +133,11 @@ During projection, code-owned Sections receive code `source_ref` values derived
 - package rows: `src-N#package:<package>@<hash>`
 - symbol rows: `src-N#symbol:<locator>:<kind>@<hash>`
 
-These refs are verified against the raw code snapshot JSONL indexes. They are separate from prose evidence refs, because code snapshots use `evidence.mode: none` and do not create raw block manifests.
+These refs are verified against the raw code snapshot JSONL indexes. They are
+separate from prose evidence refs, because code snapshots use
+`evidence.mode: none` and do not create raw block manifests. Prose/raw evidence
+uses `src-N#block:<locator> L<start>-<end>@<block-hash>` so it can keep line
+ranges for review, diffing, and re-anchoring.
 
 ## Writing a New Language Plugin
 
@@ -144,7 +152,9 @@ Minimum requirements:
 5. Emit `SymbolInfo[]` with stable `name`, `kind`, `visibility`, `file`, `line`, and `endLine`.
 6. Emit `RelationInfo[]` for imports and important type/inheritance/use edges.
 7. Keep paths module-relative inside the plugin; the repository runner prefixes them to repo-relative paths.
-8. Register the plugin in the runner input used by `context capture --code`.
+8. Register the plugin in the runner input used by the project extraction
+   phase. User-facing workspaces should declare the matching phase in
+   `src/index.ts` and execute it through `context run <phase-id>`.
 
 Example skeleton:
 
@@ -178,15 +188,29 @@ export class PythonPlugin implements ExtractionPlugin {
 }
 ```
 
-## Relationship to Code Compile and Obsidian
-
-`@c4a/extract` is upstream of code compile; it does not render knowledge itself.
-
-- `context capture --code` uses the runner to produce raw code snapshots.
-- `context compile --code` consumes `packages.jsonl`, `symbols.jsonl`, `edges.jsonl`, and `digests.jsonl` to build package/category/symbol Nodes such as `pkg`, `pkg/components`, and `pkg/symbol/button`.
-- Obsidian Render reads the compiled Markdown, `_edges.yaml`, and `_external.yaml`. It does not read raw runner snapshots directly.
-
-That means language plugins affect Obsidian only through the compiled knowledge graph: better symbols/relations produce better symbol Nodes, graph edges, source refs, and source-status chips.
+## Relationship to Context Project Extraction
+
+`@c4a/extract` is upstream of the Context project extraction phase; it does not
+render approved knowledge itself.
+
+- `context source add repo <name> --local <repo-or-subdir>` registers the code
+  source boundary.
+- `src/index.ts` declares `extractTs({ source: source("<name>"), collection: "wikis" })`.
+- `context run extract:<name>:wikis --dry-run` previews resolved modules,
+  candidate estimates, knowledge path examples, and module warnings.
+- `context run extract:<name>:wikis` writes draft candidates for review. It
+  uses `.tmp/context-runtime/extract/` for candidate snapshots, source
+  fingerprints, and symbol indexes; that scratch directory is not committed
+  state.
+- `context review html` and `context review apply` own human approval and
+  approved Markdown materialization.
+- `context close`, `context verify`, and `context build` own final package
+  readiness.
+
+That means language plugins affect published knowledge only through the current
+project flow: better symbols/relations produce better draft candidates, stable
+`repo:<source>#symbol:...` source refs, review evidence, approved Markdown, and
+package output.
 
 ## Development
 

package/bin/c4a-extract-code.js

@@ -14111,6 +14111,29 @@ var readModuleName = async (dirPath, manifestFile) => {
   }
   return null;
 };
+var readModuleVersion = async (dirPath, manifestFile) => {
+  try {
+    const content = await readFile(join(dirPath, manifestFile), "utf-8");
+    if (manifestFile === "package.json") {
+      const parsed = JSON.parse(content);
+      return typeof parsed.version === "string" && parsed.version.trim() ? parsed.version.trim() : null;
+    }
+    if (manifestFile === "pyproject.toml" || manifestFile === "Cargo.toml") {
+      const match = content.match(/^\s*version\s*=\s*["']([^"']+)["']/m);
+      return match?.[1] ?? null;
+    }
+    if (manifestFile === "setup.py") {
+      const match = content.match(/version\s*=\s*["']([^"']+)["']/);
+      return match?.[1] ?? null;
+    }
+    if (manifestFile === "pom.xml") {
+      return content.match(/<version>\s*([^<]+?)\s*<\/version>/i)?.[1] ?? null;
+    }
+  } catch {
+    return null;
+  }
+  return null;
+};
 var MANIFEST_NAMES = new Set(MANIFEST_FILES.map((f) => f.toLowerCase()));
 var findSubModuleDirs = async (rootDir, excludeDirs, gitFiles, pathFilter2) => {
   const root = resolve(rootDir);
@@ -14228,6 +14251,35 @@ var detectModules = async (repoPath, ref, pathFilter2) => {
   }
   return modules.sort((a, b) => a.name.localeCompare(b.name));
 };
+var detectModuleBoundaries = async (repoPath, ref, pathFilter2) => {
+  const repoRoot = resolve(repoPath);
+  const gitFiles = await tryGitListFiles(repoRoot, ref);
+  const subModuleDirs = await findSubModuleDirs(repoRoot, undefined, gitFiles, pathFilter2);
+  const boundaries = [];
+  const rootManifest = await findManifestFile(repoRoot);
+  if (rootManifest) {
+    const version2 = await readModuleVersion(repoRoot, rootManifest);
+    boundaries.push({
+      name: await readModuleName(repoRoot, rootManifest) ?? basename(repoRoot),
+      path: ".",
+      manifest: rootManifest,
+      ...version2 !== null ? { version: version2 } : {}
+    });
+  }
+  for (const subDir of subModuleDirs) {
+    const manifest = await findManifestFile(subDir);
+    if (!manifest)
+      continue;
+    const version2 = await readModuleVersion(subDir, manifest);
+    boundaries.push({
+      name: await readModuleName(subDir, manifest) ?? basename(subDir),
+      path: toPosixPath(relative(repoRoot, subDir)) || ".",
+      manifest,
+      ...version2 !== null ? { version: version2 } : {}
+    });
+  }
+  return boundaries.sort((left, right) => left.path.localeCompare(right.path));
+};
 var detectModuleAt = async (repoPath, modulePath, ref, pathFilter2) => {
   const repoRoot = resolve(repoPath);
   const moduleDir = modulePath === "." ? repoRoot : resolve(repoRoot, modulePath);

package/index.js

@@ -8660,7 +8660,7 @@ var require_picomatch2 = __commonJS((exports, module2) => {
 
 // ../../node_modules/.bun/web-tree-sitter@0.20.8/node_modules/web-tree-sitter/tree-sitter.js
 var require_tree_sitter = __commonJS((exports, module2) => {
-  var __dirname = "/Users/bytedance/github/c4a/node_modules/.bun/web-tree-sitter@0.20.8/node_modules/web-tree-sitter";
+  var __dirname = "/Users/bytedance/github/c4a/context/node_modules/.bun/web-tree-sitter@0.20.8/node_modules/web-tree-sitter";
   var Module = Module !== undefined ? Module : {};
   var TreeSitter = function() {
     var initPromise, document = typeof window == "object" ? { currentScript: window.document.currentScript } : null;
@@ -15263,7 +15263,9 @@ var symbolInfoSchema = exports_external.lazy(() => exports_external.object({
   implements: exports_external.array(exports_external.string().min(1)).optional(),
   doc: exports_external.string().nullable().optional(),
   propsType: exports_external.string().min(1).nullable().optional(),
-  unionValues: exports_external.array(exports_external.string()).optional()
+  unionValues: exports_external.array(exports_external.string()).optional(),
+  initializer: exports_external.string().min(1).nullable().optional(),
+  signature: exports_external.string().min(1).nullable().optional()
 }));
 var relationInfoSchema = exports_external.object({
   type: relationTypeSchema,
@@ -15463,6 +15465,530 @@ var generateSymbolDiff = (current, previous) => {
     removed: previousNames.filter((name) => !currentSet.has(name))
   };
 };
+// src/documentEvidence.ts
+import { Buffer as Buffer2 } from "node:buffer";
+import { createHash as createHash2 } from "node:crypto";
+var DOCUMENT_EVIDENCE_NORMALIZER_VERSION = "document-evidence-normalizer.v1";
+var DOCUMENT_SNAPSHOT_MANIFEST_SCHEMA_VERSION = "document.snapshot.v2";
+var DEFAULT_SOURCE_SPAN_HASH_LENGTH = 12;
+var BOM = "\uFEFF";
+var HASH_ID_RE = /^(?:sha256:)?[a-f0-9]{64}$/u;
+var SOURCE_SPAN_HASH_RE = /^[a-f0-9]{8,64}$/u;
+var DOCUMENT_SOURCE_NAME_RE = /^[a-z0-9][a-z0-9._-]*$/u;
+function isRecord(value) {
+  return value !== null && typeof value === "object" && !Array.isArray(value);
+}
+function bytesOf(value) {
+  return typeof value === "string" ? Buffer2.from(value, "utf8") : Buffer2.from(value);
+}
+function sha256Hex(value) {
+  return createHash2("sha256").update(bytesOf(value)).digest("hex");
+}
+function normalizeHashId(value) {
+  const trimmed = value.trim().toLowerCase();
+  const hex = trimmed.startsWith("sha256:") ? trimmed.slice("sha256:".length) : trimmed;
+  if (!/^[a-f0-9]{64}$/u.test(hex)) {
+    throw new TypeError(`invalid sha256 hash id: ${value}`);
+  }
+  return `sha256:${hex}`;
+}
+function assertDocumentSourceType(value) {
+  if (value !== "file" && value !== "lark") {
+    throw new TypeError(`document source_type must be file or lark: ${value}`);
+  }
+}
+function normalizeDocumentSourceName(name) {
+  const value = name.trim();
+  if (!DOCUMENT_SOURCE_NAME_RE.test(value)) {
+    throw new TypeError(`document source name must be a lowercase path-safe slug: ${name}`);
+  }
+  return value;
+}
+function normalizeSnapshotRelativePath(path2) {
+  const value = path2.trim();
+  if (value.length === 0 || value.includes("\x00") || value.startsWith("/") || /^[a-zA-Z]:[\\/]/u.test(value)) {
+    throw new TypeError(`snapshot path must be a POSIX relative path: ${path2}`);
+  }
+  if (value.includes("\\")) {
+    throw new TypeError(`snapshot path must use POSIX separators: ${path2}`);
+  }
+  const segments = value.split("/");
+  if (segments.some((segment) => segment.length === 0 || segment === "." || segment === "..")) {
+    throw new TypeError(`snapshot path must not contain empty, dot, or traversal segments: ${path2}`);
+  }
+  return value;
+}
+function encodeSnapshotLocatorPath(path2) {
+  return normalizeSnapshotRelativePath(path2).split("/").map((segment) => encodeURIComponent(segment)).join("/");
+}
+function decodeSnapshotLocatorPath(path2) {
+  try {
+    return normalizeSnapshotRelativePath(path2.split("/").map((segment) => decodeURIComponent(segment)).join("/"));
+  } catch (error) {
+    const message = error instanceof Error ? error.message : String(error);
+    throw new TypeError(`invalid encoded snapshot locator path: ${path2}: ${message}`);
+  }
+}
+function parseDocumentSourceLocator(source2) {
+  const match = /^(file|lark):([^/]+)\/(.+)$/u.exec(source2);
+  if (match?.[1] === undefined || match[2] === undefined || match[3] === undefined)
+    return null;
+  try {
+    return {
+      sourceType: match[1],
+      sourceName: normalizeDocumentSourceName(match[2]),
+      documentPath: decodeSnapshotLocatorPath(match[3])
+    };
+  } catch {
+    return null;
+  }
+}
+function normalizeMarkdownDocument(input) {
+  let text = input.normalize("NFC");
+  if (text.startsWith(BOM)) {
+    text = text.slice(BOM.length);
+  }
+  text = text.replace(/\r\n?/g, `
+`);
+  text = text.split(`
+`).map((line) => line.replace(/[ \t]+$/u, "")).join(`
+`);
+  text = text.replace(/\n{3,}/g, `
+
+`);
+  text = text.replace(/^(\s*)[*+]([ \t])/gmu, "$1-$2");
+  const fenceCount = (text.match(/^```/gmu) ?? []).length;
+  if (fenceCount % 2 === 1) {
+    if (!text.endsWith(`
+`))
+      text += `
+`;
+    text += "```\n";
+  }
+  return text;
+}
+function computeDocumentContentHash(bytes) {
+  return `sha256:${sha256Hex(bytes)}`;
+}
+function countMarkdownLines(bytes) {
+  const text = bytesOf(bytes).toString("utf8");
+  if (text.length === 0)
+    return 0;
+  const normalized = text.replace(/\r\n?/g, `
+`);
+  return normalized.endsWith(`
+`) ? normalized.split(`
+`).length - 1 : normalized.split(`
+`).length;
+}
+function computeLogicalRawHash(files) {
+  const hash2 = createHash2("sha256");
+  const sorted = [...files].map((file) => ({
+    path: normalizeSnapshotRelativePath(file.path),
+    bytes: bytesOf(file.bytes)
+  })).sort((left, right) => left.path < right.path ? -1 : left.path > right.path ? 1 : 0);
+  for (const file of sorted) {
+    hash2.update(Buffer2.from(file.path, "utf8"));
+    hash2.update("\x00");
+    hash2.update(Buffer2.from(String(file.bytes.byteLength), "utf8"));
+    hash2.update("\x00");
+    hash2.update(file.bytes);
+    hash2.update(`
+`);
+  }
+  return `sha256:${hash2.digest("hex")}`;
+}
+function slugSegment(value) {
+  return value.trim().toLowerCase().normalize("NFC").replace(/[`*_~[\]()]/gu, "").replace(/[^\p{Letter}\p{Number}]+/gu, "-").replace(/^-+|-+$/gu, "").slice(0, 64) || "document";
+}
+function headingFromLine(line) {
+  const match = /^(#{1,6})\s+(.+?)\s*#*\s*$/u.exec(line);
+  if (!match)
+    return null;
+  const title = (match[2] ?? "").trim();
+  if (title.length === 0)
+    return null;
+  return { level: match[1].length, title };
+}
+function fenceMarker(line) {
+  const match = /^\s{0,3}(`{3,}|~{3,})/u.exec(line);
+  if (!match?.[1])
+    return null;
+  return { marker: match[1][0], length: match[1].length };
+}
+function truncateSourceSpanHash(hash2, length = DEFAULT_SOURCE_SPAN_HASH_LENGTH) {
+  if (!/^[a-f0-9]{64}$/u.test(hash2)) {
+    throw new TypeError(`source span hash must be a 64-character lowercase sha256 hex digest: ${hash2}`);
+  }
+  if (!Number.isInteger(length) || length < 8 || length > 64) {
+    throw new TypeError(`source span hash length must be an integer between 8 and 64: ${length}`);
+  }
+  return hash2.slice(0, length);
+}
+function snapshotTextLines(markdown) {
+  if (markdown.length === 0)
+    return [];
+  const lines = markdown.split(`
+`);
+  if (markdown.endsWith(`
+`))
+    lines.pop();
+  return lines;
+}
+function headingContextForLine(markdown, lineStart) {
+  const lines = snapshotTextLines(markdown);
+  const headings = [];
+  let fence = null;
+  const maxIndex = Math.min(Math.max(lineStart, 1), lines.length);
+  for (let index = 0;index < maxIndex; index += 1) {
+    const line = lines[index] ?? "";
+    const marker = fenceMarker(line);
+    if (fence !== null) {
+      if (marker !== null && marker.marker === fence.marker && marker.length >= fence.length) {
+        fence = null;
+      }
+      continue;
+    }
+    if (marker !== null) {
+      fence = marker;
+      continue;
+    }
+    const heading = headingFromLine(line);
+    if (heading !== null) {
+      headings.splice(heading.level - 1);
+      headings[heading.level - 1] = {
+        ...heading,
+        headingSlug: slugSegment(heading.title)
+      };
+    }
+  }
+  const headingPath = headings.flatMap((heading) => heading === undefined ? [] : [heading.title]);
+  const headingHint = [...headings].reverse().find((heading) => heading !== undefined)?.headingSlug ?? "document";
+  return {
+    heading_hint: headingHint,
+    heading_path: headingPath
+  };
+}
+function createDocumentSourceSpan(markdown, input) {
+  const lines = snapshotTextLines(markdown);
+  const lineStart = input.lineStart;
+  const lineEnd = input.lineEnd;
+  if (!Number.isInteger(lineStart) || !Number.isInteger(lineEnd) || lineStart < 1 || lineEnd < lineStart) {
+    throw new TypeError(`source span line range must be positive and ordered: L${lineStart}-${lineEnd}`);
+  }
+  if (lineEnd > lines.length) {
+    throw new TypeError(`source span line range exceeds document line count: L${lineStart}-${lineEnd}`);
+  }
+  const text = lines.slice(lineStart - 1, lineEnd).join(`
+`);
+  const fullSpanHash = sha256Hex(text);
+  const heading = headingContextForLine(markdown, lineStart);
+  return {
+    heading_hint: heading.heading_hint,
+    heading_path: heading.heading_path,
+    line_start: lineStart,
+    line_end: lineEnd,
+    line_range: `L${lineStart}-${lineEnd}`,
+    span_hash: truncateSourceSpanHash(fullSpanHash, input.hashLength ?? DEFAULT_SOURCE_SPAN_HASH_LENGTH),
+    full_span_hash: fullSpanHash,
+    text,
+    text_preview: text.replace(/\s+/gu, " ").slice(0, 160)
+  };
+}
+function formatSpanSourceRef(span, options = {}) {
+  const hash2 = options.hashLength === undefined ? span.span_hash : truncateSourceSpanHash(span.full_span_hash, options.hashLength);
+  const body2 = `#span:${span.heading_hint} L${span.line_start}-${span.line_end}@${hash2}`;
+  return options.locator !== undefined ? `${options.locator}${body2}` : body2;
+}
+function formatCanonicalProseSourceRef(input) {
+  const sourceName = normalizeDocumentSourceName(input.sourceName);
+  const documentPath = normalizeSnapshotRelativePath(input.documentPath);
+  return formatSpanSourceRef(input.span, {
+    locator: `${input.sourceType}:${sourceName}/${encodeSnapshotLocatorPath(documentPath)}`,
+    ...input.hashLength !== undefined ? { hashLength: input.hashLength } : {}
+  });
+}
+function parseSpanSourceRef(value) {
+  const normalized = value.trim();
+  const match = /^(?:(?<locator>[^#\s]+))?#span:(?<heading>[^\s#]+)\s+L(?<start>\d+)-(?<end>\d+)@(?<hash>[a-f0-9]{8,64})$/u.exec(normalized);
+  if (!match?.groups)
+    return null;
+  const lineStart = Number(match.groups.start);
+  const lineEnd = Number(match.groups.end);
+  if (!Number.isInteger(lineStart) || !Number.isInteger(lineEnd) || lineStart < 1 || lineEnd < lineStart) {
+    return null;
+  }
+  return {
+    ...match.groups.locator !== undefined ? { locator: match.groups.locator } : {},
+    heading_hint: match.groups.heading,
+    line_start: lineStart,
+    line_end: lineEnd,
+    span_hash: match.groups.hash
+  };
+}
+function sourceSpanHashMatches(refHash, fullSpanHash) {
+  return SOURCE_SPAN_HASH_RE.test(refHash) && /^[a-f0-9]{64}$/u.test(fullSpanHash) && fullSpanHash.startsWith(refHash);
+}
+function createDocumentSnapshotFileEntry(input) {
+  const path2 = normalizeSnapshotRelativePath(input.path);
+  if ((input.title?.trim().length ?? 0) === 0 && (input.locator?.trim().length ?? 0) === 0) {
+    throw new TypeError(`snapshot file "${path2}" must include title or locator`);
+  }
+  return {
+    path: path2,
+    content_hash: computeDocumentContentHash(input.bytes),
+    line_count: countMarkdownLines(input.bytes),
+    ...input.title !== undefined ? { title: input.title.trim() } : {},
+    ...input.locator !== undefined ? { locator: input.locator.trim() } : {}
+  };
+}
+function createDocumentSnapshotManifest(input) {
+  const files = input.files.map(createDocumentSnapshotFileEntry);
+  const sourceName = normalizeDocumentSourceName(input.sourceName);
+  return parseDocumentSnapshotManifest({
+    schema_version: DOCUMENT_SNAPSHOT_MANIFEST_SCHEMA_VERSION,
+    source_type: input.sourceType,
+    source_name: sourceName,
+    captured_at: input.capturedAt,
+    snapshot_hash: computeLogicalRawHash(input.files),
+    normalizer_version: input.normalizerVersion ?? DOCUMENT_EVIDENCE_NORMALIZER_VERSION,
+    files,
+    ...input.assets !== undefined ? { assets: input.assets } : {},
+    ...input.metadata !== undefined ? { metadata: input.metadata } : {}
+  });
+}
+function parseFileEntry(value, index) {
+  if (!isRecord(value)) {
+    throw new TypeError(`snapshot manifest files[${index}] must be an object`);
+  }
+  if (typeof value.path !== "string") {
+    throw new TypeError(`snapshot manifest files[${index}].path must be a string`);
+  }
+  if (typeof value.content_hash !== "string" || !HASH_ID_RE.test(value.content_hash)) {
+    throw new TypeError(`snapshot manifest files[${index}].content_hash must be sha256:<64 hex>`);
+  }
+  if (!Number.isInteger(value.line_count) || value.line_count < 0) {
+    throw new TypeError(`snapshot manifest files[${index}].line_count must be a non-negative integer`);
+  }
+  const title = typeof value.title === "string" && value.title.trim().length > 0 ? value.title.trim() : undefined;
+  const locator = typeof value.locator === "string" && value.locator.trim().length > 0 ? value.locator.trim() : undefined;
+  if (title === undefined && locator === undefined) {
+    throw new TypeError(`snapshot manifest files[${index}] must include title or locator`);
+  }
+  return {
+    path: normalizeSnapshotRelativePath(value.path),
+    content_hash: normalizeHashId(value.content_hash),
+    line_count: value.line_count,
+    ...title !== undefined ? { title } : {},
+    ...locator !== undefined ? { locator } : {}
+  };
+}
+function parseAssetEntry(value, index) {
+  if (!isRecord(value)) {
+    throw new TypeError(`snapshot manifest assets[${index}] must be an object`);
+  }
+  if (typeof value.path !== "string") {
+    throw new TypeError(`snapshot manifest assets[${index}].path must be a string`);
+  }
+  const contentHash2 = typeof value.content_hash === "string" && HASH_ID_RE.test(value.content_hash) ? normalizeHashId(value.content_hash) : undefined;
+  const mediaType = typeof value.media_type === "string" && value.media_type.trim().length > 0 ? value.media_type.trim() : undefined;
+  const source2 = parseStringRecord(value.source, `snapshot manifest assets[${index}].source`);
+  return {
+    path: normalizeSnapshotRelativePath(value.path),
+    ...contentHash2 !== undefined ? { content_hash: contentHash2 } : {},
+    ...mediaType !== undefined ? { media_type: mediaType } : {},
+    ...source2 !== undefined ? { source: source2 } : {}
+  };
+}
+function parseStringRecord(value, field) {
+  if (value === undefined)
+    return;
+  if (!isRecord(value)) {
+    throw new TypeError(`${field} must be an object`);
+  }
+  const result = {};
+  for (const [key, item] of Object.entries(value)) {
+    if (typeof item !== "string") {
+      throw new TypeError(`${field}.${key} must be a string`);
+    }
+    const trimmed = item.trim();
+    if (trimmed.length > 0)
+      result[key] = trimmed;
+  }
+  return Object.keys(result).length > 0 ? result : undefined;
+}
+function optionalMetadataString(value, field) {
+  if (value === undefined)
+    return;
+  if (typeof value !== "string") {
+    throw new TypeError(`${field} must be a string`);
+  }
+  const trimmed = value.trim();
+  return trimmed.length > 0 ? trimmed : undefined;
+}
+function optionalMetadataStringArray(value, field) {
+  if (value === undefined)
+    return;
+  if (!Array.isArray(value)) {
+    throw new TypeError(`${field} must be an array`);
+  }
+  return value.map((item, index) => {
+    if (typeof item !== "string") {
+      throw new TypeError(`${field}[${index}] must be a string`);
+    }
+    const trimmed = item.trim();
+    if (trimmed.length === 0) {
+      throw new TypeError(`${field}[${index}] must be a non-empty string`);
+    }
+    return trimmed;
+  });
+}
+function optionalRouteFiles(value, field) {
+  if (value === undefined)
+    return;
+  if (!Array.isArray(value)) {
+    throw new TypeError(`${field} must be an array`);
+  }
+  const result = value.map((item, index) => {
+    if (!isRecord(item)) {
+      throw new TypeError(`${field}[${index}] must be an object`);
+    }
+    const path2 = optionalMetadataString(item.path, `${field}[${index}].path`);
+    if (path2 === undefined) {
+      throw new TypeError(`${field}[${index}].path must be a non-empty string`);
+    }
+    const routes = optionalMetadataStringArray(item.routes, `${field}[${index}].routes`) ?? [];
+    return {
+      path: normalizeSnapshotRelativePath(path2),
+      routes
+    };
+  }).filter((item) => item.routes.length > 0);
+  return result.length > 0 ? result : undefined;
+}
+function optionalRouteHints(value, field) {
+  if (value === undefined)
+    return;
+  if (!Array.isArray(value)) {
+    throw new TypeError(`${field} must be an array`);
+  }
+  const result = value.map((item, index) => {
+    if (!isRecord(item)) {
+      throw new TypeError(`${field}[${index}] must be an object`);
+    }
+    const documentPath = optionalMetadataString(item.documentPath, `${field}[${index}].documentPath`);
+    const route = optionalMetadataString(item.route, `${field}[${index}].route`);
+    const metadataPath = optionalMetadataString(item.metadataPath, `${field}[${index}].metadataPath`);
+    if (documentPath === undefined || route === undefined || metadataPath === undefined) {
+      throw new TypeError(`${field}[${index}] must include documentPath, route, and metadataPath`);
+    }
+    return {
+      documentPath: normalizeSnapshotRelativePath(documentPath),
+      route,
+      metadataPath: normalizeSnapshotRelativePath(metadataPath)
+    };
+  });
+  return result.length > 0 ? result : undefined;
+}
+function parseManifestMetadata(value) {
+  if (value === undefined)
+    return;
+  if (!isRecord(value)) {
+    throw new TypeError("snapshot manifest metadata must be an object");
+  }
+  let source2;
+  if (value.source !== undefined) {
+    if (!isRecord(value.source)) {
+      throw new TypeError("snapshot manifest metadata.source must be an object");
+    }
+    const url = optionalMetadataString(value.source.url, "snapshot manifest metadata.source.url");
+    const docToken = optionalMetadataString(value.source.docToken, "snapshot manifest metadata.source.docToken");
+    const wikiToken = optionalMetadataString(value.source.wikiToken, "snapshot manifest metadata.source.wikiToken");
+    const title = optionalMetadataString(value.source.title, "snapshot manifest metadata.source.title");
+    const revisionId = optionalMetadataString(value.source.revisionId, "snapshot manifest metadata.source.revisionId");
+    source2 = {
+      ...url !== undefined ? { url } : {},
+      ...docToken !== undefined ? { docToken } : {},
+      ...wikiToken !== undefined ? { wikiToken } : {},
+      ...title !== undefined ? { title } : {},
+      ...revisionId !== undefined ? { revisionId } : {}
+    };
+    if (Object.keys(source2).length === 0)
+      source2 = undefined;
+  }
+  let capture;
+  if (value.capture !== undefined) {
+    if (!isRecord(value.capture)) {
+      throw new TypeError("snapshot manifest metadata.capture must be an object");
+    }
+    const include = optionalMetadataStringArray(value.capture.include, "snapshot manifest metadata.capture.include");
+    const documentExtensions = optionalMetadataStringArray(value.capture.documentExtensions, "snapshot manifest metadata.capture.documentExtensions");
+    const routeFiles = optionalRouteFiles(value.capture.routeFiles, "snapshot manifest metadata.capture.routeFiles");
+    const routeHints = optionalRouteHints(value.capture.routeHints, "snapshot manifest metadata.capture.routeHints");
+    capture = {
+      ...include !== undefined ? { include } : {},
+      ...documentExtensions !== undefined ? { documentExtensions } : {},
+      ...routeFiles !== undefined ? { routeFiles } : {},
+      ...routeHints !== undefined ? { routeHints } : {}
+    };
+    if (Object.keys(capture).length === 0)
+      capture = undefined;
+  }
+  return source2 !== undefined || capture !== undefined ? {
+    ...source2 !== undefined ? { source: source2 } : {},
+    ...capture !== undefined ? { capture } : {}
+  } : {};
+}
+function parseDocumentSnapshotManifest(value) {
+  if (!isRecord(value)) {
+    throw new TypeError("snapshot manifest must be an object");
+  }
+  if (value.schema_version !== DOCUMENT_SNAPSHOT_MANIFEST_SCHEMA_VERSION) {
+    throw new TypeError(`snapshot manifest schema_version must be ${DOCUMENT_SNAPSHOT_MANIFEST_SCHEMA_VERSION}`);
+  }
+  if (typeof value.source_type !== "string") {
+    throw new TypeError("snapshot manifest source_type must be a string");
+  }
+  assertDocumentSourceType(value.source_type);
+  if (typeof value.source_name !== "string" || value.source_name.trim().length === 0) {
+    throw new TypeError("snapshot manifest source_name must be a non-empty string");
+  }
+  const sourceName = normalizeDocumentSourceName(value.source_name);
+  if (typeof value.captured_at !== "string" || value.captured_at.trim().length === 0) {
+    throw new TypeError("snapshot manifest captured_at must be a non-empty string");
+  }
+  if (typeof value.snapshot_hash !== "string" || !HASH_ID_RE.test(value.snapshot_hash)) {
+    throw new TypeError("snapshot manifest snapshot_hash must be sha256:<64 hex>");
+  }
+  if (typeof value.normalizer_version !== "string" || value.normalizer_version.trim().length === 0) {
+    throw new TypeError("snapshot manifest normalizer_version must be a non-empty string");
+  }
+  if (!Array.isArray(value.files)) {
+    throw new TypeError("snapshot manifest files must be an array");
+  }
+  const assets = value.assets === undefined ? undefined : Array.isArray(value.assets) ? value.assets.map(parseAssetEntry) : (() => {
+    throw new TypeError("snapshot manifest assets must be an array");
+  })();
+  const metadata2 = parseManifestMetadata(value.metadata);
+  const files = value.files.map(parseFileEntry);
+  const seenPaths = new Set;
+  for (const file of files) {
+    if (seenPaths.has(file.path)) {
+      throw new TypeError(`snapshot manifest files contains duplicate path: ${file.path}`);
+    }
+    seenPaths.add(file.path);
+  }
+  return {
+    schema_version: DOCUMENT_SNAPSHOT_MANIFEST_SCHEMA_VERSION,
+    source_type: value.source_type,
+    source_name: sourceName,
+    captured_at: value.captured_at,
+    snapshot_hash: normalizeHashId(value.snapshot_hash),
+    normalizer_version: value.normalizer_version,
+    files,
+    ...assets !== undefined ? { assets } : {},
+    ...metadata2 !== undefined ? { metadata: metadata2 } : {}
+  };
+}
 // src/repository.ts
 import { readFile as readFile2, readdir as readdir2, stat } from "node:fs/promises";
 import path2 from "node:path";
@@ -15688,6 +16214,29 @@ var readModuleName = async (dirPath, manifestFile) => {
   }
   return null;
 };
+var readModuleVersion = async (dirPath, manifestFile) => {
+  try {
+    const content = await readFile(join(dirPath, manifestFile), "utf-8");
+    if (manifestFile === "package.json") {
+      const parsed = JSON.parse(content);
+      return typeof parsed.version === "string" && parsed.version.trim() ? parsed.version.trim() : null;
+    }
+    if (manifestFile === "pyproject.toml" || manifestFile === "Cargo.toml") {
+      const match = content.match(/^\s*version\s*=\s*["']([^"']+)["']/m);
+      return match?.[1] ?? null;
+    }
+    if (manifestFile === "setup.py") {
+      const match = content.match(/version\s*=\s*["']([^"']+)["']/);
+      return match?.[1] ?? null;
+    }
+    if (manifestFile === "pom.xml") {
+      return content.match(/<version>\s*([^<]+?)\s*<\/version>/i)?.[1] ?? null;
+    }
+  } catch {
+    return null;
+  }
+  return null;
+};
 var MANIFEST_NAMES = new Set(MANIFEST_FILES.map((f) => f.toLowerCase()));
 var findSubModuleDirs = async (rootDir, excludeDirs, gitFiles, pathFilter2) => {
   const root = resolve(rootDir);
@@ -15805,6 +16354,35 @@ var detectModules = async (repoPath, ref, pathFilter2) => {
   }
   return modules.sort((a, b) => a.name.localeCompare(b.name));
 };
+var detectModuleBoundaries = async (repoPath, ref, pathFilter2) => {
+  const repoRoot = resolve(repoPath);
+  const gitFiles = await tryGitListFiles(repoRoot, ref);
+  const subModuleDirs = await findSubModuleDirs(repoRoot, undefined, gitFiles, pathFilter2);
+  const boundaries = [];
+  const rootManifest = await findManifestFile(repoRoot);
+  if (rootManifest) {
+    const version2 = await readModuleVersion(repoRoot, rootManifest);
+    boundaries.push({
+      name: await readModuleName(repoRoot, rootManifest) ?? basename(repoRoot),
+      path: ".",
+      manifest: rootManifest,
+      ...version2 !== null ? { version: version2 } : {}
+    });
+  }
+  for (const subDir of subModuleDirs) {
+    const manifest = await findManifestFile(subDir);
+    if (!manifest)
+      continue;
+    const version2 = await readModuleVersion(subDir, manifest);
+    boundaries.push({
+      name: await readModuleName(subDir, manifest) ?? basename(subDir),
+      path: toPosixPath(relative(repoRoot, subDir)) || ".",
+      manifest,
+      ...version2 !== null ? { version: version2 } : {}
+    });
+  }
+  return boundaries.sort((left, right) => left.path.localeCompare(right.path));
+};
 var detectModuleAt = async (repoPath, modulePath, ref, pathFilter2) => {
   const repoRoot = resolve(repoPath);
   const moduleDir = modulePath === "." ? repoRoot : resolve(repoRoot, modulePath);
@@ -16578,10 +17156,12 @@ var detectTechStack = async (repoPath) => {
   return labels;
 };
 export {
+  truncateSourceSpanHash,
   symbolParamSchema,
   symbolInfoSchema,
   symbolDiffSchema,
   stableStringify,
+  sourceSpanHashMatches,
   sourceInfoSchema,
   scanSourceFiles,
   runRepositoryExtraction,
@@ -16596,9 +17176,15 @@ export {
   prefixExtractionPaths,
   prefixEntryDetectionPaths,
   patternDetectionResultSchema,
+  parseSpanSourceRef,
   parseFile,
+  parseDocumentSourceLocator,
+  parseDocumentSnapshotManifest,
+  normalizeSnapshotRelativePath,
   normalizeRelativePath,
+  normalizeMarkdownDocument,
   normalizeExtractionPaths,
+  normalizeDocumentSourceName,
   metaText,
   manifestTypeSchema,
   manifestText,
@@ -16612,6 +17198,8 @@ export {
   getGitCommitHash,
   generateSymbolDiff,
   generateDigest,
+  formatSpanSourceRef,
+  formatCanonicalProseSourceRef,
   flattenSymbols,
   fileInfoSchema,
   extractionStatsSchema,
@@ -16620,12 +17208,21 @@ export {
   extractionMetaSchema,
   entryFileSchema,
   entryDetectionResultSchema,
+  encodeSnapshotLocatorPath,
   digestStatsSchema,
   digestDataSchema,
   detectTechStack,
   detectModules,
+  detectModuleBoundaries,
   detectModuleAt,
+  decodeSnapshotLocatorPath,
   createModuleFileSystem,
+  createDocumentSourceSpan,
+  createDocumentSnapshotManifest,
+  createDocumentSnapshotFileEntry,
+  countMarkdownLines,
+  computeLogicalRawHash,
+  computeDocumentContentHash,
   codeExtractRunnerInputSchema,
   canonicalizeCodeVersionLabel,
   buildSourceFileHashId,
@@ -16637,5 +17234,8 @@ export {
   SCAN_EXCLUDED_DIRS,
   PACKAGE_JSON,
   ExtractionPluginRegistry,
+  DOCUMENT_SNAPSHOT_MANIFEST_SCHEMA_VERSION,
+  DOCUMENT_EVIDENCE_NORMALIZER_VERSION,
+  DEFAULT_SOURCE_SPAN_HASH_LENGTH,
   CODE_SNAPSHOT_META_SCHEMA_VERSION
 };

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@c4a/extract",
-  "version": "0.5.41-beta.7",
+  "version": "0.6.0-alpha.1",
   "type": "module",
   "bin": {
     "c4a-extract-code": "./bin/c4a-extract-code.js"