@c4a/extract 0.5.35-beta.1 → 0.5.36

package/README.md

@@ -1,61 +1,162 @@
 # @c4a/extract
 
-Code extraction framework: plugin protocol, registry, digest generation, and shared parsing infrastructure.
+Code extraction framework for C4A. It owns the language-plugin protocol, repository runner, raw code snapshot contract, digest generation, and shared Tree-sitter parsing utilities.
 
-## Role in the monorepo
+## Role in the Monorepo
 
-Defines the standard extraction protocol (`ExtractionPlugin` interface) and provides:
-- Plugin registry for language-specific extraction plugins
-- ExtractionResult v2 schema (SymbolInfo + RelationInfo)
-- Digest generation (exported/internal symbol partitioning) and symbol diff
-- Shared Tree-sitter parsing and file scanning utilities
+`@c4a/extract` is the protocol and runner layer under `context capture --code`.
 
-**Depends on:** `core`
-**Depended on by:** `extract-ts`, `daemon`, `api`, `e2e`
+- Language plugins implement `ExtractionPlugin` and return `ExtractionResult` v2.
+- The runner loads one or more plugins, scans repository modules, emits progress/module-error/summary events, and can build a raw code snapshot payload.
+- `@c4a/context-cli` writes the snapshot under `.context/raw/aspect/code/<source-slug>/<snapshot-id>/`.
+- `context compile --code <source-slug>` reads that snapshot and materializes package/category/symbol knowledge Nodes.
 
-## Key exports
+**Depends on:** `@c4a/core`, `web-tree-sitter`, `zod`
 
-### Plugin protocol
+**Depended on by:** `@c4a/extract-ts`, `@c4a/context-cli`, `@c4a/daemon`, `@c4a/e2e`
 
-- `ExtractionPlugin` — interface that language plugins implement
-- `ExtractionPluginRegistry` — register/resolve plugins by source manifest
-- `FileSystem` — abstraction for file access (plugins don't touch disk directly)
-- `SourceInfo` / `ManifestInfo` — source metadata passed to `canHandle`/`detectEntries`
-- `EntryDetectionResult` — package info + entry files (supports monorepo recursion via `subPackages`)
-- `PatternDetectionResult` — framework pattern detection (endpoints, implicit deps)
+## Protocol Layers
 
-### Types (ExtractionResult v2)
+### 1. Language Plugin Protocol
 
-- `ExtractionResult` — standard output format: `{ version: "2", meta, package, files, symbols, relations, stats }`
-- `SymbolInfo` — extracted symbol: name, kind, visibility, file, line, members, params, returnType, etc.
-- `RelationInfo` — extracted relation: type, from, to, grounding, confidence, source
+Language packages implement `ExtractionPlugin` from `protocol.ts`.
 
-### Digest & diff
+```ts
+interface ExtractionPlugin {
+  id: string;
+  languages: string[];
+  packageManagers: string[];
+  canHandle(source: SourceInfo): boolean;
+  detectEntries(manifest: ManifestInfo, fs: FileSystem): Promise<EntryDetectionResult>;
+  extractSymbols(entries: EntryFile[], fs: FileSystem): Promise<ExtractionResult>;
+  detectPatterns?(fs: FileSystem): Promise<PatternDetectionResult>;
+}
+```
+
+Important constraints:
+
+- Plugins read through `FileSystem`; they should not access `node:fs` directly.
+- `detectEntries()` must return package identity, package kind, language, optional version, and entry files.
+- `extractSymbols()` must return `ExtractionResult` v2 with stable symbols and relations.
+- `detectEntries()` is called before `extractSymbols()`; plugins may keep per-detection package context between those calls.
+
+### 2. ExtractionResult v2
+
+Every plugin returns:
+
+```ts
+{
+  version: "2",
+  meta: { extractedAt, pluginId, commitHash, language },
+  package: { name, kind, language, version? },
+  files: [{ path, language, lines }],
+  symbols: SymbolInfo[],
+  relations: RelationInfo[],
+  stats: { files, lines, exportedSymbols, internalSymbols, relations }
+}
+```
+
+`SymbolInfo` supports:
+
+- identity: `name`, `kind`, `visibility`, `file`, `line`, `endLine`
+- structure: nested `members`
+- type surfaces: `params`, `returnType`, `typeAnnotation`, `extends`, `implements`, `propsType`, `unionValues`
+- source documentation: `doc`
+
+`RelationInfo` supports code edges such as `imports`, `imports_type`, `calls`, `extends`, `implements`, `param_type`, `return_type`, `of_type`, `depends_on`, and `contains`.
+
+### 3. Repository Runner Protocol
+
+The package exposes `c4a-extract-code`, a NDJSON runner used by `context capture --code`.
+
+Input is JSON on stdin:
+
+```json
+{
+  "repoPath": "/path/to/repo",
+  "modules": ["packages/example"],
+  "commitHash": "abc123",
+  "pathFilter": {},
+  "plugins": [{ "package": "@c4a/extract-ts", "exportName": "TypeScriptPlugin" }],
+  "snapshot": {
+    "sourceId": "aspect:code:example",
+    "sourceSlug": "example",
+    "snapshotId": "code-abc123-deadbeef",
+    "codeSnapshotContractVersion": "<contract-version>",
+    "scriptHash": "sha256:...",
+    "toolchain": {
+      "manager_package": "@c4a/context-cli",
+      "manager_version": "<manager-version>",
+      "runner_package": "@c4a/extract",
+      "runner_package_version": "<runner-version>",
+      "runner_bin": "c4a-extract-code",
+      "plugin_package": "@c4a/extract-ts",
+      "plugin_package_version": "<plugin-version>",
+      "plugin_export": "TypeScriptPlugin"
+    }
+  }
+}
+```
+
+Output is one JSON object per line:
+
+- `{ "type": "progress", "phase": "scanning|parsing|uploading", ... }`
+- `{ "type": "module_error", "module_name": "...", "module_path": "...", "error": "..." }`
+- `{ "type": "summary", "extraction": ..., "snapshot": ... }`
+- `{ "type": "error", "code": "runner-failed", "message": "..." }`
+
+The runner does not write `.context` directly. It returns snapshot files in the summary; `@c4a/context-cli` validates and writes them atomically.
+
+### 4. Raw Code Snapshot Contract
 
-- `generateDigest(result)` — ExtractionResult -> DigestData (exported/internal symbol partitioning)
-- `generateSymbolDiff(current, previous)` — compare two digests, output `{ added, removed }`
-- `DigestData` / `SymbolDiff` types
+When `snapshot` input is provided, the runner builds these files:
 
-### Shared utilities
+| File | Purpose |
+|---|---|
+| `source.yaml` | Source manifest for the code aspect source |
+| `manifest.json` | Snapshot manifest: contract version, toolchain, counts, hash, dirty state |
+| `_meta.yaml` | Backward-compatible snapshot metadata and input summary |
+| `digests.jsonl` | Per-module digest rows with version, hash, dirty state, and digest payload |
+| `source-files.jsonl` | Source-to-module/digest mapping |
+| `packages.jsonl` | Package rows: name, kind, language, module path, optional version/description |
+| `symbols.jsonl` | Flat symbol rows; nested members are flattened and retain package/module fields |
+| `edges.jsonl` | Code relation rows with package/module/version/hash fields |
 
-- `initParser()` / `parseFile()` — Tree-sitter parser initialization and per-file AST parsing
-- `scanSourceFiles()` / `detectModules()` / `detectModuleAt()` — file system scanning and module detection
-- `getGitCommitHash()` — reads the current git commit hash
-- `detectTechStack()` — identifies languages and frameworks in a repository
+`@c4a/context-cli` validates this contract before projection. Required fields include package/module identity, version labels on digest/source-file/edge rows, symbol identity fields, and matching edge/digest versions.
 
-## Writing a language plugin
+During projection, code-owned Sections receive code `source_ref` values derived from these rows:
 
-To add support for a new language (e.g., Python), create a new package `@c4a/extract-python` that implements `ExtractionPlugin`:
+- package rows: `src-N#package:<package>@<hash>`
+- symbol rows: `src-N#symbol:<locator>:<kind>@<hash>`
 
-```typescript
+These refs are verified against the raw code snapshot JSONL indexes. They are separate from prose evidence refs, because code snapshots use `evidence.mode: none` and do not create raw block manifests.
+
+## Writing a New Language Plugin
+
+Create a package such as `@c4a/extract-python` and export an `ExtractionPlugin`.
+
+Minimum requirements:
+
+1. Detect the language manifest in `canHandle()` and `detectEntries()`.
+2. Return stable package identity: package name, kind, language, and version when available.
+3. Return `subPackages` when one manifest represents a nested package layout.
+4. Resolve public entry files so exported symbols can be distinguished from internal symbols.
+5. Emit `SymbolInfo[]` with stable `name`, `kind`, `visibility`, `file`, `line`, and `endLine`.
+6. Emit `RelationInfo[]` for imports and important type/inheritance/use edges.
+7. Keep paths module-relative inside the plugin; the repository runner prefixes them to repo-relative paths.
+8. Register the plugin in the runner input used by `context capture --code`.
+
+Example skeleton:
+
+```ts
 import type {
+  EntryDetectionResult,
+  EntryFile,
   ExtractionPlugin,
   ExtractionResult,
-  EntryDetectionResult,
   FileSystem,
   ManifestInfo,
   SourceInfo,
-  EntryFile,
 } from "@c4a/extract";
 
 export class PythonPlugin implements ExtractionPlugin {
@@ -64,85 +165,28 @@ export class PythonPlugin implements ExtractionPlugin {
   readonly packageManagers = ["pip"];
 
   canHandle(source: SourceInfo): boolean {
-    // Return true if this source has a pyproject.toml or setup.py
-    return source.manifests.some(
-      (m) => m.type === "pyproject.toml",
-    );
+    return source.manifests.some((manifest) => manifest.type === "pyproject.toml");
   }
 
-  async detectEntries(
-    manifest: ManifestInfo,
-    fs: FileSystem,
-  ): Promise<EntryDetectionResult> {
-    // Parse pyproject.toml to find package name, kind, entry points
-    // Return: { package: { name, kind, language }, entries: [...] }
+  async detectEntries(manifest: ManifestInfo, fs: FileSystem): Promise<EntryDetectionResult> {
+    // Parse pyproject.toml/setup metadata and return package + entry files.
   }
 
-  async extractSymbols(
-    entries: EntryFile[],
-    fs: FileSystem,
-  ): Promise<ExtractionResult> {
-    // Use Tree-sitter with Python grammar to parse AST
-    // Extract SymbolInfo[] (functions, classes, etc.)
-    // Extract RelationInfo[] (imports, inheritance, etc.)
-    // Return ExtractionResult v2
+  async extractSymbols(entries: EntryFile[], fs: FileSystem): Promise<ExtractionResult> {
+    // Parse entry graph, classify exported/internal symbols, emit relations.
   }
 }
 ```
 
-### Plugin interface
-
-| Method | When called | Returns |
-|--------|------------|---------|
-| `canHandle(source)` | Registry resolves which plugin handles a source | `boolean` |
-| `detectEntries(manifest, fs)` | Layer 0: identify package info + entry files | `EntryDetectionResult` |
-| `extractSymbols(entries, fs)` | Layer 1: AST parsing + symbol extraction | `ExtractionResult` |
-| `detectPatterns?(fs)` | Layer 2 (optional): framework-specific patterns | `PatternDetectionResult` |
-
-### Registration
-
-Plugins are currently registered statically in the daemon:
-
-```typescript
-import { ExtractionPluginRegistry } from "@c4a/extract";
-import { TypeScriptPlugin } from "@c4a/extract-ts";
-
-const registry = new ExtractionPluginRegistry();
-registry.register(new TypeScriptPlugin());
-
-const plugin = registry.resolve(sourceInfo);
-if (plugin) {
-  const entries = await plugin.detectEntries(manifest, fs);
-  const result = await plugin.extractSymbols(entries.entries, fs);
-}
-```
-
-### FileSystem abstraction
-
-Plugins access files through the `FileSystem` interface, not directly via `node:fs`. This enables future API-based file loading without changing plugin code.
+## Relationship to Code Compile and Obsidian
 
-```typescript
-interface FileSystem {
-  readFile(path: string): Promise<string>;
-  readdir(path: string): Promise<string[]>;
-  exists(path: string): Promise<boolean>;
-  readJson<T = unknown>(path: string): Promise<T>;
-}
-```
+`@c4a/extract` is upstream of code compile; it does not render knowledge itself.
 
-### ExtractionResult v2 format
+- `context capture --code` uses the runner to produce raw code snapshots.
+- `context compile --code` consumes `packages.jsonl`, `symbols.jsonl`, `edges.jsonl`, and `digests.jsonl` to build package/category/symbol Nodes such as `pkg`, `pkg/components`, and `pkg/symbol/button`.
+- Obsidian Render reads the compiled Markdown, `_edges.yaml`, and `_external.yaml`. It does not read raw runner snapshots directly.
 
-```typescript
-{
-  version: "2",
-  meta: { extractedAt, pluginId, commitHash, language },
-  package: { name, kind, language },
-  files: [{ path, language, lines }],
-  symbols: SymbolInfo[],   // exported + internal
-  relations: RelationInfo[], // imports, calls, type refs, etc.
-  stats: { files, lines, exportedSymbols, internalSymbols, relations },
-}
-```
+That means language plugins affect Obsidian only through the compiled knowledge graph: better symbols/relations produce better symbol Nodes, graph edges, source refs, and source-status chips.
 
 ## Development
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@c4a/extract",
-  "version": "0.5.35-beta.1",
+  "version": "0.5.36",
   "type": "module",
   "bin": {
     "c4a-extract-code": "./bin/c4a-extract-code.js"