@kiyeonjeon21/datacontext 0.3.2 → 0.4.0

package/docs/KNOWLEDGE_GRAPH.md

@@ -0,0 +1,540 @@
+# Knowledge Graph Architecture
+
+> **Last Updated:** 2026-01-01  
+> **Version:** 2.0.0  
+> **Status:** Implemented (Phase 1.5)
+
+This document describes the Knowledge Graph architecture in DataContext, including data structures, graph traversal methods, and usage patterns.
+
+---
+
+## Overview
+
+DataContext's Knowledge Store has evolved from a simple key-value storage to a **graph-based knowledge model**. This enables:
+
+1. **Relationship Discovery**: Automatically find how tables connect
+2. **Join Path Finding**: Calculate optimal join paths between tables
+3. **Context Enrichment**: Provide AI with relationship information
+4. **Schema Understanding**: Help AI understand data model structure
+
+```
+┌─────────────────────────────────────────────────────────────────┐
+│                     Knowledge Graph                             │
+│                                                                 │
+│   [users] ◄────────── [orders] ────────────► [products]        │
+│      │                    │                       │             │
+│      │                    ▼                       │             │
+│      │              [order_items] ◄───────────────┘             │
+│      │                                                          │
+│      ▼                                                          │
+│   [profiles]                                                    │
+│                                                                 │
+│   Legend:                                                       │
+│   ────► Foreign Key relationship                                │
+│   ◄──── Reverse relationship                                    │
+│                                                                 │
+└─────────────────────────────────────────────────────────────────┘
+```
+
+---
+
+## Data Model
+
+### Graph Structure
+
+| Concept | Implementation | Description |
+|---------|---------------|-------------|
+| **Nodes** | `TableDescription` | Tables with business context |
+| **Edges** | `TableRelationship` | Relationships between tables |
+| **Annotations** | `QueryExample`, `BusinessRule`, `BusinessTerm` | Context attached to nodes |
+
+### Core Types
+
+#### TableRef — Qualified Table Reference
+
+```typescript
+interface TableRef {
+  database?: string;  // Optional for multi-DB (future)
+  schema: string;     // e.g., "public"
+  table: string;      // e.g., "users"
+}
+```
+
+Enables fully qualified references like `analytics.public.events` for future multi-database support.
+
+#### TableRelationship — Graph Edge
+
+```typescript
+interface TableRelationship extends KnowledgeMeta {
+  type: 'table_relationship';
+  
+  // Graph edge endpoints
+  from: TableRef;                    // Source table (FK side)
+  to: TableRef;                      // Target table (referenced side)
+  
+  // Relationship metadata
+  relationshipType: RelationshipType;  // 'foreign_key' | 'implicit_join' | 'manual'
+  joinCondition: string;               // SQL: "orders.user_id = users.id"
+  cardinality?: RelationshipCardinality;  // 'one-to-one' | 'one-to-many' | etc.
+  
+  // Column details
+  fromColumns?: string[];            // ["user_id"]
+  toColumns?: string[];              // ["id"]
+  
+  // Additional context
+  isPreferred: boolean;              // Preferred join path when multiple exist
+  constraintName?: string;           // FK constraint name from DB
+  notes?: string;                    // Human-readable notes
+}
+```
+
+### Relationship Types
+
+| Type | Source | Description |
+|------|--------|-------------|
+| `foreign_key` | Auto-harvested | From DB FK constraints |
+| `implicit_join` | Learned | Inferred from query patterns (future) |
+| `manual` | User-defined | Manually added by user |
+
+### Cardinality
+
+| Value | Description | Example |
+|-------|-------------|---------|
+| `one-to-one` | Each row maps to exactly one row | user ↔ profile |
+| `one-to-many` | One row maps to many rows | user → orders |
+| `many-to-one` | Many rows map to one row | orders → user |
+| `many-to-many` | Requires junction table | students ↔ courses |
+
+---
+
+## Usage
+
+### Auto-Harvesting Relationships
+
+Foreign keys are automatically converted to relationships during harvesting:
+
+```typescript
+// Harvest database metadata
+const result = await service.harvest('public');
+
+console.log(`Relationships added: ${result.relationshipsAdded}`);
+
+// Result includes:
+// - tablesProcessed: 10
+// - descriptionsAdded: 5
+// - columnsAdded: 20
+// - relationshipsAdded: 8  ← NEW
+```
+
+### Querying Relationships
+
+#### Get Related Tables (1-hop)
+
+```typescript
+// Find all tables connected to 'orders'
+const relationships = knowledge.getRelatedTables('orders', 'public');
+
+// Returns:
+// - orders → users (user_id → id)
+// - orders → products (product_id → id)
+// - order_items → orders (order_id → id)
+```
+
+#### Find Join Path (Multi-hop)
+
+```typescript
+// Find shortest path from order_items to customers
+const path = knowledge.findJoinPath('order_items', 'customers', 'public');
+
+// Returns array of relationships:
+// [
+//   { from: 'order_items', to: 'orders', joinCondition: '...' },
+//   { from: 'orders', to: 'customers', joinCondition: '...' }
+// ]
+```
+
+#### Get Relationship Between Tables
+
+```typescript
+const rel = knowledge.getRelationshipBetween('orders', 'users', 'public');
+
+if (rel) {
+  console.log(`Join: ${rel.joinCondition}`);
+  // "orders.user_id = users.id"
+}
+```
+
+### Adding Manual Relationships
+
+```typescript
+// Add an implicit join (no FK in database)
+await knowledge.addRelationship({
+  ...createKnowledgeMeta('user', schemaHash),
+  type: 'table_relationship',
+  from: createTableRef('events', 'public'),
+  to: createTableRef('users', 'public'),
+  relationshipType: 'implicit_join',
+  joinCondition: 'events.actor_id = users.id',
+  cardinality: 'many-to-one',
+  isPreferred: false,
+  notes: 'Legacy system - no FK constraint exists',
+});
+
+// Or use the convenience method for FK-style relationships
+await knowledge.addRelationshipFromFK({
+  fromTable: 'orders',
+  fromColumn: 'user_id',
+  toTable: 'users',
+  toColumn: 'id',
+});
+```
+
+### Graph Summary
+
+```typescript
+const summary = knowledge.getGraphSummary();
+
+// Returns:
+// {
+//   nodeCount: 15,              // Tables with descriptions
+//   edgeCount: 12,              // Relationships
+//   tablesWithRelationships: 10,
+//   isolatedTables: 5,          // Tables with no relationships
+//   relationshipsByType: {
+//     foreign_key: 10,
+//     implicit_join: 2,
+//     manual: 0
+//   }
+// }
+```
+
+### Building AI Context with Relationships
+
+```typescript
+// Standard context (descriptions, rules, examples)
+const context = knowledge.buildContext(['orders', 'users']);
+
+// Enhanced context including relationships
+const fullContext = knowledge.buildContextWithRelationships(['orders', 'users']);
+
+// Output includes:
+// ## Table Relationships
+// Use these relationships when joining tables:
+//
+// - **public.orders** → **public.users**
+//   Join: `orders.user_id = users.id`
+//   Cardinality: many-to-one
+```
+
+---
+
+## Data Flow
+
+### Harvesting Flow
+
+```
+┌──────────────┐     ┌──────────────┐     ┌──────────────┐
+│   Database   │     │   Harvester  │     │  Knowledge   │
+│              │     │              │     │    Store     │
+└──────┬───────┘     └──────┬───────┘     └──────┬───────┘
+       │                    │                    │
+       │  1. Query FK       │                    │
+       │◄───────────────────│                    │
+       │                    │                    │
+       │  2. Return FK info │                    │
+       │────────────────────►                    │
+       │                    │                    │
+       │                    │ 3. Convert to      │
+       │                    │    Relationship    │
+       │                    │────────────────────►
+       │                    │                    │
+       │                    │                    │ 4. Store
+       │                    │                    │    (dedupe)
+       │                    │                    │
+```
+
+### Query Assistance Flow
+
+```
+┌──────────────┐     ┌──────────────┐     ┌──────────────┐
+│     AI       │     │  DataContext │     │  Knowledge   │
+│              │     │   Service    │     │    Store     │
+└──────┬───────┘     └──────┬───────┘     └──────┬───────┘
+       │                    │                    │
+       │ "Join orders and   │                    │
+       │  customers"        │                    │
+       │────────────────────►                    │
+       │                    │                    │
+       │                    │ findJoinPath()     │
+       │                    │────────────────────►
+       │                    │                    │
+       │                    │ [orders→users→     │
+       │                    │  customers]        │
+       │                    │◄───────────────────│
+       │                    │                    │
+       │ Context with       │                    │
+       │ join conditions    │                    │
+       │◄───────────────────│                    │
+```
+
+---
+
+## Storage Format
+
+Knowledge is stored in JSON files at `~/.datacontext/{database_id}.json`:
+
+```json
+{
+  "version": "2.0.0",
+  "databaseId": "mydb",
+  "schemaHash": "abc123...",
+  "lastSyncAt": "2026-01-01T00:00:00.000Z",
+  
+  "tableDescriptions": [...],
+  
+  "tableRelationships": [
+    {
+      "id": "1234-abc",
+      "type": "table_relationship",
+      "from": { "schema": "public", "table": "orders" },
+      "to": { "schema": "public", "table": "users" },
+      "relationshipType": "foreign_key",
+      "joinCondition": "orders.user_id = users.id",
+      "cardinality": "many-to-one",
+      "fromColumns": ["user_id"],
+      "toColumns": ["id"],
+      "isPreferred": true,
+      "source": "auto",
+      "confidence": 0.8,
+      "schemaHash": "abc123...",
+      "createdAt": "2026-01-01T00:00:00.000Z",
+      "updatedAt": "2026-01-01T00:00:00.000Z",
+      "lastVerifiedAt": "2026-01-01T00:00:00.000Z"
+    }
+  ],
+  
+  "queryExamples": [...],
+  "businessRules": [...],
+  "businessTerms": [...]
+}
+```
+
+---
+
+## Performance Considerations
+
+### Current Implementation (JSON)
+
+- **Node count**: < 500 tables
+- **Edge count**: < 2,000 relationships
+- **Query complexity**: 2-hop traversal in < 10ms
+
+### Future: SQLite Migration
+
+For larger graphs:
+
+```sql
+CREATE TABLE relationships (
+  id TEXT PRIMARY KEY,
+  from_schema TEXT NOT NULL,
+  from_table TEXT NOT NULL,
+  to_schema TEXT NOT NULL,
+  to_table TEXT NOT NULL,
+  relationship_type TEXT NOT NULL,
+  join_condition TEXT,
+  -- ... metadata
+);
+
+CREATE INDEX idx_rel_from ON relationships(from_table);
+CREATE INDEX idx_rel_to ON relationships(to_table);
+```
+
+---
+
+## Migration from v1.x
+
+Data files from v1.x are automatically migrated:
+
+1. `tableRelationships` array is initialized if missing
+2. Version is updated to `2.0.0`
+3. Existing data is preserved
+
+```typescript
+// Migration happens automatically on load()
+await knowledge.load();
+
+// Old files without tableRelationships work fine
+// Run harvest to populate relationships
+await service.harvest('public');
+```
+
+---
+
+## API Reference
+
+### KnowledgeStore Methods
+
+| Method | Description |
+|--------|-------------|
+| `getRelationships()` | Get all relationships |
+| `getRelatedTables(table, schema)` | Get 1-hop related tables |
+| `getOutgoingRelationships(table, schema)` | Get relationships where table is source |
+| `getIncomingRelationships(table, schema)` | Get relationships where table is target |
+| `findJoinPath(from, to, schema)` | Find shortest path (BFS) |
+| `getRelationshipBetween(from, to, schema)` | Get direct relationship |
+| `addRelationship(rel)` | Add a relationship |
+| `addRelationshipFromFK(params)` | Convenience method for FK |
+| `addRelationships(rels)` | Batch add |
+| `deleteRelationship(id)` | Delete by ID |
+| `getGraphSummary()` | Get graph statistics |
+| `buildContextWithRelationships(tables)` | Build AI context with relationships |
+
+### Factory Functions
+
+| Function | Description |
+|----------|-------------|
+| `createTableRef(table, schema?, database?)` | Create qualified table reference |
+| `tableRefToString(ref)` | Convert to string "schema.table" |
+| `tableRefsEqual(a, b)` | Compare two TableRefs |
+| `createRelationshipFromFK(params)` | Create relationship from FK info |
+
+---
+
+## MCP Tools
+
+Knowledge Graph functionality is exposed via MCP tools for AI assistants:
+
+### `get_context` (Enhanced)
+
+Now includes relationship information:
+
+```json
+{
+  "context": "## Table: orders\n...\n\n## Table Relationships\n...",
+  "tables": ["orders", "users"],
+  "relatedTables": ["order_items", "products"],
+  "relationships": [
+    {
+      "from": "public.orders",
+      "to": "public.users",
+      "type": "foreign_key",
+      "joinCondition": "orders.user_id = users.id",
+      "cardinality": "many-to-one"
+    }
+  ]
+}
+```
+
+### `find_join_path`
+
+Find optimal path between two tables:
+
+```json
+// Request
+{ "fromTable": "order_items", "toTable": "users" }
+
+// Response
+{
+  "found": true,
+  "hops": 2,
+  "path": [
+    { "from": "order_items", "to": "orders", "joinCondition": "..." },
+    { "from": "orders", "to": "users", "joinCondition": "..." }
+  ],
+  "sqlHint": "FROM order_items\nJOIN orders ON ...\nJOIN users ON ..."
+}
+```
+
+### `get_relationships`
+
+Get all relationships, optionally filtered:
+
+```json
+// Request
+{ "table": "orders" }
+
+// Response
+{
+  "count": 3,
+  "relationships": [...]
+}
+```
+
+### `get_graph_summary`
+
+Get graph statistics:
+
+```json
+{
+  "nodeCount": 15,
+  "edgeCount": 12,
+  "tablesWithRelationships": 10,
+  "isolatedTables": 5
+}
+```
+
+---
+
+## REST API Endpoints
+
+### `GET /api/relationships?table=orders`
+
+Returns all relationships, optionally filtered by table.
+
+### `GET /api/relationships/path?from=order_items&to=users`
+
+Find join path between two tables.
+
+### `GET /api/graph/summary`
+
+Get Knowledge Graph summary statistics.
+
+---
+
+## VS Code Extension Commands
+
+| Command | Description |
+|---------|-------------|
+| `DataContext: Show Table Relationships` | Show all relationships in a markdown view |
+| `DataContext: Find Join Path` | Interactive picker to find path between tables |
+| `DataContext: Show Knowledge Graph Summary` | Display graph statistics |
+
+---
+
+## Future Enhancements
+
+### Phase 2: Implicit Join Learning
+
+Learn relationships from query patterns:
+
+```typescript
+// When queries frequently join on events.actor_id = users.id
+// but no FK exists, create implicit_join relationship
+```
+
+### Phase 2+: Multi-Database
+
+```typescript
+// Cross-database relationships
+const relationship: TableRelationship = {
+  from: { database: 'orders_db', schema: 'public', table: 'orders' },
+  to: { database: 'users_db', schema: 'public', table: 'users' },
+  // ...
+};
+```
+
+### Phase 3: Graph Visualization
+
+- Generate Mermaid diagrams
+- Interactive graph explorer
+- Lineage visualization
+
+---
+
+## Related Documentation
+
+- [ARCHITECTURE.md](./ARCHITECTURE.md) — Overall system architecture
+- [KNOWLEDGE_TYPES.md](./KNOWLEDGE_TYPES.md) — All knowledge type definitions
+- [API.md](./API.md) — REST API reference
+
+