@webneat/codewiki 0.0.5 → 0.0.6

package/README.md

@@ -1,14 +1,20 @@
 # @webneat/codewiki
 
-Core library for parsing, analyzing, and documenting TypeScript codebases.
+Core library for parsing, analyzing, and documenting TypeScript codebases with dependency tracking and topological ordering.
+
+## Overview
+
+`@webneat/codewiki` is a TypeScript codebase analysis engine that builds a comprehensive knowledge graph of your code. It extracts symbols, files, directories, and dependencies, then provides topological ordering to enable AI tools to generate contextual documentation with maximum context for each entity.
 
 ## Features
 
-- 🔍 **TypeScript AST Parsing** - Extract elements, imports, and exports
-- 📊 **Dependency Analysis** - Build complete dependency graphs
-- 🗄️ **SQLite Storage** - Fast, portable database for codebase metadata
-- 🤖 **AI-Powered Descriptions** - Generate contextual documentation
+- 🔍 **TypeScript AST Parsing** - Extract symbols, imports, exports, and file structure
+- 📊 **Dependency Analysis** - Build complete dependency graphs between files, symbols, and external packages
+- 🗄️ **SQLite Storage** - Fast, portable database with Drizzle ORM for codebase metadata
+- 📝 **Description Management** - Store and manage contextual documentation with smart ordering
 - ⚡ **Incremental Updates** - Git-based change detection for fast re-indexing
+- 🎯 **Topological Ordering** - Intelligent entity ordering to enable optimal AI context
+- 🔗 **Relationship Tracking** - Complete import/export relationships and internal symbol links
 
 ## Installation
 
@@ -16,44 +22,187 @@ Core library for parsing, analyzing, and documenting TypeScript codebases.
 pnpm add @webneat/codewiki
 ```
 
-## Usage
+## Quick Start
 
 ```typescript
-import { index_project, queries } from '@webneat/codewiki'
+import { create, load } from '@webneat/codewiki'
 
-// Index a TypeScript project
-await index_project('/path/to/project', {
+// Create a new wiki instance
+const wiki = await create({
   db_path: './codewiki.db',
-  excluded_paths: ['node_modules', 'dist'],
-  progress_callback: (status) => {
-    console.log(`${status.phase}: ${status.current}/${status.total}`)
-  },
+  project_path: './my-typescript-project',
+  description: 'My awesome TypeScript project'
 })
 
-// Query the database
-const element = await queries.find_element(db, 'MyFunction')
-const deps = await queries.element_dependencies(db, element.id)
+// Or load an existing wiki database
+const wiki = await load('./codewiki.db')
+
+// Update the wiki with latest changes
+await wiki.update()
+
+// Get counts of entities in different states
+const counts = wiki.counts()
+console.log(`Ready: ${counts.symbols.ready} symbols, ${counts.files.ready} files`)
+
+// Get next entities to describe (in topological order)
+const toDescribe = wiki.to_describe(10)
+if (toDescribe.type === 'symbols') {
+  console.log(`Next symbols to describe: ${toDescribe.items.map(s => s.name).join(', ')}`)
+}
 ```
 
-## Architecture
+## Core API
+
+### CodeWiki Interface
+
+The main interface provides access to all entity types:
+
+```typescript
+interface CodeWiki {
+  readonly config: Config
+  update(): Promise<void>                    // Re-index the project
+  counts(): WikiCounts                       // Get entity counts by state
+  to_describe(limit: number): EntitiesToDescribe // Get next entities for AI description
+  
+  files: FilesWiki           // File operations
+  symbols: SymbolsWiki       // Symbol/function/class operations  
+  directories: DirectoriesWiki // Directory operations
+  dependencies: DependenciesWiki // External dependency operations
+}
+```
+
+### Entity Operations
+
+Each entity type provides a consistent API:
+
+```typescript
+// Search and retrieve
+const files = wiki.files.search({ where: { state: 'ready' }, limit: 10 })
+const file = wiki.files.get(fileId)
+
+// Set AI-generated descriptions
+wiki.files.set_description(fileId, 'This file contains utility functions...')
+
+// Query relationships
+const importedFiles = wiki.files.imported_by_file(fileId)
+const usingSymbols = wiki.symbols.using_dependency(dependencyId)
+```
+
+## Entity Types
+
+### Files
+- **States**: `to_scan` → `to_describe` → `ready`
+- **Tracks**: File metadata, imports, exports, symbols
+- **Relationships**: Import relationships with other files and external dependencies
+
+### Symbols  
+- **States**: `to_link` → `to_describe` → `ready`
+- **Tracks**: Functions, classes, variables, types
+- **Relationships**: Internal symbol references and external dependency usage
+
+### Directories
+- **States**: `to_describe` → `ready`
+- **Tracks**: Directory structure and contents
+- **Relationships**: Parent-child relationships and contained files
+
+### External Dependencies
+- **States**: `to_describe` → `ready`  
+- **Tracks**: npm packages, built-in modules
+- **Relationships**: Usage by files and symbols
+
+## Topological Description Order
 
-This is the core library used by:
-- **codewiki-cli** - Command-line interface with MCP server
-- **codewiki-vscode** - VSCode extension
+The `to_describe()` method returns entities in an intelligent order that maximizes AI context:
+
+1. **External Dependencies** (no dependencies)
+   - Built-in packages first, then by usage count
+2. **Symbols** (depend on other symbols and dependencies)  
+   - Fewest missing dependencies first, then by complexity
+3. **Files** (contain symbols and have imports)
+   - Fewest missing descriptions first, then by file size
+4. **Directories** (contain files and subdirectories)
+   - Fewest missing content first, then by total size
+
+This ensures that when describing an entity, all its dependencies are already described.
+
+## CLI Usage
+
+The package includes a comprehensive CLI:
+
+```bash
+# Initialize and index a project
+codewiki update
+
+# Show current status
+codewiki status
+
+# Search entities
+codewiki files search --where.state=ready
+codewiki symbols search --where.name="myFunction"
+
+# Set descriptions
+codewiki symbols set-description 123 "This function does X..."
+
+# Query relationships
+codewiki files imported-by-file 456
+codewiki dependencies used-by-file 456
+```
+
+## Database Schema
+
+Uses SQLite with Drizzle ORM. Key tables:
+- `files` - File metadata and state
+- `symbols` - Function/class/variable definitions  
+- `directories` - Directory structure
+- `external_dependencies` - npm packages and built-ins
+- `imports` - File-to-file import relationships
+- `symbol_*_links` - Symbol dependency relationships
+
+## Configuration
+
+```typescript
+type Config = {
+  db_path: string              // SQLite database file path
+  project_path: string         // Root directory of TypeScript project
+  description?: string         // Optional project description
+  package_json_path?: string   // Optional package.json location
+  node_modules_paths?: string[] // Optional node_modules directories
+}
+```
 
 ## Development
 
 ```bash
-# Build
+# Install dependencies
+pnpm install
+
+# Build the library
 pnpm build
 
-# Test
+# Run tests
 pnpm test
 
-# Type check
+# Type checking
 pnpm typecheck
+
+# Database operations
+pnpm db:generate  # Generate migrations
+pnpm db:push      # Push schema changes
+pnpm db:studio    # Open Drizzle Studio
 ```
 
+## Architecture
+
+This is the core library that powers:
+- **codewiki-cli** - Command-line interface and MCP server
+- **codewiki-vscode** - VSCode extension (planned)
+
+The library is designed to be:
+- **Incremental** - Only processes changed files
+- **Performant** - Uses SQLite and efficient AST parsing
+- **Extensible** - Clean separation between parsing, storage, and API layers
+- **Type-safe** - Full TypeScript support throughout
+
 ## License
 
-MIT
+MIT © [Amine Ben hammou](https://github.com/webNeat)

package/dist/bin.cjs

@@ -567,7 +567,7 @@ var symbols2 = {
   where: {}
 };
 var external_dependencies2 = {
-  select: ["id", "package", "subpath", "name", "state", "error"],
+  select: ["id", "package", "subpath", "name", "is_builtin", "state", "error"],
   order_by: ["id", "asc"],
   limit: 20,
   offset: 0,
@@ -2928,12 +2928,23 @@ var describe = new import_commander3.Command("describe");
 describe.description("Show the next item to describe");
 describe.action(async () => {
   const wiki = await load_wiki();
-  const next = get_next_to_describe(wiki);
-  if (!next) return env.log("All items are described!");
-  if (next.type === "dependency") return describe_dependency(next.entity);
-  env.log(
-    `All external dependencies are described, and we don't support describing other items yet (we are implementing the right order ...). You are done for now!`
-  );
+  const result = wiki.to_describe(1);
+  if (result.type === "none") {
+    return env.log("All items are described!");
+  }
+  if (result.type === "dependencies" && result.items.length > 0) {
+    return describe_dependency(result.items[0]);
+  }
+  if (result.type === "symbols" && result.items.length > 0) {
+    return describe_symbol(wiki, result.items[0]);
+  }
+  if (result.type === "files" && result.items.length > 0) {
+    return describe_file(result.items[0]);
+  }
+  if (result.type === "directories" && result.items.length > 0) {
+    return describe_directory(result.items[0]);
+  }
+  env.log("No items to describe.");
 });
 function describe_dependency(dep) {
   env.log(import_dedent2.default`
@@ -2949,40 +2960,42 @@ function describe_dependency(dep) {
       \`codewiki dependencies set-description ${dep.id} <markdown-file>\`
   `);
 }
-function get_next_to_describe(wiki) {
-  const deps = wiki.dependencies.search({
-    where: { state: "to_describe" },
-    limit: 1,
-    select: ["id", "package", "subpath", "name", "version", "github_repo"]
-  });
-  if (deps.length > 0) {
-    return { type: "dependency", entity: deps[0] };
-  }
-  const symbols4 = wiki.symbols.search({
-    where: { state: "to_describe" },
-    limit: 1,
-    select: ["id", "name", "type", "file_id", "is_exported"]
-  });
-  if (symbols4.length > 0) {
-    return { type: "symbol", entity: symbols4[0] };
-  }
-  const files5 = wiki.files.search({
-    where: { state: "to_describe" },
-    limit: 1,
-    select: ["id", "path"]
-  });
-  if (files5.length > 0) {
-    return { type: "file", entity: files5[0] };
-  }
-  const directories4 = wiki.directories.search({
-    where: { state: "to_describe" },
-    limit: 1,
-    select: ["id", "path"]
-  });
-  if (directories4.length > 0) {
-    return { type: "directory", entity: directories4[0] };
-  }
-  return null;
+function describe_symbol(wiki, sym) {
+  const file = wiki.files.get(sym.file_id);
+  env.log(import_dedent2.default`
+    Your task is to describe the following symbol:
+
+    name: ${sym.name}
+    type: ${sym.type}
+    file: ${file?.path || "unknown"}
+    exported: ${sym.is_exported}
+
+    Write a concise developer documentation for this ${sym.type} into a markdown file, then run:
+
+    codewiki symbols set-description ${sym.id} <markdown-file>
+  `);
+}
+function describe_file(file) {
+  env.log(import_dedent2.default`
+    Your task is to describe the following file:
+
+    path: ${file.path}
+
+    Write a concise description of this file's purpose and contents into a markdown file, then run:
+
+    codewiki files set-description ${file.id} <markdown-file>
+  `);
+}
+function describe_directory(dir) {
+  env.log(import_dedent2.default`
+    Your task is to describe the following directory:
+
+    path: ${dir.path}
+
+    Write a concise description of this directory's purpose and contents into a markdown file, then run:
+
+    codewiki directories set-description ${dir.id} <markdown-file>
+  `);
 }
 
 // src/cli/commands/files/index.ts