@claudetools/tools 0.8.10 → 0.9.0

package/docs/research/2026-01-02-codedna-il-specification.md

@@ -0,0 +1,639 @@
+# CodeDNA-IL: Universal Intent Language Specification
+
+**Version:** 2.0.0-draft
+**Date:** 2026-01-02
+**Status:** Extended for Full Universality
+
+## Executive Summary
+
+CodeDNA-IL is a **universal standard language** for AI-driven code generation across ALL code types - not just web applications, but game engines, shaders, physics simulations, ML models, compilers, embedded systems, and literally anything that can be coded.
+
+### Why a New Language?
+
+1. **Current Entity DSL is too narrow** - Only describes data structures, can't express algorithms, flows, or transformations
+2. **No existing DSL is universal** - Every DSL is domain-specific (Prisma for DBs, GraphQL for APIs, Terraform for infra)
+3. **Different code requires different paradigms** - Sequential steps don't work for shaders, game loops, or physics
+4. **Token efficiency requires purpose-built syntax** - JSON/YAML are 10x more verbose than necessary
+5. **AI agents need structured intent, not code** - Natural language is ambiguous; formal languages are too syntactic
+
+### Research Foundation
+
+This specification is informed by:
+- [IRCoder research](https://arxiv.org/abs/2407.05411): "Natural language consistently emerges as the most effective intermediate representation"
+- [CrossTL architecture](https://arxiv.org/abs/2508.21256): Universal IR enabling bidirectional translation
+- [OpenRewrite's LST](https://www.moderne.ai/blog/automated-javascript-refactoring-at-enterprise-scale): Lossless Semantic Trees for cross-language refactoring
+- [IBM AI Refactoring](https://www.ibm.com/think/topics/ai-code-refactoring): Semantic-level code transformation patterns
+
+---
+
+## Three Dimensions of Code
+
+CodeDNA-IL addresses code across three orthogonal dimensions:
+
+### 1. DOMAIN (What are we building?)
+The `type:` field indicates the problem domain:
+
+| Type | Domain | Examples |
+|------|--------|----------|
+| `api` | Web services | REST, GraphQL, gRPC |
+| `service` | Business logic | Auth, payments, notifications |
+| `component` | UI elements | React, Vue, Svelte components |
+| `model` | Data structures | Entities, DTOs, schemas |
+| `shader` | GPU programs | Vertex, fragment, compute |
+| `layer` | ML building blocks | Attention, convolution, normalization |
+| `scene` | 3D/2D worlds | Game scenes, VR environments |
+| `physics` | Simulations | Rigid body, fluid, cloth |
+| `engine` | Game/app engines | Game loops, state machines |
+| `pipeline` | Data processing | ETL, streaming, batch |
+| `infrastructure` | Cloud resources | AWS, GCP, Kubernetes |
+| `protocol` | Network protocols | TCP handlers, WebSocket, custom |
+| `compiler` | Language tools | Lexers, parsers, code gen |
+| `embedded` | Hardware/IoT | Firmware, drivers, sensors |
+| `cli` | Command-line tools | Commands, flags, prompts |
+| `library` | Reusable modules | SDKs, utilities, frameworks |
+
+### 2. PARADIGM (How does it execute?)
+Different code requires different expression modes:
+
+| Keyword | Paradigm | Use Case |
+|---------|----------|----------|
+| `step:` | Sequential | Do A, then B, then C (APIs, scripts) |
+| `formula:` | Mathematical | Compute relationships (shaders, ML) |
+| `on:` | Event-driven | React to events (games, UI) |
+| `parallel:` | Concurrent | Do simultaneously (GPU, workers) |
+| `solve:` | Constraint-based | Make this true (physics, layout) |
+| `loop:` | Continuous | Frame-based execution (games, servers) |
+| `stream:` | Reactive | Data flow (pipelines, observables) |
+
+### 3. STYLE (How does it look/feel?)
+For visual/UX code, the `style:` block captures design intent:
+
+```
+style: {
+  design_system: shadcn
+  palette: { primary: amber, accent: slate }
+  typography: { heading: inter, body: system }
+  motion: subtle_spring
+  art_direction: stylized_cel_shading  # For games
+}
+```
+
+---
+
+## Design Principles
+
+### 1. Intent-Based, Not Syntax-Based
+CodeDNA-IL describes WHAT the code should do, not HOW it's written. The AI agent determines implementation details based on target language, framework, and constraints.
+
+### 2. Multi-Paradigm Expression
+Different types of code require different expression modes. A shader uses `formula:`, a game uses `loop:` + `on:`, an API uses `step:`. The language supports all paradigms natively.
+
+### 3. Universal Domain Coverage
+One language works for ANY code type - web apps, games, ML, infrastructure, embedded, compilers, protocols - everything.
+
+### 4. Dual-Mode Operation
+- **DEFINE mode**: Create new code units
+- **TRANSFORM mode**: Modify existing code
+
+### 5. Design-Aware
+For visual code (games, UI, graphics), the `style:` block captures aesthetic intent alongside behavior.
+
+### 6. Token Efficiency
+Every token must carry meaning. No redundant syntax, no verbose delimiters.
+
+---
+
+## Core Syntax
+
+### Unit Block (Universal Building Block)
+
+```
+unit <name> {
+  type: <unit_type>
+  purpose: "<description>"
+
+  <attributes>
+  <operations>
+  <constraints>
+}
+```
+
+**Unit Types:**
+- `api` - REST/GraphQL endpoint
+- `service` - Business logic component
+- `model` - Data model/entity
+- `component` - UI component
+- `function` - Standalone function
+- `pipeline` - Data processing pipeline
+- `ml_model` - Machine learning model
+- `infrastructure` - Cloud resources
+- `cli` - Command-line tool
+- `library` - Reusable module
+
+### Inputs and Outputs
+
+```
+inputs: (name:type, name:type:modifier)
+outputs: (name:type)
+```
+
+**Type Shortcuts:**
+- `str` = string
+- `int` = integer
+- `dec` = decimal
+- `bool` = boolean
+- `dt` = datetime
+- `[]` suffix = array
+- `?` suffix = optional
+
+**Modifiers:**
+- `unique` - Must be unique
+- `required` - Cannot be null
+- `hashed` - Should be hashed
+- `email` - Email format
+- `min(n)` / `max(n)` - Value constraints
+- `default(v)` - Default value
+
+### Operations
+
+```
+op <name> {
+  step: <natural language description>
+  step: <natural language description>
+  return: <value>
+}
+```
+
+Steps use natural language that AI understands semantically. Multiple steps indicate sequence.
+
+### Constraints
+
+```
+constraint: <requirement>
+```
+
+Constraints are domain-specific requirements that the generated code must satisfy.
+
+---
+
+## Examples by Domain
+
+### Web Application (API Service)
+
+```
+unit UserService {
+  type: service
+  purpose: "Handle user authentication and session management"
+
+  inputs: (email:str:email, password:str)
+  outputs: (token:str, user:User)
+
+  op authenticate {
+    step: validate credentials against database
+    step: generate JWT with user claims, 24h expiry
+    step: record login timestamp
+    return: token, user
+  }
+
+  op logout {
+    step: invalidate token in blacklist
+    step: clear user session
+  }
+
+  op refresh {
+    step: verify refresh token validity
+    step: generate new access token
+    return: token
+  }
+
+  constraint: bcrypt(12) for password hashing
+  constraint: rate_limit(5/min) on authenticate
+  constraint: https_only
+}
+```
+
+### Systems Programming (Rust TCP Handler)
+
+```
+unit tcp_handler {
+  type: function
+  purpose: "Handle incoming TCP connections with HTTP parsing"
+  language_hint: rust
+
+  inputs: (stream:TcpStream)
+  outputs: (Result<Response, IoError>)
+
+  lifecycle: async, owned
+
+  op handle {
+    step: read request bytes from stream with timeout
+    step: parse HTTP method and headers
+    step: route to appropriate handler based on path
+    step: serialize response and write to stream
+    return: response
+  }
+
+  constraint: timeout(30s)
+  constraint: max_request_size(10mb)
+  constraint: graceful_shutdown_aware
+}
+```
+
+### Data Pipeline (ETL)
+
+```
+unit user_analytics_pipeline {
+  type: pipeline
+  purpose: "Extract user data, compute metrics, load to analytics warehouse"
+
+  source: postgres.public.users
+  sink: bigquery.analytics.user_metrics
+  schedule: daily(02:00 UTC)
+
+  stage extract {
+    select: id, email, created_at, last_login_at
+    filter: created_at > {7_days_ago}
+  }
+
+  stage transform {
+    derive: days_active = count_distinct(activity_dates)
+    derive: engagement_score = days_active / days_since_signup
+    normalize: email -> lowercase
+    drop: created_at
+  }
+
+  stage load {
+    mode: upsert
+    key: id
+    partition_by: date(loaded_at)
+  }
+
+  constraint: idempotent
+  constraint: retry(3, exponential_backoff)
+}
+```
+
+### Machine Learning (Model Definition)
+
+```
+unit image_classifier {
+  type: ml_model
+  purpose: "Classify images into 10 categories"
+  framework_hint: pytorch
+
+  architecture: cnn {
+    layer: conv2d(in:3, out:64, kernel:3, padding:1)
+    layer: batch_norm(64)
+    layer: relu
+    layer: max_pool(2)
+    layer: conv2d(in:64, out:128, kernel:3)
+    layer: relu
+    layer: adaptive_avg_pool(1)
+    layer: flatten
+    layer: linear(128, 10)
+  }
+
+  training {
+    loss: cross_entropy
+    optimizer: adamw(lr:0.001, weight_decay:0.01)
+    scheduler: cosine_annealing(T_max:100)
+    epochs: 100
+    batch_size: 64
+  }
+
+  data {
+    source: imagenet_subset
+    split: train(0.8), val(0.1), test(0.1)
+    augment: random_crop(224), horizontal_flip, color_jitter
+    normalize: imagenet_stats
+  }
+
+  constraint: mixed_precision(fp16)
+  constraint: gradient_clipping(1.0)
+}
+```
+
+### Infrastructure (Cloud Resources)
+
+```
+unit production_cluster {
+  type: infrastructure
+  purpose: "Scalable web application infrastructure on AWS"
+  provider: aws
+  region: us-east-1
+
+  resource vpc:main {
+    cidr: 10.0.0.0/16
+    availability_zones: 3
+    enable_dns_hostnames: true
+  }
+
+  resource ecs_cluster:app {
+    name: "production"
+    capacity_providers: [fargate, fargate_spot]
+    default_provider: fargate_spot(weight:80)
+  }
+
+  resource service:web {
+    cluster: ecs_cluster:app
+    task_definition: web_task
+    desired_count: 3
+    deployment: rolling(max_percent:200, min_healthy:100)
+  }
+
+  resource alb:public {
+    subnets: vpc:main.public_subnets
+    security_group: allow_https
+  }
+
+  scaling web {
+    metric: cpu_utilization
+    target: 70%
+    scale_in_cooldown: 300s
+    scale_out_cooldown: 60s
+    min_capacity: 2
+    max_capacity: 20
+  }
+
+  constraint: encrypted_at_rest
+  constraint: private_subnets_only(for:ecs)
+  constraint: waf_enabled
+}
+```
+
+### CLI Tool
+
+```
+unit backup_cli {
+  type: cli
+  purpose: "Backup and restore files with incremental support"
+
+  command backup {
+    args: (source:path:required, destination:path:required)
+    flags: {
+      --compress: "Enable gzip compression"
+      --incremental: "Only backup changed files"
+      --exclude: "Glob patterns to exclude":str[]
+    }
+
+    op run {
+      step: validate source path exists
+      step: scan source directory for files
+      step: if incremental, load previous manifest and diff
+      step: copy files to destination, compressing if enabled
+      step: generate new manifest with checksums
+      step: write manifest to destination
+    }
+  }
+
+  command restore {
+    args: (backup:path:required, target:path:required)
+    flags: {
+      --verify: "Verify checksums after restore"
+      --dry-run: "Show what would be restored"
+    }
+
+    op run {
+      step: read and validate manifest
+      step: if dry-run, print file list and exit
+      step: decompress and copy files to target
+      step: if verify, check all checksums
+    }
+  }
+
+  constraint: cross_platform(linux, macos, windows)
+  constraint: progress_bar
+}
+```
+
+---
+
+## Transform Mode (Modifications)
+
+Transform mode allows modifying existing code without redefining everything.
+
+### Syntax
+
+```
+transform @<unit_reference> {
+  <operations>
+}
+```
+
+### Operations
+
+| Operation | Description |
+|-----------|-------------|
+| `add field: name:type` | Add a new field/property |
+| `remove field: name` | Remove a field/property |
+| `rename field: old -> new` | Rename a field/property |
+| `change field: name -> type` | Change field type |
+| `add op: name { ... }` | Add new operation |
+| `remove op: name` | Remove operation |
+| `modify op: name { ... }` | Modify existing operation |
+| `add constraint: ...` | Add constraint |
+| `remove constraint: ...` | Remove constraint |
+
+### Examples
+
+```
+# Add email verification to UserService
+transform @UserService {
+  add field: emailVerified:bool:default(false)
+  add field: verificationToken:str?
+
+  add op: sendVerificationEmail {
+    step: generate random verification token
+    step: save token to user record
+    step: send email with verification link
+  }
+
+  add op: verifyEmail {
+    inputs: (token:str)
+    step: find user by verification token
+    step: set emailVerified to true
+    step: clear verification token
+  }
+
+  modify op: authenticate {
+    after "validate credentials": check emailVerified is true
+  }
+}
+
+# Optimize database queries
+transform @UserRepository {
+  add constraint: index(email)
+  add constraint: index(created_at)
+
+  modify op: findByEmail {
+    add: use prepared statement
+    add: connection pooling
+  }
+}
+
+# Add caching layer
+transform @ProductService {
+  add field: cache:RedisCache
+
+  modify op: getById {
+    before: check cache for product
+    after "fetch from database": store in cache(ttl:1h)
+  }
+}
+```
+
+---
+
+## Compose Mode (Combining Units)
+
+For building larger systems from smaller units.
+
+```
+compose OrderSystem {
+  units: [
+    UserService,
+    ProductService,
+    CartService,
+    PaymentService,
+    OrderService
+  ]
+
+  flow: checkout {
+    step: CartService.getItems -> items
+    step: PaymentService.charge(items.total) -> payment
+    step: OrderService.create(items, payment) -> order
+    step: notify(UserService.current, order)
+  }
+
+  integration: {
+    UserService -> ProductService: share user context
+    CartService -> ProductService: lookup prices
+    OrderService -> PaymentService: payment webhooks
+  }
+}
+```
+
+---
+
+## Token Efficiency Analysis
+
+| Representation | UserService Example | Token Count |
+|----------------|---------------------|-------------|
+| Natural language | "Create a user authentication service with JWT..." | ~150 tokens |
+| TypeScript code | Full implementation with types, error handling | ~400 tokens |
+| JSON Schema | Detailed API schema | ~300 tokens |
+| Entity DSL (current) | `User(email:str, password:str:hashed)` | ~30 tokens |
+| CodeDNA-IL | Full spec with operations and constraints | ~80 tokens |
+
+**CodeDNA-IL is 5x more efficient than generated code while capturing 10x more semantic information than Entity DSL.**
+
+---
+
+## Language Hints and Target Mapping
+
+The `language_hint` and `framework_hint` attributes guide code generation:
+
+```
+unit UserController {
+  type: api
+  framework_hint: express  # Generates Express.js
+  # OR
+  framework_hint: fastapi  # Generates FastAPI Python
+  # OR
+  framework_hint: hono     # Generates Hono TypeScript
+}
+```
+
+Without hints, the AI infers from project context or asks.
+
+---
+
+## Implementation Roadmap
+
+### Phase 1: Parser
+- [ ] Define formal grammar (PEG or similar)
+- [ ] Build parser to AST
+- [ ] Validate semantic correctness
+
+### Phase 2: Code Generators
+- [ ] Extend existing 20+ generators to accept CodeDNA-IL
+- [ ] Add transform support to generators
+- [ ] Add compose support
+
+### Phase 3: IDE Integration
+- [ ] Syntax highlighting
+- [ ] Autocomplete
+- [ ] Validation
+
+### Phase 4: AI Training
+- [ ] Create CodeDNA-IL ↔ code pairs dataset
+- [ ] Fine-tune on generation tasks
+- [ ] Benchmark token efficiency
+
+---
+
+## Comparison with Alternatives
+
+| Feature | Entity DSL | GraphQL | Prisma | CodeDNA-IL |
+|---------|------------|---------|--------|------------|
+| Data models | ✓ | ✓ | ✓ | ✓ |
+| Operations/behavior | ✗ | Partial | ✗ | ✓ |
+| Constraints | Limited | ✗ | ✓ | ✓ |
+| Transformations | ✗ | ✗ | ✗ | ✓ |
+| Systems code | ✗ | ✗ | ✗ | ✓ |
+| ML models | ✗ | ✗ | ✗ | ✓ |
+| Infrastructure | ✗ | ✗ | ✗ | ✓ |
+| Token efficient | ✓ | ✗ | ✗ | ✓ |
+
+---
+
+## Conclusion
+
+CodeDNA-IL represents a fundamental shift from describing DATA to describing INTENT. By combining:
+
+1. **Structured blocks** for clarity
+2. **Natural language steps** for AI comprehension
+3. **Universal type system** for any domain
+4. **Transform operations** for modifications
+5. **Token-minimal syntax** for efficiency
+
+We create a specification language that can handle ANY code type, not just web applications.
+
+The key insight is that AI agents don't need a programming language - they need a SEMANTIC INTENT FORMAT that clearly expresses what to build, with enough structure to be unambiguous but enough flexibility to work across all domains.
+
+---
+
+## Appendix: Formal Grammar (Draft)
+
+```
+program     = unit* | transform* | compose*
+
+unit        = "unit" IDENT "{" unit_body "}"
+unit_body   = type_decl? purpose? hints* ios* op* constraint*
+
+type_decl   = "type:" UNIT_TYPE
+purpose     = "purpose:" STRING
+hints       = ("language_hint:" | "framework_hint:") IDENT
+ios         = ("inputs:" | "outputs:") "(" param_list ")"
+param_list  = param ("," param)*
+param       = IDENT ":" type_expr modifiers?
+type_expr   = BASE_TYPE | IDENT | type_expr "[]" | type_expr "?"
+modifiers   = (":" MODIFIER)+
+
+op          = "op" IDENT "{" op_body "}"
+op_body     = step* return?
+step        = "step:" TEXT
+return      = "return:" IDENT ("," IDENT)*
+
+constraint  = "constraint:" TEXT
+
+transform   = "transform" "@" IDENT "{" transform_op* "}"
+transform_op = add_op | remove_op | rename_op | modify_op
+
+compose     = "compose" IDENT "{" compose_body "}"
+compose_body = "units:" "[" IDENT* "]" flow* integration?
+```
+
+---
+
+*This specification is ready for implementation review. Feedback welcomed.*