kbs 0.0.1 → 0.1.0

data/docs/architecture/index.md

@@ -0,0 +1,277 @@
+# Architecture Overview
+
+KBS is built on a layered architecture that separates concerns while maintaining high performance.
+
+## System Layers
+
+![KBS System Layers](../assets/images/system-layers.svg)
+
+*KBS uses a layered architecture where facts flow from your application through the DSL, RETE engine, and working memory to one of two storage backends.*
+
+## Core Components
+
+### 1. RETE Engine
+
+The heart of KBS. Implements Charles Forgy's RETE algorithm with modern optimizations.
+
+**Key Files:**
+- `lib/kbs/rete_engine.rb` - Main engine coordinator
+- `lib/kbs/alpha_memory.rb` - Pattern-level fact storage
+- `lib/kbs/beta_memory.rb` - Token (partial match) storage
+- `lib/kbs/join_node.rb` - Inter-condition joins
+- `lib/kbs/negation_node.rb` - Negated condition handling
+- `lib/kbs/production_node.rb` - Rule firing coordination
+
+**Responsibilities:**
+- Compile rules into discrimination networks
+- Propagate fact changes through the network
+- Maintain partial matches (tokens)
+- Fire rules when all conditions are satisfied
+
+**Learn more**: [RETE Algorithm Details](rete-algorithm.md)
+
+### 2. Working Memory
+
+Stores facts and notifies the RETE engine of changes using the Observer pattern.
+
+**Variants:**
+- **`WorkingMemory`**: Transient in-memory storage
+- **`Blackboard::Memory`**: Persistent storage with audit trails
+
+**Responsibilities:**
+- Store facts
+- Notify observers when facts are added/removed
+- Support queries and bulk operations
+
+**Learn more**: [Blackboard Architecture](blackboard.md)
+
+### 3. DSL Layer
+
+Provides a Ruby-native interface for defining rules, conditions, and patterns.
+
+**Key Classes:**
+- `Rule` - Production rule with conditions and actions
+- `Condition` - Pattern specification for fact matching
+- `Fact` - Knowledge representation unit
+
+**Example:**
+```ruby
+Rule.new("alert") do |r|
+  r.conditions = [
+    Condition.new(:sensor, { temp: :t? }),
+    Condition.new(:threshold, { max: :max? })
+  ]
+
+  r.action = lambda { |facts, bindings|
+    puts "Alert!" if bindings[:t?] > bindings[:max?]
+  }
+end
+```
+
+### 4. Blackboard System
+
+Multi-agent collaboration framework with persistent shared memory.
+
+**Components:**
+- **Memory** - Central workspace for facts
+- **MessageQueue** - Priority-based agent communication
+- **AuditLog** - Complete history of changes
+- **Persistence** - Pluggable storage backends (SQLite, Redis, Hybrid)
+
+**Use Cases:**
+- Multi-agent problem solving
+- Audit requirements
+- Long-running systems
+- Distributed reasoning
+
+**Learn more**: [Blackboard System Details](blackboard.md)
+
+## Data Flow
+
+### Adding a Fact
+
+```
+User Code
+   │
+   ├─→ engine.add_fact(:stock, price: 150)
+   │
+   ▼
+WorkingMemory.add_fact(fact)
+   │
+   ├─→ @facts << fact
+   └─→ notify_observers(:add, fact)
+          │
+          ▼
+       Engine.update(:add, fact)
+          │
+          └─→ For each AlphaMemory:
+                 if fact.matches?(pattern)
+                    │
+                    ▼
+                 AlphaMemory.activate(fact)
+                    │
+                    └─→ JoinNode.right_activate(fact)
+                           │
+                           └─→ Create tokens, propagate...
+                                  │
+                                  ▼
+                               ProductionNode
+```
+
+### Firing Rules
+
+```
+User Code
+   │
+   ├─→ engine.run()
+   │
+   ▼
+For each ProductionNode:
+   │
+   ├─→ For each token:
+   │      │
+   │      └─→ rule.fire(token.facts)
+   │             │
+   │             └─→ Extract bindings
+   │                    │
+   │                    └─→ Execute action lambda
+   │                           │
+   │                           └─→ User code in action
+   │
+   └─→ Mark tokens as fired
+```
+
+## Network Compilation
+
+When you add a rule, KBS compiles it into a discrimination network:
+
+```ruby
+rule = Rule.new("example") do |r|
+  r.conditions = [
+    Condition.new(:stock, { symbol: :sym? }),
+    Condition.new(:alert, { symbol: :sym? }, negated: true)
+  ]
+  r.action = ->(facts, bindings) { puts bindings[:sym?] }
+end
+
+engine.add_rule(rule)
+```
+
+**Compiled Network:**
+
+![Compiled RETE Network](../assets/images/compiled-network.svg)
+
+*The rule compiles into a network with alpha memories for each condition type, join nodes to combine matches, a negation node for the NOT condition, and a production node that fires when all conditions are satisfied.*
+
+**Learn more**: [Network Structure](network-structure.md)
+
+## Performance Characteristics
+
+| Operation | Complexity | Notes |
+|-----------|-----------|-------|
+| Add rule | O(C × F) | C = conditions, F = existing facts |
+| Add fact | O(N) | N = activated nodes (typically << total) |
+| Remove fact | O(T) | T = tokens containing fact |
+| Fire rules | O(M) | M = complete matches |
+| Network sharing | O(1) | Same pattern → same alpha memory |
+
+## Design Principles
+
+### 1. Algorithm Fidelity
+Maintain RETE correctness per Forgy's specifications. No shortcuts that break semantics.
+
+### 2. Separation of Concerns
+- Engine: Pattern matching
+- Memory: Storage
+- DSL: User interface
+- Blackboard: Collaboration
+
+Each component is independently testable and swappable.
+
+### 3. Performance Through Clarity
+Optimize algorithm first (unlinking, network sharing), then profile before micro-optimizations.
+
+### 4. Testability
+Every method testable in isolation. Dependency injection for external services.
+
+### 5. Graceful Degradation
+Optional features (Redis, AI) don't block core functionality. Fallback to SQLite or in-memory.
+
+### 6. Auditability
+Complete audit trails for production systems. Know *why* a rule fired.
+
+## Extension Points
+
+### Custom Persistence
+
+Implement `KBS::Blackboard::Persistence::Store`:
+
+```ruby
+class MyStore
+  def save_fact(fact) ... end
+  def load_facts(type) ... end
+  def delete_fact(id) ... end
+  # ...
+end
+
+engine = KBS::Blackboard::Engine.new(store: MyStore.new)
+```
+
+### Custom Pattern Matching
+
+Override `Fact#matches?`:
+
+```ruby
+class MyFact < KBS::Fact
+  def matches?(pattern)
+    # Custom matching logic
+  end
+end
+```
+
+### Custom Rule Actions
+
+Actions are lambdas - inject any Ruby code:
+
+```ruby
+r.action = lambda do |facts, bindings|
+  send_email(bindings[:alert?])
+  log_to_database(facts)
+  trigger_api_call(bindings)
+end
+```
+
+## File Organization
+
+```
+lib/kbs/
+├── rete_engine.rb         # Main engine
+├── working_memory.rb      # Fact storage
+├── fact.rb                # Fact representation
+├── rule.rb                # Rule definition
+├── condition.rb           # Pattern specification
+├── token.rb               # Partial match
+├── alpha_memory.rb        # Pattern-level cache
+├── beta_memory.rb         # Token storage
+├── join_node.rb           # Inter-condition joins
+├── negation_node.rb       # Negated conditions
+├── production_node.rb     # Rule firing
+└── blackboard/            # Persistent memory
+    ├── engine.rb          # Blackboard-aware RETE
+    ├── memory.rb          # Central workspace
+    ├── fact.rb            # Persisted fact
+    ├── message_queue.rb   # Agent communication
+    ├── audit_log.rb       # Change history
+    └── persistence/       # Storage backends
+        ├── store.rb       # Abstract interface
+        ├── sqlite_store.rb
+        ├── redis_store.rb
+        └── hybrid_store.rb
+```
+
+## Next Steps
+
+- **[RETE Algorithm](rete-algorithm.md)** - Deep dive into pattern matching
+- **[Blackboard System](blackboard.md)** - Persistent memory architecture
+- **[Network Structure](network-structure.md)** - How rules compile into networks
+- **[API Reference](../api/index.md)** - Complete class documentation

data/docs/architecture/network-structure.md

@@ -0,0 +1,343 @@
+# Network Structure
+
+How RETE compiles rules into an efficient discrimination network.
+
+## Overview
+
+When you add a rule to the engine, KBS compiles it into a discrimination network—a directed acyclic graph (DAG) of nodes that efficiently matches patterns against facts. This document explains the compilation process, node types, and optimization strategies.
+
+## Network Compilation Process
+
+### Step 1: Parse Rule Conditions
+
+```ruby
+rule = Rule.new("example") do |r|
+  r.conditions = [
+    Condition.new(:stock, { symbol: :sym?, price: :price? }),
+    Condition.new(:threshold, { symbol: :sym?, max: :max? })
+  ]
+  r.action = lambda { |facts, bindings| ... }
+end
+```
+
+The engine extracts:
+- Condition types (`:stock`, `:threshold`)
+- Patterns (attribute constraints)
+- Variable bindings (`:sym?`, `:price?`, `:max?`)
+- Join tests (`:sym?` appears in both conditions)
+
+### Step 2: Create or Reuse Alpha Memories
+
+For each condition, the engine creates or reuses an `AlphaMemory` node:
+
+```ruby
+# Pattern for first condition
+pattern1 = { type: :stock, symbol: :sym?, price: :price? }
+alpha1 = get_or_create_alpha_memory(pattern1)
+
+# Pattern for second condition
+pattern2 = { type: :threshold, symbol: :sym?, max: :max? }
+alpha2 = get_or_create_alpha_memory(pattern2)
+```
+
+**Network Sharing**: If another rule has the same pattern, they share the same alpha memory node.
+
+### Step 3: Build Join Network
+
+Connect conditions through join nodes:
+
+```ruby
+# Start with root beta memory (contains dummy token)
+current_beta = @root_beta_memory
+
+# For each condition
+rule.conditions.each do |condition|
+  alpha_memory = get_or_create_alpha_memory(condition.pattern)
+
+  # Build join tests for variable consistency
+  tests = extract_join_tests(condition)
+
+  # Create join or negation node
+  if condition.negated
+    node = NegationNode.new(alpha_memory, current_beta, tests)
+  else
+    node = JoinNode.new(alpha_memory, current_beta, tests)
+  end
+
+  # Create beta memory to store results
+  new_beta = BetaMemory.new
+  node.successors << new_beta
+  current_beta = new_beta
+end
+```
+
+### Step 4: Attach Production Node
+
+```ruby
+production_node = ProductionNode.new(rule)
+current_beta.successors << production_node
+@production_nodes[rule.name] = production_node
+```
+
+## Node Types
+
+### Alpha Memory Nodes
+
+**Purpose**: Store facts matching a specific pattern
+
+**Structure**:
+```ruby
+class AlphaMemory
+  attr_accessor :items      # Facts that match pattern
+  attr_accessor :successors # Join nodes using this alpha
+  attr_accessor :pattern    # Pattern to match
+  attr_reader :linked       # Unlinking state
+end
+```
+
+**Example**:
+```
+AlphaMemory(stock, symbol: "AAPL")
+  items: [stock(symbol: "AAPL", price: 150), ...]
+  successors: [JoinNode1, JoinNode2, ...]
+```
+
+### Beta Memory Nodes
+
+**Purpose**: Store partial matches (tokens) as they propagate
+
+**Structure**:
+```ruby
+class BetaMemory
+  attr_accessor :tokens     # Partial matches
+  attr_accessor :successors # Next nodes in network
+  attr_reader :linked       # Unlinking state
+end
+```
+
+**Example**:
+```
+BetaMemory
+  tokens: [
+    Token(parent: root, fact: stock(...)),
+    Token(parent: root, fact: stock(...))
+  ]
+  successors: [JoinNode2]
+```
+
+### Join Nodes
+
+**Purpose**: Combine facts from alpha memory with tokens from beta memory
+
+**Structure**:
+```ruby
+class JoinNode
+  attr_accessor :alpha_memory  # Right input
+  attr_accessor :beta_memory   # Left input
+  attr_accessor :tests         # Join tests to perform
+  attr_accessor :successors    # Beta memory nodes
+end
+```
+
+**Join Tests**:
+```ruby
+{
+  token_field_index: 0,    # Check first fact in token
+  token_field: :symbol,    # Get its :symbol attribute
+  fact_field: :symbol,     # Compare with new fact's :symbol
+  operation: :eq           # Must be equal
+}
+```
+
+### Negation Nodes
+
+**Purpose**: Implement negated conditions (match when pattern is absent)
+
+**Structure**:
+```ruby
+class NegationNode
+  attr_accessor :alpha_memory       # Pattern to check
+  attr_accessor :beta_memory        # Tokens to test
+  attr_accessor :tests              # Join tests
+  attr_accessor :tokens_with_matches # Track inhibiting facts
+end
+```
+
+**Behavior**:
+- Token arrives → check alpha memory for matches
+- No matches found → propagate token (condition satisfied)
+- Matches found → block token (condition not satisfied)
+- Match removed → unblock token
+
+### Production Nodes
+
+**Purpose**: Fire rule actions when all conditions match
+
+**Structure**:
+```ruby
+class ProductionNode
+  attr_accessor :rule   # Rule to fire
+  attr_accessor :tokens # Complete matches ready to fire
+end
+```
+
+## Complete Example
+
+### Rule Definition
+
+```ruby
+rule = Rule.new("trading_signal") do |r|
+  r.conditions = [
+    Condition.new(:stock, { symbol: :sym?, price: :price? }),
+    Condition.new(:threshold, { symbol: :sym?, buy_below: :threshold? }),
+    Condition.new(:order, { symbol: :sym? }, negated: true)
+  ]
+
+  r.action = lambda do |facts, bindings|
+    if bindings[:price?] < bindings[:threshold?]
+      puts "BUY #{bindings[:sym?]}"
+    end
+  end
+end
+```
+
+### Compiled Network
+
+![Trading Signal Network](../assets/images/trading-signal-network.svg)
+
+*The trading signal rule compiles into a network with three join points. The first two join nodes combine stock and threshold facts based on matching symbols. The negation node ensures no existing order for that symbol. Tokens propagate through beta memories, carrying partial matches until reaching the production node.*
+
+## Optimization Strategies
+
+### Network Sharing
+
+Multiple rules with common patterns share alpha memory nodes:
+
+```ruby
+# Rule 1
+Condition.new(:stock, { symbol: "AAPL" })
+
+# Rule 2
+Condition.new(:stock, { symbol: "AAPL" })
+
+# Both use the same AlphaMemory node
+# Only one pattern match, one fact storage
+```
+
+### Unlinking
+
+Empty nodes automatically disconnect to avoid wasted computation:
+
+```ruby
+# BetaMemory becomes empty
+beta_memory.remove_token(last_token)
+# => Calls unlink!
+
+# Downstream nodes stop processing
+join_node.left_activate(token)  # Returns early if !@left_linked
+```
+
+### Condition Ordering
+
+Place selective conditions first to minimize beta memory size:
+
+```ruby
+# Good: Specific condition first
+conditions = [
+  Condition.new(:critical_alert, { severity: "critical" }),  # Few matches
+  Condition.new(:stock, { symbol: :sym? })                   # Many matches
+]
+
+# Bad: General condition first
+conditions = [
+  Condition.new(:stock, { symbol: :sym? }),                  # Many matches
+  Condition.new(:critical_alert, { severity: "critical" })   # Few matches
+]
+```
+
+### Variable Binding Extraction
+
+Variables create join tests automatically:
+
+```ruby
+# Rule with :sym? in two conditions
+tests = [
+  {
+    token_field_index: 0,     # First fact in token (stock)
+    token_field: :symbol,
+    fact_field: :symbol,      # New fact (threshold)
+    operation: :eq
+  }
+]
+```
+
+## Network Inspection
+
+### Debugging Network Structure
+
+```ruby
+# See all alpha memories
+engine.alpha_memories.each do |pattern, memory|
+  puts "Pattern: #{pattern}"
+  puts "  Facts: #{memory.items.size}"
+  puts "  Linked: #{memory.linked}"
+  puts "  Successors: #{memory.successors.size}"
+end
+
+# See production nodes
+engine.production_nodes.each do |name, node|
+  puts "Rule: #{name}"
+  puts "  Tokens: #{node.tokens.size}"
+end
+```
+
+### Visualizing Token Flow
+
+Enable tracing in actions:
+
+```ruby
+r.action = lambda do |facts, bindings|
+  puts "Rule '#{rule.name}' fired"
+  puts "  Facts: #{facts.map(&:to_s).join(', ')}"
+  puts "  Bindings: #{bindings.inspect}"
+end
+```
+
+## Performance Implications
+
+### Time Complexity
+
+| Operation | Complexity | Notes |
+|-----------|-----------|-------|
+| Add rule | O(C × F) | C = conditions, F = facts |
+| Network sharing lookup | O(1) | Hash-based pattern cache |
+| Join test | O(T) | T = number of tests |
+| Token propagation | O(S) | S = successors |
+
+### Space Complexity
+
+| Structure | Space | Notes |
+|-----------|-------|-------|
+| Alpha memories | O(P) | P = unique patterns across all rules |
+| Beta memories | O(R × C) | R = rules, C = avg conditions |
+| Tokens | O(M × C) | M = complete matches |
+| Join nodes | O(R × C) | One per condition |
+
+### Optimization Tips
+
+1. **Maximize network sharing**: Design rules to reuse common patterns
+2. **Order conditions by selectivity**: Specific first, general last
+3. **Minimize negations**: Expensive to maintain
+4. **Use predicates sparingly**: Can't be shared across rules
+5. **Profile your rules**: Use debugging to identify bottlenecks
+
+## Next Steps
+
+- **[RETE Algorithm](rete-algorithm.md)** - Understand the full execution cycle
+- **[Blackboard System](blackboard.md)** - Persistent network state
+- **[Performance Tuning](../advanced/performance.md)** - Optimize for production
+- **[Debugging Guide](../advanced/debugging.md)** - Inspect network state
+
+---
+
+*This document describes implementation details in `lib/kbs/rete_engine.rb:58` (network compilation) and related node classes.*