openclacky 0.7.0 → 0.7.2

data/docs/time_machine_design.md

@@ -0,0 +1,247 @@
+# Time Machine Design Documentation
+
+## Overview
+
+Time Machine is a feature that allows users to navigate through the agent's task execution history, providing undo/redo capabilities and branch exploration. Users can access it via ESC key or `/undo` command to view an interactive menu of past tasks.
+
+## Core Data Structure Design
+
+### Task History Graph
+
+The Time Machine uses a minimal tree-based data structure to track task relationships:
+
+**Three Core State Variables:**
+1. **task_parents** (Hash): Maps each task_id to its parent_id
+   - Forms a tree structure where each task points to its predecessor
+   - Root tasks have parent_id = 0
+   - Enables traversal in both directions (parent→children, child→parent)
+
+2. **current_task_id** (Integer): The latest created task ID
+   - Always increments when new tasks are created
+   - Never decreases, even during undo operations
+   - Represents the "tip" of the execution timeline
+
+3. **active_task_id** (Integer): The current active position in history
+   - Can move backward/forward during undo/redo
+   - Determines which messages are visible to the LLM
+   - When active_task_id < current_task_id, we're viewing "past" state
+
+### Task Metadata Structure
+
+Each task in the history contains:
+- **task_id**: Unique identifier (auto-incrementing integer)
+- **summary**: Brief description (first 80 chars of user's message)
+- **status**: One of three states
+  - `:past` - Task is before the current active position
+  - `:current` - Task is the active position (marked with `→`)
+  - `:future` - Task exists but is after active position (marked with `↯`)
+- **has_branches**: Boolean indicating if multiple children exist (marked with `⎇`)
+
+## Snapshot Strategy
+
+### File State Preservation
+
+**Complete AFTER-State Snapshots:**
+- After each successful task execution, all modified files are saved
+- Storage location: `~/.clacky/snapshots/{session_id}/task-{id}/`
+- Each file is stored with its full relative path from working directory
+- Only files modified during that task are snapshotted
+
+**Why AFTER-state instead of BEFORE-state:**
+- Simpler restoration logic (just copy files back)
+- No need to track "what changed" - the snapshot IS the state
+- Easier to verify correctness (snapshot = expected state)
+
+**File Restoration Process:**
+- When switching to a task, iterate through all its snapshotted files
+- Copy each file from snapshot directory to working directory
+- File permissions and timestamps are preserved
+
+### Message Filtering
+
+**Active Messages Concept:**
+- Messages array contains ALL messages (past, current, future)
+- `active_messages()` method filters out "future" messages
+- LLM only sees messages with `task_id <= active_task_id`
+- This creates the illusion of time travel without data deletion
+
+**Why Keep All Messages:**
+- Enables redo operations (future messages preserved)
+- Allows branch switching (alternative futures available)
+- Simplifies session serialization (single source of truth)
+
+## Session Persistence
+
+### State Serialization
+
+Time Machine state is saved under `:time_machine` key in session data:
+- task_parents hash (complete tree structure)
+- current_task_id (latest task number)
+- active_task_id (current viewing position)
+
+**Restoration Guarantees:**
+- Complete task tree is rebuilt
+- Active position is restored
+- Snapshot files remain available across sessions
+- User can continue undo/redo from where they left off
+
+## Critical Test Scenarios
+
+### 1. Basic Undo/Redo Flow
+
+**Test Focus:**
+- Sequential task creation increments task IDs correctly
+- Undo moves active_task_id backward (current_task_id unchanged)
+- Redo moves active_task_id forward
+- File snapshots are correctly restored at each step
+- Cannot undo beyond root task (task_id = 0)
+- Cannot redo beyond current_task_id
+
+**Edge Cases:**
+- Undoing at root task should fail gracefully
+- Redoing when already at tip should fail gracefully
+- Multiple consecutive undos should work correctly
+
+### 2. Branching Scenarios
+
+**Test Focus:**
+- After undo, creating new task creates a branch
+- New branch starts from active_task_id, not current_task_id
+- Original future branch is preserved (for potential redo)
+- Parent task is marked with `has_branches: true`
+- Child tasks list should include both branches
+
+**Branch Navigation:**
+- Switching between branches restores correct file states
+- Each branch maintains independent history
+- Message filtering correctly shows only relevant messages
+
+### 3. Message Filtering and Task IDs
+
+**Test Focus:**
+- Every message is tagged with task_id (user, assistant, tool results)
+- Active messages only include those with task_id <= active_task_id
+- LLM never sees "future" messages during undo state
+- After redo, future messages become visible again
+- New tasks created after undo get fresh task IDs (not reused)
+
+**Message Consistency:**
+- Tool results are associated with correct task
+- Multi-turn conversations maintain task association
+- Error messages don't break task ID tagging
+
+### 4. File Snapshot Integrity
+
+**Test Focus:**
+- Only modified files are snapshotted (not entire project)
+- File content is exactly preserved (byte-for-byte)
+- Nested directory structures are correctly recreated
+- Multiple files in single task are all snapshotted
+- Snapshot directory naming prevents collisions
+
+**Restoration Accuracy:**
+- After undo + file restore, file content matches expected state
+- Subsequent task execution works with restored files
+- Binary files are handled correctly (not corrupted)
+
+### 5. Session Persistence and Recovery
+
+**Test Focus:**
+- Save session, restart, restore session preserves Time Machine state
+- Task tree structure is fully rebuilt
+- Active position is correctly restored
+- Snapshot files are accessible after restart
+- Undo/redo operations work identically after restore
+
+**Persistence Edge Cases:**
+- Empty task history (new session)
+- Session with complex branching
+- Session saved while in "undo" state (active_task_id < current_task_id)
+
+### 6. AI Tool Integration
+
+**Test Focus:**
+- Tools are correctly registered in tool registry
+- AI can invoke undo_task, redo_task, list_tasks
+- Agent parameter is correctly injected (similar to TodoManager pattern)
+- Tool execution returns success/failure messages
+- Tools respect permission modes (confirm_all, auto_approve, etc.)
+
+**Tool Interaction:**
+- AI calling undo_task modifies agent state correctly
+- Subsequent AI responses use filtered messages
+- Tool results are included in task history
+- Multiple tool calls in sequence work correctly
+
+### 7. UI and User Interaction
+
+**Test Focus:**
+- ESC key triggers time machine menu
+- `/undo` command works identically to ESC
+- Menu displays correct task list with status indicators
+- Visual markers: `→` current, `↯` future, `⎇` branches
+- User selection triggers correct task switch
+- Menu updates after undo/redo operations
+
+**User Experience:**
+- Task summaries are readable (truncated to 80 chars)
+- Menu is responsive with large task histories
+- Cancel/exit returns to normal operation
+- Error messages are clear and actionable
+
+### 8. Integration with Existing Features
+
+**Test Focus:**
+- Works with message compression (no dependency on tool_calls)
+- Compatible with session serialization
+- Doesn't interfere with cost tracking
+- Works with both UI modes (UI1 and UI2)
+- Subagent forking doesn't inherit Time Machine state
+
+**Feature Compatibility:**
+- Todo manager works normally during undo state
+- Web search tools work correctly
+- File tools (write, edit) trigger snapshots
+- Shell commands can be undone via file snapshots
+
+## Design Principles
+
+### Minimal Invasiveness
+- Only 3 new instance variables in Agent class
+- No changes to core message structure (only adds task_id field)
+- Existing tools unaware of Time Machine existence
+- No performance impact when not in use
+
+### Data Integrity
+- Never delete messages or snapshots (immutable history)
+- File restoration is idempotent (can redo multiple times)
+- Task IDs never reused (prevents confusion)
+- Snapshot isolation (each task has independent directory)
+
+### User Control
+- Explicit user action required (ESC or /undo)
+- Clear visual feedback on current position
+- Cannot accidentally lose work (future preserved)
+- Can explore branches without commitment
+
+### Developer Friendly
+- Simple tree data structure (easy to reason about)
+- Comprehensive test coverage (55 test cases)
+- Clear separation of concerns (module-based design)
+- Well-documented edge cases
+
+## Future Enhancement Possibilities
+
+### Potential Improvements
+- Automatic snapshot garbage collection (old sessions)
+- Diff view between task states
+- Named checkpoints (user-defined bookmarks)
+- Merge branches functionality
+- Export task history as replay script
+- Snapshot compression for large files
+
+### Scalability Considerations
+- Large file handling (incremental snapshots)
+- Long session histories (pagination in UI)
+- Multiple simultaneous branches (better visualization)
+- Remote collaboration (shared task history)

data/docs/why-openclacky.md

@@ -79,7 +79,6 @@ A command-line AI assistant that's approachable for non-technical users but powe
    |------|----------|----------|
    | `auto_approve` | Execute all tools automatically | Batch operations |
    | `confirm_safes` | Auto-approve safe operations | Daily development |
-   | `confirm_edits` | Confirm file modifications | Careful work |
    | `plan_only` | Generate plans only | Code review |
 
 5. **Session Recovery**

data/lib/clacky/agent/cost_tracker.rb

@@ -0,0 +1,180 @@
+# frozen_string_literal: true
+
+module Clacky
+  class Agent
+    # Cost tracking and token usage statistics
+    # Manages cost calculation, token estimation, and usage display
+    module CostTracker
+      # Track cost from API usage
+      # Updates total cost and displays iteration statistics
+      # @param usage [Hash] Usage data from API response
+      # @param raw_api_usage [Hash, nil] Raw API usage data for debugging
+      def track_cost(usage, raw_api_usage: nil)
+        # Priority 1: Use API-provided cost if available (OpenRouter, LiteLLM, etc.)
+        iteration_cost = nil
+        if usage[:api_cost]
+          @total_cost += usage[:api_cost]
+          @cost_source = :api
+          @task_cost_source = :api
+          iteration_cost = usage[:api_cost]
+          @ui&.log("Using API-provided cost: $#{usage[:api_cost]}", level: :debug) if @config.verbose
+        else
+          # Priority 2: Calculate from tokens using ModelPricing
+          result = ModelPricing.calculate_cost(model: current_model, usage: usage)
+          cost = result[:cost]
+          pricing_source = result[:source]
+
+          @total_cost += cost
+          iteration_cost = cost
+          # Map pricing source to cost source: :price or :default
+          @cost_source = pricing_source
+          @task_cost_source = pricing_source
+
+          if @config.verbose
+            source_label = pricing_source == :price ? "model pricing" : "default pricing"
+            @ui&.log("Calculated cost for #{@config.model_name} using #{source_label}: $#{cost.round(6)}", level: :debug)
+            @ui&.log("Usage breakdown: prompt=#{usage[:prompt_tokens]}, completion=#{usage[:completion_tokens]}, cache_write=#{usage[:cache_creation_input_tokens] || 0}, cache_read=#{usage[:cache_read_input_tokens] || 0}", level: :debug)
+          end
+        end
+
+        # Display token usage statistics for this iteration
+        display_iteration_tokens(usage, iteration_cost)
+
+        # Update session bar cost in real-time (don't wait for agent.run to finish)
+        @ui&.update_sessionbar(cost: @total_cost)
+
+        # Track cache usage statistics (global)
+        @cache_stats[:total_requests] += 1
+
+        if usage[:cache_creation_input_tokens]
+          @cache_stats[:cache_creation_input_tokens] += usage[:cache_creation_input_tokens]
+        end
+
+        if usage[:cache_read_input_tokens]
+          @cache_stats[:cache_read_input_tokens] += usage[:cache_read_input_tokens]
+          @cache_stats[:cache_hit_requests] += 1
+        end
+
+        # Store raw API usage samples (keep last 3 for debugging)
+        if raw_api_usage
+          @cache_stats[:raw_api_usage_samples] ||= []
+          @cache_stats[:raw_api_usage_samples] << raw_api_usage
+          @cache_stats[:raw_api_usage_samples] = @cache_stats[:raw_api_usage_samples].last(3)
+        end
+
+        # Track cache usage for current task
+        if @task_cache_stats
+          @task_cache_stats[:total_requests] += 1
+
+          if usage[:cache_creation_input_tokens]
+            @task_cache_stats[:cache_creation_input_tokens] += usage[:cache_creation_input_tokens]
+          end
+
+          if usage[:cache_read_input_tokens]
+            @task_cache_stats[:cache_read_input_tokens] += usage[:cache_read_input_tokens]
+            @task_cache_stats[:cache_hit_requests] += 1
+          end
+        end
+      end
+
+      # Estimate token count for a message content
+      # Simple approximation: characters / 4 (English text)
+      # For Chinese/other languages, characters / 2 is more accurate
+      # This is a rough estimate for compression triggering purposes
+      # @param content [String, Array, Object] Message content
+      # @return [Integer] Estimated token count
+      def estimate_tokens(content)
+        return 0 if content.nil?
+
+        text = if content.is_a?(String)
+                 content
+               elsif content.is_a?(Array)
+                 # Handle content arrays (e.g., with images)
+                 # Add safety check to prevent nil.compact error
+                 mapped = content.map { |c| c[:text] if c.is_a?(Hash) }
+                 (mapped || []).compact.join
+               else
+                 content.to_s
+               end
+
+        return 0 if text.empty?
+
+        # Detect language mix - count non-ASCII characters
+        ascii_count = text.bytes.count { |b| b < 128 }
+        total_bytes = text.bytes.length
+
+        # Mix ratio (1.0 = all English, 0.5 = all Chinese)
+        mix_ratio = total_bytes > 0 ? ascii_count.to_f / total_bytes : 1.0
+
+        # English: ~4 chars/token, Chinese: ~2 chars/token
+        base_chars_per_token = mix_ratio * 4 + (1 - mix_ratio) * 2
+
+        (text.length / base_chars_per_token).to_i + 50 # Add overhead for message structure
+      end
+
+      # Calculate total token count for all messages
+      # Returns estimated tokens and breakdown by category
+      # @return [Hash] Token counts by role and total
+      def total_message_tokens
+        system_tokens = 0
+        user_tokens = 0
+        assistant_tokens = 0
+        tool_tokens = 0
+        summary_tokens = 0
+
+        @messages.each do |msg|
+          tokens = estimate_tokens(msg[:content])
+          case msg[:role]
+          when "system"
+            system_tokens += tokens
+          when "user"
+            user_tokens += tokens
+          when "assistant"
+            assistant_tokens += tokens
+          when "tool"
+            tool_tokens += tokens
+          end
+        end
+
+        {
+          total: system_tokens + user_tokens + assistant_tokens + tool_tokens,
+          system: system_tokens,
+          user: user_tokens,
+          assistant: assistant_tokens,
+          tool: tool_tokens
+        }
+      end
+
+      private
+
+      # Display token usage for current iteration
+      # @param usage [Hash] Usage data from API
+      # @param cost [Float] Cost for this iteration
+      def display_iteration_tokens(usage, cost)
+        prompt_tokens = usage[:prompt_tokens] || 0
+        completion_tokens = usage[:completion_tokens] || 0
+        total_tokens = usage[:total_tokens] || (prompt_tokens + completion_tokens)
+        cache_write = usage[:cache_creation_input_tokens] || 0
+        cache_read = usage[:cache_read_input_tokens] || 0
+
+        # Calculate token delta from previous iteration
+        delta_tokens = total_tokens - @previous_total_tokens
+        @previous_total_tokens = total_tokens  # Update for next iteration
+
+        # Prepare data for UI to format and display
+        token_data = {
+          delta_tokens: delta_tokens,
+          prompt_tokens: prompt_tokens,
+          completion_tokens: completion_tokens,
+          total_tokens: total_tokens,
+          cache_write: cache_write,
+          cache_read: cache_read,
+          cost: cost
+        }
+
+        # Let UI handle formatting and display
+        @ui&.show_token_usage(token_data)
+      end
+    end
+  end
+end

data/lib/clacky/agent/llm_caller.rb

@@ -0,0 +1,54 @@
+# frozen_string_literal: true
+
+module Clacky
+  class Agent
+    # LLM API call management
+    # Handles API calls with retry logic and progress indication
+    module LlmCaller
+      # Execute LLM API call with progress indicator, retry logic, and cost tracking
+      # This method is shared by both normal think() and compression flows
+      # @return [Hash] API response with :content, :tool_calls, :usage, etc.
+      private def call_llm
+        @ui&.show_progress
+
+        tools_to_send = @tool_registry.all_definitions
+
+        # Retry logic for network failures
+        max_retries = 10
+        retry_delay = 5
+        retries = 0
+
+        begin
+          # Use active_messages to filter out "future" messages after undo
+          messages_to_send = respond_to?(:active_messages) ? active_messages : @messages
+          
+          response = @client.send_messages_with_tools(
+            messages_to_send,
+            model: current_model,
+            tools: tools_to_send,
+            max_tokens: @config.max_tokens,
+            enable_caching: @config.enable_prompt_caching
+          )
+        rescue Faraday::ConnectionFailed, Faraday::TimeoutError, Errno::ECONNREFUSED, Errno::ETIMEDOUT => e
+          @ui&.clear_progress
+          retries += 1
+          if retries <= max_retries
+            @ui&.show_warning("Network failed: #{e.message}. Retry #{retries}/#{max_retries}...")
+            sleep retry_delay
+            retry
+          else
+            @ui&.show_error("Network failed after #{max_retries} retries: #{e.message}")
+            raise AgentError, "Network connection failed after #{max_retries} retries: #{e.message}"
+          end
+        ensure
+          @ui&.clear_progress
+        end
+
+        # Track cost for all LLM calls
+        track_cost(response[:usage], raw_api_usage: response[:raw_api_usage])
+
+        response
+      end
+    end
+  end
+end

data/lib/clacky/{message_compressor.rb → agent/message_compressor.rb}

@@ -57,6 +57,12 @@ module Clacky
 
     # Generate compression instruction message to be inserted into conversation
     # This enables cache reuse by using the same API call with tools
+    # 
+    # SIMPLIFIED APPROACH:
+    # - Don't duplicate conversation history in the compression message
+    # - LLM can already see all messages, just ask it to compress
+    # - Keep the instruction small for better cache efficiency
+    #
     # @param messages [Array<Hash>] Original conversation messages
     # @param recent_messages [Array<Hash>] Recent messages to keep uncompressed (optional)
     # @return [Hash] Compression instruction message to insert, or nil if nothing to compress
@@ -67,12 +73,12 @@ module Clacky
       # If nothing to compress, return nil
       return nil if messages_to_compress.empty?
 
-      # Build compression prompt with instruction and conversation
-      content = build_compression_content(messages_to_compress)
-      full_prompt = "#{COMPRESSION_PROMPT}\n\nConversation to compress:\n\n#{content}"
-
-      # Return the compression instruction as a user message with system_injected marker
-      { role: "user", content: full_prompt, system_injected: true }
+      # Simple compression instruction - LLM can see the history already
+      { 
+        role: "user", 
+        content: COMPRESSION_PROMPT,
+        system_injected: true
+      }
     end
 
     # Parse LLM response and rebuild message list with compression
@@ -98,36 +104,6 @@ module Clacky
 
     private
 
-    def build_compression_content(messages)
-      # Format messages as readable text for compression
-      messages.map do |msg|
-        role = msg[:role]
-        content = format_content(msg[:content])
-        "[#{role.upcase}] #{content}"
-      end.join("\n\n")
-    end
-
-    def format_content(content)
-      return content if content.is_a?(String)
-
-      if content.is_a?(Array)
-        content.map do |block|
-          case block[:type]
-          when "text"
-            block[:text]
-          when "tool_use"
-            "TOOL: #{block[:name]}(#{block[:input]})"
-          when "tool_result"
-            "RESULT: #{block[:content]}"
-          else
-            block.to_s
-          end
-        end.join("\n")
-      else
-        content.to_s
-      end
-    end
-
     def parse_compressed_result(result)
       # Return the compressed result as a single assistant message
       # Keep the <analysis> or <summary> tags as they provide semantic context