agentpool 2.2.3__py3-none-any.whl → 2.5.0__py3-none-any.whl

agentpool_storage/claude_provider/ARCHITECTURE.md

@@ -0,0 +1,433 @@
+# Claude Code Storage Architecture
+
+This document explains how Claude Code persists conversation data, sessions, and agent state to the filesystem.
+
+## Directory Structure
+
+```
+~/.claude/
+├── projects/                          # Per-project conversation storage
+│   ├── {encoded-project-path}/       # One directory per project
+│   │   ├── {sessionId}.jsonl         # Main session conversation log
+│   │   ├── agent-{shortId}.jsonl     # Sub-agent conversation logs
+│   │   └── ...                       # Multiple sessions per project
+│   └── ...
+├── todos/                             # Per-agent todo/plan persistence
+│   ├── {sessionId}-agent-{sessionId}.json      # Main agent todos
+│   ├── {sessionId}-agent-{agentId}.json        # Sub-agent todos
+│   └── ...
+├── session-env/                       # Session environment state
+│   └── {sessionId}/                   # Environment per session (often empty)
+├── plans/                             # Standalone plan documents
+│   └── {name}.md                      # Named plans (legacy?)
+├── file-history/                      # File edit history tracking
+├── history.jsonl                      # Global command history
+└── settings.json                      # User settings
+```
+
+## Core Concepts
+
+### 1. Projects
+
+**Project Directory**: `~/.claude/projects/{encoded-project-path}/`
+
+- The project path is URL-encoded to create a safe directory name
+- Example: `/home/user/myproject` → `-home-user-myproject`
+- All sessions for a given working directory are stored together
+- Multiple sessions can exist per project (different conversations over time)
+
+### 2. Sessions
+
+**Session ID**: UUID (e.g., `e8d973c7-481e-43d9-809d-9a4880e8cebc`)
+
+A session represents a single conversation thread with Claude. Each session:
+
+- Has a unique UUID identifier
+- Is tied to a specific project/working directory
+- Contains a sequence of messages (user, assistant, tool calls, etc.)
+- Has one **main agent** and zero or more **sub-agents**
+- Persists to a JSONL file: `{sessionId}.jsonl`
+
+**Session File Format**: JSONL (JSON Lines)
+- Each line is a separate JSON object representing one entry
+- Entries are appended chronologically
+- Enables efficient streaming writes and incremental reads
+
+### 3. Agents
+
+#### Main Agent
+- **Agent ID** = **Session ID** (same UUID)
+- The primary Claude instance handling the conversation
+- Session file: `{sessionId}.jsonl`
+- Todo file: `{sessionId}-agent-{sessionId}.json`
+
+#### Sub-Agents (Delegated Tasks)
+- **Agent ID**: Short hex identifier (e.g., `a753668` - 7 characters)
+- Created when tasks are delegated via the Task tool
+- Run independently with their own context
+- Session file: `agent-{shortId}.jsonl`
+- Todo file: `{sessionId}-agent-{shortId}.json`
+- Still belong to parent session (share sessionId in metadata)
+
+**Key Pattern**: All agents in a session share the parent `sessionId`:
+```
+Session: e8d973c7-481e-43d9-809d-9a4880e8cebc
+├── Main Agent: e8d973c7-481e-43d9-809d-9a4880e8cebc
+│   └── Todo: e8d973c7-481e-43d9-809d-9a4880e8cebc-agent-e8d973c7-481e-43d9-809d-9a4880e8cebc.json
+└── Sub-Agent: a753668
+    └── Todo: e8d973c7-481e-43d9-809d-9a4880e8cebc-agent-a753668.json
+```
+
+### 4. Todo/Plan Storage
+
+**Location**: `~/.claude/todos/`
+
+**Naming**: `{sessionId}-agent-{agentId}.json`
+
+**Format**: JSON array of todo entries
+```json
+[
+  {
+    "content": "Task description",
+    "status": "pending" | "in_progress" | "completed",
+    "priority": "high" | "medium" | "low",
+    "activeForm": "Current action description (optional)"
+  }
+]
+```
+
+**Persistence Pattern**:
+- Todos are scoped to a specific agent within a session
+- Main agent's todos use redundant naming: `{sessionId}-agent-{sessionId}.json`
+- Sub-agent todos use: `{sessionId}-agent-{shortId}.json`
+- All todos for a session share the session ID prefix
+- Survives across conversation resumes
+
+## Entry Types
+
+Claude Code uses different entry types in JSONL session files:
+
+### User Entry
+```json
+{
+  "type": "user",
+  "sessionId": "uuid",
+  "uuid": "message-uuid",
+  "parentUuid": "parent-message-uuid",
+  "timestamp": "ISO-8601",
+  "message": {
+    "role": "user",
+    "content": "text or array of content blocks"
+  }
+}
+```
+
+### Assistant Entry
+```json
+{
+  "type": "assistant",
+  "sessionId": "uuid",
+  "uuid": "message-uuid",
+  "parentUuid": "parent-message-uuid",
+  "timestamp": "ISO-8601",
+  "message": {
+    "role": "assistant",
+    "content": [
+      {"type": "text", "text": "..."},
+      {"type": "tool_use", "id": "...", "name": "...", "input": {...}}
+    ]
+  },
+  "requestId": "req_...",
+  "model": "claude-..."
+}
+```
+
+### Queue Operation Entry
+```json
+{
+  "type": "queue-operation",
+  "operation": "enqueue" | "dequeue",
+  "sessionId": "uuid",
+  "timestamp": "ISO-8601"
+}
+```
+
+Marks session lifecycle events (start/end of processing).
+
+### System Entry
+```json
+{
+  "type": "system",
+  "sessionId": "uuid",
+  "timestamp": "ISO-8601",
+  "message": {
+    "role": "system",
+    "content": "System message text"
+  }
+}
+```
+
+### Tool Result Entry
+Embedded within user entries as tool_result content blocks:
+```json
+{
+  "type": "user",
+  "message": {
+    "role": "user",
+    "content": [
+      {
+        "type": "tool_result",
+        "tool_use_id": "toolu_...",
+        "content": "result text or structured data"
+      }
+    ]
+  },
+  "toolUseResult": ["..."]  // Duplicate for UI
+}
+```
+
+### Summary Entry
+```json
+{
+  "type": "summary",
+  "sessionId": "uuid",
+  "timestamp": "ISO-8601",
+  "content": {
+    "summary": "Condensed conversation summary",
+    "messageCount": 10,
+    "summarizedUntil": "message-uuid"
+  }
+}
+```
+
+Used for conversation compaction/context management.
+
+### File History Entry
+```json
+{
+  "type": "file-history",
+  "sessionId": "uuid",
+  "timestamp": "ISO-8601",
+  "path": "/absolute/path/to/file",
+  "operation": "edit" | "create" | "delete",
+  "snippet": "Preview of changes..."
+}
+```
+
+Tracks file operations for undo/history features.
+
+## Message Flow & Ancestry
+
+### Parent-Child Relationships
+
+Every message (except the first) has a `parentUuid` that references the previous message:
+
+```
+User Message 1 (uuid: a)
+  └─> Assistant Reply (uuid: b, parentUuid: a)
+       └─> Tool Results (uuid: c, parentUuid: b)
+            └─> Assistant Response (uuid: d, parentUuid: c)
+```
+
+This creates a **linked list** of messages forming the conversation thread.
+
+### Branching (Sidechains)
+
+Field: `isSidechain: boolean`
+
+When a conversation is forked or edited:
+- Original messages remain unchanged
+- New branch starts with `isSidechain: true`
+- Allows exploring alternative conversation paths
+- Main chain: `isSidechain: false` or omitted
+
+### Conversation Reconstruction
+
+To read a conversation:
+1. Parse all entries from `{sessionId}.jsonl`
+2. Filter by `type` (user, assistant, etc.)
+3. Follow the parent chain from latest message backward
+4. Stop at session start or desired depth
+5. Reverse to get chronological order
+
+## Storage Provider Implementation
+
+The `ClaudeStorageProvider` class bridges between:
+- **Claude's storage format** (JSONL entries)
+- **Agentpool's abstractions** (ChatMessage, ConversationStats, etc.)
+
+### Key Responsibilities
+
+1. **Path Management**
+   - Encode/decode project paths to safe directory names
+   - Ensure project directories exist
+
+2. **Session Management**
+   - List all sessions for a project
+   - Read session entries (with optional filtering)
+   - Write new entries atomically
+
+3. **Message Conversion**
+   - JSONL Entry → `ChatMessage` (for agentpool)
+   - `ChatMessage` → JSONL Entry (for persistence)
+   - Handle tool calls, results, and metadata
+
+4. **Conversation Queries**
+   - Get message count, token usage stats
+   - Retrieve specific messages by UUID
+   - Trace message ancestry
+   - Fork conversations at specific points
+
+5. **Filtering & Compaction**
+   - Filter by message type, date range
+   - Skip tool calls or system messages
+   - Enable conversation summarization
+
+## Data Consistency
+
+### Atomic Writes
+- Each entry is appended as a complete line
+- JSONL format allows crash-safe appends
+- No partial writes visible to readers
+
+### Concurrent Access
+- Multiple sessions can write to different files safely
+- Same session should have single writer (main process)
+- Readers can stream entries while writing continues
+
+### Cleanup
+- Empty session files may accumulate
+- No automatic garbage collection (user-managed)
+- `reset()` method can delete project history
+
+## Integration Points
+
+### With Agentpool
+- `ClaudeStorageProvider` implements `StorageProvider` protocol
+- Converts between storage formats and domain models
+- Enables conversation persistence for agents
+
+### With TodoTracker
+- Separate persistence in `~/.claude/todos/`
+- Not integrated with main conversation storage
+- Must be loaded/saved independently
+- File naming links todos to session + agent
+
+### With File History
+- Separate tracking in `~/.claude/file-history/`
+- Records file operations during conversation
+- Enables undo/diff features
+- Cross-referenced by session ID
+
+## Usage Patterns
+
+### Starting a New Session
+```python
+provider = ClaudeStorageProvider(base_dir="~/.claude")
+session_id = str(uuid.uuid4())
+
+# Log initial user message
+await provider.log_message(
+    project_path="/path/to/project",
+    message=ChatMessage(role="user", content="Hello"),
+    session_id=session_id,
+)
+```
+
+### Resuming a Session
+```python
+# Get existing sessions
+conversations = await provider.get_conversations(
+    project_path="/path/to/project"
+)
+
+# Load latest session
+messages = await provider.get_conversation_messages(
+    project_path="/path/to/project",
+    session_id=conversations[0]["sessionId"],
+    limit=50  # Last 50 messages
+)
+```
+
+### Creating a Sub-Agent
+```python
+# Sub-agent gets own short ID
+agent_id = "a" + secrets.token_hex(3)  # e.g., "a753668"
+
+# But shares parent sessionId
+await provider.log_message(
+    project_path="/path/to/project",
+    message=agent_message,
+    session_id=parent_session_id,  # Same as parent!
+    agent_id=agent_id,  # Different agent ID
+)
+
+# Creates: agent-a753668.jsonl in project dir
+# Creates: {sessionId}-agent-a753668.json in todos/
+```
+
+### Saving Todos
+```python
+# Not part of ClaudeStorageProvider!
+# Separate JSON file management:
+
+todo_path = (
+    f"~/.claude/todos/{session_id}-agent-{agent_id}.json"
+)
+with open(todo_path, "w") as f:
+    json.dump(todos, f)
+```
+
+## Design Rationale
+
+### Why JSONL?
+- **Streamable**: Can read/write incrementally
+- **Crash-safe**: Each line is atomic
+- **Human-readable**: Easy debugging
+- **Flexible schema**: Each entry can evolve independently
+
+### Why Separate Agent Files?
+- **Isolation**: Sub-agents run independently
+- **Parallelism**: Multiple agents can write concurrently
+- **Clarity**: Easy to see which agent did what
+- **Performance**: Don't need to parse entire session for sub-agent context
+
+### Why Redundant Session ID in Main Agent Todos?
+- **Consistency**: All todos follow `{sessionId}-agent-{agentId}` pattern
+- **Simplicity**: Single naming rule, no special cases
+- **Glob-friendly**: Easy to find all todos for a session: `{sessionId}-agent-*.json`
+
+### Why Separate Todos from Conversation Log?
+- **Orthogonal concerns**: Todos are working memory, not conversation history
+- **Update frequency**: Todos change frequently, conversations append-only
+- **Size**: Todos stay small, conversations grow large
+- **Format**: Todos are mutable array, conversations are immutable log
+
+## Future Considerations
+
+### Scalability
+- Large projects accumulate many session files
+- May need session archival/cleanup strategies
+- Consider session indexing for faster queries
+
+### Consistency
+- No transactional guarantees across files
+- Todo updates not synchronized with conversation log
+- Sub-agent todos could become orphaned
+
+### Migration
+- Current format lacks version markers
+- Schema evolution requires careful handling
+- Consider adding format version to entries
+
+### Compression
+- JSONL files can grow large
+- Consider gzip compression for archived sessions
+- Balance between size and read performance
+
+---
+
+**Related Files:**
+- Implementation: [`claude_provider.py`](./claude_provider.py)
+- Models: [`models.py`](./models.py)
+- Base Protocol: [`../agentpool/storage/storage_provider.py`](../agentpool/storage/storage_provider.py)

agentpool_storage/claude_provider/__init__.py

@@ -0,0 +1,42 @@
+"""Claude Code storage provider.
+
+This package implements the storage backend compatible with Claude Code's
+filesystem-based persistence format.
+
+See ARCHITECTURE.md for detailed documentation of the storage format and
+design decisions.
+"""
+
+from __future__ import annotations
+
+from agentpool_storage.claude_provider.provider import (
+    ClaudeApiMessage,
+    ClaudeAssistantEntry,
+    ClaudeBaseModel,
+    ClaudeFileHistoryEntry,
+    ClaudeMessageContent,
+    ClaudeMessageEntryBase,
+    ClaudeQueueOperationEntry,
+    ClaudeStorageProvider,
+    ClaudeSummaryEntry,
+    ClaudeSystemEntry,
+    ClaudeUsage,
+    ClaudeUserEntry,
+    ClaudeUserMessage,
+)
+
+__all__ = [
+    "ClaudeApiMessage",
+    "ClaudeAssistantEntry",
+    "ClaudeBaseModel",
+    "ClaudeFileHistoryEntry",
+    "ClaudeMessageContent",
+    "ClaudeMessageEntryBase",
+    "ClaudeQueueOperationEntry",
+    "ClaudeStorageProvider",
+    "ClaudeSummaryEntry",
+    "ClaudeSystemEntry",
+    "ClaudeUsage",
+    "ClaudeUserEntry",
+    "ClaudeUserMessage",
+]

agentpool_storage/{claude_provider.py → claude_provider/provider.py}

@@ -1,4 +1,16 @@
-"""Claude Code storage provider - reads/writes to ~/.claude format."""
+"""Claude Code storage provider.
+
+This module implements storage compatible with Claude Code's filesystem format,
+enabling interoperability between agentpool and Claude Code.
+
+Key features:
+- JSONL-based conversation logs per project
+- Multi-agent support (main + sub-agents)
+- Message ancestry tracking
+- Conversation forking and branching
+
+See ARCHITECTURE.md for detailed documentation of the storage format.
+"""
 
 from __future__ import annotations
 
@@ -556,14 +568,14 @@ class ClaudeStorageProvider(StorageProvider):
             return ClaudeUserEntry(
                 type="user",
                 uuid=msg_uuid,
-                parentUuid=parent_uuid,
+                parent_uuid=parent_uuid,
                 sessionId=session_id,
                 timestamp=timestamp,
                 message=user_msg,
                 cwd=cwd or "",
                 version="agentpool",
-                userType="external",
-                isSidechain=False,
+                user_type="external",
+                is_sidechain=False,
             )
 
         # Assistant message
@@ -584,14 +596,14 @@ class ClaudeStorageProvider(StorageProvider):
         return ClaudeAssistantEntry(
             type="assistant",
             uuid=msg_uuid,
-            parentUuid=parent_uuid,
+            parent_uuid=parent_uuid,
             sessionId=session_id,
             timestamp=timestamp,
             message=assistant_msg,
             cwd=cwd or "",
             version="agentpool",
-            userType="external",
-            isSidechain=False,
+            user_type="external",
+            is_sidechain=False,
         )
 
     async def filter_messages(self, query: SessionQuery) -> list[ChatMessage[str]]:
@@ -652,7 +664,6 @@ class ClaudeStorageProvider(StorageProvider):
         cost_info: TokenCost | None = None,
         model: str | None = None,
         response_time: float | None = None,
-        forwarded_from: list[str] | None = None,
         provider_name: str | None = None,
         provider_response_id: str | None = None,
         messages: str | None = None,
@@ -905,3 +916,174 @@ class ClaudeStorageProvider(StorageProvider):
                 msg_count += len(message_entries)
 
         return conv_count, msg_count
+
+    async def get_conversation_messages(
+        self,
+        conversation_id: str,
+        *,
+        include_ancestors: bool = False,
+    ) -> list[ChatMessage[str]]:
+        """Get all messages for a conversation.
+
+        Args:
+            conversation_id: Session ID (conversation ID in Claude format)
+            include_ancestors: If True, traverse parent_uuid chain to include
+                messages from ancestor conversations
+
+        Returns:
+            List of messages ordered by timestamp
+        """
+        # Find the session file
+        sessions = self._list_sessions()
+        session_path = None
+        for sid, spath in sessions:
+            if sid == conversation_id:
+                session_path = spath
+                break
+
+        if not session_path:
+            return []
+
+        # Read entries and convert to messages
+        entries = self._read_session(session_path)
+        tool_mapping = self._build_tool_id_mapping(entries)
+
+        messages: list[ChatMessage[str]] = []
+        for entry in entries:
+            msg = self._entry_to_chat_message(entry, conversation_id, tool_mapping)
+            if msg:
+                messages.append(msg)
+
+        # Sort by timestamp
+        messages.sort(key=lambda m: m.timestamp or get_now())
+
+        if not include_ancestors or not messages:
+            return messages
+
+        # Get ancestor chain if first message has parent_id
+        first_msg = messages[0]
+        if first_msg.parent_id:
+            ancestors = await self.get_message_ancestry(first_msg.parent_id)
+            return ancestors + messages
+
+        return messages
+
+    async def get_message(self, message_id: str) -> ChatMessage[str] | None:
+        """Get a single message by ID.
+
+        Args:
+            message_id: UUID of the message
+
+        Returns:
+            The message if found, None otherwise
+        """
+        # Search all sessions for the message
+        sessions = self._list_sessions()
+
+        for session_id, session_path in sessions:
+            entries = self._read_session(session_path)
+            tool_mapping = self._build_tool_id_mapping(entries)
+
+            for entry in entries:
+                if (
+                    isinstance(entry, (ClaudeUserEntry, ClaudeAssistantEntry))
+                    and entry.uuid == message_id
+                ):
+                    return self._entry_to_chat_message(entry, session_id, tool_mapping)
+
+        return None
+
+    async def get_message_ancestry(self, message_id: str) -> list[ChatMessage[str]]:
+        """Get the ancestry chain of a message.
+
+        Traverses parent_uuid chain to build full history.
+
+        Args:
+            message_id: UUID of the message
+
+        Returns:
+            List of messages from oldest ancestor to the specified message
+        """
+        ancestors: list[ChatMessage[str]] = []
+        current_id: str | None = message_id
+
+        while current_id:
+            msg = await self.get_message(current_id)
+            if not msg:
+                break
+            ancestors.append(msg)
+            current_id = msg.parent_id
+
+        # Reverse to get oldest first
+        ancestors.reverse()
+        return ancestors
+
+    async def fork_conversation(
+        self,
+        *,
+        source_conversation_id: str,
+        new_conversation_id: str,
+        fork_from_message_id: str | None = None,
+        new_agent_name: str | None = None,
+    ) -> str | None:
+        """Fork a conversation at a specific point.
+
+        Creates a new session file. The fork point message_id is returned
+        so callers can set it as parent_uuid for new messages.
+
+        Args:
+            source_conversation_id: Source session ID
+            new_conversation_id: New session ID
+            fork_from_message_id: UUID to fork from. If None, forks from last message
+            new_agent_name: Not used in Claude format (no agent metadata in sessions)
+
+        Returns:
+            The UUID of the fork point message
+        """
+        # Find source session
+        sessions = self._list_sessions()
+        source_path = None
+        for sid, spath in sessions:
+            if sid == source_conversation_id:
+                source_path = spath
+                break
+
+        if not source_path:
+            msg = f"Source conversation not found: {source_conversation_id}"
+            raise ValueError(msg)
+
+        # Read source entries
+        entries = self._read_session(source_path)
+
+        # Find fork point
+        fork_point_id: str | None = None
+        if fork_from_message_id:
+            # Verify message exists
+            found = False
+            for entry in entries:
+                if (
+                    isinstance(entry, (ClaudeUserEntry, ClaudeAssistantEntry))
+                    and entry.uuid == fork_from_message_id
+                ):
+                    found = True
+                    fork_point_id = fork_from_message_id
+                    break
+            if not found:
+                err = f"Message {fork_from_message_id} not found in conversation"
+                raise ValueError(err)
+        else:
+            # Find last message
+            message_entries = [
+                e for e in entries if isinstance(e, (ClaudeUserEntry, ClaudeAssistantEntry))
+            ]
+            if message_entries:
+                fork_point_id = message_entries[-1].uuid
+
+        # Create new session file (empty for now - will be populated when messages added)
+        # Determine project from source path structure
+        project_name = source_path.parent.name
+        new_path = self.projects_path / project_name / f"{new_conversation_id}.jsonl"
+        new_path.parent.mkdir(parents=True, exist_ok=True)
+        new_path.touch()
+
+        return fork_point_id