ag-cortex 0.1.0

package/.agent/skills/agent-native-architecture/references/shared-workspace-architecture.md

@@ -0,0 +1,680 @@
+<overview>
+Agents and users should work in the same data space, not separate sandboxes. When the agent writes a file, the user can see it. When the user edits something, the agent can read the changes. This creates transparency, enables collaboration, and eliminates the need for sync layers.
+
+**Core principle:** The agent operates in the same filesystem as the user, not a walled garden.
+</overview>
+
+<why_shared_workspace>
+## Why Shared Workspace?
+
+### The Sandbox Anti-Pattern
+
+Many agent implementations isolate the agent:
+
+```
+┌─────────────────┐     ┌─────────────────┐
+│   User Space    │     │   Agent Space   │
+├─────────────────┤     ├─────────────────┤
+│ Documents/      │     │ agent_output/   │
+│ user_files/     │  ←→ │ temp_files/     │
+│ settings.json   │sync │ cache/          │
+└─────────────────┘     └─────────────────┘
+```
+
+Problems:
+- Need a sync layer to move data between spaces
+- User can't easily inspect agent work
+- Agent can't build on user contributions
+- Duplication of state
+- Complexity in keeping spaces consistent
+
+### The Shared Workspace Pattern
+
+```
+┌─────────────────────────────────────────┐
+│           Shared Workspace              │
+├─────────────────────────────────────────┤
+│ Documents/                              │
+│ ├── Research/                           │
+│ │   └── {bookId}/        ← Agent writes │
+│ │       ├── full_text.txt               │
+│ │       ├── introduction.md  ← User can edit │
+│ │       └── sources/                    │
+│ ├── Chats/               ← Both read/write │
+│ └── profile.md           ← Agent generates, user refines │
+└─────────────────────────────────────────┘
+         ↑                    ↑
+       User                 Agent
+       (UI)               (Tools)
+```
+
+Benefits:
+- Users can inspect, edit, and extend agent work
+- Agents can build on user contributions
+- No synchronization layer needed
+- Complete transparency
+- Single source of truth
+</why_shared_workspace>
+
+<directory_structure>
+## Designing Your Shared Workspace
+
+### Structure by Domain
+
+Organize by what the data represents, not who created it:
+
+```
+Documents/
+├── Research/
+│   └── {bookId}/
+│       ├── full_text.txt        # Agent downloads
+│       ├── introduction.md      # Agent generates, user can edit
+│       ├── notes.md             # User adds, agent can read
+│       └── sources/
+│           └── {source}.md      # Agent gathers
+├── Chats/
+│   └── {conversationId}.json    # Both read/write
+├── Exports/
+│   └── {date}/                  # Agent generates for user
+└── profile.md                   # Agent generates from photos
+```
+
+### Don't Structure by Actor
+
+```
+# BAD - Separates by who created it
+Documents/
+├── user_created/
+│   └── notes.md
+├── agent_created/
+│   └── research.md
+└── system/
+    └── config.json
+```
+
+This creates artificial boundaries and makes collaboration harder.
+
+### Use Conventions for Metadata
+
+If you need to track who created/modified something:
+
+```markdown
+<!-- introduction.md -->
+---
+created_by: agent
+created_at: 2024-01-15
+last_modified_by: user
+last_modified_at: 2024-01-16
+---
+
+# Introduction to Moby Dick
+
+This personalized introduction was generated by your reading assistant
+and refined by you on January 16th.
+```
+</directory_structure>
+
+<file_tools>
+## File Tools for Shared Workspace
+
+Give the agent the same file primitives the app uses:
+
+```swift
+// iOS/Swift implementation
+struct FileTools {
+    static func readFile() -> AgentTool {
+        tool(
+            name: "read_file",
+            description: "Read a file from the user's documents",
+            parameters: ["path": .string("File path relative to Documents/")],
+            execute: { params in
+                let path = params["path"] as! String
+                let documentsURL = FileManager.default.urls(for: .documentDirectory, in: .userDomainMask)[0]
+                let fileURL = documentsURL.appendingPathComponent(path)
+                let content = try String(contentsOf: fileURL)
+                return ToolResult(text: content)
+            }
+        )
+    }
+
+    static func writeFile() -> AgentTool {
+        tool(
+            name: "write_file",
+            description: "Write a file to the user's documents",
+            parameters: [
+                "path": .string("File path relative to Documents/"),
+                "content": .string("File content")
+            ],
+            execute: { params in
+                let path = params["path"] as! String
+                let content = params["content"] as! String
+                let documentsURL = FileManager.default.urls(for: .documentDirectory, in: .userDomainMask)[0]
+                let fileURL = documentsURL.appendingPathComponent(path)
+
+                // Create parent directories if needed
+                try FileManager.default.createDirectory(
+                    at: fileURL.deletingLastPathComponent(),
+                    withIntermediateDirectories: true
+                )
+
+                try content.write(to: fileURL, atomically: true, encoding: .utf8)
+                return ToolResult(text: "Wrote \(path)")
+            }
+        )
+    }
+
+    static func listFiles() -> AgentTool {
+        tool(
+            name: "list_files",
+            description: "List files in a directory",
+            parameters: ["path": .string("Directory path relative to Documents/")],
+            execute: { params in
+                let path = params["path"] as! String
+                let documentsURL = FileManager.default.urls(for: .documentDirectory, in: .userDomainMask)[0]
+                let dirURL = documentsURL.appendingPathComponent(path)
+                let contents = try FileManager.default.contentsOfDirectory(atPath: dirURL.path)
+                return ToolResult(text: contents.joined(separator: "\n"))
+            }
+        )
+    }
+
+    static func searchText() -> AgentTool {
+        tool(
+            name: "search_text",
+            description: "Search for text across files",
+            parameters: [
+                "query": .string("Text to search for"),
+                "path": .string("Directory to search in").optional()
+            ],
+            execute: { params in
+                // Implement text search across documents
+                // Return matching files and snippets
+            }
+        )
+    }
+}
+```
+
+### TypeScript/Node.js Implementation
+
+```typescript
+const fileTools = [
+  tool(
+    "read_file",
+    "Read a file from the workspace",
+    { path: z.string().describe("File path") },
+    async ({ path }) => {
+      const content = await fs.readFile(path, 'utf-8');
+      return { text: content };
+    }
+  ),
+
+  tool(
+    "write_file",
+    "Write a file to the workspace",
+    {
+      path: z.string().describe("File path"),
+      content: z.string().describe("File content")
+    },
+    async ({ path, content }) => {
+      await fs.mkdir(dirname(path), { recursive: true });
+      await fs.writeFile(path, content, 'utf-8');
+      return { text: `Wrote ${path}` };
+    }
+  ),
+
+  tool(
+    "list_files",
+    "List files in a directory",
+    { path: z.string().describe("Directory path") },
+    async ({ path }) => {
+      const files = await fs.readdir(path);
+      return { text: files.join('\n') };
+    }
+  ),
+
+  tool(
+    "append_file",
+    "Append content to a file",
+    {
+      path: z.string().describe("File path"),
+      content: z.string().describe("Content to append")
+    },
+    async ({ path, content }) => {
+      await fs.appendFile(path, content, 'utf-8');
+      return { text: `Appended to ${path}` };
+    }
+  ),
+];
+```
+</file_tools>
+
+<ui_integration>
+## UI Integration with Shared Workspace
+
+The UI should observe the same files the agent writes to:
+
+### Pattern 1: File-Based Reactivity (iOS)
+
+```swift
+class ResearchViewModel: ObservableObject {
+    @Published var researchFiles: [ResearchFile] = []
+
+    private var watcher: DirectoryWatcher?
+
+    func startWatching(bookId: String) {
+        let researchPath = documentsURL
+            .appendingPathComponent("Research")
+            .appendingPathComponent(bookId)
+
+        watcher = DirectoryWatcher(url: researchPath) { [weak self] in
+            // Reload when agent writes new files
+            self?.loadResearchFiles(from: researchPath)
+        }
+
+        loadResearchFiles(from: researchPath)
+    }
+}
+
+// SwiftUI automatically updates when files change
+struct ResearchView: View {
+    @StateObject var viewModel = ResearchViewModel()
+
+    var body: some View {
+        List(viewModel.researchFiles) { file in
+            ResearchFileRow(file: file)
+        }
+    }
+}
+```
+
+### Pattern 2: Shared Data Store
+
+When file-watching isn't practical, use a shared data store:
+
+```swift
+// Shared service that both UI and agent tools use
+class BookLibraryService: ObservableObject {
+    static let shared = BookLibraryService()
+
+    @Published var books: [Book] = []
+    @Published var analysisRecords: [AnalysisRecord] = []
+
+    func addAnalysisRecord(_ record: AnalysisRecord) {
+        analysisRecords.append(record)
+        // Persists to shared storage
+        saveToStorage()
+    }
+}
+
+// Agent tool writes through the same service
+tool("publish_to_feed", async ({ bookId, content, headline }) => {
+    let record = AnalysisRecord(bookId: bookId, content: content, headline: headline)
+    BookLibraryService.shared.addAnalysisRecord(record)
+    return { text: "Published to feed" }
+})
+
+// UI observes the same service
+struct FeedView: View {
+    @StateObject var library = BookLibraryService.shared
+
+    var body: some View {
+        List(library.analysisRecords) { record in
+            FeedItemRow(record: record)
+        }
+    }
+}
+```
+
+### Pattern 3: Hybrid (Files + Index)
+
+Use files for content, database for indexing:
+
+```
+Documents/
+├── Research/
+│   └── book_123/
+│       └── introduction.md   # Actual content (file)
+
+Database:
+├── research_index
+│   └── { bookId: "book_123", path: "Research/book_123/introduction.md", ... }
+```
+
+```swift
+// Agent writes file
+await writeFile("Research/\(bookId)/introduction.md", content)
+
+// And updates index
+await database.insert("research_index", {
+    bookId: bookId,
+    path: "Research/\(bookId)/introduction.md",
+    title: extractTitle(content),
+    createdAt: Date()
+})
+
+// UI queries index, then reads files
+let items = database.query("research_index", where: bookId == "book_123")
+for item in items {
+    let content = readFile(item.path)
+    // Display...
+}
+```
+</ui_integration>
+
+<collaboration_patterns>
+## Agent-User Collaboration Patterns
+
+### Pattern: Agent Drafts, User Refines
+
+```
+1. Agent generates introduction.md
+2. User opens in Files app or in-app editor
+3. User makes refinements
+4. Agent can see changes via read_file
+5. Future agent work builds on user refinements
+```
+
+The agent's system prompt should acknowledge this:
+
+```markdown
+## Working with User Content
+
+When you create content (introductions, research notes, etc.), the user may
+edit it afterward. Always read existing files before modifying them—the user
+may have made improvements you should preserve.
+
+If a file exists and has been modified by the user (check the metadata or
+compare to your last known version), ask before overwriting.
+```
+
+### Pattern: User Seeds, Agent Expands
+
+```
+1. User creates notes.md with initial thoughts
+2. User asks: "Research more about this"
+3. Agent reads notes.md to understand context
+4. Agent adds to notes.md or creates related files
+5. User continues building on agent additions
+```
+
+### Pattern: Append-Only Collaboration
+
+For chat logs or activity streams:
+
+```markdown
+<!-- activity.md - Both append, neither overwrites -->
+
+## 2024-01-15
+
+**User:** Started reading "Moby Dick"
+
+**Agent:** Downloaded full text and created research folder
+
+**User:** Added highlight about whale symbolism
+
+**Agent:** Found 3 academic sources on whale symbolism in Melville's work
+```
+</collaboration_patterns>
+
+<security_considerations>
+## Security in Shared Workspace
+
+### Scope the Workspace
+
+Don't give agents access to the entire filesystem:
+
+```swift
+// GOOD: Scoped to app's documents
+let documentsURL = FileManager.default.urls(for: .documentDirectory, in: .userDomainMask)[0]
+
+tool("read_file", { path }) {
+    // Path is relative to documents, can't escape
+    let fileURL = documentsURL.appendingPathComponent(path)
+    guard fileURL.path.hasPrefix(documentsURL.path) else {
+        throw ToolError("Invalid path")
+    }
+    return try String(contentsOf: fileURL)
+}
+
+// BAD: Absolute paths allow escape
+tool("read_file", { path }) {
+    return try String(contentsOf: URL(fileURLWithPath: path))  // Can read /etc/passwd!
+}
+```
+
+### Protect Sensitive Files
+
+```swift
+let protectedPaths = [".env", "credentials.json", "secrets/"]
+
+tool("read_file", { path }) {
+    if protectedPaths.any({ path.contains($0) }) {
+        throw ToolError("Cannot access protected file")
+    }
+    // ...
+}
+```
+
+### Audit Agent Actions
+
+Log what the agent reads/writes:
+
+```swift
+func logFileAccess(action: String, path: String, agentId: String) {
+    logger.info("[\(agentId)] \(action): \(path)")
+}
+
+tool("write_file", { path, content }) {
+    logFileAccess(action: "WRITE", path: path, agentId: context.agentId)
+    // ...
+}
+```
+</security_considerations>
+
+<examples>
+## Real-World Example: Every Reader
+
+The Every Reader app uses shared workspace for research:
+
+```
+Documents/
+├── Research/
+│   └── book_moby_dick/
+│       ├── full_text.txt           # Agent downloads from Gutenberg
+│       ├── introduction.md         # Agent generates, personalized
+│       ├── sources/
+│       │   ├── whale_symbolism.md  # Agent researches
+│       │   └── melville_bio.md     # Agent researches
+│       └── user_notes.md           # User can add their own notes
+├── Chats/
+│   └── 2024-01-15.json             # Chat history
+└── profile.md                       # Agent generated from photos
+```
+
+**How it works:**
+
+1. User adds "Moby Dick" to library
+2. User starts research agent
+3. Agent downloads full text to `Research/book_moby_dick/full_text.txt`
+4. Agent researches and writes to `sources/`
+5. Agent generates `introduction.md` based on user's reading profile
+6. User can view all files in the app or Files.app
+7. User can edit `introduction.md` to refine it
+8. Chat agent can read all of this context when answering questions
+</examples>
+
+<icloud_sync>
+## iCloud File Storage for Multi-Device Sync (iOS)
+
+For agent-native iOS apps, use iCloud Drive's Documents folder for your shared workspace. This gives you **free, automatic multi-device sync** without building a sync layer or running a server.
+
+### Why iCloud Documents?
+
+| Approach | Cost | Complexity | Offline | Multi-Device |
+|----------|------|------------|---------|--------------|
+| Custom backend + sync | $$$ | High | Manual | Yes |
+| CloudKit database | Free tier limits | Medium | Manual | Yes |
+| **iCloud Documents** | Free (user's storage) | Low | Automatic | Automatic |
+
+iCloud Documents:
+- Uses user's existing iCloud storage (free 5GB, most users have more)
+- Automatic sync across all user's devices
+- Works offline, syncs when online
+- Files visible in Files.app for transparency
+- No server costs, no sync code to maintain
+
+### Implementation Pattern
+
+```swift
+// Get the iCloud Documents container
+func iCloudDocumentsURL() -> URL? {
+    FileManager.default.url(forUbiquityContainerIdentifier: nil)?
+        .appendingPathComponent("Documents")
+}
+
+// Your shared workspace lives in iCloud
+class SharedWorkspace {
+    let rootURL: URL
+
+    init() {
+        // Use iCloud if available, fall back to local
+        if let iCloudURL = iCloudDocumentsURL() {
+            self.rootURL = iCloudURL
+        } else {
+            // Fallback to local Documents (user not signed into iCloud)
+            self.rootURL = FileManager.default.urls(for: .documentDirectory, in: .userDomainMask).first!
+        }
+    }
+
+    // All file operations go through this root
+    func researchPath(for bookId: String) -> URL {
+        rootURL.appendingPathComponent("Research/\(bookId)")
+    }
+
+    func journalPath() -> URL {
+        rootURL.appendingPathComponent("Journal")
+    }
+}
+```
+
+### Directory Structure in iCloud
+
+```
+iCloud Drive/
+└── YourApp/                          # Your app's container
+    └── Documents/                    # Visible in Files.app
+        ├── Journal/
+        │   ├── user/
+        │   │   └── 2025-01-15.md     # Syncs across devices
+        │   └── agent/
+        │       └── 2025-01-15.md     # Agent observations sync too
+        ├── Experiments/
+        │   └── magnesium-sleep/
+        │       ├── config.json
+        │       └── log.json
+        └── Research/
+            └── {topic}/
+                └── sources.md
+```
+
+### Handling Sync Conflicts
+
+iCloud handles conflicts automatically, but you should design for it:
+
+```swift
+// Check for conflicts when reading
+func readJournalEntry(at url: URL) throws -> JournalEntry {
+    // iCloud may create .icloud placeholder files for not-yet-downloaded content
+    if url.pathExtension == "icloud" {
+        // Trigger download
+        try FileManager.default.startDownloadingUbiquitousItem(at: url)
+        throw FileNotYetAvailableError()
+    }
+
+    let data = try Data(contentsOf: url)
+    return try JSONDecoder().decode(JournalEntry.self, from: data)
+}
+
+// For writes, use coordinated file access
+func writeJournalEntry(_ entry: JournalEntry, to url: URL) throws {
+    let coordinator = NSFileCoordinator()
+    var error: NSError?
+
+    coordinator.coordinate(writingItemAt: url, options: .forReplacing, error: &error) { newURL in
+        let data = try? JSONEncoder().encode(entry)
+        try? data?.write(to: newURL)
+    }
+
+    if let error = error {
+        throw error
+    }
+}
+```
+
+### What This Enables
+
+1. **User starts experiment on iPhone** → Agent creates `Experiments/sleep-tracking/config.json`
+2. **User opens app on iPad** → Same experiment visible, no sync code needed
+3. **Agent logs observation on iPhone** → Syncs to iPad automatically
+4. **User edits journal on iPad** → iPhone sees the edit
+
+### Entitlements Required
+
+Add to your app's entitlements:
+
+```xml
+<key>com.apple.developer.icloud-container-identifiers</key>
+<array>
+    <string>iCloud.com.yourcompany.yourapp</string>
+</array>
+<key>com.apple.developer.icloud-services</key>
+<array>
+    <string>CloudDocuments</string>
+</array>
+<key>com.apple.developer.ubiquity-container-identifiers</key>
+<array>
+    <string>iCloud.com.yourcompany.yourapp</string>
+</array>
+```
+
+### When NOT to Use iCloud Documents
+
+- **Sensitive data** - Use Keychain or encrypted local storage instead
+- **High-frequency writes** - iCloud sync has latency; use local + periodic sync
+- **Large media files** - Consider CloudKit Assets or on-demand resources
+- **Shared between users** - iCloud Documents is single-user; use CloudKit for sharing
+</icloud_sync>
+
+<checklist>
+## Shared Workspace Checklist
+
+Architecture:
+- [ ] Single shared directory for agent and user data
+- [ ] Organized by domain, not by actor
+- [ ] File tools scoped to workspace (no escape)
+- [ ] Protected paths for sensitive files
+
+Tools:
+- [ ] `read_file` - Read any file in workspace
+- [ ] `write_file` - Write any file in workspace
+- [ ] `list_files` - Browse directory structure
+- [ ] `search_text` - Find content across files (optional)
+
+UI Integration:
+- [ ] UI observes same files agent writes
+- [ ] Changes reflect immediately (file watching or shared store)
+- [ ] User can edit agent-created files
+- [ ] Agent reads user modifications before overwriting
+
+Collaboration:
+- [ ] System prompt acknowledges user may edit files
+- [ ] Agent checks for user modifications before overwriting
+- [ ] Metadata tracks who created/modified (optional)
+
+Multi-Device (iOS):
+- [ ] Use iCloud Documents for shared workspace (free sync)
+- [ ] Fallback to local Documents if iCloud unavailable
+- [ ] Handle `.icloud` placeholder files (trigger download)
+- [ ] Use NSFileCoordinator for conflict-safe writes
+</checklist>