@prmichaelsen/acp-visualizer 0.1.0

package/agent/tasks/milestone-1-project-scaffold-data-pipeline/task-3-build-server-api-data-loading.md

@@ -0,0 +1,193 @@
+# Task 3: Build Server API & Data Loading
+
+**Milestone**: [M1 - Project Scaffold & Data Pipeline](../../milestones/milestone-1-project-scaffold-data-pipeline.md)
+**Design Reference**: [Server API & Auto-Refresh](../../design/local.server-api-auto-refresh.md)
+**Estimated Time**: 2 hours
+**Dependencies**: Task 2
+**Status**: Not Started
+
+---
+
+## Objective
+
+Create the server-side data loading pipeline using TanStack Start server functions and the library-services pattern. The pipeline reads progress.yaml from disk, parses it through the lenient YAML parser, and delivers typed ProgressData to the client via SSR preloading.
+
+---
+
+## Context
+
+TanStack Start supports server-side data loading via `beforeLoad` in route definitions, which enables SSR preloading so the dashboard renders with data on first paint (no loading spinner). The data loading follows the library-services pattern: a static service class (`ProgressDatabaseService`) encapsulates all data access logic, and routes call service methods in their `beforeLoad` hooks. The file path is configurable via environment variable, CLI argument, or default convention.
+
+---
+
+## Steps
+
+### 1. Create configuration module
+
+Create `app/lib/config.ts` to resolve the progress.yaml file path:
+
+```typescript
+// app/lib/config.ts
+
+export function getProgressYamlPath(): string {
+  // Priority: env var > CLI arg > default
+  if (process.env.PROGRESS_YAML_PATH) {
+    return process.env.PROGRESS_YAML_PATH;
+  }
+
+  const cliArg = process.argv.find((arg) => arg.startsWith("--progress-yaml="));
+  if (cliArg) {
+    return cliArg.split("=")[1];
+  }
+
+  return "./agent/progress.yaml";
+}
+```
+
+### 2. Create the ProgressDatabaseService
+
+Create `app/services/progress-database.service.ts` following the library-services pattern (static methods, no instantiation):
+
+```typescript
+// app/services/progress-database.service.ts
+import fs from "node:fs";
+import { getProgressYamlPath } from "../lib/config";
+import { parseProgressYaml } from "../lib/yaml-loader";
+import type { ProgressData } from "../lib/types";
+
+export interface ProgressResult {
+  success: true;
+  data: ProgressData;
+} | {
+  success: false;
+  error: "FILE_NOT_FOUND" | "PARSE_ERROR";
+  message: string;
+}
+
+export class ProgressDatabaseService {
+  static getProgressData(): ProgressResult {
+    const filePath = getProgressYamlPath();
+
+    try {
+      const raw = fs.readFileSync(filePath, "utf-8");
+      const data = parseProgressYaml(raw);
+      return { success: true, data };
+    } catch (err: unknown) {
+      if ((err as NodeJS.ErrnoException).code === "ENOENT") {
+        return {
+          success: false,
+          error: "FILE_NOT_FOUND",
+          message: `progress.yaml not found at: ${filePath}`,
+        };
+      }
+      return {
+        success: false,
+        error: "PARSE_ERROR",
+        message: `Failed to parse progress.yaml: ${(err as Error).message}`,
+      };
+    }
+  }
+}
+```
+
+### 3. Implement error handling with structured errors
+
+The service returns a discriminated union (`ProgressResult`) so the UI can render appropriate error states:
+
+- `FILE_NOT_FOUND`: Show a helpful message with the expected file path and how to configure it
+- `PARSE_ERROR`: Show the error message so the user can fix the YAML
+
+### 4. Update the index route with SSR data preloading
+
+Update `app/routes/index.tsx` to load data server-side using `beforeLoad`:
+
+```typescript
+// app/routes/index.tsx
+import { createFileRoute } from "@tanstack/react-router";
+import { ProgressDatabaseService } from "../services/progress-database.service";
+
+export const Route = createFileRoute("/")({
+  beforeLoad: async () => {
+    const result = ProgressDatabaseService.getProgressData();
+    return { progressResult: result };
+  },
+  component: HomePage,
+});
+
+function HomePage() {
+  const { progressResult } = Route.useRouteContext();
+
+  if (!progressResult.success) {
+    return (
+      <div className="p-8">
+        <h1 className="text-xl font-bold text-red-400">
+          Error: {progressResult.error}
+        </h1>
+        <p className="mt-2 text-gray-400 font-mono text-sm">
+          {progressResult.message}
+        </p>
+      </div>
+    );
+  }
+
+  return (
+    <div className="p-8">
+      <h1 className="text-2xl font-bold">ACP Progress Visualizer</h1>
+      <pre className="mt-4 p-4 bg-gray-900 rounded text-xs font-mono overflow-auto max-h-[80vh]">
+        {JSON.stringify(progressResult.data, null, 2)}
+      </pre>
+    </div>
+  );
+}
+```
+
+### 5. Access data via Route.useRouteContext()
+
+The `beforeLoad` return value is available in the component via `Route.useRouteContext()`. This pattern ensures:
+
+- Data is loaded server-side during SSR (no loading spinner)
+- The component receives typed data
+- Error states are handled before rendering
+
+### 6. Display raw JSON for pipeline verification
+
+The initial implementation renders raw JSON of the parsed data. This serves as a visual verification that the entire pipeline works: file reading, YAML parsing, SSR delivery, and client rendering. This will be replaced with proper dashboard components in Milestone 2.
+
+---
+
+## Verification
+
+- [ ] `app/lib/config.ts` exists and exports `getProgressYamlPath()`
+- [ ] `app/services/progress-database.service.ts` exists and exports `ProgressDatabaseService`
+- [ ] Page loads with SSR data (no loading spinner, data visible on first paint)
+- [ ] Setting `PROGRESS_YAML_PATH` env var changes the file path
+- [ ] Missing file shows structured FILE_NOT_FOUND error with the attempted path
+- [ ] Malformed YAML shows PARSE_ERROR with error message
+- [ ] Valid progress.yaml renders as formatted JSON on the page
+- [ ] No client-side fetch waterfall (data arrives with SSR)
+
+---
+
+## Expected Output
+
+**Key Files Created**:
+- `app/lib/config.ts`: Configuration module for resolving progress.yaml path
+- `app/services/progress-database.service.ts`: Service class for reading and parsing progress data
+
+**Key Files Modified**:
+- `app/routes/index.tsx`: Updated with `beforeLoad` SSR data preloading and JSON display
+
+---
+
+## Notes
+
+- The library-services pattern uses static methods on a class rather than module-level functions. This provides a clear namespace and makes it easy to add methods later (e.g., `getProgressDataForMilestone`)
+- The `beforeLoad` hook runs on the server during SSR, which means `fs.readFileSync` is safe to use here
+- The raw JSON display is temporary and will be replaced by dashboard components in Milestone 2 tasks
+- Consider adding caching to `ProgressDatabaseService` if performance becomes a concern (not needed for local dev tool)
+
+---
+
+**Next Task**: [Task 4: Add Auto-Refresh via SSE](./task-4-add-auto-refresh-sse.md)
+**Related Design Docs**: [Server API & Auto-Refresh](../../design/local.server-api-auto-refresh.md)
+**Estimated Completion Date**: TBD

package/agent/tasks/milestone-1-project-scaffold-data-pipeline/task-4-add-auto-refresh-sse.md

@@ -0,0 +1,262 @@
+# Task 4: Add Auto-Refresh via SSE
+
+**Milestone**: [M1 - Project Scaffold & Data Pipeline](../../milestones/milestone-1-project-scaffold-data-pipeline.md)
+**Design Reference**: [Server API & Auto-Refresh](../../design/local.server-api-auto-refresh.md)
+**Estimated Time**: 2 hours
+**Dependencies**: Task 3
+**Status**: Not Started
+
+---
+
+## Objective
+
+Implement file watching and Server-Sent Events (SSE) so the dashboard auto-refreshes when progress.yaml changes on disk. This enables a live development experience where agents updating progress.yaml see their changes reflected in the browser without manual reload.
+
+---
+
+## Context
+
+The progress.yaml file is updated frequently by agents during work sessions. Rather than requiring manual browser refresh, the dashboard should detect file changes and re-render automatically. The implementation uses Node.js `fs.watch` on the server to detect changes, SSE to push notifications to the browser, and TanStack Router's `router.invalidate()` to trigger data re-fetching. This is a local development tool, so the SSE approach is appropriate (no need for WebSocket complexity or production-scale polling).
+
+---
+
+## Steps
+
+### 1. Create the file watcher module
+
+Create `app/lib/file-watcher.ts` that manages file watching and connected SSE clients:
+
+```typescript
+// app/lib/file-watcher.ts
+import fs from "node:fs";
+import { getProgressYamlPath } from "./config";
+
+type ClientCallback = () => void;
+
+let clients: Set<ClientCallback> = new Set();
+let watcher: fs.FSWatcher | null = null;
+
+export function createFileWatcher() {
+  if (watcher) return; // Already watching
+
+  const filePath = getProgressYamlPath();
+
+  try {
+    watcher = fs.watch(filePath, { persistent: false }, (eventType) => {
+      if (eventType === "change") {
+        // Notify all connected clients
+        for (const client of clients) {
+          client();
+        }
+      }
+    });
+
+    watcher.on("error", (err) => {
+      console.error("[file-watcher] Watch error:", err.message);
+      watcher?.close();
+      watcher = null;
+    });
+  } catch (err) {
+    console.error("[file-watcher] Failed to start watching:", (err as Error).message);
+  }
+}
+
+export function registerClient(callback: ClientCallback): () => void {
+  clients.add(callback);
+  // Return unregister function
+  return () => {
+    clients.delete(callback);
+  };
+}
+
+export function getClientCount(): number {
+  return clients.size;
+}
+```
+
+Key design decisions:
+- Uses `fs.watch` (not `fs.watchFile`) for efficiency since this is a single known file path
+- `persistent: false` so the watcher does not keep the Node process alive
+- Singleton pattern: only one watcher instance regardless of how many clients connect
+- Returns an unregister function for clean client removal
+
+### 2. Create the SSE API route
+
+Create `app/routes/api/watch.ts` that serves the SSE endpoint:
+
+```typescript
+// app/routes/api/watch.ts
+import { createAPIFileRoute } from "@tanstack/react-start/api";
+import { createFileWatcher, registerClient } from "../../lib/file-watcher";
+
+export const APIRoute = createAPIFileRoute("/api/watch")({
+  GET: async () => {
+    // Ensure watcher is running
+    createFileWatcher();
+
+    const stream = new ReadableStream({
+      start(controller) {
+        const encoder = new TextEncoder();
+
+        // Send initial connection confirmation
+        controller.enqueue(encoder.encode("data: connected\n\n"));
+
+        // Register for file change notifications
+        const unregister = registerClient(() => {
+          try {
+            controller.enqueue(encoder.encode("data: refresh\n\n"));
+          } catch {
+            // Client disconnected
+            unregister();
+          }
+        });
+
+        // Handle client disconnect (stream cancellation)
+        // Note: cleanup is handled when enqueue throws
+      },
+    });
+
+    return new Response(stream, {
+      headers: {
+        "Content-Type": "text/event-stream",
+        "Cache-Control": "no-cache",
+        Connection: "keep-alive",
+      },
+    });
+  },
+});
+```
+
+### 3. Create the useAutoRefresh hook
+
+Create `app/lib/useAutoRefresh.ts` that connects to the SSE endpoint and triggers router invalidation:
+
+```typescript
+// app/lib/useAutoRefresh.ts
+import { useEffect } from "react";
+import { useRouter } from "@tanstack/react-router";
+
+export function useAutoRefresh() {
+  const router = useRouter();
+
+  useEffect(() => {
+    const eventSource = new EventSource("/api/watch");
+
+    eventSource.onmessage = (event) => {
+      if (event.data === "refresh") {
+        // Invalidate all route loaders, triggering re-fetch
+        router.invalidate();
+      }
+    };
+
+    eventSource.onerror = () => {
+      // EventSource will automatically reconnect on error
+      // (built-in browser behavior)
+      console.warn("[auto-refresh] SSE connection lost, reconnecting...");
+    };
+
+    return () => {
+      eventSource.close();
+    };
+  }, [router]);
+}
+```
+
+Key behaviors:
+- `EventSource` has built-in reconnection on connection loss (browser standard behavior)
+- `router.invalidate()` re-runs all `beforeLoad` hooks, which re-reads and re-parses progress.yaml
+- Cleanup on unmount prevents memory leaks
+
+### 4. Add useAutoRefresh to root layout
+
+Update `app/routes/__root.tsx` to activate auto-refresh on all pages:
+
+```typescript
+// app/routes/__root.tsx
+import { createRootRoute, Outlet } from "@tanstack/react-router";
+import { useAutoRefresh } from "../lib/useAutoRefresh";
+
+export const Route = createRootRoute({
+  component: RootLayout,
+});
+
+function RootLayout() {
+  useAutoRefresh();
+
+  return (
+    <div className="dark min-h-screen bg-gray-950 text-gray-100">
+      <Outlet />
+    </div>
+  );
+}
+```
+
+### 5. Test the auto-refresh flow
+
+End-to-end verification:
+
+1. Start the dev server with `npm run dev`
+2. Open the dashboard in a browser
+3. Verify the SSE connection is established (check Network tab for `/api/watch` request with `text/event-stream` type)
+4. Modify the project's `progress.yaml` file (e.g., change a task status)
+5. Verify the browser updates automatically without manual refresh
+6. Check that there are no errors in the browser console or server logs
+
+---
+
+## Verification
+
+- [ ] `app/lib/file-watcher.ts` exists and exports `createFileWatcher`, `registerClient`, `getClientCount`
+- [ ] `app/routes/api/watch.ts` exists and serves SSE responses
+- [ ] `app/lib/useAutoRefresh.ts` exists and exports `useAutoRefresh` hook
+- [ ] SSE endpoint responds with `Content-Type: text/event-stream`
+- [ ] SSE endpoint sends `data: connected` on initial connection
+- [ ] Modifying progress.yaml triggers `data: refresh` SSE event
+- [ ] Browser re-renders with updated data after file change
+- [ ] EventSource reconnects automatically on connection loss (verify by restarting server)
+- [ ] No errors in browser console during normal operation
+- [ ] Root layout includes `useAutoRefresh()` call
+
+---
+
+## Expected Output
+
+**Key Files Created**:
+- `app/lib/file-watcher.ts`: Node.js file watcher with client management
+- `app/routes/api/watch.ts`: SSE API endpoint for pushing file change notifications
+- `app/lib/useAutoRefresh.ts`: React hook that connects to SSE and triggers router invalidation
+
+**Key Files Modified**:
+- `app/routes/__root.tsx`: Updated to include `useAutoRefresh()` hook
+
+---
+
+## Common Issues and Solutions
+
+### Issue 1: fs.watch fires multiple events for a single save
+**Symptom**: Dashboard refreshes multiple times when progress.yaml is saved once
+**Solution**: Add a debounce (e.g., 100ms) to the file watcher notification. Ignore events that fire within the debounce window of the last event.
+
+### Issue 2: fs.watch not available or not working on certain filesystems
+**Symptom**: File changes are not detected
+**Solution**: Fall back to `fs.watchFile` (polling-based) if `fs.watch` fails. Log a warning about reduced performance.
+
+### Issue 3: SSE connection shows as pending indefinitely in DevTools
+**Symptom**: Network tab shows `/api/watch` request stuck in "pending" state
+**Solution**: This is expected behavior for SSE connections. The connection stays open to receive events. It is not an error.
+
+---
+
+## Notes
+
+- This is a local development tool, so SSE is appropriate. For production, consider WebSocket or polling with ETags
+- The file watcher is a singleton: multiple browser tabs share the same watcher but each has its own SSE connection
+- `fs.watch` behavior varies by OS; test on the target platform (Linux with inotify is reliable)
+- Consider adding a debounce to the file watcher if double-fires become an issue during testing
+- The `useAutoRefresh` hook is placed in the root layout so it works across all routes, including future milestone and task detail pages
+
+---
+
+**Next Task**: [Milestone 2 Tasks](../milestone-2-dashboard-views-interaction/)
+**Related Design Docs**: [Server API & Auto-Refresh](../../design/local.server-api-auto-refresh.md)
+**Estimated Completion Date**: TBD

package/agent/tasks/milestone-2-dashboard-views-interaction/task-10-polish-integration-testing.md

@@ -0,0 +1,156 @@
+# Task 10: Polish & Integration Testing
+
+**Milestone**: [M2 - Dashboard Views & Interaction](../../milestones/milestone-2-dashboard-views-interaction.md)
+**Design Reference**: [Visualizer Requirements](../../design/local.visualizer-requirements.md)
+**Estimated Time**: 2 hours
+**Dependencies**: Task 9
+**Status**: Not Started
+
+---
+
+## Objective
+
+Final polish pass — ensure all views work together, error states are handled, empty states are graceful, and the full data flow works end-to-end. This task validates that all M2 features are functional and ready for use.
+
+---
+
+## Context
+
+Tasks 5–9 build individual features (layout, overview, table, tree, search/filter) in isolation. This task validates they work together as a cohesive application. It focuses on integration testing, edge cases, error handling, and visual polish. Real-world progress.yaml files can be large (1800+ lines) and may contain unexpected data shapes, so testing with real data is critical. This is the final task in Milestone 2 and should result in a fully functional dashboard.
+
+---
+
+## Steps
+
+### 1. Test full data flow end-to-end
+
+Verify the complete data pipeline works across all views:
+
+- `progress.yaml` file is read from configured path
+- YAML is parsed into typed ProgressData
+- Data is served via SSR (server function → route loader)
+- Overview page renders project metadata, milestones, next steps, blockers
+- Milestones page renders table view with all milestones and tasks
+- Milestones page renders tree view when toggled
+- Search page finds and displays matching results
+- Auto-refresh updates all views when progress.yaml changes
+
+### 2. Verify error states
+
+Test error handling for common failure scenarios:
+
+- **File not found**: When `PROGRESS_YAML_PATH` points to a non-existent file, display a configuration prompt explaining how to set the path
+- **Parse error**: When progress.yaml contains invalid YAML, display a helpful error message showing the parse error details
+- **Missing fields**: When progress.yaml is valid but missing expected fields, degrade gracefully (show available data, skip missing sections)
+
+### 3. Verify empty states
+
+Test empty data scenarios:
+
+- **No milestones**: Overview and milestones pages show "No milestones found" message
+- **No tasks in milestone**: Expanded milestone row shows "No tasks" message
+- **No search results**: Search page shows "No results found for '{query}'" message
+- **No blockers**: BlockersDisplay renders nothing (no empty card)
+- **No next steps**: NextSteps renders nothing (no empty card)
+
+### 4. Verify auto-refresh across all views
+
+- File watcher detects changes to progress.yaml
+- All currently visible views update with new data
+- Expanded state in table/tree views is preserved after refresh
+- Search/filter state is preserved after refresh
+- No flicker or layout shift during refresh
+
+### 5. Test with real progress.yaml data
+
+Load a real-world progress.yaml file (e.g., from ACP core project, 1800+ lines):
+
+- Verify all milestones and tasks render without errors
+- Verify performance is acceptable (no lag when scrolling, sorting, searching)
+- Verify ExtraFieldsBadge shows correct counts for tasks with custom fields
+- Verify long text (names, notes) truncates gracefully
+- Verify large numbers of tasks don't break layout
+
+### 6. Fix styling inconsistencies
+
+Review all views for visual polish:
+
+- Consistent spacing between sections (space-y-6)
+- Consistent card styling (bg-gray-900/50, border-gray-800, rounded-xl)
+- Consistent text hierarchy (headings, subheadings, body, captions)
+- Hover states on interactive elements (nav links, table rows, buttons)
+- Focus states for keyboard navigation (search input, filter buttons)
+- No orphaned borders, extra padding, or alignment issues
+
+### 7. Verify route navigation
+
+Test all navigation paths:
+
+- Sidebar links navigate to correct pages
+- Browser back/forward navigation works
+- Direct URL access works (e.g., navigating directly to `/milestones`)
+- Active nav item in sidebar matches current route
+- No 404 or blank pages for any defined route
+
+### 8. Test edge cases
+
+Verify the app handles unusual data shapes:
+
+- Single milestone with no tasks
+- Single task with no notes
+- Milestone with many tasks (20+)
+- Task with many extra fields (10+)
+- Very long milestone/task names (100+ characters)
+- Special characters in names and notes (quotes, angle brackets, unicode)
+- Status values beyond the standard set (should render as gray/unknown)
+
+---
+
+## Verification
+
+- [ ] All P0 features from requirements are functional (overview, table, tree, search, filter)
+- [ ] No console errors or warnings in browser developer tools
+- [ ] All views render correctly with real progress.yaml data (1800+ lines)
+- [ ] Error state: file-not-found shows configuration prompt
+- [ ] Error state: parse error shows helpful message with details
+- [ ] Empty state: no milestones shows appropriate message
+- [ ] Empty state: no search results shows appropriate message
+- [ ] Auto-refresh works across overview, table, tree, and search views
+- [ ] Search finds milestones and tasks by name with fuzzy matching
+- [ ] Status filter narrows results correctly
+- [ ] All sidebar navigation links work correctly
+- [ ] Browser back/forward navigation works
+- [ ] No styling inconsistencies or visual bugs
+- [ ] Edge cases handled: long text, special characters, missing fields, many items
+
+---
+
+## Expected Output
+
+**Key Files Modified**:
+- Bug fixes and polish across existing files created in Tasks 5–9
+- No major new files expected — this is a testing and polish task
+
+**Potential files touched**:
+- `app/routes/__root.tsx` — error boundary, layout adjustments
+- `app/routes/index.tsx` — empty state handling
+- `app/routes/milestones.tsx` — error/empty states
+- `app/routes/search.tsx` — empty state messaging
+- `app/components/*.tsx` — styling fixes, edge case handling
+- `app/lib/*.ts` — search/filter edge case fixes
+
+---
+
+## Notes
+
+- This is a testing and polish task, not a feature-building task — the goal is quality, not new functionality
+- Keep a list of issues found and fixed for the commit message
+- If any P0 feature is broken or missing, fix it before marking this task complete
+- Performance testing with large files is important — if search or rendering is slow, optimize before closing
+- This task marks the end of Milestone 2; after completion, the dashboard should be fully usable for viewing progress.yaml data
+
+---
+
+**Next Task**: Milestone 3 tasks (TBD)
+**Related Design Docs**: [Visualizer Requirements](../../design/local.visualizer-requirements.md)
+**Estimated Completion Date**: TBD