spooling 0.2.0__tar.gz → 0.2.1__tar.gz

{spooling-0.2.0/spooling.egg-info → spooling-0.2.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: spooling
-Version: 0.2.0
+Version: 0.2.1
 Summary: Local session tracker and semantic search for AI coding assistants
 Author: Parsed Analytics, Inc.
 License: MIT
@@ -38,7 +38,7 @@ Dynamic: license-file
 
 Local session tracker and semantic search for AI coding assistants.
 
-Track your AI coding sessions across **OpenAI Codex CLI**, **GitHub Copilot**, **Cursor**, **Windsurf**, **Kiro**, **Google Antigravity**, and **opencode**, all in one place. Get usage stats, cost estimates, per-provider breakdowns, semantic search via pgvector, and a built-in AI chat agent to explore your history.
+Track your AI coding sessions across **OpenAI Codex CLI**, **GitHub Copilot**, **Cursor**, **Windsurf**, **Kiro**, **Google Antigravity**, and **opencode**, all in one place. Get usage stats, cost estimates, per-provider breakdowns, semantic search, and a built-in AI chat agent to explore your history.
 
 **Website:** [spooling.ai](https://spooling.ai)
 
@@ -47,67 +47,50 @@ Track your AI coding sessions across **OpenAI Codex CLI**, **GitHub Copilot**, *
 ## Prerequisites
 
 - **Python 3.11+**
-- **Node.js 18+**
-- **Docker** (for PostgreSQL + pgvector)
 - **pipx** or **uv** (optional, alternative install methods)
 - **Ollama** (optional, for free local AI chat) or an **Anthropic API key**
 
+No Docker, no PostgreSQL, no Node.js required.
+
 ---
 
 ## How it works
 
 **Four commands. Zero cloud.**
 
-### 01   Clone & start the database
+### 01   Install
 
 ```bash
-git clone https://github.com/sashimiboi/spooling && cd spooling
-docker compose up -d   # postgres + pgvector :5434
-```
-
-### 02   Install backend + UI
+# pip
+pip install spooling
 
-Choose one:
-
-```bash
-# pip (recommended)
-python3 -m venv .venv && source .venv/bin/activate
-pip install -e .
-
-# pipx
-pipx install .
+# pipx (recommended — keeps it isolated)
+pipx install spooling
 
 # uv
-uv sync
-```
-
-Then install the UI:
-
-```bash
-cd ui && npm install && cd ..
+uv tool install spooling
 ```
 
-### 03   Detect providers & sync
+### 02   Detect providers & sync
 
 ```bash
 spooling init      # scan for available providers
-spooling sync      # embed every session into pgvector
+spooling sync      # embed every session for semantic search
 ```
 
-### 04   Search & explore
+### 03   Search & explore
 
 ```bash
-spooling ui        # API :3002 · MCP :3004 · GUI :3003
 spooling search "that redis race condition"
+spooling stats
+spooling mcp       # MCP server at http://127.0.0.1:3004/mcp
 ```
 
-Open **http://localhost:3003** and you're in.
-
 ---
 
 ## Connect Spooling to your AI coding agent
 
-`spooling ui` automatically exposes an MCP server at `http://127.0.0.1:3004/mcp` (HTTP streamable transport). Any MCP-speaking agent can connect to it and pull context from your local KB mid-conversation. The agent gets tools like `spooling_search`, `spooling_recent_sessions`, `spooling_get_session`, `spooling_workspace_stats`, and `spooling_top_projects`.
+`spooling mcp` exposes an MCP server at `http://127.0.0.1:3004/mcp` (HTTP streamable transport). Any MCP-speaking agent can connect to it and pull context from your local KB mid-conversation. The agent gets tools like `spooling_search`, `spooling_recent_sessions`, `spooling_get_session`, `spooling_workspace_stats`, and `spooling_top_projects`.
 
 ### Cursor / Windsurf / Codex / Antigravity
 
@@ -140,12 +123,6 @@ Should return the five `spool_*` tools. If you get "connection refused", `spooli
 
 ## CLI Usage
 
-All CLI commands require the venv to be active and the database running.
-
-```bash
-source .venv/bin/activate
-```
-
 ### `spooling init`
 
 Check database connection and show which AI coding tool providers are detected on your system. This scans default paths for Codex CLI, GitHub Copilot, Cursor, and Windsurf session data.
@@ -156,7 +133,7 @@ spooling init
 
 ### `spooling sync`
 
-Parse and ingest sessions from all connected providers into the database. Chunks and embeds message content into pgvector for semantic search.
+Parse and ingest sessions from all connected providers into the local database. Chunks and embeds message content for semantic search.
 
 ```bash
 spooling sync                      # Full sync with embeddings
@@ -199,7 +176,7 @@ spooling watch
 
 ### `spooling serve`
 
-Start the API server only (for when you want to run the GUI separately).
+Start the API server.
 
 ```bash
 spooling serve                    # Default: http://127.0.0.1:3002
@@ -209,10 +186,10 @@ spooling serve --host 0.0.0.0     # Bind to all interfaces
 
 ### `spooling ui`
 
-Start the API server, the MCP HTTP server, and the Next.js UI together.
+Start the API server, the MCP HTTP server, and the Next.js UI together. Requires the `ui/` directory from the [source repo](https://github.com/sashimiboi/spooling) to be present. Set `SPOOLING_UI_DIR` to point at it if running from a non-standard location.
 
 ```bash
-spooling ui
+spooling ui        # API :3002 · MCP :3004 · GUI :3003
 ```
 
 ### `spooling mcp`
@@ -523,7 +500,7 @@ Or use `spooling ui` to start both at once.
 
 ## Chat Agent Setup
 
-The chat page lets you ask questions about your coding sessions in natural language. It uses RAG - retrieves relevant context from pgvector before answering.
+The chat page lets you ask questions about your coding sessions in natural language. It uses RAG — retrieves relevant context from the local database before answering.
 
 ### Option A: Ollama (free, local)
 
@@ -552,24 +529,23 @@ Available models: Sonnet, Haiku, Opus.
 
 ```
 spooling/
-├── docker-compose.yml       # PostgreSQL + pgvector
-├── init.sql                 # Database schema
 ├── pyproject.toml           # Python package config
-├── spooling/                   # Python backend
+├── spooling/                # Python backend
 │   ├── cli.py               # Click CLI
 │   ├── config.py            # Configuration
-│   ├── db.py                # Database connection
-│   ├── providers/           # Provider plugins (codex, copilot, cursor, windsurf)
+│   ├── db.py                # SQLite connection (PostgreSQL shim)
+│   ├── schema.py            # SQLite schema DDL
+│   ├── providers/           # Provider plugins (codex, copilot, cursor, windsurf, ...)
 │   ├── parser.py            # Session JSONL parser
 │   ├── embeddings.py        # sentence-transformers (all-MiniLM-L6-v2)
 │   ├── ingest.py            # Sync pipeline
-│   ├── search.py            # pgvector semantic search
+│   ├── search.py            # Numpy cosine similarity search
 │   ├── stats.py             # Usage statistics
 │   ├── watcher.py           # File watcher (watchdog)
 │   ├── agent.py             # Chat agent (Ollama + Anthropic)
 │   ├── tunnel.py            # Cloudflare quick tunnel support
 │   └── server.py            # FastAPI API server
-└── ui/                      # Next.js frontend
+└── ui/                      # Next.js frontend (source repo only)
     ├── next.config.js       # API proxy to :3002
     └── src/
         ├── components/      # shadcn/ui components
@@ -581,8 +557,9 @@ spooling/
 
 | Layer | Technology |
 |-------|-----------|
-| Database | PostgreSQL 16 + pgvector (Docker) |
+| Database | SQLite (no external dependencies) |
 | Embeddings | sentence-transformers / all-MiniLM-L6-v2 (local) |
+| Search | numpy cosine similarity |
 | Backend | Python, FastAPI, Click |
 | Frontend | Next.js 14, shadcn/ui, Tailwind CSS, AG Charts |
 | Chat AI | Ollama (local) or Anthropic API |
@@ -591,7 +568,6 @@ spooling/
 
 | Service | Port |
 |---------|------|
-| PostgreSQL | 5434 |
 | API Server | 3002 |
 | GUI | 3003 |
 | MCP Server (streamable-HTTP) | 3004 |
@@ -600,19 +576,16 @@ spooling/
 
 ## Environment Variables
 
-All optional - defaults work out of the box for local development.
+All optional — defaults work out of the box.
 
 | Variable | Default | Description |
 |----------|---------|-------------|
-| `SPOOLING_DB_HOST` | `localhost` | Database host |
-| `SPOOLING_DB_PORT` | `5434` | Database port |
-| `SPOOLING_DB_NAME` | `spooling` | Database name |
-| `SPOOLING_DB_USER` | `spooling` | Database user |
-| `SPOOLING_DB_PASSWORD` | `spooling` | Database password |
+| `SPOOLING_DB` | `~/.local/share/spooling/spooling.db` | Path to the SQLite database file (`%APPDATA%\spooling\spooling.db` on Windows) |
 | `SPOOLING_EMBEDDING_MODEL` | `all-MiniLM-L6-v2` | Sentence transformer model |
 | `SPOOLING_UI_HOST` | `127.0.0.1` | API server host |
-| `ANTHROPIC_API_KEY` | - | Anthropic API key (alternative to setting in UI) |
-| `API_URL` | `http://127.0.0.1:3002` | Backend API URL for Next.js rewrites (`/api/*`→`<API_URL>/api/*`). Set to `https://api.spooling.ai` in production. |
+| `SPOOLING_UI_PORT` | `3001` | API server port (used by `spooling serve`) |
+| `SPOOLING_UI_DIR` | *(auto-detected)* | Path to the Next.js `ui/` directory (required for `spooling ui`) |
+| `ANTHROPIC_API_KEY` | - | Anthropic API key (alternative to setting in the GUI) |
 
 ---
 

{spooling-0.2.0 → spooling-0.2.1}/README.md

@@ -2,7 +2,7 @@
 
 Local session tracker and semantic search for AI coding assistants.
 
-Track your AI coding sessions across **OpenAI Codex CLI**, **GitHub Copilot**, **Cursor**, **Windsurf**, **Kiro**, **Google Antigravity**, and **opencode**, all in one place. Get usage stats, cost estimates, per-provider breakdowns, semantic search via pgvector, and a built-in AI chat agent to explore your history.
+Track your AI coding sessions across **OpenAI Codex CLI**, **GitHub Copilot**, **Cursor**, **Windsurf**, **Kiro**, **Google Antigravity**, and **opencode**, all in one place. Get usage stats, cost estimates, per-provider breakdowns, semantic search, and a built-in AI chat agent to explore your history.
 
 **Website:** [spooling.ai](https://spooling.ai)
 
@@ -11,67 +11,50 @@ Track your AI coding sessions across **OpenAI Codex CLI**, **GitHub Copilot**, *
 ## Prerequisites
 
 - **Python 3.11+**
-- **Node.js 18+**
-- **Docker** (for PostgreSQL + pgvector)
 - **pipx** or **uv** (optional, alternative install methods)
 - **Ollama** (optional, for free local AI chat) or an **Anthropic API key**
 
+No Docker, no PostgreSQL, no Node.js required.
+
 ---
 
 ## How it works
 
 **Four commands. Zero cloud.**
 
-### 01   Clone & start the database
+### 01   Install
 
 ```bash
-git clone https://github.com/sashimiboi/spooling && cd spooling
-docker compose up -d   # postgres + pgvector :5434
-```
-
-### 02   Install backend + UI
+# pip
+pip install spooling
 
-Choose one:
-
-```bash
-# pip (recommended)
-python3 -m venv .venv && source .venv/bin/activate
-pip install -e .
-
-# pipx
-pipx install .
+# pipx (recommended — keeps it isolated)
+pipx install spooling
 
 # uv
-uv sync
-```
-
-Then install the UI:
-
-```bash
-cd ui && npm install && cd ..
+uv tool install spooling
 ```
 
-### 03   Detect providers & sync
+### 02   Detect providers & sync
 
 ```bash
 spooling init      # scan for available providers
-spooling sync      # embed every session into pgvector
+spooling sync      # embed every session for semantic search
 ```
 
-### 04   Search & explore
+### 03   Search & explore
 
 ```bash
-spooling ui        # API :3002 · MCP :3004 · GUI :3003
 spooling search "that redis race condition"
+spooling stats
+spooling mcp       # MCP server at http://127.0.0.1:3004/mcp
 ```
 
-Open **http://localhost:3003** and you're in.
-
 ---
 
 ## Connect Spooling to your AI coding agent
 
-`spooling ui` automatically exposes an MCP server at `http://127.0.0.1:3004/mcp` (HTTP streamable transport). Any MCP-speaking agent can connect to it and pull context from your local KB mid-conversation. The agent gets tools like `spooling_search`, `spooling_recent_sessions`, `spooling_get_session`, `spooling_workspace_stats`, and `spooling_top_projects`.
+`spooling mcp` exposes an MCP server at `http://127.0.0.1:3004/mcp` (HTTP streamable transport). Any MCP-speaking agent can connect to it and pull context from your local KB mid-conversation. The agent gets tools like `spooling_search`, `spooling_recent_sessions`, `spooling_get_session`, `spooling_workspace_stats`, and `spooling_top_projects`.
 
 ### Cursor / Windsurf / Codex / Antigravity
 
@@ -104,12 +87,6 @@ Should return the five `spool_*` tools. If you get "connection refused", `spooli
 
 ## CLI Usage
 
-All CLI commands require the venv to be active and the database running.
-
-```bash
-source .venv/bin/activate
-```
-
 ### `spooling init`
 
 Check database connection and show which AI coding tool providers are detected on your system. This scans default paths for Codex CLI, GitHub Copilot, Cursor, and Windsurf session data.
@@ -120,7 +97,7 @@ spooling init
 
 ### `spooling sync`
 
-Parse and ingest sessions from all connected providers into the database. Chunks and embeds message content into pgvector for semantic search.
+Parse and ingest sessions from all connected providers into the local database. Chunks and embeds message content for semantic search.
 
 ```bash
 spooling sync                      # Full sync with embeddings
@@ -163,7 +140,7 @@ spooling watch
 
 ### `spooling serve`
 
-Start the API server only (for when you want to run the GUI separately).
+Start the API server.
 
 ```bash
 spooling serve                    # Default: http://127.0.0.1:3002
@@ -173,10 +150,10 @@ spooling serve --host 0.0.0.0     # Bind to all interfaces
 
 ### `spooling ui`
 
-Start the API server, the MCP HTTP server, and the Next.js UI together.
+Start the API server, the MCP HTTP server, and the Next.js UI together. Requires the `ui/` directory from the [source repo](https://github.com/sashimiboi/spooling) to be present. Set `SPOOLING_UI_DIR` to point at it if running from a non-standard location.
 
 ```bash
-spooling ui
+spooling ui        # API :3002 · MCP :3004 · GUI :3003
 ```
 
 ### `spooling mcp`
@@ -487,7 +464,7 @@ Or use `spooling ui` to start both at once.
 
 ## Chat Agent Setup
 
-The chat page lets you ask questions about your coding sessions in natural language. It uses RAG - retrieves relevant context from pgvector before answering.
+The chat page lets you ask questions about your coding sessions in natural language. It uses RAG — retrieves relevant context from the local database before answering.
 
 ### Option A: Ollama (free, local)
 
@@ -516,24 +493,23 @@ Available models: Sonnet, Haiku, Opus.
 
 ```
 spooling/
-├── docker-compose.yml       # PostgreSQL + pgvector
-├── init.sql                 # Database schema
 ├── pyproject.toml           # Python package config
-├── spooling/                   # Python backend
+├── spooling/                # Python backend
 │   ├── cli.py               # Click CLI
 │   ├── config.py            # Configuration
-│   ├── db.py                # Database connection
-│   ├── providers/           # Provider plugins (codex, copilot, cursor, windsurf)
+│   ├── db.py                # SQLite connection (PostgreSQL shim)
+│   ├── schema.py            # SQLite schema DDL
+│   ├── providers/           # Provider plugins (codex, copilot, cursor, windsurf, ...)
 │   ├── parser.py            # Session JSONL parser
 │   ├── embeddings.py        # sentence-transformers (all-MiniLM-L6-v2)
 │   ├── ingest.py            # Sync pipeline
-│   ├── search.py            # pgvector semantic search
+│   ├── search.py            # Numpy cosine similarity search
 │   ├── stats.py             # Usage statistics
 │   ├── watcher.py           # File watcher (watchdog)
 │   ├── agent.py             # Chat agent (Ollama + Anthropic)
 │   ├── tunnel.py            # Cloudflare quick tunnel support
 │   └── server.py            # FastAPI API server
-└── ui/                      # Next.js frontend
+└── ui/                      # Next.js frontend (source repo only)
     ├── next.config.js       # API proxy to :3002
     └── src/
         ├── components/      # shadcn/ui components
@@ -545,8 +521,9 @@ spooling/
 
 | Layer | Technology |
 |-------|-----------|
-| Database | PostgreSQL 16 + pgvector (Docker) |
+| Database | SQLite (no external dependencies) |
 | Embeddings | sentence-transformers / all-MiniLM-L6-v2 (local) |
+| Search | numpy cosine similarity |
 | Backend | Python, FastAPI, Click |
 | Frontend | Next.js 14, shadcn/ui, Tailwind CSS, AG Charts |
 | Chat AI | Ollama (local) or Anthropic API |
@@ -555,7 +532,6 @@ spooling/
 
 | Service | Port |
 |---------|------|
-| PostgreSQL | 5434 |
 | API Server | 3002 |
 | GUI | 3003 |
 | MCP Server (streamable-HTTP) | 3004 |
@@ -564,19 +540,16 @@ spooling/
 
 ## Environment Variables
 
-All optional - defaults work out of the box for local development.
+All optional — defaults work out of the box.
 
 | Variable | Default | Description |
 |----------|---------|-------------|
-| `SPOOLING_DB_HOST` | `localhost` | Database host |
-| `SPOOLING_DB_PORT` | `5434` | Database port |
-| `SPOOLING_DB_NAME` | `spooling` | Database name |
-| `SPOOLING_DB_USER` | `spooling` | Database user |
-| `SPOOLING_DB_PASSWORD` | `spooling` | Database password |
+| `SPOOLING_DB` | `~/.local/share/spooling/spooling.db` | Path to the SQLite database file (`%APPDATA%\spooling\spooling.db` on Windows) |
 | `SPOOLING_EMBEDDING_MODEL` | `all-MiniLM-L6-v2` | Sentence transformer model |
 | `SPOOLING_UI_HOST` | `127.0.0.1` | API server host |
-| `ANTHROPIC_API_KEY` | - | Anthropic API key (alternative to setting in UI) |
-| `API_URL` | `http://127.0.0.1:3002` | Backend API URL for Next.js rewrites (`/api/*`→`<API_URL>/api/*`). Set to `https://api.spooling.ai` in production. |
+| `SPOOLING_UI_PORT` | `3001` | API server port (used by `spooling serve`) |
+| `SPOOLING_UI_DIR` | *(auto-detected)* | Path to the Next.js `ui/` directory (required for `spooling ui`) |
+| `ANTHROPIC_API_KEY` | - | Anthropic API key (alternative to setting in the GUI) |
 
 ---
 

{spooling-0.2.0 → spooling-0.2.1}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "spooling"
-version = "0.2.0"
+version = "0.2.1"
 description = "Local session tracker and semantic search for AI coding assistants"
 readme = "README.md"
 authors = [{ name = "Parsed Analytics, Inc." }]

{spooling-0.2.0 → spooling-0.2.1}/spooling/agent.py

@@ -156,7 +156,7 @@ def _build_context(query: str) -> dict:
         context_parts.append("## Recent sessions")
         for r in recent:
             title = (r["title"] or "")[:60]
-            started = r["started_at"].isoformat()[:10] if r["started_at"] else ""
+            started = (r["started_at"] if isinstance(r["started_at"], str) else r["started_at"].isoformat())[:10] if r["started_at"] else ""
             context_parts.append(
                 f"- [{r['id']}] {r['provider_id']} {r['project'] or ''} {started}: {title}"
             )
@@ -508,7 +508,7 @@ async def chat_stream(
         context_parts.append("## Recent sessions")
         for r in recent:
             title = (r["title"] or "")[:60]
-            started = r["started_at"].isoformat()[:10] if r["started_at"] else ""
+            started = (r["started_at"] if isinstance(r["started_at"], str) else r["started_at"].isoformat())[:10] if r["started_at"] else ""
             context_parts.append(
                 f"- [{r['id']}] {r['provider_id']} {r['project'] or ''} {started}: {title}"
             )

{spooling-0.2.0 → spooling-0.2.1}/spooling/cloud.py

@@ -23,6 +23,15 @@ _MAX_TITLE = 4096
 _MAX_SPAN_NAME = 512
 
 
+def _to_iso(val) -> str | None:
+    """Return an ISO-format string for *val*, whether it's already a str or a datetime."""
+    if val is None:
+        return None
+    if isinstance(val, str):
+        return val
+    return val.isoformat()
+
+
 def _clean_str(raw: str | None, max_len: int) -> str | None:
     if not raw:
         return raw
@@ -179,8 +188,8 @@ the agent was launched from the home dir so the cwd
                 "project": r["project"],
                 "title": _clean_str(r["title"], _MAX_TITLE),
                 "cwd": r["cwd"],
-                "started_at": r["started_at"].isoformat() if r["started_at"] else None,
-                "ended_at": r["ended_at"].isoformat() if r["ended_at"] else None,
+                "started_at": _to_iso(r["started_at"]),
+                "ended_at": _to_iso(r["ended_at"]),
                 "message_count": r["message_count"] or 0,
                 "tool_call_count": r["tool_call_count"] or 0,
                 "input_tokens": r["estimated_input_tokens"] or 0,
@@ -192,7 +201,7 @@ the agent was launched from the home dir so the cwd
                         "role": m["role"],
                         "content": (m["content"] or "")[:20000],
                         "sequence": int(m["seq"]),
-                        "timestamp": m["timestamp"].isoformat() if m["timestamp"] else None,
+                        "timestamp": _to_iso(m["timestamp"]),
                     }
                     for m in msgs
                 ],

{spooling-0.2.0 → spooling-0.2.1}/spooling/db.py

@@ -14,6 +14,7 @@ codebase to their SQLite equivalents at query time:
 
 import re
 import sqlite3
+from functools import lru_cache
 from pathlib import Path
 
 from spooling.config import DB_PATH
@@ -29,6 +30,7 @@ _CAST_RE = re.compile(r"::\w+")
 _ILIKE_RE = re.compile(r"\bILIKE\b", re.IGNORECASE)
 
 
+@lru_cache(maxsize=512)
 def _adapt_sql(sql: str) -> str:
     """Translate PostgreSQL SQL idioms → SQLite equivalents."""
     sql = _PLACEHOLDER_RE.sub("?", sql)
@@ -148,6 +150,8 @@ def get_connection() -> SpoolingConnection:
     DB_PATH.parent.mkdir(parents=True, exist_ok=True)
     raw = sqlite3.connect(str(DB_PATH), check_same_thread=False)
     raw.execute("PRAGMA journal_mode=WAL")
+    raw.execute("PRAGMA synchronous=NORMAL")   # safe with WAL; much faster than FULL
+    raw.execute("PRAGMA cache_size=-32000")    # 32 MB page cache
     raw.execute("PRAGMA foreign_keys=ON")
     _ensure_schema(raw)
     return SpoolingConnection(raw)

{spooling-0.2.0 → spooling-0.2.1}/spooling/ingest.py

@@ -92,42 +92,52 @@ def _store_session(conn, session: ParsedSession):
         ),
     )
 
-    # Upsert messages
+    # Batch-upsert messages and tool calls
+    msg_rows = []
+    tool_rows_full = []
+    tool_rows_simple = []
     for msg in session.messages:
         if not msg.uuid:
             continue
-        conn.execute(
+        msg_rows.append((
+            msg.uuid, session.session_id, msg.role, _scrub(msg.content),
+            msg.timestamp, json.dumps(msg.tools_used), _scrub(msg.cwd),
+            _scrub(msg.git_branch), msg.estimated_tokens,
+        ))
+        if getattr(msg, "tool_details", None):
+            for td in msg.tool_details:
+                tool_rows_full.append((
+                    session.session_id, msg.uuid, _scrub(td.name),
+                    _scrub(td.input_summary),
+                    _scrub(td.result_preview) or None,
+                    msg.timestamp,
+                ))
+        else:
+            for tool_name in msg.tools_used:
+                tool_rows_simple.append((
+                    session.session_id, msg.uuid, _scrub(tool_name), msg.timestamp,
+                ))
+
+    if msg_rows:
+        conn.executemany(
             """INSERT INTO messages (id, session_id, role, content, timestamp, tools_used,
                cwd, git_branch, estimated_tokens)
                VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s)
                ON CONFLICT (id) DO NOTHING""",
-            (
-                msg.uuid, session.session_id, msg.role, _scrub(msg.content),
-                msg.timestamp, json.dumps(msg.tools_used), _scrub(msg.cwd),
-                _scrub(msg.git_branch), msg.estimated_tokens,
-            ),
+            msg_rows,
+        )
+    if tool_rows_full:
+        conn.executemany(
+            """INSERT INTO tool_calls (session_id, message_id, tool_name, tool_input, tool_result_preview, timestamp)
+               VALUES (%s, %s, %s, %s, %s, %s)""",
+            tool_rows_full,
+        )
+    if tool_rows_simple:
+        conn.executemany(
+            """INSERT INTO tool_calls (session_id, message_id, tool_name, timestamp)
+               VALUES (%s, %s, %s, %s)""",
+            tool_rows_simple,
         )
-
-        # Store tool calls (with rich details when available).
-        if getattr(msg, "tool_details", None):
-            for td in msg.tool_details:
-                conn.execute(
-                    """INSERT INTO tool_calls (session_id, message_id, tool_name, tool_input, tool_result_preview, timestamp)
-                       VALUES (%s, %s, %s, %s, %s, %s)""",
-                    (
-                        session.session_id, msg.uuid, _scrub(td.name),
-                        _scrub(td.input_summary),
-                        _scrub(td.result_preview) or None,
-                        msg.timestamp,
-                    ),
-                )
-        else:
-            for tool_name in msg.tools_used:
-                conn.execute(
-                    """INSERT INTO tool_calls (session_id, message_id, tool_name, timestamp)
-                       VALUES (%s, %s, %s, %s)""",
-                    (session.session_id, msg.uuid, _scrub(tool_name), msg.timestamp),
-                )
 
 
 def _store_trace(conn, trace: Trace):
@@ -195,9 +205,26 @@ def _store_trace(conn, trace: Trace):
         ),
     )
 
-    # Insert spans in sequence order so parent rows always exist first.
-    for span in sorted(trace.spans, key=lambda s: s.sequence):
-        conn.execute(
+    # Batch-insert spans (already in sequence order by construction) and events.
+    span_rows = []
+    event_rows = []
+    for span in trace.spans:
+        span_rows.append((
+            span.id, span.trace_id, span.parent_id, span.kind.value, _scrub(span.name), span.status.value,
+            span.started_at, span.ended_at, span.duration_ms, span.depth, span.sequence,
+            span.input_tokens, span.output_tokens, span.cache_read_tokens, span.cache_write_tokens,
+            span.cost_usd, span.model, _scrub(span.tool_name),
+            json.dumps(span.tool_input, default=str) if span.tool_input is not None else None,
+            _scrub(span.tool_output), span.tool_is_error,
+            _scrub(span.agent_type), _scrub(span.agent_prompt),
+            _scrub(span.vendor), _scrub(span.category),
+            json.dumps(span.attrs or {}, default=str),
+        ))
+        for ev in span.events:
+            event_rows.append((span.id, trace.id, ev.name, ev.timestamp, json.dumps(ev.attrs or {})))
+
+    if span_rows:
+        conn.executemany(
             """INSERT INTO spans (
                 id, trace_id, parent_id, kind, name, status,
                 started_at, ended_at, duration_ms, depth, sequence,
@@ -205,27 +232,15 @@ def _store_trace(conn, trace: Trace):
                 cost_usd, model, tool_name, tool_input, tool_output, tool_is_error,
                 agent_type, agent_prompt, vendor, category, attrs
             ) VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s,
-                      %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s)
-            """,
-            (
-                span.id, span.trace_id, span.parent_id, span.kind.value, _scrub(span.name), span.status.value,
-                span.started_at, span.ended_at, span.duration_ms, span.depth, span.sequence,
-                span.input_tokens, span.output_tokens, span.cache_read_tokens, span.cache_write_tokens,
-                span.cost_usd, span.model, _scrub(span.tool_name),
-                json.dumps(span.tool_input, default=str) if span.tool_input is not None else None,
-                _scrub(span.tool_output), span.tool_is_error,
-                _scrub(span.agent_type), _scrub(span.agent_prompt),
-                _scrub(span.vendor), _scrub(span.category),
-                json.dumps(span.attrs or {}, default=str),
-            ),
+                      %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s)""",
+            span_rows,
+        )
+    if event_rows:
+        conn.executemany(
+            """INSERT INTO span_events (span_id, trace_id, name, timestamp, attrs)
+               VALUES (%s, %s, %s, %s, %s)""",
+            event_rows,
         )
-
-        for ev in span.events:
-            conn.execute(
-                """INSERT INTO span_events (span_id, trace_id, name, timestamp, attrs)
-                   VALUES (%s, %s, %s, %s, %s)""",
-                (span.id, trace.id, ev.name, ev.timestamp, json.dumps(ev.attrs or {})),
-            )
 
     # After spans land, sync the session's headline cost to the sum of its
     # llm_call span costs. Span cost is computed from real per-turn usage
@@ -272,19 +287,14 @@ def _serialize_embedding(vec: list[float]) -> bytes:
     return struct.pack(f"{len(vec)}f", *vec)
 
 
-def _embed_session(conn, session: ParsedSession):
-    """Chunk and embed session messages, storing vectors as BLOBs."""
-    # Delete existing chunks for this session (re-embed on update)
-    conn.execute("DELETE FROM chunks WHERE session_id = %s", (session.session_id,))
-
-    all_chunks = []
-    chunk_meta = []
-
+def _collect_session_chunks(session: ParsedSession) -> tuple[list[str], list[dict]]:
+    """Collect all text chunks and their metadata for a session (no embedding)."""
+    all_chunks: list[str] = []
+    chunk_meta: list[dict] = []
     for msg in session.messages:
         if not msg.content.strip():
             continue
-        chunks = chunk_text(msg.content)
-        for chunk in chunks:
+        for chunk in chunk_text(msg.content):
             all_chunks.append(chunk)
             chunk_meta.append({
                 "session_id": session.session_id,
@@ -293,25 +303,36 @@ def _embed_session(conn, session: ParsedSession):
                 "project": session.project,
                 "timestamp": msg.timestamp,
             })
+    return all_chunks, chunk_meta
 
-    if not all_chunks:
-        return 0
-
-    # Batch embed
-    vectors = embed_texts(all_chunks)
 
-    for chunk, vec, meta in zip(all_chunks, vectors, chunk_meta):
-        conn.execute(
+def _write_chunk_rows(conn, all_chunks: list[str], vectors, chunk_meta: list[dict]) -> None:
+    """Write pre-embedded chunks to the DB using executemany."""
+    rows = [
+        (
+            meta["session_id"], meta["message_id"], chunk,
+            meta["role"], meta["project"],
+            meta["timestamp"].isoformat() if hasattr(meta["timestamp"], "isoformat") else meta["timestamp"],
+            _serialize_embedding(vec),
+        )
+        for chunk, vec, meta in zip(all_chunks, vectors, chunk_meta)
+    ]
+    if rows:
+        conn.executemany(
             """INSERT INTO chunks (session_id, message_id, content, role, project, timestamp, embedding)
                VALUES (%s, %s, %s, %s, %s, %s, %s)""",
-            (
-                meta["session_id"], meta["message_id"], chunk,
-                meta["role"], meta["project"],
-                meta["timestamp"].isoformat() if hasattr(meta["timestamp"], "isoformat") else meta["timestamp"],
-                _serialize_embedding(vec),
-            ),
+            rows,
         )
 
+
+def _embed_session(conn, session: ParsedSession):
+    """Chunk and embed session messages, storing vectors as BLOBs."""
+    conn.execute("DELETE FROM chunks WHERE session_id = %s", (session.session_id,))
+    all_chunks, chunk_meta = _collect_session_chunks(session)
+    if not all_chunks:
+        return 0
+    vectors = embed_texts(all_chunks)
+    _write_chunk_rows(conn, all_chunks, vectors, chunk_meta)
     return len(all_chunks)
 
 
@@ -454,6 +475,10 @@ def sync(embed: bool = True, provider_filter: str | None = None):
         total_chunks = 0
         total_sessions = 0
 
+        # Process files in batches: parse+store all, then embed all at once.
+        # One model.encode() call per batch instead of one per session.
+        _EMBED_BATCH = 20
+
         with Progress(
             SpinnerColumn(),
             TextColumn("[progress.description]{task.description}"),
@@ -463,22 +488,40 @@ def sync(embed: bool = True, provider_filter: str | None = None):
         ) as progress:
             task = progress.add_task(f"Syncing {provider.name}...", total=len(to_process))
 
-            for f in to_process:
-                sessions = provider.parse_session_file(f)
-                for session in sessions:
-                    _store_session(conn, session)
-                    if session.trace is not None:
-                        _store_trace(conn, session.trace)
-                    total_messages += session.message_count
-                    total_sessions += 1
-
-                    if embed:
-                        chunks = _embed_session(conn, session)
-                        total_chunks += chunks
-
-                _mark_synced(conn, str(f), f.stat().st_size, prov_info["type"])
+            for batch_start in range(0, len(to_process), _EMBED_BATCH):
+                batch_files = to_process[batch_start : batch_start + _EMBED_BATCH]
+                batch_sessions: list[tuple] = []  # (file, [sessions])
+
+                for f in batch_files:
+                    sessions = provider.parse_session_file(f)
+                    for session in sessions:
+                        _store_session(conn, session)
+                        if session.trace is not None:
+                            _store_trace(conn, session.trace)
+                        total_messages += session.message_count
+                        total_sessions += 1
+                    batch_sessions.append((f, sessions))
+                    progress.advance(task)
+
+                if embed:
+                    # Collect chunks from every session in the batch, then embed once.
+                    all_chunks: list[str] = []
+                    all_metas: list[dict] = []
+                    for _f, sessions in batch_sessions:
+                        for session in sessions:
+                            conn.execute("DELETE FROM chunks WHERE session_id = %s", (session.session_id,))
+                            c, m = _collect_session_chunks(session)
+                            all_chunks.extend(c)
+                            all_metas.extend(m)
+
+                    if all_chunks:
+                        vectors = embed_texts(all_chunks)
+                        _write_chunk_rows(conn, all_chunks, vectors, all_metas)
+                        total_chunks += len(all_chunks)
+
+                for f, _ in batch_sessions:
+                    _mark_synced(conn, str(f), f.stat().st_size, prov_info["type"])
                 conn.commit()
-                progress.advance(task)
 
         # Update provider stats
         conn.execute(

{spooling-0.2.0 → spooling-0.2.1}/spooling/parser.py

@@ -95,8 +95,13 @@ def _summarize_tool_input(name: str, inp: dict) -> str:
         path = inp.get("file_path", "")
         offset = inp.get("offset")
         limit = inp.get("limit")
-        if offset and limit:
-            return f"{path}:{offset}-{offset + limit}"
+        if offset is not None and limit is not None:
+            try:
+                offset = int(offset[0]) if isinstance(offset, list) else int(offset)
+                limit = int(limit[0]) if isinstance(limit, list) else int(limit)
+                return f"{path}:{offset}-{offset + limit}"
+            except (TypeError, ValueError, IndexError):
+                pass
         return path
     if name in ("Edit", "Write"):
         return inp.get("file_path", "")

{spooling-0.2.0 → spooling-0.2.1/spooling.egg-info}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: spooling
-Version: 0.2.0
+Version: 0.2.1
 Summary: Local session tracker and semantic search for AI coding assistants
 Author: Parsed Analytics, Inc.
 License: MIT
@@ -38,7 +38,7 @@ Dynamic: license-file
 
 Local session tracker and semantic search for AI coding assistants.
 
-Track your AI coding sessions across **OpenAI Codex CLI**, **GitHub Copilot**, **Cursor**, **Windsurf**, **Kiro**, **Google Antigravity**, and **opencode**, all in one place. Get usage stats, cost estimates, per-provider breakdowns, semantic search via pgvector, and a built-in AI chat agent to explore your history.
+Track your AI coding sessions across **OpenAI Codex CLI**, **GitHub Copilot**, **Cursor**, **Windsurf**, **Kiro**, **Google Antigravity**, and **opencode**, all in one place. Get usage stats, cost estimates, per-provider breakdowns, semantic search, and a built-in AI chat agent to explore your history.
 
 **Website:** [spooling.ai](https://spooling.ai)
 
@@ -47,67 +47,50 @@ Track your AI coding sessions across **OpenAI Codex CLI**, **GitHub Copilot**, *
 ## Prerequisites
 
 - **Python 3.11+**
-- **Node.js 18+**
-- **Docker** (for PostgreSQL + pgvector)
 - **pipx** or **uv** (optional, alternative install methods)
 - **Ollama** (optional, for free local AI chat) or an **Anthropic API key**
 
+No Docker, no PostgreSQL, no Node.js required.
+
 ---
 
 ## How it works
 
 **Four commands. Zero cloud.**
 
-### 01   Clone & start the database
+### 01   Install
 
 ```bash
-git clone https://github.com/sashimiboi/spooling && cd spooling
-docker compose up -d   # postgres + pgvector :5434
-```
-
-### 02   Install backend + UI
+# pip
+pip install spooling
 
-Choose one:
-
-```bash
-# pip (recommended)
-python3 -m venv .venv && source .venv/bin/activate
-pip install -e .
-
-# pipx
-pipx install .
+# pipx (recommended — keeps it isolated)
+pipx install spooling
 
 # uv
-uv sync
-```
-
-Then install the UI:
-
-```bash
-cd ui && npm install && cd ..
+uv tool install spooling
 ```
 
-### 03   Detect providers & sync
+### 02   Detect providers & sync
 
 ```bash
 spooling init      # scan for available providers
-spooling sync      # embed every session into pgvector
+spooling sync      # embed every session for semantic search
 ```
 
-### 04   Search & explore
+### 03   Search & explore
 
 ```bash
-spooling ui        # API :3002 · MCP :3004 · GUI :3003
 spooling search "that redis race condition"
+spooling stats
+spooling mcp       # MCP server at http://127.0.0.1:3004/mcp
 ```
 
-Open **http://localhost:3003** and you're in.
-
 ---
 
 ## Connect Spooling to your AI coding agent
 
-`spooling ui` automatically exposes an MCP server at `http://127.0.0.1:3004/mcp` (HTTP streamable transport). Any MCP-speaking agent can connect to it and pull context from your local KB mid-conversation. The agent gets tools like `spooling_search`, `spooling_recent_sessions`, `spooling_get_session`, `spooling_workspace_stats`, and `spooling_top_projects`.
+`spooling mcp` exposes an MCP server at `http://127.0.0.1:3004/mcp` (HTTP streamable transport). Any MCP-speaking agent can connect to it and pull context from your local KB mid-conversation. The agent gets tools like `spooling_search`, `spooling_recent_sessions`, `spooling_get_session`, `spooling_workspace_stats`, and `spooling_top_projects`.
 
 ### Cursor / Windsurf / Codex / Antigravity
 
@@ -140,12 +123,6 @@ Should return the five `spool_*` tools. If you get "connection refused", `spooli
 
 ## CLI Usage
 
-All CLI commands require the venv to be active and the database running.
-
-```bash
-source .venv/bin/activate
-```
-
 ### `spooling init`
 
 Check database connection and show which AI coding tool providers are detected on your system. This scans default paths for Codex CLI, GitHub Copilot, Cursor, and Windsurf session data.
@@ -156,7 +133,7 @@ spooling init
 
 ### `spooling sync`
 
-Parse and ingest sessions from all connected providers into the database. Chunks and embeds message content into pgvector for semantic search.
+Parse and ingest sessions from all connected providers into the local database. Chunks and embeds message content for semantic search.
 
 ```bash
 spooling sync                      # Full sync with embeddings
@@ -199,7 +176,7 @@ spooling watch
 
 ### `spooling serve`
 
-Start the API server only (for when you want to run the GUI separately).
+Start the API server.
 
 ```bash
 spooling serve                    # Default: http://127.0.0.1:3002
@@ -209,10 +186,10 @@ spooling serve --host 0.0.0.0     # Bind to all interfaces
 
 ### `spooling ui`
 
-Start the API server, the MCP HTTP server, and the Next.js UI together.
+Start the API server, the MCP HTTP server, and the Next.js UI together. Requires the `ui/` directory from the [source repo](https://github.com/sashimiboi/spooling) to be present. Set `SPOOLING_UI_DIR` to point at it if running from a non-standard location.
 
 ```bash
-spooling ui
+spooling ui        # API :3002 · MCP :3004 · GUI :3003
 ```
 
 ### `spooling mcp`
@@ -523,7 +500,7 @@ Or use `spooling ui` to start both at once.
 
 ## Chat Agent Setup
 
-The chat page lets you ask questions about your coding sessions in natural language. It uses RAG - retrieves relevant context from pgvector before answering.
+The chat page lets you ask questions about your coding sessions in natural language. It uses RAG — retrieves relevant context from the local database before answering.
 
 ### Option A: Ollama (free, local)
 
@@ -552,24 +529,23 @@ Available models: Sonnet, Haiku, Opus.
 
 ```
 spooling/
-├── docker-compose.yml       # PostgreSQL + pgvector
-├── init.sql                 # Database schema
 ├── pyproject.toml           # Python package config
-├── spooling/                   # Python backend
+├── spooling/                # Python backend
 │   ├── cli.py               # Click CLI
 │   ├── config.py            # Configuration
-│   ├── db.py                # Database connection
-│   ├── providers/           # Provider plugins (codex, copilot, cursor, windsurf)
+│   ├── db.py                # SQLite connection (PostgreSQL shim)
+│   ├── schema.py            # SQLite schema DDL
+│   ├── providers/           # Provider plugins (codex, copilot, cursor, windsurf, ...)
 │   ├── parser.py            # Session JSONL parser
 │   ├── embeddings.py        # sentence-transformers (all-MiniLM-L6-v2)
 │   ├── ingest.py            # Sync pipeline
-│   ├── search.py            # pgvector semantic search
+│   ├── search.py            # Numpy cosine similarity search
 │   ├── stats.py             # Usage statistics
 │   ├── watcher.py           # File watcher (watchdog)
 │   ├── agent.py             # Chat agent (Ollama + Anthropic)
 │   ├── tunnel.py            # Cloudflare quick tunnel support
 │   └── server.py            # FastAPI API server
-└── ui/                      # Next.js frontend
+└── ui/                      # Next.js frontend (source repo only)
     ├── next.config.js       # API proxy to :3002
     └── src/
         ├── components/      # shadcn/ui components
@@ -581,8 +557,9 @@ spooling/
 
 | Layer | Technology |
 |-------|-----------|
-| Database | PostgreSQL 16 + pgvector (Docker) |
+| Database | SQLite (no external dependencies) |
 | Embeddings | sentence-transformers / all-MiniLM-L6-v2 (local) |
+| Search | numpy cosine similarity |
 | Backend | Python, FastAPI, Click |
 | Frontend | Next.js 14, shadcn/ui, Tailwind CSS, AG Charts |
 | Chat AI | Ollama (local) or Anthropic API |
@@ -591,7 +568,6 @@ spooling/
 
 | Service | Port |
 |---------|------|
-| PostgreSQL | 5434 |
 | API Server | 3002 |
 | GUI | 3003 |
 | MCP Server (streamable-HTTP) | 3004 |
@@ -600,19 +576,16 @@ spooling/
 
 ## Environment Variables
 
-All optional - defaults work out of the box for local development.
+All optional — defaults work out of the box.
 
 | Variable | Default | Description |
 |----------|---------|-------------|
-| `SPOOLING_DB_HOST` | `localhost` | Database host |
-| `SPOOLING_DB_PORT` | `5434` | Database port |
-| `SPOOLING_DB_NAME` | `spooling` | Database name |
-| `SPOOLING_DB_USER` | `spooling` | Database user |
-| `SPOOLING_DB_PASSWORD` | `spooling` | Database password |
+| `SPOOLING_DB` | `~/.local/share/spooling/spooling.db` | Path to the SQLite database file (`%APPDATA%\spooling\spooling.db` on Windows) |
 | `SPOOLING_EMBEDDING_MODEL` | `all-MiniLM-L6-v2` | Sentence transformer model |
 | `SPOOLING_UI_HOST` | `127.0.0.1` | API server host |
-| `ANTHROPIC_API_KEY` | - | Anthropic API key (alternative to setting in UI) |
-| `API_URL` | `http://127.0.0.1:3002` | Backend API URL for Next.js rewrites (`/api/*`→`<API_URL>/api/*`). Set to `https://api.spooling.ai` in production. |
+| `SPOOLING_UI_PORT` | `3001` | API server port (used by `spooling serve`) |
+| `SPOOLING_UI_DIR` | *(auto-detected)* | Path to the Next.js `ui/` directory (required for `spooling ui`) |
+| `ANTHROPIC_API_KEY` | - | Anthropic API key (alternative to setting in the GUI) |
 
 ---