si-deckhand 0.1.0__tar.gz

si_deckhand-0.1.0/PKG-INFO

@@ -0,0 +1,106 @@
+Metadata-Version: 2.4
+Name: si-deckhand
+Version: 0.1.0
+Summary: Lightweight local agent that indexes, retrieves, and wikis the workspace. The crew member who knows where everything is.
+Author: SuperInstance
+License-Expression: MIT
+Keywords: agent,retriever,wiki,index,exo-filed,local
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Programming Language :: Python :: 3
+Classifier: Topic :: System :: Systems Administration
+Classifier: Topic :: Text Processing :: Indexing
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+Provides-Extra: vector
+Requires-Dist: numpy>=1.20; extra == "vector"
+Provides-Extra: dev
+Requires-Dist: pytest>=7.0; extra == "dev"
+Requires-Dist: pytest-cov; extra == "dev"
+
+# deckhand
+
+*The crew member who knows where everything is.*
+
+---
+
+**deckhand** is a lightweight local agent that indexes, retrieves, and wikis your workspace. File-first (exo-filed). Human-readable. No hidden database.
+
+## What it does
+
+- **Indexes** every file in your workspace (code, docs, configs)
+- **Builds wiki pages** that a human can `cat` — REPOS.md, CONNECTIONS.md, GLOSSARY.md, STATUS.md
+- **Searches** via BM25 (pure Python, no dependencies)
+- **Batches tasks** from PLANNER_QUEUE.md, OPEN_QUESTIONS.md, and recent file changes
+- **Studies** repos off-watch — writes daily observation logs
+- **Runs forever** in a loop: index → wiki → batch → study → sleep → repeat
+
+## Install
+
+```bash
+pip install deckhand
+```
+
+## Use
+
+```bash
+# Index the workspace and generate wiki
+deckhand index /path/to/workspace
+
+# Search for something
+deckhand search /path/to/workspace "conservation law"
+
+# Generate just the wiki
+deckhand wiki /path/to/workspace
+
+# Collect tasks into a batch
+deckhand batch /path/to/workspace
+
+# Run the deckhand loop forever (index → wiki → batch → study → sleep)
+deckhand run /path/to/workspace --sleep 300
+```
+
+## The wiki output
+
+After indexing, deckhand writes to `deckhand-wiki/`:
+
+- **REPOS.md** — table of every repo, one line each. Health indicators (✅ ok, ⚠️ untested, 🔩 stub, ❌ no-readme)
+- **CONNECTIONS.md** — cross-reference graph. Which repos reference which.
+- **GLOSSARY.md** — top 100 terms across the workspace
+- **STATUS.md** — health summary. Lists repos that need tests, need READMEs, or may be abandoned
+- **INDEX.md** — machine-readable JSON index
+- **BATCH.md** — collected tasks from PLANNER_QUEUE + OPEN_QUESTIONS + recent changes
+- **logs/YYYY-MM-DD.md** — daily study observations
+
+## Why exo-filed?
+
+A vector database is opaque. You can't `cat` it. You can't grep it. You can't diff it in git.
+
+deckhand's index is markdown + JSON. A human can open REPOS.md and understand the fleet in 30 seconds. An agent can read CONNECTIONS.md instead of re-reading 4,000 repos.
+
+When the corpus gets too large for file-based search (>10K files), build the optional vector twin:
+
+```python
+from deckhand.vector_twin import VectorTwin  # coming in v0.2
+```
+
+The vector twin is ALWAYS accompanied by its file-twin. You can always fall back to the human-readable version.
+
+## The maritime metaphor
+
+On a boat, the deckhand is the crew member who:
+- Knows where every line, every shackle, every tool is stored
+- Prepares tasks for the captain (batches)
+- Maintains the logbook (study logs)
+- Studies charts off-watch (studies repos when idle)
+- Doesn't need to be told what to do — knows the direction of the voyage
+
+The deckhand doesn't replace the captain (the director model). The deckhand makes the captain more efficient by pre-processing the workspace into a form the captain can act on in fewer calls.
+
+## License
+
+MIT
+
+---
+
+*Built by SuperInstance. The deckhand knows where everything is.*

si_deckhand-0.1.0/README.md

@@ -0,0 +1,86 @@
+# deckhand
+
+*The crew member who knows where everything is.*
+
+---
+
+**deckhand** is a lightweight local agent that indexes, retrieves, and wikis your workspace. File-first (exo-filed). Human-readable. No hidden database.
+
+## What it does
+
+- **Indexes** every file in your workspace (code, docs, configs)
+- **Builds wiki pages** that a human can `cat` — REPOS.md, CONNECTIONS.md, GLOSSARY.md, STATUS.md
+- **Searches** via BM25 (pure Python, no dependencies)
+- **Batches tasks** from PLANNER_QUEUE.md, OPEN_QUESTIONS.md, and recent file changes
+- **Studies** repos off-watch — writes daily observation logs
+- **Runs forever** in a loop: index → wiki → batch → study → sleep → repeat
+
+## Install
+
+```bash
+pip install deckhand
+```
+
+## Use
+
+```bash
+# Index the workspace and generate wiki
+deckhand index /path/to/workspace
+
+# Search for something
+deckhand search /path/to/workspace "conservation law"
+
+# Generate just the wiki
+deckhand wiki /path/to/workspace
+
+# Collect tasks into a batch
+deckhand batch /path/to/workspace
+
+# Run the deckhand loop forever (index → wiki → batch → study → sleep)
+deckhand run /path/to/workspace --sleep 300
+```
+
+## The wiki output
+
+After indexing, deckhand writes to `deckhand-wiki/`:
+
+- **REPOS.md** — table of every repo, one line each. Health indicators (✅ ok, ⚠️ untested, 🔩 stub, ❌ no-readme)
+- **CONNECTIONS.md** — cross-reference graph. Which repos reference which.
+- **GLOSSARY.md** — top 100 terms across the workspace
+- **STATUS.md** — health summary. Lists repos that need tests, need READMEs, or may be abandoned
+- **INDEX.md** — machine-readable JSON index
+- **BATCH.md** — collected tasks from PLANNER_QUEUE + OPEN_QUESTIONS + recent changes
+- **logs/YYYY-MM-DD.md** — daily study observations
+
+## Why exo-filed?
+
+A vector database is opaque. You can't `cat` it. You can't grep it. You can't diff it in git.
+
+deckhand's index is markdown + JSON. A human can open REPOS.md and understand the fleet in 30 seconds. An agent can read CONNECTIONS.md instead of re-reading 4,000 repos.
+
+When the corpus gets too large for file-based search (>10K files), build the optional vector twin:
+
+```python
+from deckhand.vector_twin import VectorTwin  # coming in v0.2
+```
+
+The vector twin is ALWAYS accompanied by its file-twin. You can always fall back to the human-readable version.
+
+## The maritime metaphor
+
+On a boat, the deckhand is the crew member who:
+- Knows where every line, every shackle, every tool is stored
+- Prepares tasks for the captain (batches)
+- Maintains the logbook (study logs)
+- Studies charts off-watch (studies repos when idle)
+- Doesn't need to be told what to do — knows the direction of the voyage
+
+The deckhand doesn't replace the captain (the director model). The deckhand makes the captain more efficient by pre-processing the workspace into a form the captain can act on in fewer calls.
+
+## License
+
+MIT
+
+---
+
+*Built by SuperInstance. The deckhand knows where everything is.*

si_deckhand-0.1.0/pyproject.toml

@@ -0,0 +1,35 @@
+[build-system]
+requires = ["setuptools>=68.0", "wheel"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "si-deckhand"
+version = "0.1.0"
+description = "Lightweight local agent that indexes, retrieves, and wikis the workspace. The crew member who knows where everything is."
+readme = "README.md"
+license = "MIT"
+requires-python = ">=3.10"
+authors = [{ name = "SuperInstance" }]
+keywords = ["agent", "retriever", "wiki", "index", "exo-filed", "local"]
+classifiers = [
+    "Development Status :: 3 - Alpha",
+    "Intended Audience :: Developers",
+    "Programming Language :: Python :: 3",
+    "Topic :: System :: Systems Administration",
+    "Topic :: Text Processing :: Indexing",
+]
+
+dependencies = []
+
+[project.optional-dependencies]
+vector = ["numpy>=1.20"]
+dev = ["pytest>=7.0", "pytest-cov"]
+
+[project.scripts]
+deckhand = "deckhand.cli:main"
+
+[tool.setuptools.packages.find]
+where = ["src"]
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]

si_deckhand-0.1.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

si_deckhand-0.1.0/src/deckhand/__init__.py

@@ -0,0 +1,23 @@
+"""
+deckhand — Lightweight local agent that indexes, retrieves, and wikis the workspace.
+
+The deckhand is the crew member who knows where everything is.
+File-first (exo-filed). Human-readable. No hidden vector DB.
+"""
+
+from __future__ import annotations
+
+__version__ = "0.1.0"
+__all__ = [
+    "Indexer",
+    "Retriever",
+    "WikiGenerator",
+    "TaskBatcher",
+    "Deckhand",
+]
+
+from deckhand.agent import Deckhand
+from deckhand.indexer import Indexer
+from deckhand.retriever import Retriever
+from deckhand.wiki import WikiGenerator
+from deckhand.task_batcher import TaskBatcher

si_deckhand-0.1.0/src/deckhand/agent.py

@@ -0,0 +1,143 @@
+"""Agent — the main deckhand loop.
+
+Index → Study → Batch → Sleep → Repeat.
+When no tasks are queued, study the workspace and update the wiki.
+"""
+
+from __future__ import annotations
+
+import time
+from pathlib import Path
+from typing import Any
+
+from deckhand.indexer import Indexer
+from deckhand.retriever import Retriever
+from deckhand.wiki import WikiGenerator
+from deckhand.task_batcher import TaskBatcher
+
+
+class Deckhand:
+    """The main agent loop."""
+
+    def __init__(
+        self,
+        workspace_root: str,
+        wiki_dir: str | None = None,
+        sleep_seconds: float = 300.0,
+    ) -> None:
+        self.root = workspace_root
+        self.wiki_dir = wiki_dir or str(Path(workspace_root) / "deckhand-wiki")
+        self.sleep_seconds = sleep_seconds
+        self.indexer = Indexer(workspace_root)
+        self.retriever: Retriever | None = None
+        self.wiki_gen: WikiGenerator | None = None
+        self.batcher = TaskBatcher(workspace_root)
+        self._running = False
+
+    def index(self) -> dict[str, Any]:
+        """Run a full index pass."""
+        stats = self.indexer.index()
+        self.retriever = Retriever(self.indexer.files)
+        self.wiki_gen = WikiGenerator(self.indexer)
+        return stats
+
+    def search(self, query: str, top_k: int = 10) -> list[dict[str, Any]]:
+        """Search the workspace."""
+        if not self.retriever:
+            self.index()
+        assert self.retriever is not None
+        results = self.retriever.search(query, top_k=top_k)
+        return [r.to_dict() for r in results]
+
+    def generate_wiki(self) -> list[str]:
+        """Generate all wiki pages."""
+        if not self.wiki_gen:
+            self.index()
+        assert self.wiki_gen is not None
+        return self.wiki_gen.generate(self.wiki_dir)
+
+    def batch_tasks(self) -> str:
+        """Collect and batch tasks."""
+        batch_path = str(Path(self.wiki_dir) / "BATCH.md")
+        return self.batcher.write_batch(batch_path)
+
+    def study(self) -> str:
+        """Study a random repo and write observations."""
+        if not self.indexer.repos:
+            self.index()
+
+        # Pick a repo that hasn't been studied recently
+        repos_by_age = sorted(
+            self.indexer.repos.values(),
+            key=lambda r: r.last_modified,
+        )
+        if not repos_by_age:
+            return "nothing to study"
+
+        target = repos_by_age[0]  # oldest modified
+
+        log_dir = Path(self.wiki_dir) / "logs"
+        log_dir.mkdir(parents=True, exist_ok=True)
+        log_path = log_dir / f"{time.strftime('%Y-%m-%d')}.md"
+
+        lines = [
+            f"# Deckhand Study Log — {time.strftime('%Y-%m-%d')}",
+            f"",
+            f"## Studied: `{target.name}`",
+            f"",
+            f"- Files: {target.file_count}",
+            f"- Health: {target.health}",
+            f"- Has tests: {target.has_tests}",
+            f"- Has README: {target.has_readme}",
+            f"- Languages: {target.languages}",
+            f"",
+        ]
+
+        if target.health == "untested":
+            lines.append("⚠️ This repo has code but no tests. Needs a test suite.")
+        elif target.health == "no-readme":
+            lines.append("❌ This repo has no README. Needs documentation.")
+        elif target.health == "stub":
+            lines.append("🔩 This repo is a stub (< 3 files). May be abandoned.")
+        else:
+            lines.append("✅ This repo looks healthy.")
+
+        lines.append("")
+
+        with open(log_path, "a") as fh:
+            fh.write("\n".join(lines))
+
+        return str(log_path)
+
+    def run_once(self) -> dict[str, Any]:
+        """Run one full cycle: index → wiki → batch → study."""
+        stats = self.index()
+        wiki_files = self.generate_wiki()
+        batch_path = self.batch_tasks()
+        study_log = self.study()
+
+        return {
+            "index": stats,
+            "wiki_files": wiki_files,
+            "batch": batch_path,
+            "study_log": study_log,
+        }
+
+    def run_forever(self) -> None:
+        """Run the main loop forever."""
+        self._running = True
+        while self._running:
+            try:
+                result = self.run_once()
+                print(
+                    f"[deckhand] indexed {result['index']['total_files']} files, "
+                    f"wrote {len(result['wiki_files'])} wiki pages, "
+                    f"batch at {result['batch']}"
+                )
+            except Exception as e:
+                print(f"[deckhand] error: {e}")
+
+            time.sleep(self.sleep_seconds)
+
+    def stop(self) -> None:
+        self._running = False

si_deckhand-0.1.0/src/deckhand/cli.py

@@ -0,0 +1,98 @@
+"""CLI for deckhand."""
+
+from __future__ import annotations
+
+import argparse
+import sys
+
+from deckhand.agent import Deckhand
+
+
+def main(argv: list[str] | None = None) -> int:
+    parser = argparse.ArgumentParser(
+        prog="deckhand",
+        description="🐑 The deckhand — indexes, retrieves, and wikis the workspace.",
+    )
+
+    sub = parser.add_subparsers(dest="command")
+
+    # index
+    p_index = sub.add_parser("index", help="Index the workspace")
+    p_index.add_argument("workspace", help="Workspace root directory")
+    p_index.add_argument("--wiki", default=None, help="Wiki output directory")
+
+    # search
+    p_search = sub.add_parser("search", help="Search the workspace")
+    p_search.add_argument("workspace", help="Workspace root directory")
+    p_search.add_argument("query", help="Search query")
+    p_search.add_argument("--top", type=int, default=10, help="Number of results")
+
+    # wiki
+    p_wiki = sub.add_parser("wiki", help="Generate wiki pages")
+    p_wiki.add_argument("workspace", help="Workspace root directory")
+    p_wiki.add_argument("--output", default=None, help="Wiki output directory")
+
+    # batch
+    p_batch = sub.add_parser("batch", help="Collect and batch tasks")
+    p_batch.add_argument("workspace", help="Workspace root directory")
+
+    # study
+    p_study = sub.add_parser("study", help="Study a repo and write observations")
+    p_study.add_argument("workspace", help="Workspace root directory")
+
+    # run
+    p_run = sub.add_parser("run", help="Run the deckhand loop forever")
+    p_run.add_argument("workspace", help="Workspace root directory")
+    p_run.add_argument("--sleep", type=float, default=300, help="Sleep seconds between cycles")
+
+    args = parser.parse_args(argv)
+
+    if not args.command:
+        parser.print_help()
+        return 0
+
+    dh = Deckhand(
+        workspace_root=args.workspace,
+        wiki_dir=getattr(args, "wiki", None) or getattr(args, "output", None),
+        sleep_seconds=getattr(args, "sleep", 300),
+    )
+
+    if args.command == "index":
+        stats = dh.index()
+        print(f"Indexed {stats['total_files']} files across {stats['total_repos']} repos in {stats['elapsed_seconds']}s")
+        dh.generate_wiki()
+        print(f"Wiki written to {dh.wiki_dir}")
+
+    elif args.command == "search":
+        results = dh.search(args.query, top_k=args.top)
+        if not results:
+            print("No results found.")
+        for i, r in enumerate(results, 1):
+            print(f"{i}. [{r['score']:.2f}] {r['rel_path']}")
+            if r["snippet"]:
+                print(f"   {r['snippet'][:150]}...")
+
+    elif args.command == "wiki":
+        dh.index()
+        files = dh.generate_wiki()
+        print(f"Wiki pages written:")
+        for f in files:
+            print(f"  {f}")
+
+    elif args.command == "batch":
+        path = dh.batch_tasks()
+        print(f"Batch written to {path}")
+
+    elif args.command == "study":
+        log = dh.study()
+        print(f"Study log written to {log}")
+
+    elif args.command == "run":
+        print(f"Deckhand running on {args.workspace} (sleep={args.sleep}s)")
+        dh.run_forever()
+
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())