drydock-cli 1.5.3__tar.gz → 1.5.5__tar.gz

{drydock_cli-1.5.3 → drydock_cli-1.5.5}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: drydock-cli
-Version: 1.5.3
+Version: 1.5.5
 Summary: Drydock — nautical CLI coding agent
 Project-URL: Homepage, https://github.com/fbobe321/drydock
 Project-URL: Repository, https://github.com/fbobe321/drydock

{drydock_cli-1.5.3 → drydock_cli-1.5.5}/PRD.md

@@ -1,9 +1,12 @@
 # DryDock — Local CLI Coding Agent
 
 **Repository:** https://github.com/fbobe321/drydock
-**PyPI:** https://pypi.org/project/drydock-cli/ (v1.5.0)
+**PyPI:** https://pypi.org/project/drydock-cli/ (v1.5.3)
 **License:** Apache 2.0 (fork of [mistralai/mistral-vibe](https://github.com/mistralai/mistral-vibe))
-**Status:** Active development — continuous improvement running
+**Status:** Active development
+**Current version:** 1.5.3
+**Hardware:** 2x RTX 4060 Ti 16GB, devstral-24B-AWQ-4bit via vLLM at localhost:8000
+**Server:** remus (Ubuntu 22.04, user: bobef)
 
 ---
 
@@ -30,31 +33,143 @@ Scripts:
 
 ---
 
+## Configuration Management
+
+**CRITICAL: Every script and cron MUST follow these rules.**
+
+### Python Path
+All scripts use explicit Python path, never bare `python3`:
+```bash
+export PATH="/home/bobef/miniconda3/bin:$PATH"
+PYTHON="/home/bobef/miniconda3/bin/python3"
+```
+**Why:** Cron environment doesn't inherit shell PATH. Bare `python3` resolves to `/usr/bin/python3` (system Python, no packages). This caused an 8-hour crash loop where the test battery restarted every 10 minutes and immediately failed.
+
+### Version Tracking
+- `pyproject.toml` is the single source of truth for version
+- `publish_to_pypi.sh` creates git tags (`v1.5.3`, etc.)
+- `publish_to_pypi.sh` runs post-publish integration test (install in venv, verify version)
+- Test scripts log version before running
+
+### Cron Jobs (DryDock)
+| Schedule | Script | Purpose |
+|---|---|---|
+| 2 AM daily | `test_full.sh` | Nightly regression (smoke + real backend) |
+| 3 AM daily | `backup.sh` | rsync to NAS |
+| 4 AM daily | `deploy_to_github.sh` | Sync code to GitHub |
+| Every 10 min | `monitor_test_battery.sh` | Monitor/restart long test runs |
+
+### Pre-Publish Checklist
+1. All smoke tests pass (169 tests)
+2. Version bumped in pyproject.toml
+3. Build succeeds
+4. Upload to PyPI
+5. Git tag created
+6. Integration test: install from wheel in temp venv, verify version matches
+7. Deploy to GitHub
+
+---
+
+## File Locations (CRITICAL for session continuity)
+
+```
+/data3/drydock/                    ← Main repo (git)
+├── drydock/                       ← Python package (source code)
+│   ├── core/
+│   │   ├── agent_loop.py          ← THE most important file. Loop detection, circuit breaker, auto-delegation
+│   │   ├── prompts/cli.md         ← System prompt (debugging rules, tool usage, import rules)
+│   │   ├── tools/builtins/        ← 24 builtin tools
+│   │   ├── agents/                ← Subagent definitions (explore, diagnostic, planner)
+│   │   └── skills/                ← Skill framework
+│   ├── cli/textual_ui/app.py     ← TUI application
+│   └── skills/                    ← 7 bundled skill markdown files
+├── tests/
+│   ├── test_smoke.py              ← 20 fast tests (imports, branding, safety)
+│   ├── test_loop_detection.py     ← 38 loop/circuit breaker tests
+│   ├── test_drydock_*.py          ← ~100 regression tests
+│   ├── test_bank_prd.py           ← 15 PRD-driven tests (CORE usability tests)
+│   ├── test_bank_prd_extended.py  ← 50 extended PRD tests (overnight battery)
+│   ├── test_bank_build.py         ← 25 build tests
+│   ├── test_bank_debug.py         ← 20 debug tests
+│   ├── test_bank_update.py        ← 15 update tests
+│   ├── test_bank_multiagent.py    ← 10 multi-agent tests
+│   ├── test_bank_tools.py         ← 13 tool integration tests
+│   └── testbank_helpers.py        ← Shared test infrastructure
+├── scripts/
+│   ├── deploy_to_github.sh        ← Clone GitHub, sync, push (daily 4 AM)
+│   ├── publish_to_pypi.sh         ← Tests → build → PyPI → tag → integration test
+│   ├── test_full.sh               ← Smoke + real backend regression (nightly 2 AM)
+│   ├── test_bank.sh               ← 83-test bank runner with category selection
+│   ├── monitor_test_battery.sh    ← Cron monitor: restart stuck tests, skip repeat failures
+│   ├── backup.sh                  ← rsync to NAS (daily 3 AM)
+│   └── install.sh                 ← User installation script
+├── test_bank_results/             ← Test logs (overnight runs, partial results)
+├── pyproject.toml                 ← Version, dependencies, build config
+├── CLAUDE.md                      ← Instructions for AI assistants working on this repo
+└── PRD.md                         ← This file
+```
+
+```
+/data3/swe_bench_runs/             ← SWE-bench infrastructure (SEPARATE from drydock repo)
+├── harness.py                     ← Runs drydock on SWE-bench tasks
+├── continuous_bench.sh            ← Continuous improvement loop (CURRENTLY PAUSED)
+├── analyze_batch.py               ← Post-batch analysis
+├── auto_fix.py                    ← Auto-apply prompt fixes
+└── monitor_health.sh              ← Worktree cleanup (CURRENTLY PAUSED)
+```
+
+```
+~/.config/drydock/
+├── github_token                   ← GitHub PAT for push/deploy
+└── pypi_token                     ← PyPI API token for publishing
+```
+
+### Key Paths
+- **vLLM server:** http://localhost:8000 (devstral-24B-AWQ-4bit)
+- **NAS backup:** 192.168.50.183 via rsync
+- **Conda env:** `/home/bobef/miniconda3/bin/python3` (ALWAYS use this, never bare `python3`)
+- **Git remotes:** `drydock` → github.com/fbobe321/drydock, `origin` → github.com/mistralai/mistral-vibe
+- **Main branch:** `main` (push to `drydock` remote, not `origin`)
+
+---
+
 ## Test Suite
 
-| Tier | File | Tests | Backend | Time | When |
-|------|------|-------|---------|------|------|
+| Tier | File(s) | Tests | Backend | Time | When |
+|------|---------|-------|---------|------|------|
 | **Smoke** | `test_smoke.py` | 20 | None | <1s | Every deploy |
+| **Loop Detection** | `test_loop_detection.py` | 38 | None | <1s | Every deploy |
+| **Regression** | `test_drydock_*.py` | ~100 | None | ~1min | Every deploy |
 | **Full Regression** | `test_full_regression.py` | 13 | Real vLLM | 5-10 min | Nightly 2 AM |
+| **PRD Battery** | `test_bank_prd.py` | 15 | Real vLLM | ~20 min | On demand |
+| **Extended PRD** | `test_bank_prd_extended.py` | 50 | Real vLLM | ~3-6 hours | Overnight |
+| **Build/Debug/Update** | `test_bank_*.py` | 83 | Real vLLM | ~90 min | On demand |
+
+**Run commands:**
+```bash
+# Quick smoke (every change)
+python3 -m pytest tests/test_smoke.py tests/test_loop_detection.py -q
+
+# All fast tests (before commit)
+python3 -m pytest tests/test_drydock_regression.py tests/test_drydock_tasks.py tests/test_loop_detection.py tests/test_smoke.py -q
+
+# PRD battery (user-realistic, needs vLLM)
+python3 -m pytest tests/test_bank_prd.py -v -s
+
+# Overnight (needs vLLM, 3-6 hours)
+nohup python3 -m pytest tests/test_bank_prd.py tests/test_bank_prd_extended.py -v --tb=short > test_bank_results/run.log 2>&1 &
+```
 
 ---
 
-## Continuous Improvement
+## SWE-bench (CURRENTLY PAUSED)
 
-| System | Schedule | What it does |
-|--------|----------|-------------|
-| `continuous_bench.sh` | Always running | SWE-bench batches (20 tasks, 600s timeout) |
-| `analyze_batch.py` | After each batch | Detects crash patterns, multi-file misses, test edits |
-| `auto_fix.py` | After analysis | Applies safe prompt fixes based on patterns |
-| `monitor_health.sh` | Every 30 min | Prunes worktrees, checks vLLM, disk space, pass rates |
-| `deploy_to_github.sh` | Daily 4 AM | Smoke tests → push to GitHub |
-| `backup.sh` | Daily 3 AM | rsync to NAS (192.168.50.183) |
-| `@reboot` cron | On restart | Restarts bench loop after 2 min |
+Infrastructure at `/data3/swe_bench_runs/` (not in drydock repo). Paused to focus on usability testing.
 
-**Latest SWE-bench results (Mar 28):**
+**Latest results (Mar 28):**
 - 2,220/2,294 unique tasks tested (97%)
-- Recent pass rate: **39-42%** (up from 17% baseline)
-- Net improvement: **+22% pass rate** after bash abuse fix and nudge improvements
+- Pass rate: **39-42%** (up from 17% baseline)
+- Crons removed — re-enable with `continuous_bench.sh` when ready
 
 ---
 
@@ -279,3 +394,20 @@ User reported DryDock was unusable — model looped 12+ times running the same c
 Run with: `./scripts/test_bank.sh` (full) or `./scripts/test_bank.sh quick` (easy only, ~2h)
 
 **Removed:** False-positive test file guard in search_replace that blocked any file in a directory containing "test" in its name.
+
+### Phase 12 (Mar 29): Configuration Management + Auto-Delegation + PRD Tests
+
+**Config management crisis:** Overnight test battery crashed for 8 hours because cron used system Python (no pytest). All scripts now use explicit `/home/bobef/miniconda3/bin/python3`.
+
+**Auto-delegation:** Model wasn't using subagents despite prompt instructions. Fixed by injecting context automatically before the model's first turn:
+- Project file listing (replaces explore subagent)
+- Skill content injection (matches prompt keywords to skills)
+- Planning prompt for complex builds (forces absolute imports, python3 -m)
+
+**Circuit breaker reset:** After write_file/search_replace, circuit breaker history clears. Previously-failing commands can be retried after code changes.
+
+**Post-success stop:** After 3 successful bash runs, model is told to stop testing and summarize.
+
+**PRD-driven test bank:** 65 tests (15 core + 50 extended) that give DryDock a PRD and verify the built project RUNS. Core 15 pass at 100%.
+
+**Relative import auto-fix:** When model gets "relative import with no known parent package", agent loop explains absolute imports vs python3 -m and forces search_replace.

{drydock_cli-1.5.3 → drydock_cli-1.5.5}/drydock/core/agent_loop.py

@@ -474,12 +474,29 @@ class AgentLoop:
 
         yield UserMessageEvent(content=user_msg, message_id=user_message.message_id)
 
-        # === AUTO-DELEGATION ===
-        # Inject context automatically so the model doesn't have to choose to delegate.
-        # This replaces relying on the model to call task()/invoke_skill().
-        auto_context = self._build_auto_context(user_msg)
-        if auto_context:
-            self._inject_system_note(auto_context)
+        # === BUILD ORCHESTRATION ===
+        # For complex build tasks (PRD + multiple files), run the multi-phase
+        # pipeline instead of letting the model do everything in one context.
+        if self._is_build_task(user_msg):
+            try:
+                from drydock.core.build_orchestrator import run_build_pipeline
+                summary = await run_build_pipeline(
+                    user_prompt=user_msg,
+                    config=self.config,
+                    base_dir=Path.cwd(),
+                )
+                self._inject_system_note(summary)
+                logger.info("Build orchestrator completed successfully")
+            except Exception as e:
+                logger.warning("Build orchestrator failed (%s), falling back to normal flow", e)
+                auto_context = self._build_auto_context(user_msg)
+                if auto_context:
+                    self._inject_system_note(auto_context)
+        else:
+            # Normal flow: inject context for non-build tasks
+            auto_context = self._build_auto_context(user_msg)
+            if auto_context:
+                self._inject_system_note(auto_context)
 
         try:
             should_break_loop = False
@@ -585,6 +602,10 @@ class AgentLoop:
                                     except (json.JSONDecodeError, AttributeError):
                                         edit_path = ""
 
+                                    # AUTO-FIX: Fix packaging issues the model gets wrong
+                                    if edit_path and edit_path.endswith(".py"):
+                                        self._auto_fix_package(edit_path)
+
                                     # Auto-run diagnostics on edited Python files
                                     if edit_path and edit_path.endswith(".py"):
                                         try:
@@ -1658,6 +1679,98 @@ class AgentLoop:
 
         return None
 
+    def _is_build_task(self, user_msg: str) -> bool:
+        """Detect if a user message is a complex build task that needs orchestration."""
+        msg_lower = user_msg.lower()
+        has_build_verb = any(kw in msg_lower for kw in (
+            "build", "create a", "implement", "get started",
+            "set up", "scaffold", "review the prd", "review prd",
+        ))
+        has_complexity = (
+            len(user_msg) > 200
+            or any(kw in msg_lower for kw in (
+                "prd", "multiple", "package", "modules", "cli",
+                "features", "commands", "architecture", "structure",
+            ))
+        )
+        # Check if a PRD.md exists in the working directory
+        has_prd = Path.cwd().joinpath("PRD.md").exists() or Path.cwd().joinpath("prd.md").exists()
+        return has_build_verb and (has_complexity or has_prd)
+
+    def _auto_fix_package(self, file_path: str) -> None:
+        """Silently fix common packaging mistakes after model writes a file.
+
+        The model (devstral-24B) consistently fails at:
+        1. Creating __main__.py for packages
+        2. Using relative imports instead of absolute
+        This runs after every write_file/search_replace and fixes both.
+        """
+        try:
+            fp = Path(file_path)
+            if not fp.exists() or not fp.is_file():
+                return
+
+            pkg_dir = fp.parent
+            init_file = pkg_dir / "__init__.py"
+
+            # Only fix files inside a package (has __init__.py)
+            if not init_file.exists():
+                return
+
+            pkg_name = pkg_dir.name
+
+            # 1. Create __main__.py if missing
+            main_file = pkg_dir / "__main__.py"
+            if not main_file.exists():
+                # Find the most likely entry point (cli.py, main.py, app.py)
+                entry = None
+                entry_func = "main"
+                for candidate in ["cli.py", "main.py", "app.py", "__init__.py"]:
+                    cand_path = pkg_dir / candidate
+                    if cand_path.exists() and candidate != "__init__.py":
+                        # Check if it has a main() function
+                        try:
+                            content = cand_path.read_text()
+                            if "def main(" in content:
+                                entry = candidate[:-3]  # strip .py
+                                break
+                        except Exception:
+                            pass
+                if entry:
+                    main_content = (
+                        f"from {pkg_name}.{entry} import {entry_func}\n\n"
+                        f"if __name__ == \"__main__\":\n"
+                        f"    {entry_func}()\n"
+                    )
+                    main_file.write_text(main_content)
+                    logger.info("Auto-created %s/__main__.py (entry: %s.%s)", pkg_name, entry, entry_func)
+
+            # 2. Fix relative imports → absolute imports
+            try:
+                content = fp.read_text()
+                if f"from .{'' if content else ''}" in content:
+                    import re
+                    # Replace from .module import X → from pkg.module import X
+                    fixed = re.sub(
+                        r"from \.([\w.]+) import",
+                        f"from {pkg_name}.\\1 import",
+                        content,
+                    )
+                    # Replace from . import X → from pkg import X
+                    fixed = re.sub(
+                        r"from \. import",
+                        f"from {pkg_name} import",
+                        fixed,
+                    )
+                    if fixed != content:
+                        fp.write_text(fixed)
+                        logger.info("Auto-fixed relative imports in %s", fp.name)
+            except Exception:
+                pass
+
+        except Exception as e:
+            logger.debug("Auto-fix package failed for %s: %s", file_path, e)
+
     def _build_auto_context(self, user_msg: str) -> str | None:
         """Build auto-delegation context based on the prompt and project state.
 
@@ -1727,17 +1840,24 @@ class AgentLoop:
         ))
         if is_build and is_complex:
             parts.append(
-                "PLANNING REQUIRED — This is a complex build task. Before writing code:\n"
-                "1. List ALL files you will create (with brief purpose)\n"
-                "2. Define the package structure\n"
-                "3. Use ABSOLUTE imports (from pkg.module import X), NOT relative imports\n"
-                "4. Create files in dependency order (utilities first, then modules that import them)\n"
-                "5. After creating all files, test with: python3 -m package_name\n"
-                "6. If tests fail, READ the error, then FIX with search_replace"
+                "MANDATORY BUILD RULES:\n"
+                "1. Plan first: list ALL files you will create\n"
+                "2. ALWAYS create __main__.py so 'python3 -m package_name' works:\n"
+                "   ```python\n"
+                "   from package_name.cli import main\n"
+                "   if __name__ == '__main__':\n"
+                "       main()\n"
+                "   ```\n"
+                "3. Use ABSOLUTE imports: 'from pkg.module import X', NOT 'from .module import X'\n"
+                "4. Create test/sample data files BEFORE testing (the tool needs input to work)\n"
+                "5. Test with: python3 -m package_name (NOT python3 package/file.py)\n"
+                "6. If a test fails, read the error, fix with search_replace, then retry\n"
+                "7. After 1-2 successful tests, STOP and tell the user it's done"
             )
         elif is_build:
             parts.append(
-                "REMINDER: Use write_file to create files. Use absolute imports. "
+                "BUILD RULES: Use write_file to create files. Use absolute imports. "
+                "Create __main__.py for packages. Create test data before testing. "
                 "Test with python3 -m package_name (not python3 package/file.py)."
             )