devflow-engine 1.1.23__tar.gz → 1.1.25__tar.gz

{devflow_engine-1.1.23 → devflow_engine-1.1.25}/.gitignore

@@ -14,5 +14,9 @@ __pycache__/
 # DevFlow runtime
 .devflow/
 
+# Agent / tooling runtime logs
+.debug.log
+.tau/
+
 # OS
 .DS_Store

devflow_engine-1.1.25/OPERATOR.md

@@ -0,0 +1,492 @@
+# DevFlow Engine Operator Guide
+
+**This document is for AI agents and operators who need to publish or modify DevFlow Engine.**
+
+---
+
+## What is DevFlow Engine?
+
+DevFlow Engine (`devflow-engine`) is a Python CLI package published to PyPI. It:
+- Provides the `devflow` command-line interface
+- Contains the core engine logic (story lifecycle, execution, worker processes)
+- Runs in per-project Docker containers managed by DevFlow Host
+- Is version-managed and can be pinned to specific versions
+
+---
+
+## Package Structure
+
+```
+devflow_engine/
+├── pyproject.toml           # Package metadata and dependencies
+├── src/
+│   ├── devflow_engine/      # Main package
+│   │   ├── cli/             # CLI entry points
+│   │   ├── story/           # Story lifecycle
+│   │   ├── idea/            # Idea management
+│   │   ├── implementation/   # Implementation logic
+│   │   ├── error/           # Error handling
+│   │   ├── review/          # Review packets
+│   │   └── ...
+│   └── devin/               # Devin agent nodes
+│       └── nodes/           # DAG nodes for different scenarios
+├── tests/                   # Test suite
+├── prompts/                 # Agent prompts
+└── docs/                    # Architecture documentation
+```
+
+---
+
+## Version Management
+
+### Current Version
+
+Check `pyproject.toml`:
+```toml
+[project]
+name = "devflow-engine"
+version = "X.Y.Z"
+```
+
+### Version Pinning in DevFlow-Host
+
+DevFlow-Host pins the engine version in `/opt/devflow-host/ENGINE_VERSION`. To update:
+
+```bash
+# Local change
+echo "X.Y.Z" > devflow-host/ENGINE_VERSION
+git add ENGINE_VERSION
+git commit -m "chore: pin devflow-engine to X.Y.Z"
+git push
+
+# Deploy
+ssh clarity "cd /opt/devflow-host && git pull && docker compose build controller && docker compose up -d controller"
+```
+
+---
+
+## Publishing to PyPI
+
+### Prerequisites
+
+1. **PyPI API Token** - Required to publish. Ask the operator for this.
+2. **uv** or **twine** - Build tools
+
+### Publishing Steps
+
+#### 1. Verify Your Changes
+
+```bash
+cd devflow_engine
+
+# Run tests
+uv pip install -e '.[dev]'
+pytest
+
+# Type check
+mypy src
+
+# Lint
+ruff check src
+```
+
+#### 2. Update Version
+
+Edit `pyproject.toml`:
+```toml
+[project]
+name = "devflow-engine"
+version = "X.Y.Z"  # Increment appropriately
+```
+
+Version bump guidelines:
+- **PATCH** (1.2.3 → 1.2.4): Bug fixes, small improvements
+- **MINOR** (1.2.3 → 1.3.0): New features, backward compatible
+- **MAJOR** (1.2.3 → 2.0.0): Breaking changes
+
+#### 3. Build the Package
+
+```bash
+cd devflow_engine
+
+# Clean old builds
+rm -rf dist/ build/ *.egg-info
+
+# Build with uv (recommended)
+uv build
+
+# Or with Python build
+python -m build
+```
+
+#### 4. Test the Package (Optional - TestPyPI)
+
+```bash
+# Upload to TestPyPI first (for verification)
+uv publish --repository testpypi --token pypi-api-token-here
+
+# Or with twine
+twine upload --repository testpypi dist/* --token pypi-api-token-here
+
+# Install from TestPyPI to verify
+uv pip install --index-url https://test.pypi.org/simple/ devflow-engine==X.Y.Z
+devflow --help  # Verify
+```
+
+#### 5. Publish to PyPI
+
+```bash
+# Publish to PyPI
+uv publish --token pypi-api-token-here
+
+# Or with twine
+twine upload dist/* --token pypi-api-token-here
+```
+
+#### 6. Tag the Release (Optional)
+
+```bash
+git tag vX.Y.Z
+git push origin vX.Y.Z
+```
+
+#### 7. Update DevFlow-Host
+
+```bash
+# Update ENGINE_VERSION
+echo "X.Y.Z" > devflow-host/ENGINE_VERSION
+
+# Commit and push
+git add ENGINE_VERSION
+git commit -m "chore: pin devflow-engine to X.Y.Z"
+git push
+
+# Deploy
+ssh clarity "cd /opt/devflow-host && git pull && docker compose build controller && docker compose up -d controller"
+```
+
+---
+
+## DevFlow Engine CLI
+
+### Installation
+
+```bash
+# From PyPI (published version)
+pip install devflow-engine
+
+# From local source (development)
+cd devflow_engine
+uv pip install -e '.[dev]'
+```
+
+### Common Commands
+
+```bash
+# Help
+devflow --help
+
+# Project management
+devflow project init --name my-repo
+devflow project import /path/to/repo --init python
+
+# Story management
+devflow plan analyze
+devflow story validate .devflow/drafts/DF-PLAN-001.yaml
+devflow story register .devflow/drafts/DF-PLAN-001.yaml
+
+# Implementation
+devflow impl gate
+./scripts/green.sh  # Run tests + typecheck + lint
+
+# Review
+devflow review run DF-PLAN-001
+
+# Worker
+devflow worker supabase-events  # Listen for execution events
+```
+
+---
+
+## Per-Project Engine Containers
+
+DevFlow-Host spawns engine containers per-project. The container:
+1. Reads version from `ENGINE_VERSION` file
+2. Pulls that version from PyPI
+3. Runs the `devflow` CLI
+
+### Container Environment
+
+Key env vars passed to engine containers:
+- `DEVFLOW_HOST_ENGINE_VERSION` - Pinned engine version
+- `DEVFLOW_API_BASE_URL` - Internal API URL
+- `CLARITY_SECRET_KEY` - For LLM provider authentication
+- `DEVFLOW_INTERNAL_SECRET` - For controller communication
+
+### Debug a Per-Project Engine
+
+```bash
+ssh clarity
+
+# Find the container
+docker ps | grep devflow-engine
+
+# View logs
+docker logs devflow-engine-YOURSLUG --tail 100 -f
+
+# Get into the container
+docker exec -it devflow-engine-YOURSLUG /bin/bash
+
+# Run commands inside
+docker exec devflow-engine-YOURSLUG devflow story list
+
+# Restart
+docker compose -p devflow-project-YOURSLUG restart engine
+```
+
+---
+
+## Local Development Workflow
+
+### 1. Make Changes to DevFlow Engine
+
+```bash
+cd devflow_engine
+
+# Make your changes
+# Edit files, add tests, etc.
+
+# Test locally
+uv pip install -e '.[dev]'
+devflow --help
+pytest
+
+# Type check
+mypy src
+
+# Lint
+ruff check src
+```
+
+### 2. Test with a Real Project
+
+```bash
+# Use a test project on the server
+ssh clarity
+
+# Navigate to project workspace
+cd /opt/devflow-host/workspaces/your-test-project
+
+# Access the engine container
+docker exec -it devflow-engine-your-test-project /bin/bash
+
+# Manually install your local version (for testing)
+# Option: Mount local source via volumes in docker-compose.override.yml
+```
+
+### 3. Publish a New Version
+
+Follow the publishing steps above.
+
+### 4. Deploy
+
+```bash
+# Update and deploy
+echo "X.Y.Z" > devflow-host/ENGINE_VERSION
+git add ENGINE_VERSION
+git commit -m "chore: pin devflow-engine to X.Y.Z"
+git push
+
+# On server
+ssh clarity "cd /opt/devflow-host && git pull && docker compose build controller && docker compose up -d controller"
+```
+
+---
+
+## Testing
+
+### Run Tests
+
+```bash
+cd devflow_engine
+
+# All tests
+pytest
+
+# Specific test file
+pytest tests/test_story.py
+
+# With verbose output
+pytest -v
+
+# With coverage
+pytest --cov=src --cov-report=html
+```
+
+### Test Markers
+
+Tests use markers for categorization:
+- `@pytest.mark.story_id("DF2-BOOT-001")` - Story-specific tests
+- `@pytest.mark.story_uuid(uuid)` - UUID-based tests
+- `@pytest.mark.plane("auth")` - Testing specific planes
+
+### Evals
+
+The `evals/` directory contains evaluation suites:
+```bash
+cd devflow_engine
+python -m pytest evals/
+```
+
+---
+
+## Architecture Overview
+
+### Key Components
+
+1. **CLI** (`src/devflow_engine/cli/`)
+   - Entry point: `devflow_engine.cli.app:app`
+   - Typer-based CLI with subcommands
+
+2. **Story Lifecycle** (`src/devflow_engine/story/`)
+   - Validation, registration, execution tracking
+
+3. **Implementation** (`src/devflow_engine/implementation/`)
+   - Green gate (tests + typecheck + lint)
+   - DAG execution for implementation
+
+4. **Worker** (`src/devflow_engine/worker.py`)
+   - Supabase event listener
+   - Dispatches workflows for queued events
+
+5. **Devin Nodes** (`src/devin/nodes/`)
+   - Ideation, insight, intake, iterate nodes
+   - Scenario-specific behavior
+
+### Execution Store
+
+SQLite database at `.devflow/execution.sqlite`:
+- Stores all runs, events, errors
+- Tracks story states and review packets
+- Query for debugging:
+```bash
+sqlite3 .devflow/execution.sqlite ".tables"
+sqlite3 .devflow/execution.sqlite "SELECT * FROM runs LIMIT 10;"
+```
+
+---
+
+## Troubleshooting
+
+### Import Errors
+
+```bash
+# Reinstall in editable mode
+uv pip install -e .
+
+# Check Python path
+python -c "import devflow_engine; print(devflow_engine.__file__)"
+```
+
+### Type Check Failures
+
+```bash
+# Run mypy
+mypy src
+
+# Fix errors before publishing
+```
+
+### Publishing Failures
+
+```bash
+# Verify you're logged in
+uv logout
+uv login  # Use PyPI token
+
+# Check package build
+ls dist/
+
+# Verify package contents
+tar -tzf dist/devflow_engine-X.Y.Z.tar.gz | head
+```
+
+### Container Not Starting
+
+```bash
+ssh clarity
+cd /opt/devflow-host/workspaces/YOURSLUG
+docker compose logs engine
+# Check: version mismatch, missing env vars, Python import errors
+```
+
+---
+
+## Emergency Rollback
+
+### Rollback to Previous Version
+
+```bash
+# 1. Determine previous version
+cd devflow_engine
+git log --oneline -10
+git show HEAD~1:pyproject.toml | grep version
+
+# 2. Reset pyproject.toml
+git checkout HEAD~1 -- pyproject.toml
+
+# 3. Rebuild and republish
+uv build
+uv publish --token pypi-api-token-here
+
+# 4. Update DevFlow-Host
+echo "PREVIOUS_VERSION" > devflow-host/ENGINE_VERSION
+git add ENGINE_VERSION
+git commit -m "chore: rollback devflow-engine to PREVIOUS_VERSION"
+git push
+ssh clarity "cd /opt/devflow-host && git pull && docker compose build controller"
+```
+
+### Emergency: Pin Older Version on Server
+
+```bash
+ssh clarity
+
+# SSH into server and manually edit
+cd /opt/devflow-host
+echo "OLD_VERSION" > ENGINE_VERSION
+docker compose restart controller
+```
+
+---
+
+## Documentation
+
+Key docs in this repo:
+- `DESIGN_NOTES.md` - System design
+- `docs/` - Detailed architecture docs
+- `docs/queue-worker-infra.md` - Worker architecture
+- `docs/story-implementation-planning-node.md` - Story planning
+
+---
+
+## Quick Reference
+
+```bash
+# Local dev setup
+cd devflow_engine
+uv venv
+uv pip install -e '.[dev]'
+
+# Run tests
+pytest
+
+# Build package
+uv build
+
+# Publish (needs PyPI token)
+uv publish --token YOUR_TOKEN_HERE
+
+# Update DevFlow-Host version
+echo "X.Y.Z" > ../devflow-host/ENGINE_VERSION
+```

{devflow_engine-1.1.23 → devflow_engine-1.1.25}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: devflow-engine
-Version: 1.1.23
+Version: 1.1.25
 Summary: DevFlow v2 CLI-first engine (projects, stories, execution store, review packets)
 Author: DevFlow
 License: Proprietary
@@ -10,10 +10,17 @@ Requires-Dist: pydantic>=2.6
 Requires-Dist: pyyaml>=6.0
 Requires-Dist: rich>=13.7
 Requires-Dist: typer>=0.12
+Provides-Extra: dev
+Requires-Dist: mypy>=1.8; extra == 'dev'
+Requires-Dist: pytest>=8.0; extra == 'dev'
+Requires-Dist: ruff>=0.6; extra == 'dev'
+Requires-Dist: types-pyyaml>=6.0; extra == 'dev'
 Description-Content-Type: text/markdown
 
 # DevFlow Engine (v2)
 
+> **For operators:** See [OPERATOR.md](./OPERATOR.md) for publishing to PyPI and deployment.
+
 CLI-first Python engine for DevFlow v2.
 
 ## Goals

{devflow_engine-1.1.23 → devflow_engine-1.1.25}/README.md

@@ -1,5 +1,7 @@
 # DevFlow Engine (v2)
 
+> **For operators:** See [OPERATOR.md](./OPERATOR.md) for publishing to PyPI and deployment.
+
 CLI-first Python engine for DevFlow v2.
 
 ## Goals

{devflow_engine-1.1.23 → devflow_engine-1.1.25}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "devflow-engine"
-version = "1.1.23"
+version = "1.1.25"
 description = "DevFlow v2 CLI-first engine (projects, stories, execution store, review packets)"
 readme = "README.md"
 requires-python = ">=3.11"
@@ -15,6 +15,14 @@ dependencies = [
   "PyYAML>=6.0",
 ]
 
+[project.optional-dependencies]
+dev = [
+  "pytest>=8.0",
+  "ruff>=0.6",
+  "mypy>=1.8",
+  "types-PyYAML>=6.0",
+]
+
 [dependency-groups]
 dev = [
   "pytest>=8.0",

devflow_engine-1.1.25/scripts/audit_story_ideas.py

@@ -0,0 +1,155 @@
+#!/usr/bin/env python3
+"""
+CLI tool to audit stories for idea associations.
+
+Usage:
+    # Audit a specific project
+    python -m devflow_engine.scripts.audit_story_ideas /path/to/project/.devflow/stories
+    
+    # Audit all workspaces on the host
+    python -m devflow_engine.scripts.audit_story_ideas --all-workspaces /opt/devflow-host/workspaces
+    
+    # Strict mode (fail if any orphaned stories found)
+    python -m devflow_engine.scripts.audit_story_ideas --strict /path/to/project/.devflow/stories
+"""
+
+from __future__ import annotations
+
+import argparse
+import sys
+from pathlib import Path
+
+# Add src to path for imports
+import sys
+from pathlib import Path
+sys.path.insert(0, str(Path(__file__).parent.parent / "src"))
+
+from devflow_engine.story_validation import (
+    audit_stories_in_directory,
+    enforce_story_idea_association,
+    get_orphaned_stories_report,
+    StoryIdeaValidationError,
+)
+
+
+def audit_workspace(workspace_path: Path, strict: bool = False) -> int:
+    """Audit stories in a single workspace."""
+    stories_dir = workspace_path / ".devflow" / "stories"
+    
+    if not stories_dir.exists():
+        print(f"  (no .devflow/stories directory)")
+        return 0
+    
+    print(f"  Auditing {stories_dir}...")
+    result = audit_stories_in_directory(stories_dir)
+    
+    print(f"    Total: {result.total_stories}, Valid: {result.valid_stories}, Orphaned: {result.orphaned_stories}")
+    
+    if result.orphaned_stories > 0:
+        print(f"    ⚠ Found {result.orphaned_stories} orphaned story(ies)")
+        for error in result.errors:
+            print(f"      - {error}")
+        
+        if strict:
+            raise StoryIdeaValidationError(
+                f"Workspace {workspace_path.name} has {result.orphaned_stories} orphaned stories. "
+                f"Run without --strict to see details."
+            )
+    
+    return result.orphaned_stories
+
+
+def main() -> int:
+    parser = argparse.ArgumentParser(
+        description="Audit stories for idea associations"
+    )
+    parser.add_argument(
+        "path",
+        nargs="?",
+        type=Path,
+        help="Path to stories directory or workspace"
+    )
+    parser.add_argument(
+        "--all-workspaces",
+        type=Path,
+        help="Path to workspaces directory (audits all workspaces)"
+    )
+    parser.add_argument(
+        "--strict",
+        action="store_true",
+        help="Exit with error code if any orphaned stories found"
+    )
+    parser.add_argument(
+        "--enforce",
+        action="store_true",
+        help="Enforce strict mode (raise exception on orphaned stories)"
+    )
+    
+    args = parser.parse_args()
+    
+    total_orphaned = 0
+    
+    if args.all_workspaces:
+        # Audit all workspaces
+        print(f"Auditing all workspaces in {args.all_workspaces}...")
+        print()
+        
+        for workspace_dir in sorted(args.all_workspaces.iterdir()):
+            if not workspace_dir.is_dir():
+                continue
+            if workspace_dir.name.startswith("."):
+                continue
+            
+            print(f"Workspace: {workspace_dir.name}")
+            orphaned = audit_workspace(workspace_dir, strict=args.strict)
+            total_orphaned += orphaned
+            print()
+    
+    elif args.path:
+        stories_dir = args.path
+        
+        # If given a workspace path, resolve to stories directory
+        if stories_dir.name == ".devflow":
+            stories_dir = stories_dir / "stories"
+        elif not stories_dir.name == "stories":
+            potential = stories_dir / ".devflow" / "stories"
+            if potential.exists():
+                stories_dir = potential
+        
+        print(f"Auditing {stories_dir}...")
+        print()
+        
+        if args.enforce:
+            try:
+                enforce_story_idea_association(stories_dir)
+                print("✓ All stories are properly associated with Ideas.")
+            except StoryIdeaValidationError as e:
+                print(f"✗ Validation failed: {e}")
+                return 1
+        else:
+            report = get_orphaned_stories_report(stories_dir)
+            print(report)
+            total_orphaned = audit_stories_in_directory(stories_dir).orphaned_stories
+    
+    else:
+        parser.print_help()
+        return 1
+    
+    print()
+    if total_orphaned > 0:
+        print(f"⚠ Total orphaned stories: {total_orphaned}")
+        return 1 if args.strict else 0
+    else:
+        print("✓ All stories properly associated with Ideas.")
+        return 0
+
+
+if __name__ == "__main__":
+    try:
+        sys.exit(main())
+    except StoryIdeaValidationError as e:
+        print(f"✗ Story validation error: {e}", file=sys.stderr)
+        sys.exit(1)
+    except KeyboardInterrupt:
+        print("\nAborted.")
+        sys.exit(130)

{devflow_engine-1.1.23 → devflow_engine-1.1.25}/src/devflow_engine/__init__.py

@@ -1,3 +1,3 @@
 __all__ = ["__version__"]
 
-__version__ = "0.1.0"
+__version__ = "1.1.24"