@orderful/droid 0.19.0 → 0.21.0

package/CHANGELOG.md

@@ -1,5 +1,24 @@
 # @orderful/droid
 
+## 0.21.0
+
+### Minor Changes
+
+- [#111](https://github.com/Orderful/droid/pull/111) [`c14a3e2`](https://github.com/Orderful/droid/commit/c14a3e2acbf7a87052c9d5dba251502b8be067f9) Thanks [@frytyler](https://github.com/frytyler)! - Add codex-document-processor agent for importing documents to the codex. The agent handles PDF/markdown processing, frontmatter generation, and file writing - keeping the main conversation context lean during document imports.
+
+## 0.20.0
+
+### Minor Changes
+
+- [#109](https://github.com/Orderful/droid/pull/109) [`1014243`](https://github.com/Orderful/droid/commit/10142438bcd75723027d12ec7bdc23f012643ee3) Thanks [@frytyler](https://github.com/frytyler)! - Add codex tool for shared organizational knowledge
+
+  New tool providing access to the orderful-codex repo containing PRDs, tech designs, patterns, and explored topics. Features include:
+  - Load project context with `/codex {name}`
+  - Search across all categories
+  - Capture decisions during implementation
+  - Add explored topics with freshness tracking
+  - Auto-generate CONTEXT.md from project artifacts
+
 ## 0.19.0
 
 ### Minor Changes

package/dist/tools/codex/TOOL.yaml

@@ -0,0 +1,33 @@
+name: codex
+description: "Shared organizational knowledge - PRDs, tech designs, patterns, and explored topics. Use when loading project context, searching codex, capturing decisions, or adding explored topics."
+version: 0.1.0
+status: beta
+
+includes:
+  skills:
+    - name: droid-codex
+      required: true
+  commands:
+    - codex
+  agents:
+    - codex-document-processor
+
+dependencies: []
+
+# Prerequisites checked on install
+prerequisites:
+  - name: gh
+    description: GitHub CLI for PR workflows
+    check: "gh --version"
+    install_hint: "brew install gh && gh auth login"
+
+config_schema:
+  codex_repo:
+    type: string
+    description: Path to local orderful-codex repository (required)
+    required: true
+    default: "~/src/github.com/orderful-codex"
+  freshness_days:
+    type: number
+    description: Days before showing staleness warning
+    default: 30

package/dist/tools/codex/agents/codex-document-processor.md

@@ -0,0 +1,137 @@
+---
+name: codex-document-processor
+description: "Process documents (PDF, markdown) into codex-ready markdown with proper frontmatter. Use when importing files to the codex - handles extraction, formatting, and writing to the correct location."
+tools:
+  - Read
+  - Write
+  - Glob
+color: yellow
+---
+
+You are a document processor for the Orderful codex. Your job is to transform source documents (PDFs, markdown files) into properly structured codex entries.
+
+## Input
+
+You will receive:
+1. **File path** - The source document to process
+2. **Document type** - One of: `prd`, `tech-design`, `topic`, `pattern`
+3. **Project/entry name** - Where this should be stored in the codex
+4. **Codex repo path** - The local path to orderful-codex repository
+
+## Validation (Do This First)
+
+Before processing, validate all inputs:
+
+1. **Document type** - Must be one of: `prd`, `tech-design`, `topic`, `pattern`. Reject anything else.
+2. **File path** - Must exist and be readable. Use Read tool to verify.
+3. **Project/entry name** - Must be safe for filesystem paths:
+   - Only allow: alphanumeric, dashes, underscores
+   - **Reject** if contains: `..`, `/`, `\`, or any shell metacharacters (`$`, backtick, `|`, `;`, `&`, `>`, `<`)
+   - If invalid, return error immediately (do not attempt to sanitize)
+4. **Codex repo path** - Must exist and contain expected structure
+
+If any validation fails, return an error (see Error Handling below).
+
+## Process
+
+### 1. Read Source Document
+
+Read the file at the provided path. For PDFs, extract all text content. For markdown, read as-is.
+
+### 2. Extract Key Information
+
+From the content, identify:
+- **Title** - The document/feature name
+- **Main content** - The substantive information
+- **Codebase paths** (if tech-design) - Any code paths mentioned
+- **Source** - Default to `confluence` for PRDs/tech-designs, `exploration` for topics
+- **Status** - Default based on type:
+  - PRD/tech-design: `draft` (new imports start as drafts)
+  - Topic/pattern: `active` (explorations are immediately usable)
+
+### 3. Read and Apply Frontmatter Template
+
+**Read the template from the codex repo** - don't hardcode frontmatter structure:
+
+| Type | Template Path |
+|------|---------------|
+| `prd` | `{codex_repo}/templates/PRD.md` |
+| `tech-design` | `{codex_repo}/templates/TECH-DESIGN.md` |
+| `topic` | `{codex_repo}/templates/TOPIC.md` |
+| `pattern` | `{codex_repo}/templates/PATTERN.md` (if exists, else use TOPIC.md) |
+
+1. Read the appropriate template file
+2. Extract its frontmatter structure
+3. Fill in the values:
+   - `title` → extracted title
+   - `created` / `explored` → today's date (YYYY-MM-DD)
+   - `updated` → today's date (YYYY-MM-DD)
+   - `status` → `draft` for PRD/tech-design, `active` for topic/pattern
+   - `codebase_paths` → extracted paths, or omit if none found
+   - Keep other fields from template as-is
+
+This ensures frontmatter stays in sync with the codex repo's templates.
+
+### 4. Structure Content
+
+Transform the source content into the codex format:
+- Preserve the original information and structure where sensible
+- Use markdown headings, lists, and tables appropriately
+- Remove any PDF artifacts or formatting noise
+- Keep the content substantive - don't pad with empty template sections
+
+### 5. Write Output
+
+Write the processed markdown to the correct location:
+
+| Type | Output Path |
+|------|-------------|
+| `prd` | `{codex_repo}/projects/{name}/PRD.md` |
+| `tech-design` | `{codex_repo}/projects/{name}/TECH-DESIGN.md` |
+| `topic` | `{codex_repo}/topics/{name}.md` |
+| `pattern` | `{codex_repo}/patterns/{name}.md` |
+
+Create the directory if it doesn't exist.
+
+**If file already exists:** Return an error. Do not overwrite without explicit confirmation from the main conversation. The error response should indicate the file exists so the user can decide whether to proceed.
+
+### 6. Return Result
+
+Return a brief JSON summary:
+
+```json
+{
+  "status": "success",
+  "file_path": "/path/to/written/file.md",
+  "title": "Extracted document title",
+  "type": "prd",
+  "summary": "One sentence describing what was processed"
+}
+```
+
+## Important Notes
+
+- **Preserve substance over form** - The goal is to capture the document's information, not perfectly replicate its layout
+- **Don't invent content** - Only include information that exists in the source document
+- **Handle missing sections gracefully** - If the source doesn't have a section (e.g., no "Testing Strategy"), omit it rather than leaving empty placeholders
+- **Omit empty frontmatter fields** - If `codebase_paths` can't be determined, omit the field entirely rather than using placeholders
+- **Canadian spelling** - Use behaviour, colour, etc.
+
+## Error Handling
+
+If anything fails, return an error response instead of the success format:
+
+```json
+{
+  "status": "error",
+  "error_type": "validation_failed" | "file_not_found" | "extraction_failed" | "file_exists" | "write_failed",
+  "message": "Human-readable description of what went wrong"
+}
+```
+
+**Error types:**
+- `validation_failed` - Invalid document type, unsafe name, or missing codex repo
+- `file_not_found` - Source file doesn't exist or isn't readable
+- `extraction_failed` - Couldn't extract meaningful content from the document
+- `file_exists` - Target file already exists (user must confirm overwrite)
+- `write_failed` - Couldn't write to target location (permissions, disk space, etc.)

package/dist/tools/codex/commands/codex.md

@@ -0,0 +1,70 @@
+---
+description: Shared organizational knowledge - PRDs, tech designs, patterns, and explored topics
+argument-hint: "[projects | patterns | topics | {name} | search {query} | new {name} | decision {text} | add topic {name}]"
+allowed-tools: Read, Write, Edit, Glob, Grep, Bash(gh:*), Bash(git:*), Bash(ls:*), Bash(mkdir:*)
+---
+
+# /codex
+
+Entry point for shared organizational knowledge. See the **codex skill** for full behaviour.
+
+> Context is AI's multiplier. This is Orderful's.
+
+## Arguments
+
+$ARGUMENTS
+
+## Prerequisites
+
+Before first use, verify:
+
+1. **Repo cloned** at configured `codex_repo` path
+2. **gh CLI** installed and authenticated
+
+## Usage
+
+```bash
+/codex                           # Show categories (projects, patterns, topics)
+/codex projects                  # List projects
+/codex patterns                  # List patterns
+/codex topics                    # List topics
+/codex {name}                    # Load entry (searches all categories)
+/codex search {query}            # Search across everything
+/codex new {name}                # Scaffold new project from templates
+/codex decision {text}           # Append to active project's DECISIONS.md
+/codex snapshot {file} {name}    # Import PDF/markdown to project
+/codex add topic {name}          # Add explored topic with freshness metadata
+```
+
+## Configuration
+
+**ALWAYS read `~/.droid/skills/codex/overrides.yaml` first.**
+
+- `codex_repo` - Path to local orderful-codex repository (required)
+- `freshness_days` - Days before showing staleness warning (default: 30)
+
+## Behaviour
+
+Refer to the codex skill for:
+- **Loading**: How to search, handle matches, check freshness, auto-generate CONTEXT.md
+- **Decisions**: How to append to active project's DECISIONS.md
+- **Topics**: How to capture explored codebase areas with freshness metadata
+- **Creating**: How to scaffold new projects from templates
+
+The skill's `references/` folder contains detailed procedures for each workflow.
+
+## Examples
+
+```bash
+# Load project context
+/codex transaction-templates
+
+# Search for webhook-related content
+/codex search webhook
+
+# Capture a decision during implementation
+/codex decision "Using UUIDv7 for IDs because it's sortable and includes timestamp"
+
+# Save today's exploration as a topic
+/codex add topic organization-hierarchy
+```

package/dist/tools/codex/skills/droid-codex/SKILL.md

@@ -0,0 +1,314 @@
+---
+name: droid-codex
+description: "Shared organizational knowledge - PRDs, tech designs, patterns, and explored topics. Use when loading project context ('load transaction-templates'), searching codex ('search webhook'), capturing decisions ('codex decision'), or adding explored topics ('add topic'). User prompts like 'check the codex', 'what's in the codex about X', 'load the PRD for Y'."
+globs:
+  - "**/PRD.md"
+  - "**/TECH-DESIGN.md"
+  - "**/CONTEXT.md"
+  - "**/DECISIONS.md"
+alwaysApply: false
+allowed-tools: Read, Write, Edit, Glob, Grep, Bash(gh:*), Bash(git:*), Bash(ls:*), Bash(mkdir:*)
+---
+
+# Codex Skill
+
+> Context is AI's multiplier. This is Orderful's.
+
+Shared organizational knowledge for humans and droids alike. Product requirements documents, tech designs, patterns, and explored topics - structured for AI-assisted development.
+
+## Prerequisites
+
+Before using codex, verify:
+
+1. **Repo cloned:** Check if `codex_repo` path exists
+2. **gh CLI:** Required for PR workflows (CONTEXT.md auto-generation)
+
+```bash
+# Check repo exists
+ls -la {codex_repo}
+
+# Check gh CLI
+gh --version
+```
+
+If prerequisites fail, guide user to fix:
+- Repo missing: `git clone git@github.com:orderful/orderful-codex.git {path}`
+- gh missing: `brew install gh && gh auth login`
+
+## Sync Before Reading
+
+**Before any read operation** (loading, searching, listing), pull latest:
+
+```bash
+git -C {codex_repo} pull --rebase --quiet
+```
+
+This ensures shared knowledge is always current. The pull is quiet to avoid noise, but if it fails (conflicts, network), warn the user.
+
+## Security
+
+### Input Validation
+
+**Always validate user-provided names before using in paths:**
+
+```bash
+# REJECT names containing path traversal attempts
+if [[ "{name}" == *".."* ]] || [[ "{name}" == *"/"* ]]; then
+  echo "Invalid name: must not contain '..' or '/'"
+  exit 1
+fi
+
+# REJECT names with shell metacharacters
+if [[ "{name}" =~ [\$\`\|\;\&\>\<] ]]; then
+  echo "Invalid name: contains unsafe characters"
+  exit 1
+fi
+```
+
+### Safe Git Commits
+
+**Use heredoc for commit messages to prevent injection:**
+
+```bash
+# GOOD - safe from injection
+git commit -F - <<EOF
+decision({active}): {summary}
+EOF
+
+# BAD - vulnerable to injection
+git commit -m "decision({active}): {summary}"
+```
+
+### Access Control
+
+**Check repo access before attempting write operations:**
+
+```bash
+# Verify push access before making changes
+if ! git -C {codex_repo} push --dry-run 2>/dev/null; then
+  echo "Error: No push access to codex repo. Check your permissions."
+  exit 1
+fi
+```
+
+## Configuration
+
+**ALWAYS read `~/.droid/skills/codex/overrides.yaml` first.**
+
+| Setting | Default | Description |
+|---------|---------|-------------|
+| `codex_repo` | `~/src/github.com/orderful-codex` | Path to local codex repo |
+| `freshness_days` | `30` | Days before staleness warning |
+
+## Categories
+
+The codex has three categories:
+
+| Category | Path | Purpose |
+|----------|------|---------|
+| `projects` | `{codex_repo}/projects/` | Feature/project context (PRD, tech design, decisions) |
+| `patterns` | `{codex_repo}/patterns/` | Cross-cutting technical patterns |
+| `topics` | `{codex_repo}/topics/` | Explored codebase areas with freshness tracking |
+
+## Commands
+
+| Command | Action |
+|---------|--------|
+| `/codex` | Show categories overview |
+| `/codex projects` | List all projects |
+| `/codex patterns` | List all patterns |
+| `/codex topics` | List all topics |
+| `/codex {name}` | Load entry (searches all categories) |
+| `/codex search {query}` | Search across everything |
+| `/codex new {name}` | Scaffold new project from templates |
+| `/codex decision {text}` | Append to active project's DECISIONS.md |
+| `/codex snapshot {type} {file} {name}` | Import PDF/markdown to codex (uses agent) |
+| `/codex add topic {name}` | Add explored topic with freshness metadata |
+
+## Loading an Entry
+
+**Trigger:** `/codex {name}` or user asks to load/check codex for something
+
+**Procedure:**
+
+1. Search for `{name}` across all categories (fuzzy match on folder/file names)
+2. If multiple matches → show list, let user pick
+3. If single match → load it
+4. For projects with multiple files → show file list, let user pick which to load
+5. Check freshness (see below)
+6. Output loaded content
+
+**If no CONTEXT.md exists for a project:**
+- Synthesise unified context from PRD.md + TECH-DESIGN.md + DECISIONS.md
+- Create branch, commit CONTEXT.md, open PR
+- Inform user: "I've created a CONTEXT.md summarising this project - PR #{number}"
+
+Full procedure: `references/loading.md`
+
+## Freshness Checking
+
+All codex files have frontmatter:
+
+```yaml
+---
+title: Feature Name
+type: prd | tech-design | context | decisions | pattern | topic
+status: draft | active | complete | archived
+created: 2026-01-07
+updated: 2026-01-07
+confidence: high | medium | low
+source: confluence | exploration | implementation
+codebase_paths:
+  - apps/platform-api/src/...
+---
+```
+
+**Status values:**
+- `draft` - Initial capture, not ready for consumption
+- `active` - Current, being used for implementation
+- `complete` - Project shipped, kept for reference
+- `archived` - No longer relevant, kept for history
+
+Note: Topics use `draft | active | archived` (no "complete"). Patterns use `active | archived` (no draft phase).
+
+**When loading any file:**
+
+1. Parse frontmatter for `updated` date
+2. Calculate days since update
+3. If > `freshness_days` (default 30):
+   - Warn: "This doc was last updated {date} ({n} days ago). Want me to verify it's still accurate?"
+4. If `codebase_paths` present:
+   - Check if those paths have commits since `updated` date
+   - If yes: "Files in {paths} have changed since this doc was updated. Want me to review?"
+5. If `confidence: low`:
+   - Note: "This was a quick exploration and may be incomplete"
+
+## Active Project Tracking
+
+Scoped operations (`decision`, `snapshot`) require an active project:
+
+- When user loads a project via `/codex {project-name}`, set it as active
+- Store active project name in session context
+- If scoped operation called with no active project → show project list, let user pick
+
+## Adding Decisions
+
+**Trigger:** `/codex decision {text}` or user wants to capture a decision
+
+**Procedure:**
+
+1. Verify active project exists (if not, prompt to select)
+2. Read `{codex_repo}/projects/{active}/DECISIONS.md`
+3. Append new decision with today's date:
+   ```markdown
+   ## {YYYY-MM-DD}: {Decision summary from text}
+
+   **Context:** {Extract from conversation}
+
+   **Decision:** {The decision}
+
+   **Rationale:** {Why this choice}
+   ```
+4. Update `updated` date in frontmatter
+5. Create branch, commit, and open PR via `gh pr create`
+
+Full procedure: `references/decisions.md`
+
+## Adding Topics
+
+**Trigger:** `/codex add topic {name}` or user wants to save exploration
+
+**Procedure:**
+
+1. Create `{codex_repo}/topics/{name}.md`
+2. Use template from `{codex_repo}/templates/TOPIC.md`
+3. Fill in frontmatter:
+   - `explored`: today's date
+   - `confidence`: ask user or infer from exploration depth
+   - `codebase_paths`: paths that were explored
+4. Fill in content from current exploration/conversation
+5. Create branch, commit, and open PR via `gh pr create`
+
+Full procedure: `references/topics.md`
+
+## Importing Documents
+
+**Trigger:** `/codex snapshot {type} {file} {name}` or user wants to add a file to the codex
+
+**IMPORTANT:** Always use the `codex-document-processor` agent to process documents. This keeps the main conversation context lean by offloading heavy document processing.
+
+**Procedure:**
+
+1. Determine the document type (required): `prd`, `tech-design`, `topic`, or `pattern`
+2. Spawn the `codex-document-processor` agent with:
+   - File path to the source document
+   - Document type
+   - Project/entry name
+   - Codex repo path from config
+3. Agent processes the document:
+   - Reads and extracts content
+   - Applies appropriate frontmatter template
+   - Writes to correct location in codex repo
+   - Returns file path and summary
+4. **Back in main conversation:** Handle git workflow
+   - Create branch: `codex/snapshot-{name}`
+   - Commit the new file
+   - Push and create PR via `gh pr create`
+5. Share PR link with user
+
+**Example invocations:**
+
+```bash
+# Explicit snapshot command
+/codex snapshot prd ~/Downloads/transaction-templates-prd.pdf transaction-templates
+
+# Natural language (still uses the agent)
+"Add this PRD to the transaction-templates project" + attach file
+"Import this tech design to generic-scenario-testing"
+```
+
+The agent handles: PDF reading, content extraction, frontmatter generation, file writing.
+The main conversation handles: git branch, commit, PR creation, user interaction.
+
+## Creating New Projects
+
+**Trigger:** `/codex new {name}` or user wants to start new project entry
+
+**Procedure:**
+
+1. Create `{codex_repo}/projects/{name}/`
+2. Copy templates from `{codex_repo}/templates/`:
+   - PRD.md
+   - TECH-DESIGN.md
+   - DECISIONS.md
+3. Fill in frontmatter with today's date
+4. Create branch, commit, and open PR via `gh pr create`
+
+Full procedure: `references/creating.md`
+
+## Integration with /project
+
+The codex holds **shared** organizational knowledge. The `/project` skill holds **personal** working memory.
+
+```
+/codex transaction-templates              → load shared context
+/project create --from codex:{name}       → seed personal PROJECT.md
+/codex publish                            → promote learnings back (future)
+```
+
+When user runs `/project create --from codex:{name}`:
+1. Load the codex entry
+2. Extract key context for personal PROJECT.md
+3. Create project in user's projects_dir
+
+## Git Workflow
+
+**All codex changes must go through PRs** (main branch is protected):
+
+1. Create branch: `codex/{action}-{name}` (e.g., `codex/decision-uuid-format`)
+2. Make changes
+3. Commit with descriptive message
+4. Push and create PR via `gh pr create`
+5. Share PR link with user
+
+Safe changes (new files, decision appends) are auto-merged by GitHub Action. Modifications to existing content require review.