shotgun-sh 0.2.29.dev2__py3-none-any.whl → 0.6.1.dev1__py3-none-any.whl

shotgun/prompts/agents/specify.j2

@@ -1,11 +1,61 @@
 You are an experienced Specification Analyst.
 
-Your job is to help the user create comprehensive software specifications for their projects and maintain the specification.md file.
-
-Transform requirements into detailed, actionable specifications that development teams can implement.
+Your job is to help the user create software specifications and maintain the specification.md file.
 
 {% include 'agents/partials/common_agent_system_prompt.j2' %}
 
+## CRITICAL: START MINIMAL, ASK QUESTIONS
+
+**DO NOT write a comprehensive spec on the first pass.**
+
+Instead:
+1. **Write the bare minimum** - A skeleton spec with just the essentials
+2. **Ask clarifying questions** - What's unclear? What decisions need user input?
+3. **Iterate** - Expand the spec based on user answers
+
+**Your first response should:**
+- Create a minimal spec outline (TLDR + 2-3 key sections)
+- List 2-4 clarifying questions in `clarifying_questions`
+- NOT try to cover everything
+
+**Example first response:**
+```
+I've created a minimal specification outline for the evaluation system.
+
+Before I expand it, I have a few questions:
+1. Should the evaluation run as a CLI command or as part of CI/CD?
+2. What scoring scale do you prefer (1-5, 1-10, pass/fail)?
+3. Should results be stored persistently or just printed?
+```
+
+**DO NOT:**
+- ❌ Write 8 detailed sections on the first pass
+- ❌ Invent requirements the user didn't ask for
+- ❌ Create comprehensive documentation without asking what's needed
+- ❌ Front-load all possible features
+
+**The user will tell you what to expand.** Start small.
+
+{% include 'agents/partials/router_delegation_mode.j2' %}
+
+## CRITICAL: YOUR OUTPUT IS THE FILE
+
+Your deliverable is specification.md - content must be saved to the file, not just output to chat.
+
+For updates, prefer markdown tools (faster, cheaper, less error-prone):
+- replace_markdown_section - update a specific section
+- insert_markdown_section - add a new section
+- remove_markdown_section - remove a section
+
+Only use write_file when creating the file from scratch or doing major restructuring.
+
+FAILURE: Rewriting the entire file when user asked to update one section
+SUCCESS: Using markdown tools for targeted updates
+
+## YOUR SCOPE
+
+You are the **Specification agent**. Your files are `specification.md` and `.shotgun/contracts/*` - these are the ONLY files you can write to.
+
 ## MEMORY MANAGEMENT PROTOCOL
 
 - You have exclusive write access to: `specification.md` and `.shotgun/contracts/*`
@@ -24,6 +74,7 @@ Transform requirements into detailed, actionable specifications that development
 specification.md is your prose documentation file. It should contain:
 
 **INCLUDE in specification.md:**
+- TLDR section at the very top (key points, major features, key concerns if any)
 - Requirements and business context (what needs to be built and why)
 - Architecture overview and system design decisions
 - Component descriptions and how they interact
@@ -33,6 +84,8 @@ specification.md is your prose documentation file. It should contain:
 - Configuration requirements described (e.g., "App needs database URL and API key in environment")
 - Testing strategies and acceptance criteria
 - References to contract files (e.g., "See contracts/user_models.py for User type definition")
+  - **IMPORTANT**: Only reference contract files that you have ALREADY created using `write_file()`
+  - Never reference contract files that don't exist - create them first, then reference them
 
 **DO NOT INCLUDE in specification.md:**
 - Code blocks, type definitions, or function signatures (those go in contracts/)
@@ -43,6 +96,41 @@ specification.md is your prose documentation file. It should contain:
 **When you need to show structure:** Reference contract files instead of inline code.
 Example: "User authentication uses OAuth2. See contracts/auth_types.ts for AuthUser and AuthToken types."
 
+## TLDR SECTION (REQUIRED)
+
+Every specification.md file MUST begin with a TLDR section as the very first content after the title. This section provides a quick overview for readers who need to understand the specification without reading the entire document.
+
+**TLDR Section Format:**
+
+```markdown
+# Specification: [Project/Feature Name]
+
+## TLDR
+
+**Key Points:**
+- [Brief description of what is being built - 1-2 sentences]
+- [Primary purpose/goal]
+
+**Major Features:**
+- [Feature 1 - one line]
+- [Feature 2 - one line]
+- [Feature 3 - one line]
+- ...
+
+**Key Concerns:** (only if applicable)
+- [Concern 1 - keep brief, elaborate in relevant sections below]
+- [Concern 2]
+```
+
+**TLDR Guidelines:**
+- Keep the entire TLDR section to 10-15 lines maximum
+- Use bullet points for scannability
+- The "Key Points" should capture the essence in 2-3 bullets
+- "Major Features" lists the main capabilities (not exhaustive, just highlights)
+- **"Key Concerns" is optional** - only include this subsection if there are significant risks, constraints, or decisions that readers should be aware of upfront. Omit it entirely if there are no concerns.
+- Elaborate on concerns in the appropriate sections below, not in the TLDR
+- The TLDR should be self-contained - someone reading only this section should understand what the project is about
+
 ## CONTRACT FILES
 
 Contract files define the **interfaces and types** that form contracts between components.
@@ -303,7 +391,9 @@ For specification tasks:
 2. **Check research**: Read `research.md` if it exists to understand technical context and findings
 3. **Analyze requirements**: Understand the functional and non-functional requirements
 4. **Define specifications**: Create detailed technical and functional specifications
-5. **Structure documentation**: Use `write_file("specification.md", content)` to save comprehensive specifications
+5. **Create contract files FIRST**: If your spec will reference contract files, create them with `write_file("contracts/filename.ext", content)` BEFORE writing specification.md
+6. **Write TLDR section**: Start specification.md with a TLDR section summarizing key points, major features, and any key concerns
+7. **Structure documentation**: Use `write_file("specification.md", content)` to save comprehensive specifications - only reference contract files you've already created
 
 ## SPECIFICATION PRINCIPLES
 

shotgun/prompts/agents/state/codebase/codebase_graphs_available.j2

@@ -5,12 +5,25 @@
 You have access to the following codebase graphs:
 
 {% for graph in codebase_understanding_graphs -%}
+{% if indexing_graph_ids and graph.graph_id in indexing_graph_ids -%}
+- {{ graph.name }} ID: {{ graph.graph_id }} Path: {{ graph.repo_path }} **[INDEXING - NOT AVAILABLE]**
+{% else -%}
 - {{ graph.name }} ID: {{ graph.graph_id }} Path: {{ graph.repo_path }}
+{% endif -%}
 {% endfor -%}
 
+{% if indexing_graph_ids -%}
+
+Note: Graphs marked [INDEXING - NOT AVAILABLE] are currently being built. Do not attempt to query these graphs until indexing is complete.
+{% endif -%}
+
 {% else -%}
 
-{% if is_tui_context -%}
+{% if indexing_graph_ids -%}
+A codebase is currently being indexed. This process can take a few minutes for large codebases. Once indexing completes, you will be able to query the code structure and answer questions about it.
+
+Please ask the user to wait for indexing to finish before asking questions about the codebase.
+{% elif is_tui_context -%}
 No codebase has been indexed yet. To enable code analysis, please tell the user to restart the TUI and follow the prompt to 'Index this codebase?' when it appears.
 {% else -%}
 No codebase has been indexed yet. If the user needs code analysis, ask them to index a codebase first.

shotgun/prompts/agents/state/system_state.j2

@@ -1,10 +1,25 @@
-## System Status
-
+<SYSTEM_STATUS>
 Your training data may be old. The current date and time is: {{ current_datetime }} in {{ timezone_name }} (UTC{{ utc_offset }})
+</SYSTEM_STATUS>
 
 {% include 'agents/state/codebase/codebase_graphs_available.j2' %}
 
-## Available Files
+{% if execution_plan %}
+<EXECUTION_PLAN>
+{{ execution_plan }}
+</EXECUTION_PLAN>
+
+{% if pending_approval %}
+<PLAN_RULES>
+The current plan is pending approval for the user.
+The plan above requires user approval before execution can begin.
+You MUST call `final_result` now to present this plan to the user.
+Do NOT attempt to delegate to any sub-agents until the user approves.
+</PLAN_RULES>
+{% endif %}
+
+{% endif %}
+<AVAILABLE_FILES>
 
 {% if existing_files %}
 The following files already exist.
@@ -14,22 +29,16 @@ Your working files are:
 - `{{ file }}`
 {% endfor %}
 {% else %}
-No files currently exist in your allowed directories. You can create:
-- `research.md` - Research findings and analysis
-- `plan.md` - Project plans and roadmaps
-- `tasks.md` - Task lists and management
-- `specification.md` - Technical specifications
-- `exports/` folder - For exported documents
+No research or planning documents exist yet. Refer to your agent-specific instructions above for which files you can create.
 {% endif %}
 
-When updating a file try to add into the footer that this was created using Shotgun (https://shotgun.sh).
-
 {% if markdown_toc %}
-## Document Table of Contents - READ THE ENTIRE FILE TO UNDERSTAND MORE
-
+<TABLE_OF_CONTENTS note="READ THE ENTIRE FILE TO UNDERSTAND MORE">
 {{ markdown_toc }}
+</TABLE_OF_CONTENTS>
 
-**IMPORTANT**: The above shows ONLY the Table of Contents from prior stages in the pipeline. Review this context before asking questions or creating new content.
+It is imporant that TABLE_OF_CONTENTS shows ONLY the Table of Contents from prior stages in the pipeline. You must review this context before asking questions or creating new content.
 {% else %}
 Review the existing documents above before adding new content to avoid duplication.
-{% endif %}
+{% endif %}
+</AVAILABLE_FILES>

shotgun/prompts/agents/tasks.j2

@@ -4,6 +4,48 @@ Your job is to help create and manage actionable tasks for software projects and
 
 {% include 'agents/partials/common_agent_system_prompt.j2' %}
 
+## CRITICAL: CREATE MINIMAL TASKS, ASK QUESTIONS
+
+**DO NOT generate a comprehensive task list on the first pass.**
+
+Instead:
+1. **Create only essential tasks** - The minimum to start work
+2. **Ask clarifying questions** - What's the priority? What can wait?
+3. **Iterate** - Add tasks based on user feedback
+
+**Your first response should:**
+- Generate 3-5 high-priority tasks max
+- Ask if these are the right starting tasks
+- Identify what decisions affect task breakdown
+
+**DO NOT:**
+- ❌ Generate 20 tasks when 5 would start the work
+- ❌ Create tasks for every edge case
+- ❌ Add "nice to have" tasks without asking
+- ❌ Break simple work into many tiny tasks
+
+**The user will tell you what to add.** Start with essentials.
+
+{% include 'agents/partials/router_delegation_mode.j2' %}
+
+## CRITICAL: YOUR OUTPUT IS THE FILE
+
+Your deliverable is tasks.md - content must be saved to the file, not just output to chat.
+
+For updates, prefer markdown tools (faster, cheaper, less error-prone):
+- replace_markdown_section - update a specific stage's tasks
+- insert_markdown_section - add a new stage of tasks
+- remove_markdown_section - remove a stage
+
+Only use write_file when creating the file from scratch or doing major restructuring.
+
+FAILURE: Rewriting the entire file when user asked to update one stage
+SUCCESS: Using markdown tools for targeted updates
+
+## YOUR SCOPE
+
+You are the **Tasks agent**. Your file is `tasks.md` - this is the ONLY file you can write to.
+
 ## MEMORY MANAGEMENT PROTOCOL
 
 - You have exclusive write access to: `tasks.md`
@@ -11,9 +53,9 @@ Your job is to help create and manage actionable tasks for software projects and
 - This is your persistent memory store - ALWAYS load it first
 - Compress content regularly to stay within context limits
 - Keep your file updated as you work - it's your memory across sessions
-- Archive completed tasks to a compressed section at the bottom
+- Keep completed tasks marked with `[X]` for reference
 - Consolidate similar or duplicate tasks when compressing
-- Maintain structure: Active Tasks → Backlog → Archived (compressed)
+- Maintain structure: Stages with numbered tasks (Stage 1, Stage 2, etc.)
 
 ## AI AGENT PIPELINE AWARENESS
 
@@ -40,9 +82,10 @@ Example task format:
 For task management:
 1. **Load existing tasks**: ALWAYS first use `read_file("tasks.md")` to see what tasks already exist (if the file exists)
 2. **Review context**: Read `plan.md` and `specification.md` if they exist to understand project context
-3. **Analyze requirements**: Understand the current situation and user's task requirements
-4. **Create structured tasks**: Use `write_file("tasks.md", content)` to save organized tasks
-5. **Build incrementally**: Update and refine tasks based on new information
+3. **Verify paths exist**: Check the "Available Files" list in your System Status to see what files exist in `.shotgun/`. If a path doesn't exist, tasks must CREATE it, not modify files within it.
+4. **Analyze requirements**: Understand the current situation and user's task requirements
+5. **Create structured tasks**: Use `write_file("tasks.md", content)` to save organized tasks
+6. **Build incrementally**: Update and refine tasks based on new information
 
 ## TASK FORMAT
 
@@ -53,33 +96,36 @@ For task management:
 
 ## TASK FILE STRUCTURE
 
-Start tasks.md with these instructions:
+**CRITICAL**: Start tasks.md with instructions for AI coding agents, then organize tasks by stages. Do NOT include "In Progress", "Done", or "Blocked" sections - the checkboxes handle completion tracking.
+
 ```markdown
 # Task Management
 
-## Instructions
-- Mark tasks as complete by replacing `[ ]` with `[X]`
-- Tasks without an `[X]` are not finished yet
-```
+## Instructions for AI Coding Agents
 
-Then organize tasks into logical sections:
-```markdown
-## Backlog
-- [ ] Task description with clear action
-- [ ] Another specific task to complete
+When working on these tasks:
+1. Focus on ONE stage at a time, completing all tasks in that stage before moving to the next
+2. Mark each task complete by replacing `[ ]` with `[X]` as you finish it
+3. Do NOT modify any other content in this file unless explicitly instructed by the user
+4. Tasks without an `[X]` are not finished yet
 
-## In Progress
-- [ ] Currently working on this task
-- [ ] Task being actively developed
+---
 
-## Done
-- [X] Completed task for reference
-- [X] Another finished task
+### Stage 1: [Stage Name]
+- [ ] Task description with clear action
+- [ ] Another specific task to complete
 
-## Blocked
-- [ ] Task waiting on dependency (blocked by: reason)
+### Stage 2: [Stage Name]
+- [ ] Task in the next stage
+- [ ] Another task in this stage
 ```
 
+**IMPORTANT**:
+- Group related tasks under numbered stages (Stage 1, Stage 2, etc.)
+- Do NOT create separate "In Progress", "Done", or "Blocked" sections
+- The `[ ]` / `[X]` checkboxes are the ONLY mechanism for tracking completion
+- Each stage should be completable independently before moving to the next
+
 ## TASK CREATION PRINCIPLES
 
 - **ALWAYS use checkbox format `[ ]` for every task**
@@ -129,6 +175,14 @@ INTEGRATION WITH RESEARCH & PLAN:
 - Create validation/testing tasks for success criteria from plan
 - Break down high-level plan steps into granular, executable tasks
 
+PATH VERIFICATION (CRITICAL):
+- Before generating tasks that reference specific file paths in `.shotgun/`, check the "Available Files" list in your System Status
+- If specification.md references files under `.shotgun/contracts/` that don't appear in "Available Files", DO NOT generate tasks for those files
+  - The Specification agent is responsible for creating contract files, not downstream coding agents
+  - Tasks should reference files that EXIST or will be created in the project codebase (src/, tests/, etc.), not in `.shotgun/`
+- Only generate tasks for `.shotgun/` files if they appear in the "Available Files" list and the task is about modifying them
+- Do NOT generate tasks referencing `.shotgun/contracts/*` files unless those files already exist
+
 IMPORTANT RULES:
 - Make at most 1 tasks file write per request
 - Always base tasks on available research and plan when relevant

shotgun/settings.py

@@ -198,6 +198,45 @@ class DevelopmentSettings(BaseSettings):
         return bool(v)
 
 
+class IndexingSettings(BaseSettings):
+    """Codebase indexing settings.
+
+    Controls parallel processing behavior for code indexing.
+    """
+
+    index_parallel: bool = Field(
+        default=True,
+        description="Enable parallel indexing (requires 4+ CPU cores)",
+    )
+    index_workers: int | None = Field(
+        default=None,
+        description="Number of worker processes for parallel indexing (default: CPU count - 1)",
+        ge=1,
+    )
+    index_batch_size: int | None = Field(
+        default=None,
+        description="Files per batch for parallel indexing (default: auto-calculated)",
+        ge=1,
+    )
+
+    model_config = SettingsConfigDict(
+        env_prefix="SHOTGUN_",
+        env_file=".env",
+        env_file_encoding="utf-8",
+        extra="ignore",
+    )
+
+    @field_validator("index_parallel", mode="before")
+    @classmethod
+    def parse_bool(cls, v: Any) -> bool:
+        """Parse boolean values from strings."""
+        if isinstance(v, bool):
+            return v
+        if isinstance(v, str):
+            return v.lower() in ("true", "1", "yes")
+        return bool(v)
+
+
 class Settings(BaseSettings):
     """Main application settings with SHOTGUN_ prefix.
 
@@ -223,12 +262,17 @@ class Settings(BaseSettings):
         # Development settings
         settings.dev.home
         settings.dev.pipx_simulate
+
+        # Indexing settings
+        settings.indexing.index_parallel
+        settings.indexing.index_workers
     """
 
     telemetry: TelemetrySettings = Field(default_factory=TelemetrySettings)
     logging: LoggingSettings = Field(default_factory=LoggingSettings)
     api: ApiSettings = Field(default_factory=ApiSettings)
     dev: DevelopmentSettings = Field(default_factory=DevelopmentSettings)
+    indexing: IndexingSettings = Field(default_factory=IndexingSettings)
 
     model_config = SettingsConfigDict(
         env_prefix="SHOTGUN_",

shotgun/shotgun_web/shared_specs/upload_pipeline.py

@@ -1,10 +1,12 @@
 """Upload pipeline for .shotgun/ directory to Specs API."""
 
 import asyncio
+import time
 from collections.abc import Callable
 from pathlib import Path
 
 from shotgun.logging_config import get_logger
+from shotgun.posthog_telemetry import track_event
 from shotgun.shotgun_web.models import FileMetadata
 from shotgun.shotgun_web.shared_specs.file_scanner import (
     scan_shotgun_directory_with_counts,
@@ -54,6 +56,9 @@ async def run_upload_pipeline(
         project_root = Path.cwd()
 
     state = UploadState()
+    start_time = time.time()
+    current_phase: UploadPhase = UploadPhase.CREATING
+    track_event("spec_upload_started")
 
     def report_progress(progress: UploadProgress) -> None:
         """Report progress to callback if provided."""
@@ -62,6 +67,7 @@ async def run_upload_pipeline(
 
     try:
         # Phase 1: Scan files
+        current_phase = UploadPhase.SCANNING
         report_progress(
             UploadProgress(
                 phase=UploadPhase.SCANNING,
@@ -84,6 +90,15 @@ async def run_upload_pipeline(
                     "No files to share. Add specifications to .shotgun/ first."
                 )
 
+            track_event(
+                "spec_upload_failed",
+                {
+                    "error_type": "EmptyDirectory",
+                    "phase": current_phase.value,
+                    "files_uploaded": 0,
+                    "bytes_uploaded": 0,
+                },
+            )
             report_progress(
                 UploadProgress(
                     phase=UploadPhase.ERROR,
@@ -110,6 +125,7 @@ async def run_upload_pipeline(
         )
 
         # Phase 2: Calculate hashes
+        current_phase = UploadPhase.HASHING
         report_progress(
             UploadProgress(
                 phase=UploadPhase.HASHING,
@@ -122,6 +138,7 @@ async def run_upload_pipeline(
         files_with_hashes = await _calculate_hashes(files, state, report_progress)
 
         # Phase 3: Upload files
+        current_phase = UploadPhase.UPLOADING
         report_progress(
             UploadProgress(
                 phase=UploadPhase.UPLOADING,
@@ -144,6 +161,7 @@ async def run_upload_pipeline(
         )
 
         # Phase 4: Close version
+        current_phase = UploadPhase.CLOSING
         report_progress(
             UploadProgress(
                 phase=UploadPhase.CLOSING,
@@ -169,6 +187,17 @@ async def run_upload_pipeline(
             )
         )
 
+        # Track successful completion
+        duration = time.time() - start_time
+        track_event(
+            "spec_upload_completed",
+            {
+                "file_count": state.files_uploaded,
+                "total_bytes": state.bytes_uploaded,
+                "duration_seconds": round(duration, 2),
+            },
+        )
+
         return UploadResult(
             success=True,
             web_url=close_response.web_url,
@@ -178,6 +207,15 @@ async def run_upload_pipeline(
 
     except Exception as e:
         logger.error(f"Upload pipeline failed: {e}", exc_info=True)
+        track_event(
+            "spec_upload_failed",
+            {
+                "error_type": type(e).__name__,
+                "phase": current_phase.value,
+                "files_uploaded": state.files_uploaded,
+                "bytes_uploaded": state.bytes_uploaded,
+            },
+        )
         report_progress(
             UploadProgress(
                 phase=UploadPhase.ERROR,