tasktree 0.0.19__tar.gz → 0.0.21__tar.gz

{tasktree-0.0.19 → tasktree-0.0.21}/.claude/settings.local.json

@@ -14,7 +14,10 @@
       "Bash(ls:*)",
       "Bash(cat:*)",
       "Bash(tt --help:*)",
-      "Bash(head:*)"
+      "Bash(head:*)",
+      "Bash(wc:*)",
+      "Bash(athena status:*)",
+      "Bash(athena sync:*)"
     ]
   }
 }

{tasktree-0.0.19 → tasktree-0.0.21}/.gitignore

@@ -154,5 +154,8 @@ example/output.txt
 example/archive.tar.gz
 example/.tasktree-state
 
-# UV lock file (optional - some prefer to commit this)
-# uv.lock
+# Requirements files copied from GitHub issues (for driving Claude Code)
+requirements/*
+
+.athena-cache
+

tasktree-0.0.21/CLAUDE.md

@@ -0,0 +1,220 @@
+# CLAUDE.md
+
+This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
+
+## Project Overview
+
+Task Tree (tt) is a task automation tool that combines simple command execution with intelligent dependency tracking and incremental execution. The project is a Python application built with a focus on:
+
+- **Intelligent incremental execution**: Tasks only run when necessary based on input changes, dependency updates, or task definition changes
+- **YAML-based task definition**: Tasks are defined in `tasktree.yaml` or `tt.yaml` files with dependencies, inputs, outputs, and commands
+- **Automatic input inheritance**: Tasks automatically inherit inputs from dependencies
+- **Parameterized tasks**: Tasks can accept typed arguments with defaults and constraints (int, float, bool, str, path, datetime, hostname, email, IP addresses)
+- **File imports**: Task definitions can be split across multiple files and namespaced
+- **Environment definitions**: Named execution environments (shell with custom configuration, or Docker containers with full image building support)
+- **Template substitution**: Rich variable system with access to arguments, environment variables, built-in variables, dependency outputs, and task inputs/outputs
+- **Docker support**: Full Docker integration with volume mounts, port mappings, build arguments, and user mapping
+- **Named inputs/outputs**: Reference specific inputs and outputs by name for better clarity and DRY principles
+
+## IMPORTANT! Development philosophies
+
+### Small, incremental changes
+The project team requires that each commit contains a small number of changes. Ideally, just the addition of a new line or statement in the code and an accompanying unit test (or tests) that validate the functionality of that line.
+
+Tickets and features should be iteratively broken down until they can be implemented as a series of small commits.
+
+When prompting the user to move to the next stage, estimate the size of the work and indicate whether you think there is sufficient usage to implement the next stage, or not. ("estimated required tokens: X, remaining usage in this session: Y tokens")
+
+#### Local Claude Code work 
+When working locally on a user's machine, Claude Code should NEVER make commits - only stop and ask the user to review and commit, before carrying on with the next incremental change.
+
+#### GitHub Claude Code integration work
+When working as a GitHub agent, claude should still BREAK DOWN THE TASK into small, incremental commits, but commit those changes to the feature branch as they are made. GitHub integration Claude Code does not need to ask for permission to commit each change.
+
+### Write tests, not ad hoc test scripts
+If you are checking that a feature you are implementing has been implemented correctly, DO NOT write a bespoke test script to check the output of the app with the new functionality. INSTEAD, write a unit/integration/end-to-end test that will confirm you have correctly implemented the feature and RUN JUST THAT TEST. If it passes, you have implemented things correctly; and you can either carry on with additional parts of the feature, or run all the tests to ensure no regressions.
+
+It is still permissible to write and run an ad hoc script to investigate/confirm the current behaviour. Although, it is better to first search for a test that does the thing that you're investigating. If one exists and is passing: then the app does the thing.
+
+### Testas we go!
+We do not plan to implement all the code (maybe even with unit tests) and then write a bunch of integration tests. We PLAN END-TO-END incremental changes. This will involve writing high-level test of the functionality as early as possible, to ensure that the new feature is progressing as expected.
+
+### Try to be efficient with token usage
+Your sponsor is not made of money! Try to minimise token useage, so that we can maximise the effectiveness of Claude Code on a features per token basis. Obviously, if a thing needs doing and it takes a bunch of tokens, that's just the way it is. Just try to consider/avoid profligacy!
+
+### Architectural philosophies
+
+- Try to follow SOLID principles
+- Try to follow the advice in "Clean Code", by Robert Martin.
+- Try to keep algorithmic logic abstracted from the TYPES that the logic can be run on. This is a restatement of the Liskov Substitution principle covered in the SOLID principles
+- **Small, named functions are preferred over comments**.  If a comment on WHAT the code is doing feels warranted, then refactor that code into a function with an indicative name.  Comment on WHY code is like it is are more permissible.
+
+## Architecture
+
+### Core Components
+
+- **`src/tasktree/parser.py`** (2,415 lines): YAML recipe parsing, task and environment definitions, circular import detection, schema validation
+- **`src/tasktree/executor.py`** (1,200 lines): Task execution logic, incremental execution engine, state tracking, built-in variables, subprocess management
+- **`src/tasktree/cli.py`** (591 lines): Typer-based CLI with commands: `--list`, `--show`, `--tree`, `--force`, `--only`, `--dry-run`, `--verbose`
+- **`src/tasktree/graph.py`** (545 lines): Dependency resolution using graphlib.TopologicalSorter, parameterized dependencies, cycle detection
+- **`src/tasktree/docker.py`** (446 lines): Docker image building and container execution, user mapping, volume mounts, build args
+- **`src/tasktree/substitution.py`** (374 lines): Template variable substitution engine supporting multiple prefixes (var, arg, env, tt, dep, self)
+- **`src/tasktree/types.py`** (139 lines): Custom Click parameter types for argument validation (hostname, email, IP, IPv4, IPv6, datetime)
+- **`src/tasktree/hasher.py`** (161 lines): Task hashing for incremental execution, cache key generation, environment definition hashing
+- **`src/tasktree/state.py`** (119 lines): State file management (.tasktree-state), task execution state tracking
+
+### Key Dependencies
+
+- **PyYAML**: For recipe parsing
+- **Typer, Click, Rich**: For CLI and rich terminal output
+- **graphlib.TopologicalSorter**: For dependency resolution
+- **pathlib**: For file operations and glob expansion
+- **docker (Python SDK)**: For Docker image building and container management
+- **jsonschema**: For YAML schema validation
+
+## Development Commands
+
+## IMPORTANT! Tool use
+### Athena Code Knowledge
+This repo uses Athena to store and retrieve knowledge about the codebase. YOU SHOULD USE THE TOOL TO MAKE INQUIRIES ABOUT THE CODE.
+
+1. Find a function/class: `athena locate <function/class name>`
+2. Find information about what a function/class does:
+   - `athena info path/to/module.py:ClassName`
+   - `athena info path/to/module.py:ClassName.method`
+   - `athena info path/to/module.py:some_function`
+3. Search for relevant code information:
+   - `athena serch "generate JWT authentication tokens`
+4. AFTER code changes, run `athena status` to check which docs should have been updated
+5. AFTER checking/updating docs, run `athena sync` to register the changes
+
+**prefer this workflow to using native `grep` and `find` tools** for understanding the code.
+
+### Testing
+```bash
+python3 -m pytest tests/
+```
+The project has tests across three categories:
+- **Unit tests** (`tests/unit/`): Should always be updated for any change. Makes up the bulk of code coverage
+- **Integration tests** (`tests/integration/`): Test changes using the Typer CliRunner to cover end-to-end workflows across multiple modules
+- **E2E tests** (`tests/e2e/`): Heavyweight tests that cover running the tool in a subprocess and/or containerized environment 
+
+### Running the Application
+```bash
+python3 main.py [task-names] [--options]
+```
+
+### Package Management
+This project uses `uv` for dependency management (indicated by `uv.lock` file). Configuration is in `pyproject.toml`.
+
+## State Management
+
+The application uses a `.tasktree-state` file at the project root to track:
+- When tasks last ran
+- Timestamps of input files at execution time
+- Task hashes based on command, outputs, working directory, arguments, and environment definitions
+- Cached results for incremental execution
+
+## Testing Approach
+
+The project uses Python's built-in `unittest` framework with:
+- `unittest.mock` for mocking subprocess calls and external dependencies
+- `click.testing.CliRunner` for CLI testing
+- Comprehensive coverage across unit, integration, and E2E test layers
+- Tests verify subprocess calls, Docker operations, state management, and CLI behavior
+
+## Task Definition Format
+
+Tasks are defined in YAML with the following structure:
+```yaml
+tasks:
+  task-name:
+    desc: Description (optional)
+    deps: [dependency-tasks]  # Can include parameterized dependencies: dep-name(arg1, key=value)
+    inputs:
+      - pattern1  # Anonymous glob patterns
+      - name: glob-pattern  # Named inputs for reference
+    outputs:
+      - pattern1  # Anonymous glob patterns
+      - name: path-or-pattern  # Named outputs for reference
+    working_dir: execution-directory
+    env: environment-name  # Reference to environment definition
+    args:
+      - name: arg-name
+        type: str|int|float|bool|path|datetime|hostname|email|ip|ipv4|ipv6
+        default: value
+        choices: [option1, option2]  # Optional constraint
+        min: value  # Optional for numeric types
+        max: value  # Optional for numeric types
+        exported: true  # Export as $ARG_NAME environment variable
+    cmd: shell-command  # Can use {{ var.name }}, {{ arg.name }}, {{ env.NAME }}, {{ tt.* }}, {{ dep.task.outputs.name }}, {{ self.inputs.name }}
+    private: true  # Hide from --list but still executable
+
+environments:
+  env-name:
+    default: true  # Make this the default environment
+    shell: /bin/bash  # Shell environment
+    # OR
+    dockerfile: path/to/Dockerfile  # Docker environment
+    context: build-context-dir
+    image: optional-image-name
+    volumes:
+      - host_path:container_path[:ro]
+    ports:
+      - "host:container"
+    build_args:
+      ARG_NAME: value
+    environment:
+      ENV_VAR: value
+
+variables:
+  var-name: value  # Simple string value
+  var-from-env: { env: ENV_VAR, default: fallback }  # From environment
+  var-from-eval: { eval: "command to run" }  # Runtime command evaluation
+  var-from-file: { read: path/to/file }  # Read file contents
+```
+
+## Built-in Variables
+
+Tasks have access to these built-in template variables:
+- `{{ tt.project_root }}`: Root directory of the project
+- `{{ tt.recipe_dir }}`: Directory containing the recipe file
+- `{{ tt.task_name }}`: Name of the current task
+- `{{ tt.working_dir }}`: Working directory for task execution
+- `{{ tt.timestamp }}`: ISO 8601 timestamp
+- `{{ tt.timestamp_unix }}`: Unix timestamp
+- `{{ tt.user_home }}`: User's home directory
+- `{{ tt.user_name }}`: Current username
+
+## Key Features
+
+### Template Substitution
+Commands and paths support template substitution with multiple prefixes:
+- `{{ var.name }}`: Variables defined in the `variables` section
+- `{{ arg.name }}`: Task arguments passed on command line
+- `{{ env.NAME }}`: Environment variables
+- `{{ tt.* }}`: Built-in variables (see above)
+- `{{ dep.task_name.outputs.output_name }}`: Outputs from dependency tasks
+- `{{ self.inputs.input_name }}`: Named inputs of the current task
+- `{{ self.outputs.output_name }}`: Named outputs of the current task
+
+### Parameterized Dependencies
+Tasks can pass arguments to their dependencies:
+```yaml
+tasks:
+  caller:
+    deps:
+      - dependency(value1, key=value2)  # Positional and named args
+```
+
+### Docker Integration
+Full Docker support with:
+- Dockerfile-based image building
+- Volume mounts (read-only and read-write)
+- Port mappings
+- User mapping (run as non-root on Unix/macOS)
+- Build arguments separate from shell arguments
+- Environment variable injection
+
+### Schema Validation
+The project includes JSON Schema definitions in `schema/` for validating recipe YAML files.

{tasktree-0.0.19 → tasktree-0.0.21}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: tasktree
-Version: 0.0.19
+Version: 0.0.21
 Summary: A task automation tool with incremental execution
 Requires-Python: >=3.11
 Requires-Dist: click>=8.1.0
@@ -833,6 +833,369 @@ Hint: Define named outputs like: outputs: [{ missing: 'path/to/file' }]
 - **Deployment pipelines**: Reference exact artifacts to deploy
 - **Configuration propagation**: Pass generated config files through build stages
 
+### Self-References
+
+Tasks can reference their own inputs and outputs using `{{ self.inputs.name }}` (named access) or `{{ self.inputs.0 }}` (positional access) templates. This eliminates repetition when paths contain variables or when tasks have multiple inputs/outputs.
+
+**Named Inputs and Outputs:**
+
+Just like dependency output references, inputs and outputs can have names:
+
+```yaml
+tasks:
+  process:
+    inputs:
+      - src: "data/input.json"      # Named input
+      - config: "config.yaml"        # Named input
+    outputs:
+      - result: "output/result.json" # Named output
+      - log: "output/process.log"    # Named output
+    cmd: |
+      process-tool \
+        --input {{ self.inputs.src }} \
+        --config {{ self.inputs.config }} \
+        --output {{ self.outputs.result }} \
+        --log {{ self.outputs.log }}
+```
+
+**Syntax:**
+
+- **Defining named inputs**: `inputs: [{ name: "path/to/file" }]`
+- **Defining named outputs**: `outputs: [{ name: "path/to/file" }]`
+- **Defining anonymous inputs**: `inputs: ["path/to/file"]`
+- **Defining anonymous outputs**: `outputs: ["path/to/file"]`
+- **Referencing by name**: `{{ self.inputs.input_name }}` or `{{ self.outputs.output_name }}`
+- **Referencing by index**: `{{ self.inputs.0 }}` or `{{ self.outputs.1 }}` (0-based)
+- **Mixed format**: Can combine named and anonymous inputs/outputs in the same task
+
+**Why Use Self-References?**
+
+Self-references follow the DRY (Don't Repeat Yourself) principle:
+
+```yaml
+# Without self-references - repetitive
+tasks:
+  build:
+    inputs: [src/app-{{ var.version }}.c]
+    outputs: [build/app-{{ var.version }}.o]
+    cmd: gcc src/app-{{ var.version }}.c -o build/app-{{ var.version }}.o
+
+# With self-references - DRY
+tasks:
+  build:
+    inputs:
+      - source: src/app-{{ var.version }}.c
+    outputs:
+      - object: build/app-{{ var.version }}.o
+    cmd: gcc {{ self.inputs.source }} -o {{ self.outputs.object }}
+```
+
+**Working with Variables:**
+
+Self-references work seamlessly with variables:
+
+```yaml
+variables:
+  platform: linux
+  arch: x86_64
+
+tasks:
+  compile:
+    inputs:
+      - src: src/{{ var.platform }}/main.c
+      - header: include/{{ var.arch }}/defs.h
+    outputs:
+      - binary: build/{{ var.platform }}-{{ var.arch }}/app
+    cmd: |
+      gcc {{ self.inputs.src }} \
+        -include {{ self.inputs.header }} \
+        -o {{ self.outputs.binary }}
+```
+
+Variables are evaluated first, then self-references substitute the expanded paths.
+
+**Working with Arguments:**
+
+Self-references work with parameterized tasks:
+
+```yaml
+tasks:
+  deploy:
+    args: [environment]
+    inputs:
+      - config: configs/{{ arg.environment }}/app.yaml
+    outputs:
+      - deployed: deployed-{{ arg.environment }}.yaml
+    cmd: |
+      validate {{ self.inputs.config }}
+      deploy {{ self.inputs.config }} > {{ self.outputs.deployed }}
+```
+
+```bash
+tt deploy production  # Uses configs/production/app.yaml
+tt deploy staging     # Uses configs/staging/app.yaml
+```
+
+**Working with Dependency Outputs:**
+
+Self-references and dependency references can be used together:
+
+```yaml
+tasks:
+  build:
+    outputs:
+      - artifact: dist/app.js
+    cmd: webpack build
+
+  package:
+    deps: [build]
+    inputs:
+      - manifest: package.json
+    outputs:
+      - tarball: release.tar.gz
+    cmd: tar czf {{ self.outputs.tarball }} \
+           {{ self.inputs.manifest }} \
+           {{ dep.build.outputs.artifact }}
+```
+
+**Mixed Named and Anonymous:**
+
+Tasks can mix named and anonymous inputs/outputs:
+
+```yaml
+tasks:
+  build:
+    inputs:
+      - config: build.yaml  # Named - can reference
+      - src/**/*.c          # Anonymous - tracked but not referenceable
+    outputs:
+      - binary: bin/app     # Named - can reference
+      - bin/app.debug       # Anonymous - tracked but not referenceable
+    cmd: build-tool --config {{ self.inputs.config }} --output {{ self.outputs.binary }}
+```
+
+**Positional Index References:**
+
+In addition to named references, you can access inputs and outputs by their positional index using `{{ self.inputs.0 }}`, `{{ self.inputs.1 }}`, etc. This provides an alternative way to reference items, especially useful for:
+
+- **Anonymous inputs/outputs**: Reference items that don't have names
+- **Simple sequential access**: When order is more important than naming
+- **Mixed with named access**: Use both styles in the same task
+
+**Syntax:**
+
+```yaml
+tasks:
+  process:
+    inputs: ["file1.txt", "file2.txt", "file3.txt"]
+    outputs: ["output1.txt", "output2.txt"]
+    cmd: |
+      cat {{ self.inputs.0 }} {{ self.inputs.1 }} > {{ self.outputs.0 }}
+      cat {{ self.inputs.2 }} > {{ self.outputs.1 }}
+```
+
+Indices follow YAML declaration order, starting from 0 (zero-based indexing):
+- First input/output = index 0
+- Second input/output = index 1
+- Third input/output = index 2, etc.
+
+**Works with Both Named and Anonymous:**
+
+```yaml
+tasks:
+  build:
+    inputs:
+      - config: "build.yaml"  # Index 0, also accessible as {{ self.inputs.config }}
+      - "src/**/*.c"          # Index 1, ONLY accessible as {{ self.inputs.1 }}
+      - headers: "include/*.h" # Index 2, also accessible as {{ self.inputs.headers }}
+    outputs:
+      - "dist/app.js"         # Index 0
+      - bundle: "dist/bundle.js" # Index 1, also accessible as {{ self.outputs.bundle }}
+    cmd: |
+      # Mix positional and named references
+      build-tool \
+        --config {{ self.inputs.0 }} \
+        --sources {{ self.inputs.1 }} \
+        --headers {{ self.inputs.headers }} \
+        --output {{ self.outputs.bundle }}
+```
+
+**Same Item, Multiple Ways:**
+
+Named items can be accessed by both name and index:
+
+```yaml
+tasks:
+  copy:
+    inputs:
+      - source: data.txt
+    cmd: |
+      # These are equivalent:
+      cat {{ self.inputs.source }} > copy1.txt
+      cat {{ self.inputs.0 }} > copy2.txt
+```
+
+**With Variables and Arguments:**
+
+Positional references work with variable-expanded paths:
+
+```yaml
+variables:
+  version: "1.0"
+
+tasks:
+  package:
+    args: [platform]
+    inputs:
+      - "dist/app-{{ var.version }}.js"
+      - "dist/lib-{{ arg.platform }}.so"
+    outputs: ["release-{{ var.version }}-{{ arg.platform }}.tar.gz"]
+    cmd: tar czf {{ self.outputs.0 }} {{ self.inputs.0 }} {{ self.inputs.1 }}
+```
+
+**Index Boundaries:**
+
+Indices are validated before execution:
+
+```yaml
+tasks:
+  build:
+    inputs: ["file1.txt", "file2.txt"]
+    cmd: cat {{ self.inputs.5 }}  # Error: index 5 out of bounds!
+```
+
+Error message:
+```
+Task 'build' references input index '5' but only has 2 inputs (indices 0-1)
+```
+
+**Empty Lists:**
+
+Referencing an index when no inputs/outputs exist:
+
+```yaml
+tasks:
+  generate:
+    cmd: echo "test" > {{ self.outputs.0 }}  # Error: no outputs defined!
+```
+
+Error message:
+```
+Task 'generate' references output index '0' but has no outputs defined
+```
+
+**When to Use Index References:**
+
+- **Anonymous items**: Only way to reference inputs/outputs without names
+- **Order-based processing**: When the sequence matters more than naming
+- **Simple tasks**: Quick access without defining names
+- **Compatibility**: Accessing items in legacy YAML that uses anonymous format
+
+**When to Use Named References:**
+
+- **Clarity**: Names make commands more readable (`{{ self.inputs.config }}` vs `{{ self.inputs.2 }}`)
+- **Maintainability**: Adding/removing items doesn't break indices
+- **Complex tasks**: Many inputs/outputs are easier to manage with names
+
+**Error Messages:**
+
+If you reference a non-existent input or output:
+
+```yaml
+tasks:
+  build:
+    inputs:
+      - src: input.txt
+    cmd: cat {{ self.inputs.missing }}  # Error!
+```
+
+You'll get a clear error before execution:
+
+```
+Task 'build' references input 'missing' but has no input named 'missing'.
+Available named inputs: src
+Hint: Define named inputs like: inputs: [{ missing: 'path/to/file' }]
+```
+
+Similarly, if you try to reference an anonymous input:
+
+```yaml
+tasks:
+  build:
+    inputs: [file.txt]  # Anonymous input
+    cmd: cat {{ self.inputs.src }}  # Error!
+```
+
+You'll get:
+
+```
+Task 'build' references input 'src' but has no input named 'src'.
+Available named inputs: (none - all inputs are anonymous)
+Hint: Define named inputs like: inputs: [{ src: 'file.txt' }]
+```
+
+**Key Behaviors:**
+
+- **Two access methods**: Reference by name (`{{ self.inputs.name }}`) or by index (`{{ self.inputs.0 }}`)
+- **Template resolution**: Self-references are resolved during dependency graph planning
+- **Substitution order**: Variables → Dependency outputs → Self-references → Arguments/Environment
+- **Fail-fast validation**: Errors are caught before execution begins (missing names, out-of-bounds indices)
+- **Clear error messages**: Lists available names/indices if reference doesn't exist
+- **Backward compatible**: Existing anonymous inputs/outputs work unchanged
+- **State tracking**: Works correctly with incremental execution and freshness checks
+- **Index order**: Positional indices follow YAML declaration order (0-based)
+
+**Limitations:**
+
+- **Anonymous not referenceable by name**: Anonymous inputs/outputs cannot be referenced by name (use positional index instead: `{{ self.inputs.0 }}`)
+- **Case sensitive**: `{{ self.inputs.Src }}` and `{{ self.inputs.src }}` are different
+- **Argument defaults**: Self-references in argument defaults are not supported (arguments are evaluated before self-references)
+- **No negative indices**: Python-style negative indexing (`{{ self.inputs.-1 }}`) is not supported
+
+**Use Cases:**
+
+- **Eliminate repetition**: Define complex paths once, use them multiple times
+- **Variable composition**: Combine variables with self-references for clean commands
+- **Multiple inputs/outputs**: Reference specific files when tasks have many
+- **Complex build pipelines**: Keep commands readable with named artifacts
+- **Glob patterns**: Use self-references with glob patterns for dynamic inputs
+
+**Example: Multi-Stage Build:**
+
+```yaml
+variables:
+  version: "2.1.0"
+  platform: "linux"
+
+tasks:
+  prepare:
+    outputs:
+      - builddir: build/{{ var.platform }}-{{ var.version }}
+    cmd: mkdir -p {{ self.outputs.builddir }}
+
+  compile:
+    deps: [prepare]
+    inputs:
+      - source: src/main.c
+      - headers: include/*.h
+    outputs:
+      - object: build/{{ var.platform }}-{{ var.version }}/main.o
+    cmd: |
+      gcc -c {{ self.inputs.source }} \
+        -I include \
+        -o {{ self.outputs.object }}
+
+  link:
+    deps: [compile]
+    outputs:
+      - executable: build/{{ var.platform }}-{{ var.version }}/app
+      - symbols: build/{{ var.platform }}-{{ var.version }}/app.sym
+    cmd: |
+      gcc build/{{ var.platform }}-{{ var.version }}/main.o \
+        -o {{ self.outputs.executable }}
+      objcopy --only-keep-debug {{ self.outputs.executable }} {{ self.outputs.symbols }}
+```
+
 
 ### Private Tasks