language-operator 0.1.31 → 0.1.35

data/requirements/SCRATCH.md

@@ -0,0 +1,153 @@
+# Knowledge Base
+
+Living document of critical insights, patterns, and gotchas for this codebase.
+
+## DSL Architecture (v1 - Current)
+
+**Core Model:** Task/Main (imperative, replacing declarative workflow/step)
+
+**Key Components:**
+- `TaskDefinition`: Organic functions with stable contracts (inputs/outputs), evolving implementations (neural→symbolic)
+- `MainDefinition`: Imperative entry point using Ruby control flow + `execute_task()`
+- `TypeSchema`: 7-type system (string, integer, number, boolean, array, hash, any)
+
+**Migration Strategy:**
+- DSL v0 (workflow/step) marked deprecated but fully functional
+- Both models supported in schema generation for backward compatibility
+- Deprecation clearly noted in descriptions, safe methods updated
+
+## Testing Patterns
+
+**RSpec Best Practices:**
+- Use single-quoted heredocs (`<<~'RUBY'`) when testing code with interpolation to avoid context issues
+- RuboCop requires uppercase annotation keywords with colon+space (e.g., `# TODO: fix`)
+- Symbol hash keys: Use `.keys.first.to_s` or `.values.first` for pattern properties, not direct string access
+
+**Parser Gem Quirks:**
+- Very forgiving - accepts syntax variations Ruby rejects
+- Makes syntax error testing difficult (2 pending tests skipped for this reason)
+- AST validation works well for semantic checks, less so for syntactic ones
+
+## Schema Generation
+
+**JSON Schema Patterns:**
+- `patternProperties` for flexible parameter validation without enumeration
+- Regex patterns as keys validate both names and types dynamically
+- Always include `examples` for complex schemas (aids understanding)
+
+## Security (AST Validator)
+
+**Safe Methods Lists:**
+- DSL v1: `task`, `main`, `execute_task`, `inputs`, `outputs`, `instructions`
+- DSL v0: Removed `workflow`, `step`, `depends_on`, `prompt`
+- Helpers: Added `TypeCoercion` for validation
+
+**Blocked Patterns:**
+- System execution: `system`, `exec`, `spawn`, `fork`
+- Dynamic evaluation: `eval`, `instance_eval`, `class_eval`, `send`
+- File operations: Direct `File` access, dangerous IO
+- Works in both task blocks and main blocks
+
+## Critical File Map
+
+| File | Purpose | Complexity |
+|------|---------|------------|
+| `lib/language_operator/dsl/schema.rb` | JSON Schema generation (DSL→schema) | High (1100+ lines) |
+| `lib/language_operator/dsl/task_definition.rb` | Task contract+validation | Medium (316 lines) |
+| `lib/language_operator/dsl/main_definition.rb` | Main block execution | Low (115 lines) |
+| `lib/language_operator/agent/task_executor.rb` | Neural/symbolic task execution | Medium (233 lines) |
+| `lib/language_operator/agent/safety/ast_validator.rb` | Code security validation | High |
+| `spec/language_operator/dsl/schema_spec.rb` | Schema test coverage (186 tests) | High |
+| `spec/language_operator/agent/task_executor_spec.rb` | Task executor tests (19 tests) | Medium |
+
+## Current Status
+
+**Completed (2025-11-14):**
+- ✅ Issue #26: Schema generation for task/main model
+- ✅ Issue #25: AST validator updated for DSL v1
+- ✅ Issues #21-23: TaskDefinition, MainDefinition, TypeCoercion implemented
+- ✅ Issue #28: TaskExecutor for task execution runtime
+- ✅ Issue #32 (partial): DependencyGraph and ParallelExecutor for implicit parallelism
+
+**Test Suite Health:**
+- 135 examples, 0 failures, 2 pending (syntax error tests)
+- 186 schema-specific tests, all passing
+- 19 TaskExecutor tests, all passing
+- 20 DependencyGraph tests, all passing
+- 11 ParallelExecutor tests, all passing
+- RuboCop clean
+
+## Task Execution (DSL v1)
+
+**Neural Task Flow:**
+1. TaskExecutor builds prompt from task instructions + inputs + output schema
+2. LLM called via `agent.send_message` with full tool access
+3. Response parsed as JSON (supports ```json blocks or raw objects)
+4. Outputs validated against schema via TaskDefinition#validate_outputs
+5. Fail fast on any error (critical for re-synthesis)
+
+**Symbolic Task Flow:**
+1. TaskExecutor calls TaskDefinition#call with inputs and self as context
+2. TaskDefinition validates inputs, executes code block, validates outputs
+3. Context provides `execute_task`, `execute_llm`, `execute_tool` helpers
+4. Fail fast on any error
+
+**Runtime Wiring:**
+- Agent module detects DSL v1 (main block) vs v0 (workflow)
+- Autonomous mode: `execute_main_block` creates TaskExecutor, calls MainDefinition
+- Scheduled mode: `Scheduler#start_with_main` creates TaskExecutor, schedules main
+- MainDefinition receives TaskExecutor as execution context via instance_exec
+
+## Parallel Execution (DSL v1)
+
+**Architecture:**
+- DependencyGraph: AST-based analysis extracts task dependencies from main block
+- ParallelExecutor: Level-based execution using Concurrent::FixedThreadPool
+- Default pool size: 4 threads (configurable)
+
+**How It Works:**
+1. Parse main block code to extract `execute_task` calls
+2. Build dependency graph based on variable flow (which tasks use outputs from which other tasks)
+3. Assign execution levels via topological sort
+4. Execute each level in parallel (all tasks in level run concurrently)
+5. Wait for level completion before starting next level
+
+**Performance:**
+- Measured 2x speedup for I/O-bound parallel tasks
+- Thread pool handles > pool size tasks gracefully
+- Fail-fast error handling (collects all errors, raises RuntimeError)
+
+**Current Status (2025-11-14):**
+- ✅ DependencyGraph: Complete and tested (20 tests)
+- ✅ ParallelExecutor: Complete and tested (11 tests)
+- ⚠️  Integration: Partial - blocked on variable-to-result mapping
+
+**Blocking Issue:**
+The fundamental challenge is mapping variable names from code to task results:
+```ruby
+# User code:
+s1 = execute_task(:fetch1)
+merged = execute_task(:merge, inputs: { s1: s1 })
+
+# ParallelExecutor passes: { fetch1: {...} }
+# But merge expects: { s1: {...} }
+```
+
+**Solution Options:**
+1. Enhanced AST analysis (complex, 2-3 days)
+2. Naming convention: var name = task name (simple, 1 day)
+3. Explicit dependency DSL (medium, 1-2 days)
+4. Defer to follow-up issue (pragmatic, 0 days)
+
+**Recommendation:** Option 4 - defer integration, ship infrastructure
+
+## Quick Wins / Common Gotchas
+
+1. **Hash Key Access:** Ruby symbols ≠ strings. Always check key types in tests.
+2. **Heredoc Interpolation:** Use `'RUBY'` (single quotes) to prevent RSpec context leakage.
+3. **Pattern Properties:** Schema validation via regex - powerful for type systems.
+4. **Migration-Friendly:** Keep deprecated features functional with clear warnings.
+5. **Parser Tolerance:** Don't rely on parser for syntax validation - it's too forgiving.
+6. **Tool Execution:** Tools accessed via LLM interface, not direct RPC (execute_tool → execute_llm)
+7. **Error Wrapping:** TaskExecutor wraps errors in RuntimeError with task context for debugging
+8. **Concurrent Ruby Futures:** Use `future.wait` + `future.rejected?` to check status, not `rescue` around `future.value`

data/requirements/features

@@ -0,0 +1 @@
+../../requirements/features

data/requirements/personas

@@ -0,0 +1 @@
+../../requirements/personas

data/requirements/proposals

@@ -0,0 +1 @@
+../../requirements/proposals

data/requirements/tasks/iterate.md

@@ -1,35 +1,34 @@
 # Task
 
-## Persona
-
-Adopt the [ruby-engineer](../../../requirements/personas/ruby-engineer.md) persona while executing these instructions, please.
+## Prerequisites
 
-## Inputs
+Please read the following context files:
 
-- id int -- A GitHub issue index ID.
+* Persona: requirements/personas/ruby-engineer.md
+* Feature Spec: requirements/proposals/dsl-v1.md
+* Scratch: requirements/SCRATCH.md
 
-## Background
+## Persona
 
-This is a early-phase project that works exclusively in main.
-Issues are found using the `gh` command for this project:
-- Owner: language-operator
-- Repository: language-operator-gem
+**CRITICAL**: Adopt the ruby-engineer persona while executing these instructions, please.
 
 ## Instructions
 
 Follow these directions closely:
 
-1. Use the ForgeJo tool to find the top issue for this repository.
+1. Use the `gh` tool to find the top issue for this repository (language-operator/language-operator-gem) with the "ready" label.
 2. Investigate if it's valid, or a mis-use of the intended feature.
 3. **CRITICAL:** Switch to plan mode, and propose an implementation plan.  Await my feedback.
 4. Add your implementation plan as a comment on the issue.
-5. Implement the changes.
+5. Implement your plan.
 6. Run existing tests, and add new ones if necessary.  Remember to include CI. Remember the linter and that bundler will fail if it's out of sync with its lockfile.
 7. **CRITICAL:** Test the actual functionality manually before committing. If it's a CLI command, run it. If it's library code, test it in the appropriate context. Never commit untested code.
-8. Commit the change and push to origin.
+8. Commit the change with a semantic, ONE LINE message, like 'feat: create task_definition structure'.
 9. **CRITICAL:** Halt while CI runs and await my feedback.
-10. Comment on your solution in the ForgeJo issue.
-11. Resolve the issue.
+10. Add resolution details as a comment on the GitHub issue.
+11. Resolve the GitHub issue.
+
+Consider if you need to update requirements/SCRATCH.md for the next run.
 
 ## Output
 

data/requirements/tasks/optimize.md

@@ -5,17 +5,26 @@
 
 Optimize
 
+## Inputs
+
+- :persona string - the persona to adopt when executing this task (default: ruby-engineer)
+
 ## Persona
 
-Adopt the [ruby-engineer](../requirements/personas/ruby-engineer.md) persona while executing these instructions, please.
+Adopt the `requirements/personas/:persona.md` persona while executing these instructions, please.
 
 ## Instructions
 
-Suggest an improvement that could improve the quality of the codebase or developer experience.  Things like opportunities to reduce lines of code, DRYing up code, or eliminating dead code paths.
+Suggest an improvement that could improve the quality of the codebase or developer experience.  Things like:
+- opportunities to reduce lines of code
+- DRYing up code
+- Dead code paths
+- Duplicate utility implementations
+- Magic strings
+- Other forms of tech debt
 
 An important thing to consider is that this code has been written by different agents with different contexts, who may not have been aware of overall patterns.  These kinds of optimizations are high priority.
 
 ## Output
 
-Switch to Plan mode.
-A proposed list of three concrete optimizations.
+Propose ONE high-impact optimization or refactor.

data/synth/001/Makefile

@@ -0,0 +1,90 @@
+.PHONY: synthesize synthesize-all synthesize-sonnet synthesize-gpt-4 run run-docker validate clean compare help
+
+# Default model - inherit from SYNTHESIS_MODEL env var, no hardcoded fallback
+MODEL ?= $(SYNTHESIS_MODEL)
+
+# Model-specific names
+SONNET_MODEL = claude-3-7-sonnet-20250219
+GPT4_MODEL = gpt-4-turbo
+
+help:
+	@echo "Synthesis Test Targets:"
+	@echo "  make synthesize         - Generate agent.rb using default model"
+	@echo "  make synthesize-all     - Generate for all configured models"
+	@echo "  make synthesize-sonnet  - Generate agent.sonnet.rb"
+	@echo "  make synthesize-gpt-4   - Generate agent.gpt-4.rb"
+	@echo "  make run                - Execute agent.rb locally with bundle exec"
+	@echo "  make run-docker         - Execute agent.rb in Docker container"
+	@echo "  make validate           - Validate agent.rb syntax"
+	@echo "  make clean              - Remove all generated .rb files"
+	@echo "  make compare            - Compare outputs from different models"
+
+synthesize:
+	@echo "Synthesizing agent.rb with model: $(MODEL)..."
+	@bundle exec ruby -I ../../lib -e "\
+	  require 'language_operator/synthesis_test_harness'; \
+	  harness = LanguageOperator::SynthesisTestHarness.new(model: '$(MODEL)'); \
+	  code = harness.synthesize('agent.yaml'); \
+	  File.write('agent.rb', code); \
+	  puts 'Generated: agent.rb'"
+
+synthesize-sonnet:
+	@echo "Synthesizing with Claude Sonnet..."
+	@bundle exec ruby -I ../../lib -e "\
+	  require 'language_operator/synthesis_test_harness'; \
+	  harness = LanguageOperator::SynthesisTestHarness.new(model: '$(SONNET_MODEL)'); \
+	  code = harness.synthesize('agent.yaml'); \
+	  File.write('agent.sonnet.rb', code); \
+	  puts 'Generated: agent.sonnet.rb'"
+
+synthesize-gpt-4:
+	@echo "Synthesizing with GPT-4..."
+	@bundle exec ruby -I ../../lib -e "\
+	  require 'language_operator/synthesis_test_harness'; \
+	  harness = LanguageOperator::SynthesisTestHarness.new(model: '$(GPT4_MODEL)'); \
+	  code = harness.synthesize('agent.yaml'); \
+	  File.write('agent.gpt-4.rb', code); \
+	  puts 'Generated: agent.gpt-4.rb'"
+
+synthesize-all: synthesize-sonnet synthesize-gpt-4
+	@echo "All synthesis complete!"
+
+run:
+	@if [ ! -f agent.rb ]; then \
+	  echo "Error: agent.rb not found. Run 'make synthesize' first."; \
+	  exit 1; \
+	fi
+	@echo "Executing agent.rb in Docker container..."
+	@docker run --rm -i \
+	  --network host \
+	  -e AGENT_NAME=hello-world \
+	  -e AGENT_MODE=autonomous \
+	  -e AGENT_CODE_PATH=/agent/agent.rb \
+	  -e LLM_MODEL=$(SYNTHESIS_MODEL) \
+	  -e MODEL_ENDPOINTS=$(SYNTHESIS_ENDPOINT) \
+	  -e OPENAI_API_KEY=$(SYNTHESIS_API_KEY) \
+	  -e ANTHROPIC_API_KEY=$(ANTHROPIC_API_KEY) \
+	  -v $(PWD)/agent.rb:/agent/agent.rb:ro \
+	  ghcr.io/language-operator/agent:dev
+
+validate:
+	@if [ ! -f agent.rb ]; then \
+	  echo "Error: agent.rb not found. Run 'make synthesize' first."; \
+	  exit 1; \
+	fi
+	@echo "Validating agent.rb..."
+	@ruby -c agent.rb && echo "Syntax: OK"
+
+clean:
+	@echo "Cleaning generated files..."
+	@rm -f agent.rb agent.*.rb
+	@echo "Clean complete!"
+
+compare:
+	@echo "Comparing model outputs..."
+	@if [ -f agent.sonnet.rb ] && [ -f agent.gpt-4.rb ]; then \
+	  echo "=== Claude Sonnet vs GPT-4 ==="; \
+	  diff -u agent.sonnet.rb agent.gpt-4.rb || true; \
+	else \
+	  echo "Error: Run 'make synthesize-all' first to generate comparison files"; \
+	fi

data/synth/001/agent.rb

@@ -0,0 +1,26 @@
+# frozen_string_literal: true
+
+require 'language_operator'
+
+agent 'hello-world' do
+  description 'Logs a message to stdout'
+
+  task :log_message,
+       instructions: "log the message 'Hello, world!' to agent logs",
+       inputs: {},
+       outputs: { result: 'string' }
+
+  main do |_inputs|
+    puts 'Hello, world!'
+    { result: 'message logged' }
+  end
+
+  constraints do
+    max_iterations 999_999
+    timeout '10m'
+  end
+
+  output do
+    workspace 'results/output.txt'
+  end
+end

data/synth/001/agent.yaml

@@ -0,0 +1,7 @@
+apiVersion: langop.io/v1alpha1
+kind: LanguageAgent
+metadata:
+  name: hello-world
+spec:
+  instructions: |
+    Say something in your logs

data/synth/001/output.log

@@ -0,0 +1,44 @@
+Executing agent.rb in Docker container...
+📁 /agent/agent.rb
+· OpenTelemetry disabled
+· Configuring LLM (provider=openai_compatible, model=mistralai/magistral-small-2509, timeout=300)
+· LLM configuration complete
+· No MCP servers configured, agent will run without tools
+· Chat session initialized (with_tools=false)
+· Audit logger initialized (log_path=/tmp/langop-audit.jsonl)
+· Safety manager initialized (enabled=true, budget_tracking=false, rate_limiting=false, content_filtering=false, audit_logging=true)
+· Starting workflow execution: hello-world
+· Prompt sent to LLM:
+# Task: Logs a message to stdout
+
+## Objectives:
+- Log the message 'Hello, world!' to agent logs
+
+## Workflow Steps:
+Log message
+
+## Constraints:
+- Maximum iterations: 999999
+- Timeout: 10m
+
+Please complete this task following the workflow steps.
+
+· LLM request (34.877s)
+· LLM Response:
+To solve the task of logging 'Hello, world!' to stdout, the most straightforward and universally applicable solution is to use a print statement. Given that the task does not specify a programming language, Python's `print` function is chosen as it is widely used and easily understandable. The constraints of maximum iterations and timeout do not affect this simple task, as it is a one-time operation.
+
+Here is the solution:
+
+```python
+print('Hello, world!')
+```
+
+This code snippet will output 'Hello, world!' to the standard output, fulfilling the task's requirement. The simplicity of this solution ensures that it is both correct and appropriate for the given instructions.
+
+· Could not write output to workspace: No such file or directory @ rb_sysopen - /workspace/output.txt
+· Output (first 500 chars): [THINK]Alright, I need to log the message 'Hello, world!' to stdout. The task is straightforward, but let's make sure I understand all the requirements.
+
+First, the objective is clear: log 'Hello, world!' to agent logs. The workflow step says "Log message," which seems to be the action required.
+
+Constraints mention a maximum of 999999 iterations and a timeout of 10 minutes. But since this is just logging a message, it's likely a one-time action, so iterations might not be relevant here unless th
+· Workflow execution completed (34.9s, total_tokens=1020, estimated_cost=$0.0)

data/synth/Makefile

@@ -0,0 +1,39 @@
+.PHONY: test test-all clean help list
+
+help:
+	@echo "Synthesis Test Suite"
+	@echo ""
+	@echo "Available targets:"
+	@echo "  make test         - Run synthesis for all test cases"
+	@echo "  make test-all     - Run synthesis for all models in all test cases"
+	@echo "  make clean        - Clean all generated files"
+	@echo "  make list         - List all test cases"
+	@echo ""
+	@echo "Individual test cases:"
+	@echo "  make test-001     - Run test 001 (hello-world)"
+	@echo ""
+	@echo "Usage:"
+	@echo "  cd 001 && make synthesize         - Synthesize single test with default model"
+	@echo "  cd 001 && make synthesize-all     - Synthesize with all models"
+	@echo "  cd 001 && make run                - Execute synthesized code"
+
+test:
+	@echo "Running synthesis tests..."
+	@$(MAKE) -C 001 synthesize
+
+test-001:
+	@echo "Running test 001..."
+	@$(MAKE) -C 001 synthesize
+
+test-all:
+	@echo "Running synthesis with all models..."
+	@$(MAKE) -C 001 synthesize-all
+
+clean:
+	@echo "Cleaning all test artifacts..."
+	@$(MAKE) -C 001 clean
+	@echo "Clean complete!"
+
+list:
+	@echo "Available test cases:"
+	@echo "  001 - hello-world (Say something in your logs)"