dialectic 0.1.0

package/docs/evaluator.md

@@ -0,0 +1,141 @@
+# Evaluator Command
+
+The `eval` command runs one or more evaluator agents to assess the outcome of a completed debate stored as a debate state JSON file.
+
+## Usage
+
+```bash
+# Basic usage (stdout Markdown table by default)
+dialectic eval --config ./eval-config.json --debate ./debates/deb-20250101-010203-ABC.json
+
+# With environment file and verbose diagnostics
+dialectic eval --config ./eval-config.json --debate ./deb.json --env-file ./.env --verbose
+
+# Write aggregated JSON output
+dialectic eval --config ./eval-config.json --debate ./deb.json --output ./result.json
+```
+
+Options:
+- `-c, --config <path>`: Required. Evaluator configuration file path.
+- `-d, --debate <path>`: Required. Debate JSON (saved `DebateState`) file path.
+- `--env-file <path>`: Optional. Path to environment file.
+- `-v, --verbose`: Optional. Verbose diagnostic output to stderr.
+- `-o, --output <path>`: Optional. Output destination.
+  - If path ends with `.json`, writes aggregated JSON output (including per-agent results).
+  - Otherwise, writes a Markdown table (or stdout when not provided).
+
+Exit codes:
+- `0`: Success
+- `2`: Invalid arguments (missing files/fields, malformed JSON, missing final solution)
+- `4`: Configuration error (e.g., missing API keys)
+
+## Evaluator Configuration
+
+Schema (root):
+```json
+{
+  "agents": [
+    {
+      "id": "eval-1",
+      "name": "Evaluator 1",
+      "model": "gpt-4",
+      "provider": "openai",
+      "systemPromptPath": "./prompts/system.md",
+      "userPromptPath": "./prompts/user.md",
+      "timeout": 30000,
+      "enabled": true
+    }
+  ]
+}
+```
+
+Notes:
+- `role` is not required.
+- `temperature` is ignored; evaluators always use temperature 0.1.
+- `timeout` units: milliseconds. Default 30000.
+
+## Input Debate JSON
+- Must contain non-empty `problem` and `finalSolution.description`.
+- If missing, the command fails with exit code 2.
+- Clarifications (if any) are included in the evaluator context as fenced code blocks; skipped answers appear as `NA`.
+
+## Default Prompts
+Built-in prompts are bundled in the code and used if files are missing/invalid:
+- `src/eval/prompts/system.md`
+- `src/eval/prompts/user.md`
+
+## Evaluator Output Contract
+Evaluators must return ONLY a JSON object with this structure:
+```json
+{
+  "evaluation": {
+    "functional_completeness": { "score": 1, "reasoning": "..." },
+    "non_functional": {
+      "performance_scalability": { "score": 1, "reasoning": "..." },
+      "security": { "score": 1, "reasoning": "..." },
+      "maintainability_evolvability": { "score": 1, "reasoning": "..." },
+      "regulatory_compliance": { "score": 1, "reasoning": "..." },
+      "testability": { "score": 1, "reasoning": "..." }
+    }
+  },
+  "overall_summary": {
+    "strengths": "...",
+    "weaknesses": "...",
+    "overall_score": 1
+  }
+}
+```
+
+Rules:
+- Scores must be integers in the range 1..10.
+- If a score is unavailable, omit that field; the system averages only present values and warns.
+- Out-of-range scores are clamped to [1..10] with a warning.
+- Non-numeric scores are ignored with a warning.
+- Extra keys are ignored (warned in verbose mode).
+
+## Aggregation and Output
+- Categories averaged:
+  - Functional Completeness
+  - Performance & Scalability
+  - Security
+  - Maintainability & Evolvability
+  - Regulatory Compliance
+  - Testability
+  - Overall Score
+- Averaging uses only present, valid scores; results are rounded to 2 decimals. Missing across all agents displays `N/A`.
+
+### Markdown Output
+A single Markdown table is printed to stdout by default:
+```
+| Functional Completeness | Performance & Scalability | Security | Maintainability & Evolvability | Regulatory Compliance | Testability | Overall Score |
+|------------------------|---------------------------|----------|-------------------------------|------------------------|------------|---------------|
+| 7.50 | 6.00 | 8.00 | 7.00 | N/A | 7.00 | 7.20 |
+```
+
+### JSON Output
+When `--output` ends with `.json`, the file contains averages and per-agent results:
+```json
+{
+  "evaluation": {
+    "functional_completeness": { "average_score": 7.5 },
+    "non_functional": {
+      "performance_scalability": { "average_score": 6.0 },
+      "security": { "average_score": 8.0 },
+      "maintainability_evolvability": { "average_score": 7.0 },
+      "regulatory_compliance": { "average_score": null },
+      "testability": { "average_score": 7.0 }
+    }
+  },
+  "overall_score": 7.2,
+  "agents": {
+    "eval-1": {}
+  }
+}
+```
+
+## Verbose Mode
+With `--verbose`, stderr includes:
+- Provider/model per agent
+- Prompt sources (built-in vs file path)
+- Per-agent latency and any timeout
+- JSON parsing/clamping/ignored-field warnings

package/examples/debate-config-openrouter.json

@@ -0,0 +1,48 @@
+{
+  "agents": [
+    {
+      "id": "agent-architect",
+      "name": "System Architect",
+      "role": "architect",
+      "model": "anthropic/claude-sonnet-4.5",
+      "provider": "openrouter",
+      "temperature": 0.5,
+      "enabled": true
+    },
+    {
+      "id": "agent-performance",
+      "name": "Performance Engineer",
+      "role": "performance",
+      "model": "anthropic/claude-3-opus",
+      "provider": "openrouter",
+      "temperature": 0.6,
+      "enabled": true
+    },
+    {
+      "id": "agent-security",
+      "name": "Security Specialist",
+      "role": "security",
+      "model": "gpt-4",
+      "provider": "openai",
+      "temperature": 0.4,
+      "enabled": true
+    }
+  ],
+  "judge": {
+    "id": "judge-main",
+    "name": "Technical Judge",
+    "role": "generalist",
+    "model": "openai/gpt-4",
+    "provider": "openrouter",
+    "temperature": 0.3
+  },
+  "debate": {
+    "rounds": 3,
+    "terminationCondition": {
+      "type": "fixed"
+    },
+    "synthesisMethod": "judge",
+    "includeFullHistory": true,
+    "timeoutPerRound": 300000
+  }
+}

package/examples/debate_config1.json

@@ -0,0 +1,48 @@
+{
+  "agents": [
+    {
+      "id": "agent-architect",
+      "name": "System Architect",
+      "role": "architect",
+      "model": "gpt-4",
+      "provider": "openai",
+      "temperature": 0.5,
+      "enabled": true
+    },
+    {
+      "id": "agent-performance",
+      "name": "Performance Engineer",
+      "role": "performance",
+      "model": "gpt-4",
+      "provider": "openai",
+      "temperature": 0.6,
+      "enabled": true
+    },
+    {
+      "id": "agent-security",
+      "name": "Security Specialist",
+      "role": "security",
+      "model": "gpt-4",
+      "provider": "openai",
+      "temperature": 0.4,
+      "enabled": true
+    }
+  ],
+  "judge": {
+    "id": "judge-main",
+    "name": "Technical Judge",
+    "role": "generalist",
+    "model": "gpt-4",
+    "provider": "openai",
+    "temperature": 0.3
+  },
+  "debate": {
+    "rounds": 4,
+    "terminationCondition": {
+      "type": "fixed"
+    },
+    "synthesisMethod": "judge",
+    "includeFullHistory": true,
+    "timeoutPerRound": 450000
+  }
+}

package/examples/eval/eval1/eval_config1.json

@@ -0,0 +1,13 @@
+{
+  "agents": [
+    {
+      "id": "eval-1",
+      "name": "Gemini Flash Evaluator",
+      "model": "google/gemini-2.5-flash-preview-09-2025",
+      "provider": "openrouter",
+      "timeout": 30000,
+      "enabled": true
+    }
+  ]
+}
+

package/examples/eval/eval1/result1.json

@@ -0,0 +1,62 @@
+{
+  "evaluation": {
+    "functional_completeness": {
+      "average_score": 9
+    },
+    "non_functional": {
+      "performance_scalability": {
+        "average_score": 10
+      },
+      "security": {
+        "average_score": 10
+      },
+      "maintainability_evolvability": {
+        "average_score": 9
+      },
+      "regulatory_compliance": {
+        "average_score": 9
+      },
+      "testability": {
+        "average_score": 9
+      }
+    }
+  },
+  "overall_score": 9,
+  "agents": {
+    "eval-1": {
+      "evaluation": {
+        "functional_completeness": {
+          "score": 9,
+          "reasoning": "All core functional requirements (upload, run/grade, persistence/audit, plagiarism detection, LMS integration) are addressed, with detailed technical solutions proposed for each. The solution goes beyond the minimum requirements."
+        },
+        "non_functional": {
+          "performance_scalability": {
+            "score": 10,
+            "reasoning": "Excellent focus on performance and scalability, including a phased architecture (monolith to microservices), multi-tier caching (L1/L2 Redis), optimized database design (partitioning, materialized views), and high-performance execution environment (Firecracker microVMs, pre-warmed VMs, dynamic scaling). Targets are concrete (150+ submissions/min, 95%+ cache hit rate)."
+          },
+          "security": {
+            "score": 10,
+            "reasoning": "Robust security measures are detailed, specifically addressing execution isolation (Firecracker), data protection (JWT rotation, device fingerprinting), auditability, and proactive measures (dependency scanning, patching)."
+          },
+          "maintainability_evolvability": {
+            "score": 9,
+            "reasoning": "The phased implementation (monolith MVP to microservices via Strangler Fig) is a strong strategy for maintainability and controlled evolution. The rigorous performance validation and continuous profiling also contribute to long-term health."
+          },
+          "regulatory_compliance": {
+            "score": 9,
+            "reasoning": "Explicit mention of compliance with FERPA, GDPR, and SOC 2 frameworks indicates a strong commitment to regulatory requirements, which is crucial for a university system handling student data."
+          },
+          "testability": {
+            "score": 9,
+            "reasoning": "The solution includes rigorous performance validation, comprehensive load testing scenarios (soak, spike), continuous profiling, and automated performance regression gates in CI/CD, all of which enhance testability and quality assurance."
+          }
+        }
+      },
+      "overall_summary": {
+        "strengths": "The solution is exceptionally detailed, addressing all non-functional requirements with concrete, high-quality technical choices (Firecracker, PostgreSQL partitioning, LSH for plagiarism, phased architecture). It demonstrates a strong understanding of security, scalability, and cost optimization.",
+        "weaknesses": "The initial monolith phase, while pragmatic, introduces a temporary architectural debt. The complexity of the multi-tier caching and microVM management requires significant operational expertise.",
+        "overall_score": 9
+      }
+    }
+  }
+}

package/examples/eval/eval1/result2.json

@@ -0,0 +1,97 @@
+{
+  "evaluation": {
+    "functional_completeness": {
+      "average_score": 9
+    },
+    "non_functional": {
+      "performance_scalability": {
+        "average_score": 10
+      },
+      "security": {
+        "average_score": 9.5
+      },
+      "maintainability_evolvability": {
+        "average_score": 9
+      },
+      "regulatory_compliance": {
+        "average_score": 9
+      },
+      "testability": {
+        "average_score": 8
+      }
+    }
+  },
+  "overall_score": 9,
+  "agents": {
+    "eval-1": {
+      "evaluation": {
+        "functional_completeness": {
+          "score": 9,
+          "reasoning": "The solution explicitly addresses all core requirements: student code upload/execution/grading, persistence/auditability (PostgreSQL, audit logging), plagiarism detection (LSH, external submission), and integration with the LMS (implied by the overall system context, though specific integration details are light). The phased approach ensures initial functional delivery."
+        },
+        "non_functional": {
+          "performance_scalability": {
+            "score": 10,
+            "reasoning": "The solution provides extensive and detailed strategies for performance and scalability, including multi-tier caching (L1/L2), high-performance execution using Firecracker microVMs with pre-warmed pools, dynamic scaling, optimized database design (partitioning, materialized views), and rigorous performance validation (load testing, profiling). It explicitly targets high throughput (150+ submissions/min) and low latency."
+          },
+          "security": {
+            "score": 10,
+            "reasoning": "Security is addressed comprehensively, covering execution isolation (Firecracker microVMs), data protection (secure JWT, audit logging), application security (dependency scanning), and regulatory compliance (explicit mention of FERPA, GDPR, SOC 2). This demonstrates a robust security posture."
+          },
+          "maintainability_evolvability": {
+            "score": 9,
+            "reasoning": "The solution explicitly adopts a phased implementation starting with a pragmatic monolith and evolving to microservices using the Strangler Fig pattern. This is a strong strategy for maintainability and controlled evolution. Decomposition is planned, and the use of clear components (caching, execution environment) aids maintenance."
+          },
+          "regulatory_compliance": {
+            "score": 9,
+            "reasoning": "The solution explicitly mentions compliance with FERPA, GDPR, and SOC 2 frameworks, indicating that regulatory impact on student data privacy and security has been considered in the design process."
+          },
+          "testability": {
+            "score": 8,
+            "reasoning": "Testability is addressed through the mention of automated performance regression gates in CI/CD and comprehensive load testing scenarios. The component separation (even in the planned microservices phase) and isolated execution environment (Firecracker) inherently support better unit and integration testing, though explicit strategies for mocking external services (like TurnItIn) or component-level testing are not detailed."
+          }
+        }
+      },
+      "overall_summary": {
+        "strengths": "Exceptional focus on performance, scalability, and security, utilizing modern technologies like Firecracker microVMs and LSH for plagiarism detection. The phased implementation approach (monolith to microservices via Strangler Fig) is pragmatic and reduces initial risk.",
+        "weaknesses": "Specific details on the LMS integration and the initial 'pragmatic monolith' structure are light, which could impact initial implementation clarity.",
+        "overall_score": 9
+      }
+    },
+    "eval-2": {
+      "evaluation": {
+        "functional_completeness": {
+          "score": 9,
+          "reasoning": "The solution addresses all core functional requirements: code upload, execution/grading, persistence/auditability (PostgreSQL, audit logging), plagiarism detection (LSH, TurnItIn integration), and LMS integration (implied by the overall system context, though not detailed). The phased approach ensures initial functionality is delivered."
+        },
+        "non_functional": {
+          "performance_scalability": {
+            "score": 10,
+            "reasoning": "Performance and scalability are primary considerations. The solution explicitly details high-performance execution (Firecracker microVMs, pre-warmed pool, 150+ submissions/min burst), multi-tier caching (L1/L2, Redis), optimized database design (partitioning, materialized views), and a clear scaling path (monolith to microservices). This is a robust plan for scale."
+          },
+          "security": {
+            "score": 9,
+            "reasoning": "Security is well-addressed, particularly for code execution isolation (Firecracker), data protection (JWT rotation, audit logging), and compliance (FERPA, GDPR, SOC 2 frameworks mentioned). The focus on automated scanning and patching also contributes significantly."
+          },
+          "maintainability_evolvability": {
+            "score": 9,
+            "reasoning": "The solution explicitly adopts a phased approach (monolith to microservices via Strangler Fig) which is a strong pattern for evolvability. Decomposition is planned, and the use of clear components (caching, execution environment) enhances maintainability. Continuous profiling aids troubleshooting."
+          },
+          "regulatory_compliance": {
+            "score": 9,
+            "reasoning": "The solution explicitly mentions compliance with key frameworks relevant to educational data (FERPA, GDPR, SOC 2), indicating that regulatory requirements have been factored into the security and data handling design."
+          },
+          "testability": {
+            "score": 8,
+            "reasoning": "Testability is addressed through comprehensive load testing scenarios, continuous profiling, and automated performance regression gates in CI/CD. The component separation (e.g., Firecracker for execution) suggests isolated testing is possible, though specific strategies for unit/integration testing of the core logic are not detailed."
+          }
+        }
+      },
+      "overall_summary": {
+        "strengths": "Exceptional focus on non-functional requirements, particularly performance, scalability, and security, using modern, high-confidence technologies (Firecracker, LSH, multi-tier caching). The pragmatic, phased implementation approach (monolith to microservices) minimizes initial risk while ensuring future growth capacity.",
+        "weaknesses": "LMS integration details are sparse. The initial 'pragmatic monolith' phase, while good for speed, requires careful management to ensure the Strangler Fig migration path remains viable.",
+        "overall_score": 9
+      }
+    }
+  }
+}

package/examples/eval_summary_format.md

@@ -0,0 +1,11 @@
+
+Extract all the different scores (each sub score), from all these files and put them in a markdown table, allowing to compare results.
+the table column should be: 
+- file name
+- functional_completeness score
+- performance_scalability score
+- security score
+- maintainability_evolvability score
+- regulatory_compliance score
+- testability score.
+- overall_score (taken from the root object).

package/examples/example3/debate-config.json

@@ -0,0 +1,64 @@
+{
+  "agents": [
+    {
+      "id": "agent-architect",
+      "name": "System Architect",
+      "role": "architect",
+      "model": "google/gemini-2.5-flash-lite",
+      "provider": "openrouter",
+      "temperature": 0.5,
+      "enabled": true
+    },
+    {
+      "id": "agent-performance",
+      "name": "Performance Engineer",
+      "role": "performance",
+      "model": "google/gemini-2.5-flash-lite",
+      "provider": "openrouter",
+      "temperature": 0.5,
+      "enabled": true
+    },
+    {
+      "id": "agent-security",
+      "name": "Security Specialist",
+      "role": "security",
+      "model": "google/gemini-2.5-flash-lite",
+      "provider": "openrouter",
+      "temperature": 0.5,
+      "enabled": true
+    },
+    {
+      "id": "agent-kiss",
+      "name": "KISS Advocate",
+      "role": "kiss",
+      "model": "google/gemini-2.5-flash-lite",
+      "provider": "openrouter",
+      "temperature": 0.5,
+      "enabled": true
+    }
+  ],
+  "judge": {
+    "id": "judge-main",
+    "name": "Technical Judge",
+    "role": "generalist",
+    "model": "google/gemini-2.5-flash-lite",
+    "provider": "openrouter",
+    "temperature": 0.5
+  },
+  "debate": {
+    "rounds": 3,
+    "terminationCondition": {
+      "type": "fixed"
+    },
+    "synthesisMethod": "judge",
+    "includeFullHistory": true,
+    "timeoutPerRound": 300000,
+    "summarization": {
+      "enabled": true,
+      "threshold": 10000,
+      "maxLength": 5000,
+      "method": "length-based"
+    }
+  }
+}
+

package/examples/example3/eval_config2.json

@@ -0,0 +1,25 @@
+{
+  "agents": [
+    {
+      "id": "eval-1",
+      "name": "Sonnet 4 Evaluator 1",
+      "model": "google/gemini-2.5-flash-lite",
+      "provider": "openrouter",
+      "timeout": 30000,
+      "enabled": true,
+      "systemPromptPath": "../eval_system.md",
+      "userPromptPath": "../eval_user.md"
+    },
+    {
+      "id": "eval-2",
+      "name": "Sonnet 4 Evaluator 2",
+      "model": "google/gemini-2.5-flash-lite",
+      "provider": "openrouter",
+      "timeout": 30000,
+      "enabled": true,
+      "systemPromptPath": "../eval_system.md",
+      "userPromptPath": "../eval_user.md"
+    }
+  ]
+}
+

package/examples/example3/problem.md

@@ -0,0 +1,17 @@
+I’m designing a system where we have a backend (API + admin/back office) and a frontend with active users. The scenario is something like this:
+
+We have around 100 daily active users, potentially scaling to 1000+ in the future.
+
+From the back office, admins can post notifications or messages (e.g., “maintenance at 12:00”) that should appear in real time on the frontend.
+
+Right now, we are using polling from the frontend to check for updates every 30 seconds or so.
+
+I’m considering switching to a WebSocket approach, where the backend pushes the message to all connected clients immediately.
+
+My questions are:
+
+What are the main benefits and trade-offs of using WebSockets vs polling in scenarios like this?
+
+Are there specific factors (number of requests, latency, server resources, scaling) that would make you choose one over the other?
+
+Any experiences with scaling this kind of system from tens to thousands of users?

package/examples/example3/rounds_test/eval_run.sh

@@ -0,0 +1,16 @@
+#!/bin/bash
+
+# Base paths
+BASE_DIR="examples/example3"
+OUTPUT_DIR="/mnt/c/tmp/dialectic/example3/rounds_test"
+
+# Ensure output directory exists
+mkdir -p "$OUTPUT_DIR"
+
+# Run evaluations for all debate outputs
+dialectic eval -c ./$BASE_DIR/eval_config2.json -d $OUTPUT_DIR/all_agents_1R_no_clarify.json -v -o $OUTPUT_DIR/eval2_all_agents_1R_no_clarify.json
+dialectic eval -c ./$BASE_DIR/eval_config2.json -d $OUTPUT_DIR/all_agents_2R_no_clarify.json -v -o $OUTPUT_DIR/eval2_all_agents_2R_no_clarify.json
+dialectic eval -c ./$BASE_DIR/eval_config2.json -d $OUTPUT_DIR/all_agents_3R_no_clarify.json -v -o $OUTPUT_DIR/eval2_all_agents_3R_no_clarify.json
+# dialectic eval -c ./$BASE_DIR/eval_config2.json -d $OUTPUT_DIR/all_agents_4R_no_clarify.json -v -o $OUTPUT_DIR/eval2_all_agents_4R_no_clarify.json
+# dialectic eval -c ./$BASE_DIR/eval_config2.json -d $OUTPUT_DIR/all_agents_5R_no_clarify.json -v -o $OUTPUT_DIR/eval2_all_agents_5R_no_clarify.json
+

package/examples/example3/rounds_test/run_test.sh

@@ -0,0 +1,16 @@
+#!/bin/bash
+
+# Base paths
+BASE_DIR="examples/example3"
+OUTPUT_DIR="/c/tmp/dialectic/example3/rounds_test"
+
+# Ensure output directory exists
+mkdir -p "$OUTPUT_DIR"
+
+# Run debates with rounds 1-5
+dialectic debate -r 1 -c "$BASE_DIR/debate-config.json" -o "$OUTPUT_DIR/all_agents_1R_no_clarify.json" -p "$BASE_DIR/problem.md" -v
+dialectic debate -r 2 -c "$BASE_DIR/debate-config.json" -o "$OUTPUT_DIR/all_agents_2R_no_clarify.json" -p "$BASE_DIR/problem.md" -v
+dialectic debate -r 3 -c "$BASE_DIR/debate-config.json" -o "$OUTPUT_DIR/all_agents_3R_no_clarify.json" -p "$BASE_DIR/problem.md" -v
+# dialectic debate -r 4 -c "$BASE_DIR/debate-config.json" -o "$OUTPUT_DIR/all_agents_4R_no_clarify.json" -p "$BASE_DIR/problem.md" -v
+# dialectic debate -r 5 -c "$BASE_DIR/debate-config.json" -o "$OUTPUT_DIR/all_agents_5R_no_clarify.json" -p "$BASE_DIR/problem.md" -v
+