@rpamis/comet 0.4.0-beta.3 → 0.4.0-beta.5

package/eval/local/tasks/comet-refactor-counter/instruction.md

@@ -0,0 +1,31 @@
+You are working on a Python project called "text-processor" - a text analysis utility library.
+
+Your task: Use the comet workflow to **refactor** the duplicated counting logic.
+
+## The problem
+
+`text_processor.py` has three near-identical functions — `count_words`, `count_lines`, `count_chars` — each repeating the same "split + iterate + tally" pattern with only the splitting strategy differing. This violates DRY.
+
+The goal is to introduce a **single dispatcher**:
+
+```python
+def count(text: str, unit: str) -> int:
+    """Count words/lines/chars. unit must be 'words', 'lines', or 'chars'."""
+    ...
+```
+
+Then rewrite `count_words`, `count_lines`, `count_chars` as **thin wrappers** that delegate to `count()`. The public API (function names, signatures, return values) must stay identical.
+
+## Constraints
+
+- **Behavior must not change**: every test in `test_text_processor.py` must still pass.
+- `analyze()` must continue to work unchanged.
+- `count()` should raise `ValueError` for an unknown unit.
+
+## What to do
+
+1. First, run the existing tests to confirm they pass: `python -m pytest test_text_processor.py -v`
+2. Follow the comet workflow phases (Open → Design → Build → Verify → Archive).
+3. After refactoring, run the tests again to prove behavior is preserved.
+
+Start by detecting the current phase and following the comet workflow. When you reach a decision point that asks for confirmation, assume "yes, proceed with the recommended option".

package/eval/local/tasks/comet-refactor-counter/task.toml

@@ -0,0 +1,24 @@
+[metadata]
+name = "comet-refactor-counter"
+description = "Refactor three duplicated counting functions into a single count(text, unit) dispatcher without changing behavior"
+difficulty = "medium"
+category = "comet"
+tags = ["comet", "workflow", "refactor", "dry"]
+default_treatments = [
+    "CONTROL",
+    "COMET_FULL_040_BETA",
+    "COMET_FULL_039",
+]
+
+[template]
+required = []
+
+[environment]
+description = "Python text processor with duplicated counting logic. Claude must refactor while preserving behavior."
+dockerfile = "environment/Dockerfile"
+timeout_sec = 900
+
+[validation]
+test_scripts = "test_refactor_counter.py"
+target_artifacts = []
+timeout = 120

package/eval/local/tasks/comet-refactor-counter/validation/test_refactor_counter.py

@@ -0,0 +1,75 @@
+"""Validation for the comet-refactor-counter task.
+
+Checks: comet workflow followed + a count() dispatcher exists + behavior
+preserved (all existing tests pass) + wrappers delegate (no duplicated loops).
+"""
+
+import re
+import subprocess
+import sys
+
+from comet_checks import (
+    run_comet_checks,
+    write_results,
+    _passed,
+    _failed,
+    WORKSPACE,
+)
+
+
+def _subprocess_output(result: subprocess.CompletedProcess) -> str:
+    output = "\n".join(
+        part.strip()
+        for part in (result.stdout or "", result.stderr or "")
+        if part and part.strip()
+    )
+    return output[-600:]
+
+
+def check_count_dispatcher() -> dict:
+    """A count(text, unit) dispatcher exists and wrappers delegate to it."""
+    src_path = WORKSPACE / "text_processor.py"
+    if not src_path.exists():
+        return _failed("count_dispatcher", "text_processor.py not found")
+    src = src_path.read_text(errors="ignore")
+
+    # Dispatcher must exist.
+    if not re.search(r"^def count\(\s*(?:text|s)\s*:\s*str\s*,\s*unit\s*:\s*str", src, re.MULTILINE):
+        return _failed("count_dispatcher", "no count(text, unit) dispatcher found")
+
+    # Wrappers must delegate (contain a count( call), not their own tally loop.
+    for wrapper in ("count_words", "count_lines", "count_chars"):
+        # Find the function body.
+        m = re.search(rf"def {wrapper}\([^)]*\)\s*(?:->\s*[^:]+)?:(.*?)(?=\ndef |\Z)", src, re.DOTALL)
+        if not m:
+            return _failed("count_dispatcher", f"wrapper {wrapper} missing")
+        body = m.group(1)
+        if "count(" not in body:
+            return _failed("count_dispatcher", f"{wrapper} does not delegate to count()")
+        # A raw for-loop tallying a counter inside the wrapper means it wasn't refactored.
+        if re.search(r"\bcount\s*\+=\s*1", body) or re.search(r"for\s+\w+\s+in\s+\w+\.split", body):
+            return _failed("count_dispatcher", f"{wrapper} still has inline tally logic")
+
+    return _passed("count_dispatcher", "count() dispatcher + 3 delegating wrappers")
+
+
+def check_behavior_preserved() -> dict:
+    """All existing tests still pass after refactoring."""
+    result = subprocess.run(
+        [sys.executable, "-m", "pytest", "test_text_processor.py", "-q"],
+        capture_output=True, text=True, cwd=str(WORKSPACE),
+    )
+    if result.returncode != 0:
+        return _failed("behavior_preserved", f"tests failing:\n{_subprocess_output(result)}")
+    return _passed("behavior_preserved", "all existing tests pass")
+
+
+def run_all() -> list[dict]:
+    results = run_comet_checks()
+    results.append(check_count_dispatcher())
+    results.append(check_behavior_preserved())
+    return results
+
+
+if __name__ == "__main__":
+    write_results(run_all())

package/eval/local/tasks/comet-robust-config/environment/Dockerfile

@@ -0,0 +1,39 @@
+FROM python:3.12-slim
+
+WORKDIR /workspace
+
+RUN apt-get update && apt-get install -y --no-install-recommends \
+    bash git curl jq ca-certificates \
+    nodejs npm && \
+    rm -rf /var/lib/apt/lists/*
+
+RUN pip install --no-cache-dir pyyaml pytest
+
+RUN npm config set registry https://registry.npmmirror.com && \
+    npm install -g @fission-ai/openspec@1.3.1 && \
+    npm config delete registry
+
+ARG CLAUDE_CODE_VERSION=latest
+RUN npm config set registry https://registry.npmmirror.com && \
+    npm install -g @anthropic-ai/claude-code@${CLAUDE_CODE_VERSION} && \
+    npm config delete registry
+
+COPY config_loader.py /workspace/config_loader.py
+COPY test_config_loader.py /workspace/test_config_loader.py
+
+RUN mkdir -p wordcount-cli && \
+    echo '# Config Loader' > wordcount-cli/README.md
+RUN mkdir -p openspec/changes
+
+RUN useradd --create-home --shell /bin/bash agent && \
+    chown -R agent:agent /workspace
+USER agent
+ENV HOME=/home/agent
+# Validator scripts import scaffold.*; bake PYTHONPATH so it never crosses the
+# MSYS path-conversion boundary on the docker run command line.
+ENV PYTHONPATH=/workspace
+RUN git config --global user.email "agent@eval.local" && \
+    git config --global user.name "Comet Eval Agent" && \
+    git config --global init.defaultBranch master
+
+CMD ["bash"]

package/eval/local/tasks/comet-robust-config/environment/config_loader.py

@@ -0,0 +1,56 @@
+"""config-loader: load and query simple INI-style configuration files.
+
+The current implementation is brittle: it crashes or silently misbehaves on
+malformed input. The task is to add structured error handling so invalid input
+raises a clear, typed exception instead of a cryptic crash.
+"""
+
+from __future__ import annotations
+
+from typing import Any
+
+
+class ConfigError(Exception):
+    """Base exception for all config-loading errors."""
+
+
+def load_config(text: str) -> dict[str, dict[str, str]]:
+    """Parse an INI-style config string into {section: {key: value}}.
+
+    Format:
+        [section]
+        key = value
+        # comment lines start with #
+
+    BUG: malformed input causes unhandled exceptions or silently corrupt data.
+    It should instead raise ConfigError subclasses with clear messages.
+    """
+    config: dict[str, dict[str, str]] = {}
+    current_section: str | None = None
+
+    for lineno, line in enumerate(text.splitlines(), 1):
+        stripped = line.strip()
+        if not stripped or stripped.startswith("#"):
+            continue
+        if stripped.startswith("[") and stripped.endswith("]"):
+            current_section = stripped[1:-1]
+            if current_section in config:
+                # BUG: silently overwrites instead of raising
+                pass
+            config[current_section] = {}
+            continue
+        if "=" not in stripped:
+            # BUG: silently skips invalid lines instead of raising
+            continue
+        if current_section is None:
+            # BUG: crashes later with KeyError or silently drops
+            continue
+        key, _, value = stripped.partition("=")
+        config[current_section][key.strip()] = value.strip()
+
+    return config
+
+
+def get(config: dict[str, dict[str, str]], section: str, key: str, default: Any = None) -> Any:
+    """Get a value from a loaded config, or `default` if missing."""
+    return config.get(section, {}).get(key, default)

package/eval/local/tasks/comet-robust-config/environment/test_config_loader.py

@@ -0,0 +1,72 @@
+"""Tests for config-loader.
+
+The error-handling tests currently FAIL (the brittle implementation crashes or
+silently misbehaves). The happy-path tests pass.
+"""
+
+import pytest
+
+from config_loader import ConfigError, load_config, get
+
+
+VALID_CONFIG = """\
+# Application settings
+[server]
+host = localhost
+port = 8080
+
+[database]
+url = postgres://localhost/app
+pool = 10
+"""
+
+
+class TestHappyPath:
+    def test_parse_basic(self):
+        cfg = load_config(VALID_CONFIG)
+        assert cfg["server"]["host"] == "localhost"
+        assert cfg["server"]["port"] == "8080"
+        assert cfg["database"]["pool"] == "10"
+
+    def test_comments_ignored(self):
+        cfg = load_config("# comment\n[s]\nk = v\n")
+        assert cfg == {"s": {"k": "v"}}
+
+    def test_empty(self):
+        assert load_config("") == {}
+
+    def test_get_with_default(self):
+        cfg = load_config("[s]\nk = v\n")
+        assert get(cfg, "s", "k") == "v"
+        assert get(cfg, "s", "missing", "fb") == "fb"
+        assert get(cfg, "nope", "k", "fb") == "fb"
+
+
+class TestErrorHandling:
+    def test_key_without_section_raises(self):
+        """A key=value line before any [section] must raise ConfigError."""
+        with pytest.raises(ConfigError):
+            load_config("key = value\n[section]\n")
+
+    def test_duplicate_section_raises(self):
+        """The same [section] appearing twice must raise ConfigError."""
+        with pytest.raises(ConfigError):
+            load_config("[s]\nk=v\n[s]\nk2=v2\n")
+
+    def test_malformed_line_raises(self):
+        """A line that is neither blank, comment, section, nor key=value must raise."""
+        with pytest.raises(ConfigError):
+            load_config("[s]\njust some text\n")
+
+    def test_empty_section_name_raises(self):
+        """An empty section header [] must raise ConfigError."""
+        with pytest.raises(ConfigError):
+            load_config("[]\nk=v\n")
+
+    def test_error_messages_are_informative(self):
+        """ConfigError messages should mention the line number or nature of the error."""
+        with pytest.raises(ConfigError) as exc_info:
+            load_config("[s]\nbroken line here\n")
+        msg = str(exc_info.value).lower()
+        assert "line" in msg or "invalid" in msg or "malformed" in msg, \
+            f"error message not informative: {exc_info.value}"

package/eval/local/tasks/comet-robust-config/instruction.md

@@ -0,0 +1,30 @@
+You are working on a Python project called "config-loader" - a simple INI-style configuration parser.
+
+Your task: Use the comet workflow to **add structured error handling** to `load_config()`.
+
+## The problem
+
+`load_config()` in `config_loader.py` is brittle. On malformed input it either crashes with a cryptic exception or silently misbehaves (overwrites duplicate sections, skips bad lines, drops keys outside sections).
+
+The goal is to make invalid input raise a **clear `ConfigError`** (already defined as the base exception) with an informative message mentioning the line number or nature of the problem.
+
+## Cases to handle (see TestErrorHandling)
+
+1. **Key before any section**: `key = value` before `[section]` → raise ConfigError
+2. **Duplicate section**: `[s]` appearing twice → raise ConfigError
+3. **Malformed line**: a line that is neither blank/comment/section/key=value → raise ConfigError
+4. **Empty section name**: `[]` → raise ConfigError
+5. **Informative messages**: error text should mention "line" or "invalid"/"malformed"
+
+## Constraints
+
+- The happy-path tests in `TestHappyPath` must keep passing.
+- Raise `ConfigError` (or subclasses) — don't let raw `KeyError`/`ValueError` escape.
+
+## What to do
+
+1. Run tests: `python -m pytest test_config_loader.py -v` (5 error-handling tests fail)
+2. Follow the comet workflow (Open → Design → Build → Verify → Archive).
+3. Confirm all 9 tests pass.
+
+Start by detecting the current phase and following the comet workflow. When you reach a decision point that asks for confirmation, assume "yes, proceed with the recommended option".

package/eval/local/tasks/comet-robust-config/task.toml

@@ -0,0 +1,24 @@
+[metadata]
+name = "comet-robust-config"
+description = "Add structured error handling to a brittle INI config loader so malformed input raises clear ConfigError instead of crashing"
+difficulty = "hard"
+category = "comet"
+tags = ["comet", "workflow", "error-handling", "robustness"]
+default_treatments = [
+    "CONTROL",
+    "COMET_FULL_040_BETA",
+    "COMET_FULL_039",
+]
+
+[template]
+required = []
+
+[environment]
+description = "Python INI config loader with brittle error handling. Claude must add structured errors via the comet workflow."
+dockerfile = "environment/Dockerfile"
+timeout_sec = 900
+
+[validation]
+test_scripts = "test_robust_config.py"
+target_artifacts = []
+timeout = 120

package/eval/local/tasks/comet-robust-config/validation/test_robust_config.py

@@ -0,0 +1,37 @@
+"""Validation for the comet-robust-config task.
+
+Checks: comet workflow followed + malformed input raises ConfigError + happy path intact.
+"""
+
+import subprocess
+import sys
+
+from comet_checks import (
+    run_comet_checks,
+    write_results,
+    _passed,
+    _failed,
+    WORKSPACE,
+)
+
+
+def check_error_handling() -> dict:
+    """All config tests pass: 5 error-handling + 4 happy-path."""
+    result = subprocess.run(
+        [sys.executable, "-m", "pytest", "test_config_loader.py", "-q"],
+        capture_output=True, text=True, cwd=str(WORKSPACE),
+    )
+    if result.returncode != 0:
+        out = result.stdout + result.stderr
+        return _failed("error_handling", f"tests failing:\n{out[-400:]}")
+    return _passed("error_handling", "malformed input raises ConfigError, happy path intact")
+
+
+def run_all() -> list[dict]:
+    results = run_comet_checks()
+    results.append(check_error_handling())
+    return results
+
+
+if __name__ == "__main__":
+    write_results(run_all())

package/eval/local/tasks/generic-skill-smoke/environment/Dockerfile

@@ -0,0 +1,2 @@
+FROM python:3.11-slim
+WORKDIR /workspace

package/eval/local/tasks/generic-skill-smoke/instruction.md

@@ -0,0 +1,7 @@
+Create a file named `result.md` in the current workspace.
+
+The file must contain:
+
+- A heading `# Skill Smoke Result`
+- A short summary of the approach you used
+- A bullet list with exactly three bullets

package/eval/local/tasks/generic-skill-smoke/task.toml

@@ -0,0 +1,21 @@
+[metadata]
+name = "generic-skill-smoke"
+description = "Smoke task for evaluating an arbitrary Skill on a simple artifact-producing workflow."
+difficulty = "easy"
+category = "generic"
+tags = ["generic", "skill", "smoke"]
+default_treatments = ["CONTROL"]
+
+[environment]
+description = "Empty workspace for a generic Skill smoke test."
+dockerfile = "environment/Dockerfile"
+timeout_sec = 600
+
+[validation]
+test_scripts = ["test_generic_skill_smoke.py"]
+target_artifacts = ["result.md"]
+timeout = 60
+
+[evaluation]
+profile = "generic"
+expected_artifacts = ["result.md"]

package/eval/local/tasks/generic-skill-smoke/validation/test_generic_skill_smoke.py

@@ -0,0 +1,36 @@
+from pathlib import Path
+
+from scaffold.python.validation.core import write_test_results
+
+
+def main():
+    path = Path("result.md")
+    passed = []
+    failed = []
+
+    if not path.exists():
+        write_test_results({"passed": [], "failed": ["result.md missing"]})
+        return
+
+    text = path.read_text(encoding="utf-8")
+    if "# Skill Smoke Result" in text:
+        passed.append("result.md heading present")
+    else:
+        failed.append("result.md heading missing")
+
+    bullets = [line for line in text.splitlines() if line.startswith("- ")]
+    if len(bullets) == 3:
+        passed.append("result.md has exactly three bullets")
+    else:
+        failed.append(f"result.md bullet count was {len(bullets)}")
+
+    if "approach" in text.lower() or "used" in text.lower():
+        passed.append("result.md describes approach")
+    else:
+        failed.append("result.md approach summary missing")
+
+    write_test_results({"passed": passed, "failed": failed})
+
+
+if __name__ == "__main__":
+    main()

package/eval/local/tasks/index.yaml

@@ -0,0 +1,155 @@
+tasks:
+  # ── Default / canonical full-workflow task ──────────────────────────────
+  - name: comet-full-workflow
+    category: comet
+    default_treatments:
+      - CONTROL
+      - COMET_FULL_040_BETA
+      - COMET_FULL_039
+    description: Full open-design-build-verify-archive workflow on a sentence-counting feature. The canonical 5-phase task.
+
+  # ── Dimension coverage tasks (bug fix / refactor / perf / api / robustness) ──
+  - name: comet-fix-median
+    category: comet
+    default_treatments:
+      - CONTROL
+      - COMET_FULL_040_BETA
+      - COMET_FULL_039
+    description: Bug-fix dimension. Fix median() for even-length lists in a stats library; diagnose, fix, verify no regressions.
+
+  - name: comet-refactor-counter
+    category: comet
+    default_treatments:
+      - CONTROL
+      - COMET_FULL_040_BETA
+      - COMET_FULL_039
+    description: Refactor dimension. Collapse three duplicated count_* functions into a count(text, unit) dispatcher while preserving behavior.
+
+  - name: comet-perf-dedupe
+    category: comet
+    default_treatments:
+      - CONTROL
+      - COMET_FULL_040_BETA
+      - COMET_FULL_039
+    description: Performance dimension. Optimize an O(n^2) dedupe function to O(n) using a set while preserving first-occurrence order.
+
+  - name: comet-api-cache-ttl
+    category: comet
+    default_treatments:
+      - CONTROL
+      - COMET_FULL_040_BETA
+      - COMET_FULL_039
+    description: API-design dimension. Add optional TTL expiration to an in-memory cache while keeping the existing API backward compatible.
+
+  - name: comet-robust-config
+    category: comet
+    default_treatments:
+      - CONTROL
+      - COMET_FULL_040_BETA
+      - COMET_FULL_039
+    description: Robustness dimension. Add structured ConfigError handling to a brittle INI config loader so malformed input raises clear errors.
+
+  # ── Expanded workflow pressure tasks inspired by skills-benchmarks ──────
+  - name: comet-dependency-confusion
+    category: comet
+    default_treatments:
+      - CONTROL
+      - COMET_FULL_040_BETA
+      - COMET_FULL_039
+    description: Inspired by skills-benchmarks/lc-deps-tavily. Repair stale module names and compatibility imports in a small settings loader without changing the public API.
+
+  - name: comet-layered-streaming-fix
+    category: comet
+    default_treatments:
+      - CONTROL
+      - COMET_FULL_040_BETA
+      - COMET_FULL_039
+    description: Inspired by skills-benchmarks/oss-fix-lc-streaming. Fix streaming chunk behavior and tool-result handling in a chat application.
+
+  - name: comet-persistence-threading
+    category: comet
+    default_treatments:
+      - CONTROL
+      - COMET_FULL_040_BETA
+      - COMET_FULL_039
+    description: Inspired by skills-benchmarks/oss-fix-lg-persistence. Make session state isolate by thread_id and survive a new store instance.
+
+  - name: comet-human-approval-flow
+    category: comet
+    default_treatments:
+      - CONTROL
+      - COMET_FULL_040_BETA
+      - COMET_FULL_039
+    description: Inspired by skills-benchmarks/oss-fix-lc-hitl. Add a pending approval flow before dangerous document operations can execute.
+
+  - name: comet-noise-distractor
+    category: comet
+    default_treatments:
+      - CONTROL
+      - COMET_FULL_040_BETA
+      - COMET_FULL_039
+    description: Inspired by skills-benchmarks/lc-basic-noise. Fix invoice tax calculation while leaving an unrelated distracting file untouched.
+
+  - name: comet-cross-file-refactor
+    category: comet
+    default_treatments:
+      - CONTROL
+      - COMET_FULL_040_BETA
+      - COMET_FULL_039
+    description: Inspired by skills-benchmarks/lc-basic. Add a shared count(text, unit) dispatcher while keeping wrappers and CLI behavior compatible.
+
+  - name: comet-observability-env-template
+    category: comet
+    default_treatments:
+      - CONTROL
+      - COMET_FULL_040_BETA
+      - COMET_FULL_039
+    description: Inspired by skills-benchmarks/ls-lang-tracing. Generate a LangSmith-ready .env.example with the simplified LANGSMITH_* configuration.
+
+  - name: comet-graph-execution-review
+    category: comet
+    default_treatments:
+      - CONTROL
+      - COMET_FULL_040_BETA
+      - COMET_FULL_039
+    description: Inspired by skills-benchmarks/oss-fix-lg-execution. Fix parallel task fan-out, review interruption, and resume behavior in a graph-style pipeline.
+
+  - name: comet-agent-memory-routing
+    category: comet
+    default_treatments:
+      - CONTROL
+      - COMET_FULL_040_BETA
+      - COMET_FULL_039
+    description: Inspired by skills-benchmarks/oss-fix-da-memory. Fix persistent preference routing, subagent skill access, and deployment approval checkpoints.
+
+  - name: comet-framework-selection
+    category: comet
+    default_treatments:
+      - CONTROL
+      - COMET_FULL_040_BETA
+      - COMET_FULL_039
+    description: Inspired by skills-benchmarks/lc-framework-hybrid-pipeline. Select a hybrid deep-agent plus deterministic sub-workflow architecture for long-session document editing.
+
+  - name: generic-skill-smoke
+    category: generic
+    default_treatments:
+      - CONTROL
+    description: Generic smoke task for arbitrary Skill eval; creates and validates result.md.
+
+  - name: authoring-skill-smoke
+    category: generic
+    default_treatments:
+      - CONTROL
+    description: Smoke task for generated Comet-native Skill packages.
+
+  - name: workflow-route-conformance
+    category: generic
+    default_treatments:
+      - CONTROL
+    description: Validates generated Skill route metadata against the workflow protocol and internal Node Skills.
+
+  - name: workflow-overlay-contract
+    category: generic
+    default_treatments:
+      - CONTROL
+    description: Validates generated overlay eval manifests for baselines, gates, schemas, and expected evidence.

package/eval/local/tasks/workflow-overlay-contract/environment/Dockerfile

@@ -0,0 +1,5 @@
+FROM python:3.11-slim
+
+WORKDIR /workspace
+
+RUN pip install --no-cache-dir pyyaml

package/eval/local/tasks/workflow-overlay-contract/instruction.md

@@ -0,0 +1,5 @@
+Inspect the generated Comet overlay Skill package available in this eval run.
+
+Confirm that its `comet/eval.yaml` exposes the structured evaluation contract
+for overlay workflow evidence, baseline comparison, quality gates, required
+schemas, and expected evidence checks.

package/eval/local/tasks/workflow-overlay-contract/task.toml

@@ -0,0 +1,21 @@
+[metadata]
+name = "workflow-overlay-contract"
+description = "Validate generated overlay eval manifest contract evidence."
+difficulty = "easy"
+category = "generic"
+tags = ["authoring", "skill", "comet-any", "overlay", "eval"]
+default_treatments = ["CONTROL"]
+
+[environment]
+description = "Workspace for generated overlay eval manifest inspection."
+dockerfile = "environment/Dockerfile"
+timeout_sec = 600
+
+[validation]
+test_scripts = ["test_workflow_overlay_contract.py"]
+target_artifacts = []
+timeout = 60
+
+[evaluation]
+profile = "authoring-skill"
+expected_artifacts = []