devsper 2.6.0__tar.gz → 2.7.1__tar.gz

{devsper-2.6.0 → devsper-2.7.1}/CHANGELOG.md

@@ -7,6 +7,52 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [2.7.1] — 2026-04-07
+
+### Added
+
+- **Eval harness** — New `devsper/evals/` module: `EvalCase`, `EvalDataset` (JSONL-backed), `EvalRunner` (async, bounded concurrency), `EvalSummary`, and `MetricFn` protocol. Built-in metrics: `exact_match`, `contains`, `regex_match`, `word_overlap`, `non_empty`, `llm_judge(model=)`.
+- **Prompt optimizer abstraction** — `PromptOptimizerBackend` ABC mirroring the `MemoryBackend` / `LLMBackend` hot-swap pattern. Active backend resolved from `DEVSPER_PROMPT_OPTIMIZER` env var → `[prompt_optimizer] provider` config → default `"noop"`.
+- **DSPy backend** (`devsper[dspy]`) — `DSPyBackend` compiles few-shot examples into an optimized system prompt using `BootstrapFewShot`, `MIPROv2`, or `BootstrapFewShotWithRandomSearch`. Configured via `[prompt_optimizer] dspy_optimizer`, `max_demos`, `num_candidates`.
+- **GEPA backend** (`devsper[gepa]`) — `GEPABackend` runs an evolutionary prompt optimization loop. Uses the `gepa` library when installed; falls back to a built-in LLM-driven mutation loop so it works without any extra dependencies.
+- **OpenEvals metric adapter** (`devsper[openevals]`) — `openevals_metric(name)` wraps any OpenEvals LLM-as-judge evaluator as a `MetricFn`. Prebuilt names: `correctness`, `conciseness`, `groundedness`, `relevance`. Accessible via `get_metric("openevals:correctness")`. Falls back to built-in `llm_judge` when the package is absent.
+- **`devsper eval` CLI** — Three subcommands: `eval stub` (generate JSONL stub datasets per role), `eval run` (score a dataset, optional `--optimize` flag, `--optimizer dspy|gepa`), `eval results` (list persisted result files).
+- **Config sections** — `[prompt_optimizer]` and `[evals]` added to `devsperConfigModel`.
+- **Optimized prompt persistence** — `EvalRunner` saves results to `.devsper/optimized_prompts/{role}.json`. `get_role_config()` auto-loads these on the next run.
+- **`devsper[evals]`, `devsper[dspy]`, `devsper[gepa]`, `devsper[openevals]` extras.**
+- **pytest `live` marker** — Gates tests that require real API keys.
+
+### Changed
+
+- **`get_metric(name)`** extended: now accepts `"openevals:<evaluator>"` in addition to built-in names.
+- **`get_role_config()`** — Checks `.devsper/optimized_prompts/{role}.json` and uses the optimized prefix if present.
+
+## [2.7.0] — 2026-04-07
+
+### Added
+
+- **MemoryProvider abstraction** — Introduced `MemoryBackend` ABC (`devsper/memory/providers/base.py`) mirroring the `LLMBackend` pattern. All memory backends implement a unified async interface: `store`, `retrieve`, `delete`, `list_memory`, `list_all_ids`, `query_similar`, `health`.
+- **MemoryProvider factory** — `get_memory_provider()` singleton factory resolves the active backend from config (`[memory] provider`), `DEVSPER_MEMORY_PROVIDER` env var, or legacy `backend` field. Defaults to Vektori (pgvector) with automatic SQLite fallback when `DATABASE_URL` is not set.
+- **Snowflake memory backend** — New `SnowflakeBackend` using `VECTOR(FLOAT, 1536)` columns and `VECTOR_COSINE_SIMILARITY` for native semantic search. Credentials resolved exclusively via the devsper credential store or `SNOWFLAKE_*` env vars — never from config files.
+- **SQLite, Redis, Vektori, Platform backends** — Existing stores wrapped as `MemoryBackend` implementations with `get_sync_store()` escape hatch for legacy sync callers.
+- **Credential store: Snowflake + Redis** — Added `snowflake` and `redis_memory` providers to keyring mappings. `devsper credentials set snowflake password` stores the Snowflake password securely; `inject_into_env()` propagates all `SNOWFLAKE_*` vars automatically.
+- **Config schema** — Added `RedisMemoryConfig` and `SnowflakeMemoryConfig` sub-models under `[memory]`. `SnowflakeMemoryConfig` has no `password` field — credentials are credential-store-only.
+- **`devsper doctor` memory provider health check** — `run_doctor()` now calls `provider.health()` and reports the active backend name and status.
+- **`devsper[snowflake]` and `devsper[redis-memory]` extras** — `snowflake-connector-python>=3.6.0` and `redis>=5.0.0` optional dependency groups.
+- **`deregister(name)` in tool registry** — Public function to remove a single tool by name without clearing the whole registry.
+- **`MemoryIndex` native search delegation** — When the active backend has `supports_native_vector_search=True`, `query_memory()` and `query_across_runs()` delegate directly to `backend.query_similar()`, bypassing in-process cosine ranking.
+
+### Changed
+
+- **`MemoryRouter` and `MemoryIndex`** accept an optional `backend: MemoryBackend` parameter. The `_build_memory_store()` fallback now calls the factory instead of constructing `MemoryStore()` directly.
+- **`get_effective_memory_store()`** preserved as a backwards-compatible alias; new `get_effective_memory_backend()` returns the full `MemoryBackend`. Async-only backends (Vektori, Snowflake) are bridged via `_AsyncBridgeStore` for sync callers.
+- **Legacy `backend` values** (`local`, `supermemory`, `hybrid`) now map to `"vektori"` as the default production store.
+
+### Fixed
+
+- **`test_pipelines.py`** — Replaced `registry._tools.pop()` (broken module-level access) with new `deregister()` function.
+- **`test_e2e_redis_loop.py`** — Import guard prevents `ModuleNotFoundError` when `redis` is not installed; test now correctly skips instead of failing.
+
 ## [2.6.0] — 2026-04-06
 
 ### Added
@@ -82,7 +128,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 - **Supermemory ranking backend** — `[memory] backend = "supermemory"` (default) uses the Rust `supermemory-core` ranker for hybrid lexical (+ optional embedding) memory retrieval, deduplication, and recency-aware context formatting.
 - **Platform memory backends** — `[memory] backend` can be `platform` or `hybrid` with `platform_api_url` and `platform_org_slug`; config resolver honors `DEVSPER_PLATFORM_API_URL` and `DEVSPER_PLATFORM_ORG` when set.
-- **Remote platform workflows** — `load_workflow("org/pkg@N")` fetches workflow specs from the Devsper Platform API when `DEVSPER_PLATFORM_API_URL`, `DEVSPER_PLATFORM_ORG`, and (when required) `DEVSPER_PLATFORM_TOKEN` are configured.
+- **Remote platform workflows** — `load_workflow("org/pkg@N")` fetches workflow specs from the Devsper Cloud API when `DEVSPER_PLATFORM_API_URL`, `DEVSPER_PLATFORM_ORG`, and (when required) `DEVSPER_PLATFORM_TOKEN` are configured.
 - **Local worker pool** — `devsper pool` subcommands to run the local worker pool manager (foreground spawner) alongside improved swarm execution when using pool-backed paths.
 - **Devsper Cloud CLI** — `devsper cloud login`, `logout`, `run`, `status`, and `logs` for JWT authentication (OS keychain) and runs against the platform API.
 

{devsper-2.6.0 → devsper-2.7.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: devsper
-Version: 2.6.0
+Version: 2.7.1
 Summary: Orchestrate distributed swarms of AI agents that collaboratively solve complex tasks.
 Project-URL: Homepage, https://github.com/devsper-com/runtime
 Project-URL: Documentation, https://docs.devsper.com
@@ -57,9 +57,14 @@ Requires-Dist: mkdocs<2,>=1.6.0; extra == 'docs'
 Provides-Extra: document
 Requires-Dist: docproc>=2.0.0; extra == 'document'
 Requires-Dist: python-docx>=1.0.0; extra == 'document'
+Provides-Extra: dspy
+Requires-Dist: dspy-ai>=2.4; extra == 'dspy'
 Provides-Extra: embeddings
 Requires-Dist: sentence-transformers>=3.0.0; extra == 'embeddings'
+Provides-Extra: evals
 Provides-Extra: explainability
+Provides-Extra: gepa
+Requires-Dist: gepa>=0.1; extra == 'gepa'
 Provides-Extra: hitl
 Requires-Dist: httpx>=0.27; extra == 'hitl'
 Provides-Extra: langgraph
@@ -68,6 +73,11 @@ Provides-Extra: mcp
 Requires-Dist: anyio>=4.0; extra == 'mcp'
 Requires-Dist: httpx>=0.27; extra == 'mcp'
 Provides-Extra: ollama
+Provides-Extra: openevals
+Requires-Dist: openevals>=0.0.6; extra == 'openevals'
+Provides-Extra: redis-memory
+Requires-Dist: hiredis>=2.3.0; extra == 'redis-memory'
+Requires-Dist: redis>=5.0.0; extra == 'redis-memory'
 Provides-Extra: research
 Requires-Dist: duckduckgo-search>=6.0.0; extra == 'research'
 Provides-Extra: server
@@ -75,6 +85,8 @@ Requires-Dist: fastapi>=0.110; extra == 'server'
 Requires-Dist: sse-starlette>=2.0; extra == 'server'
 Requires-Dist: uvicorn>=0.29; extra == 'server'
 Requires-Dist: vektori>=0.1.1; extra == 'server'
+Provides-Extra: snowflake
+Requires-Dist: snowflake-connector-python>=3.6.0; extra == 'snowflake'
 Provides-Extra: trulens
 Requires-Dist: trulens-core>=1.0; extra == 'trulens'
 Provides-Extra: worker

{devsper-2.6.0 → devsper-2.7.1}/devsper/agents/roles.py

@@ -105,10 +105,34 @@ ROLE_CONFIGS: dict[str, RoleConfig] = {
 
 
 def get_role_config(role: str | None) -> RoleConfig:
-    """Return config for the given role, or default if unknown/None."""
-    if role and role in ROLE_CONFIGS:
-        return ROLE_CONFIGS[role]
-    return ROLE_CONFIGS[DEFAULT_ROLE]
+    """Return config for the given role, or default if unknown/None.
+
+    If an optimized prompt exists at .devsper/optimized_prompts/{role}.json
+    (written by EvalRunner after prompt optimization), it is used in place of
+    the hardcoded prompt_prefix.
+    """
+    base = ROLE_CONFIGS.get(role or DEFAULT_ROLE, ROLE_CONFIGS[DEFAULT_ROLE]) if role else ROLE_CONFIGS[DEFAULT_ROLE]
+    optimized = _load_optimized_prompt(base.name)
+    if optimized:
+        from dataclasses import replace
+        return replace(base, prompt_prefix=optimized)
+    return base
+
+
+def _load_optimized_prompt(role: str) -> str | None:
+    """Load persisted optimized prompt for a role, if it exists."""
+    import json
+    from pathlib import Path
+
+    p = Path(".devsper/optimized_prompts") / f"{role}.json"
+    if not p.exists():
+        return None
+    try:
+        data = json.loads(p.read_text())
+        prompt = data.get("prompt_prefix", "")
+        return prompt if prompt else None
+    except Exception:
+        return None
 
 
 # Keywords to infer role from task description

{devsper-2.6.0 → devsper-2.7.1}/devsper/cli/init.py

@@ -534,6 +534,31 @@ def run_doctor() -> int:
     except Exception as e:
         warnings.append(f"Memory store: {e}")
 
+    # Memory provider health check
+    try:
+        import asyncio
+        from devsper.memory.providers.factory import get_memory_provider
+        provider = get_memory_provider()
+        try:
+            loop = asyncio.get_event_loop()
+            if loop.is_running():
+                import concurrent.futures
+                with concurrent.futures.ThreadPoolExecutor(max_workers=1) as pool:
+                    healthy = pool.submit(asyncio.run, provider.health()).result()
+            else:
+                healthy = loop.run_until_complete(provider.health())
+        except Exception:
+            healthy = False
+        if healthy:
+            ok.append(f"Memory provider: {provider.name} — healthy")
+        else:
+            warnings.append(
+                f"Memory provider: {provider.name} — health check failed. "
+                "Check credentials and connectivity."
+            )
+    except Exception as e:
+        warnings.append(f"Memory provider health check failed: {e}")
+
     try:
         from devsper.knowledge.knowledge_graph import KnowledgeGraph
         kg = KnowledgeGraph(store=get_default_store())

{devsper-2.6.0 → devsper-2.7.1}/devsper/cli/main.py

@@ -145,7 +145,7 @@ def _run_platform_once(task: str, args: object) -> tuple[int, dict, float]:
     api = _platform_api_builder()
     if not api.enabled():
         print(
-            "Platform routing is not configured. Run `devsper platform connect` first.",
+            "Cloud routing is not configured. Run `devsper platform connect` first.",
             file=sys.stderr,
         )
         return 2, {}, 0.0
@@ -2721,6 +2721,164 @@ def _run_version(args: object) -> int:
     return 0
 
 
+def _run_eval(args: object) -> int:
+    """Eval harness: run dataset, score results, optionally optimize prompts."""
+    import asyncio
+    import json
+    from pathlib import Path
+
+    eval_cmd = getattr(args, "eval_cmd", None)
+
+    if eval_cmd == "stub" or eval_cmd is None and not hasattr(args, "dataset"):
+        # Generate stub dataset
+        from devsper.evals.dataset import EvalDataset
+
+        role = getattr(args, "role", "general")
+        n = getattr(args, "n", 5)
+        out = getattr(args, "out", None)
+        dataset = EvalDataset.stub(role=role, n=n)
+        if out:
+            dataset.save(out)
+            print(f"Stub dataset ({len(dataset)} cases) written to {out}")
+        else:
+            for case in dataset:
+                print(json.dumps(case.to_dict()))
+        return 0
+
+    if eval_cmd == "results":
+        from devsper.config import get_config
+
+        try:
+            results_dir = Path(getattr(args, "dir", None) or get_config().evals.results_dir)
+        except Exception:
+            results_dir = Path(".devsper/eval_results")
+        if not results_dir.exists():
+            print(f"No results found in {results_dir}")
+            return 0
+        files = sorted(results_dir.glob("*.json"), key=lambda p: p.stat().st_mtime, reverse=True)
+        if not files:
+            print("No eval result files found.")
+            return 0
+        for f in files[:20]:
+            try:
+                data = json.loads(f.read_text())
+                print(
+                    f"  {f.name}  role={data.get('role', '?')}  "
+                    f"pass_rate={data.get('pass_rate', '?')}  "
+                    f"mean_score={data.get('mean_score', '?')}"
+                )
+            except Exception:
+                print(f"  {f.name}")
+        return 0
+
+    # eval_cmd == "run"
+    from devsper.evals.dataset import EvalDataset
+    from devsper.evals.metrics import get_metric
+    from devsper.evals.runner import EvalRunner
+    from devsper.config import get_config
+
+    try:
+        cfg = get_config()
+    except Exception:
+        from devsper.config.schema import devsperConfigModel
+        cfg = devsperConfigModel()
+
+    dataset_path = getattr(args, "dataset", None)
+    if not dataset_path:
+        print("Error: --dataset is required for 'eval run'")
+        return 1
+
+    dataset = EvalDataset.load(dataset_path)
+    role = getattr(args, "role", None)
+    metric_name = getattr(args, "metric", None) or cfg.evals.default_metric
+    threshold = getattr(args, "threshold", None) or cfg.evals.pass_threshold
+    concurrency = getattr(args, "concurrency", None) or cfg.evals.concurrency
+    do_optimize = getattr(args, "optimize", False)
+    optimizer_override = getattr(args, "optimizer", None)
+    out_path = getattr(args, "out", None)
+
+    metric = get_metric(metric_name)
+
+    # Build optimizer if requested
+    optimizer = None
+    if do_optimize:
+        from devsper.prompt_optimizer.factory import get_prompt_optimizer, reset_prompt_optimizer
+
+        if optimizer_override:
+            import os
+            os.environ["DEVSPER_PROMPT_OPTIMIZER"] = optimizer_override
+            reset_prompt_optimizer()
+        optimizer = get_prompt_optimizer(cfg)
+
+    # Build a minimal agent for evaluation
+    from devsper.agents.agent import Agent
+
+    agent = Agent(model_name=cfg.models.worker, use_tools=False)
+
+    runner = EvalRunner(
+        agent=agent,
+        metric=metric,
+        pass_threshold=threshold,
+        concurrency=concurrency,
+        optimize_after=do_optimize,
+        optimizer=optimizer,
+    )
+
+    try:
+        summary = asyncio.run(runner.run_async(dataset, role=role))
+    except RuntimeError:
+        loop = asyncio.new_event_loop()
+        summary = loop.run_until_complete(runner.run_async(dataset, role=role))
+
+    # Print summary
+    try:
+        from rich.console import Console
+        from rich.table import Table
+
+        console = Console()
+        console.print(
+            f"\n[bold]Eval Results[/bold]  role=[cyan]{summary.role}[/cyan]  "
+            f"metric=[cyan]{summary.metric_name}[/cyan]  "
+            f"optimizer=[cyan]{summary.optimizer_backend}[/cyan]"
+        )
+        console.print(
+            f"  Passed: [green]{summary.passed}[/green]/{summary.total}  "
+            f"Pass rate: [bold]{summary.pass_rate:.1%}[/bold]  "
+            f"Mean score: [bold]{summary.mean_score:.3f}[/bold]\n"
+        )
+        table = Table(show_header=True, header_style="bold")
+        table.add_column("ID", style="dim")
+        table.add_column("Task", max_width=40)
+        table.add_column("Score")
+        table.add_column("Pass")
+        for r in summary.results:
+            color = "green" if r.passed else "red"
+            table.add_row(
+                r.case.id,
+                r.case.task[:40],
+                f"{r.score:.2f}",
+                f"[{color}]{'✓' if r.passed else '✗'}[/{color}]",
+            )
+        console.print(table)
+    except ImportError:
+        print(f"\nEval: role={summary.role} metric={summary.metric_name}")
+        print(f"  {summary.passed}/{summary.total} passed ({summary.pass_rate:.1%})")
+        print(f"  Mean score: {summary.mean_score:.3f}")
+
+    # Persist results
+    results_dir = Path(cfg.evals.results_dir)
+    results_dir.mkdir(parents=True, exist_ok=True)
+    ts = __import__("datetime").datetime.now().strftime("%Y%m%d_%H%M%S")
+    result_file = results_dir / f"eval_{summary.role}_{ts}.json"
+    result_file.write_text(summary.to_json())
+    print(f"\nResults saved to {result_file}")
+
+    if out_path:
+        Path(out_path).write_text(summary.to_json())
+
+    return 0 if summary.pass_rate >= threshold else 1
+
+
 def _run_health(args: object) -> int:
     """Run health checks. Exit 0 if healthy, 1 otherwise. Print ✓/✗ per check."""
     import asyncio
@@ -2811,7 +2969,7 @@ def _run_upgrade(args: object) -> int:
 
 
 def _run_cloud_dispatch(args: object) -> int:
-    """Devsper Platform (cloud): login, run, status, logs."""
+    """Devsper Cloud: login, run, status, logs."""
     cmd = getattr(args, "cloud_cmd", None)
     if not cmd:
         return 0
@@ -4082,6 +4240,64 @@ Examples:
     )
     observe_parser.set_defaults(func=lambda a: _run_observe(a.port, a.db))
 
+    eval_parser = subparsers.add_parser(
+        "eval",
+        help="Eval harness and prompt optimization",
+        description="Run evals against a JSONL dataset and optionally optimize prompts.",
+        epilog="""
+Examples:
+  devsper eval run --dataset evals.jsonl --metric contains
+  devsper eval run --dataset evals.jsonl --role research --optimize --optimizer dspy
+  devsper eval stub --role research --out evals.jsonl
+  devsper eval results
+""",
+        formatter_class=argparse.RawDescriptionHelpFormatter,
+    )
+    eval_sub = eval_parser.add_subparsers(dest="eval_cmd", help="Subcommand")
+
+    eval_run_p = eval_sub.add_parser("run", help="Run eval dataset")
+    eval_run_p.add_argument("--dataset", required=True, help="Path to JSONL dataset")
+    eval_run_p.add_argument("--role", default=None, help="Filter to this agent role")
+    eval_run_p.add_argument(
+        "--metric",
+        default=None,
+        help="Metric name: exact_match | contains | regex_match | word_overlap | llm_judge (default: config)",
+    )
+    eval_run_p.add_argument(
+        "--threshold", type=float, default=None, help="Pass threshold (default: config)"
+    )
+    eval_run_p.add_argument(
+        "--optimize",
+        action="store_true",
+        help="Run prompt optimization after eval using the configured optimizer",
+    )
+    eval_run_p.add_argument(
+        "--optimizer",
+        default=None,
+        help="Override optimizer backend: noop | dspy | gepa",
+    )
+    eval_run_p.add_argument(
+        "--concurrency", type=int, default=None, help="Parallel eval cases"
+    )
+    eval_run_p.add_argument("--out", default=None, help="Save JSON results to this path")
+    eval_run_p.set_defaults(eval_cmd="run")
+
+    eval_stub_p = eval_sub.add_parser("stub", help="Generate a stub dataset")
+    eval_stub_p.add_argument(
+        "--role", default="general", help="Agent role (research/code/analysis/general)"
+    )
+    eval_stub_p.add_argument("-n", type=int, default=5, help="Number of examples")
+    eval_stub_p.add_argument(
+        "--out", default=None, help="Output JSONL path (default: prints to stdout)"
+    )
+    eval_stub_p.set_defaults(eval_cmd="stub")
+
+    eval_results_p = eval_sub.add_parser("results", help="List recent eval result files")
+    eval_results_p.add_argument("--dir", default=None, help="Results directory")
+    eval_results_p.set_defaults(eval_cmd="results")
+
+    eval_parser.set_defaults(func=_run_eval)
+
     health_parser = subparsers.add_parser(
         "health",
         help="Health and readiness check",

{devsper-2.6.0 → devsper-2.7.1}/devsper/config/schema.py

@@ -74,13 +74,46 @@ class ModelsConfig(BaseModel):
     quality: str | None = None  # v1.6: complex tier (defaults to planner)
 
 
+class RedisMemoryConfig(BaseModel):
+    """Connection config for Redis memory backend."""
+
+    redis_url: str = "redis://localhost:6379"
+    run_id: str = ""  # empty = auto-generated from os.getpid()
+
+
+class SnowflakeMemoryConfig(BaseModel):
+    """Connection config for Snowflake memory backend.
+
+    Credentials (account, user, password, etc.) are resolved from:
+      1. devsper credential store (devsper credentials set snowflake <key>)
+      2. SNOWFLAKE_* environment variables
+      3. These TOML fields as a last fallback (non-secret fields only)
+
+    NEVER put passwords in config files — use the credential store.
+    """
+
+    account: str = ""       # or SNOWFLAKE_ACCOUNT env var
+    user: str = ""          # or SNOWFLAKE_USER env var
+    database: str = ""      # or SNOWFLAKE_DATABASE env var
+    schema_name: str = ""   # "schema" is reserved by Pydantic; or SNOWFLAKE_SCHEMA
+    warehouse: str = ""     # or SNOWFLAKE_WAREHOUSE env var
+    role: str = ""          # or SNOWFLAKE_ROLE env var
+    table: str = "devsper_memory"
+
+
 class MemoryConfig(BaseModel):
     enabled: bool = True
     store_results: bool = True
     top_k: int = 5
+    # New explicit provider field. Takes precedence over legacy `backend`.
+    # Values: "sqlite" | "redis" | "snowflake" | "vektori"
+    # Empty string = auto-detect (vektori if DATABASE_URL set, else sqlite).
+    provider: str = ""
     backend: Literal["local", "platform", "hybrid", "supermemory"] = "supermemory"
     platform_api_url: str = ""
     platform_org_slug: str = ""
+    redis: RedisMemoryConfig = Field(default_factory=RedisMemoryConfig)
+    snowflake: SnowflakeMemoryConfig = Field(default_factory=SnowflakeMemoryConfig)
 
 
 class KnowledgeConfig(BaseModel):
@@ -276,6 +309,32 @@ class HitlConfig(BaseModel):
     policies: list[HitlPolicyConfig] = Field(default_factory=list)
 
 
+class PromptOptimizerConfig(BaseModel):
+    """v2.7: Prompt optimization backend config.
+
+    provider: "noop" | "dspy" | "gepa"  (override with DEVSPER_PROMPT_OPTIMIZER env var)
+    """
+
+    provider: str = "noop"
+    # DSPy-specific
+    dspy_optimizer: str = "bootstrap"  # "bootstrap" | "mipro" | "bootstrap_random"
+    max_demos: int = 4
+    num_candidates: int = 10           # MIPROv2 only
+    # GEPA-specific
+    population_size: int = 5
+    n_iterations: int = 10
+
+
+class EvalConfig(BaseModel):
+    """v2.7: Eval harness config."""
+
+    dataset_dir: str = ".devsper/evals"   # default location for JSONL datasets
+    results_dir: str = ".devsper/eval_results"
+    pass_threshold: float = 0.5
+    concurrency: int = 4
+    default_metric: str = "contains"      # metric name from devsper.evals.metrics
+
+
 class devsperConfigModel(BaseModel):
     """Full resolved configuration with Pydantic validation."""
 
@@ -300,6 +359,8 @@ class devsperConfigModel(BaseModel):
     sandbox: SandboxConfig = Field(default_factory=SandboxConfig)
     compliance: ComplianceConfig = Field(default_factory=ComplianceConfig)
     hitl: HitlConfig = Field(default_factory=HitlConfig)
+    prompt_optimizer: PromptOptimizerConfig = Field(default_factory=PromptOptimizerConfig)
+    evals: EvalConfig = Field(default_factory=EvalConfig)
 
     # Backward-compat aliases (property-style access from old devsperConfig)
     @property

{devsper-2.6.0 → devsper-2.7.1}/devsper/credentials/cli.py

@@ -22,10 +22,12 @@ KNOWN_CREDENTIALS = {
     "gemini": ["api_key"],
     "azure": ["endpoint", "api_key", "deployment", "api_version"],
     "azure_anthropic": ["endpoint", "api_key", "deployment"],
+    "snowflake": ["account", "user", "password", "database", "schema", "warehouse", "role"],
+    "redis_memory": ["url"],
 }
 
 # Keys that must never be shown in list (only "(stored)"); all others can show value
-SENSITIVE_KEYS = {"api_key", "token"}
+SENSITIVE_KEYS = {"api_key", "token", "password"}
 
 # (provider, key) -> env var name for export
 PROVIDER_KEY_TO_ENV: dict[tuple[str, str], str] = {
@@ -40,6 +42,14 @@ PROVIDER_KEY_TO_ENV: dict[tuple[str, str], str] = {
     ("azure_anthropic", "endpoint"): "AZURE_ANTHROPIC_ENDPOINT",
     ("azure_anthropic", "api_key"): "AZURE_ANTHROPIC_API_KEY",
     ("azure_anthropic", "deployment"): "AZURE_ANTHROPIC_DEPLOYMENT_NAME",
+    ("snowflake", "account"): "SNOWFLAKE_ACCOUNT",
+    ("snowflake", "user"): "SNOWFLAKE_USER",
+    ("snowflake", "password"): "SNOWFLAKE_PASSWORD",
+    ("snowflake", "database"): "SNOWFLAKE_DATABASE",
+    ("snowflake", "schema"): "SNOWFLAKE_SCHEMA",
+    ("snowflake", "warehouse"): "SNOWFLAKE_WAREHOUSE",
+    ("snowflake", "role"): "SNOWFLAKE_ROLE",
+    ("redis_memory", "url"): "REDIS_URL",
 }
 
 

{devsper-2.6.0 → devsper-2.7.1}/devsper/credentials/store.py

@@ -19,6 +19,15 @@ _PROVIDER_KEY_TO_ENV: list[tuple[str, str, str]] = [
     ("azure_anthropic", "api_key", "AZURE_ANTHROPIC_API_KEY"),
     ("azure_anthropic", "endpoint", "AZURE_ANTHROPIC_ENDPOINT"),
     ("azure_anthropic", "deployment", "AZURE_ANTHROPIC_DEPLOYMENT_NAME"),
+    # Memory backends
+    ("snowflake", "account", "SNOWFLAKE_ACCOUNT"),
+    ("snowflake", "user", "SNOWFLAKE_USER"),
+    ("snowflake", "password", "SNOWFLAKE_PASSWORD"),
+    ("snowflake", "database", "SNOWFLAKE_DATABASE"),
+    ("snowflake", "schema", "SNOWFLAKE_SCHEMA"),
+    ("snowflake", "warehouse", "SNOWFLAKE_WAREHOUSE"),
+    ("snowflake", "role", "SNOWFLAKE_ROLE"),
+    ("redis_memory", "url", "REDIS_URL"),
 ]
 
 SERVICE_NAME = "devsper"
@@ -31,6 +40,8 @@ KNOWN_CREDENTIALS = {
     "gemini": ["api_key"],
     "azure": ["endpoint", "api_key", "deployment", "api_version"],
     "azure_anthropic": ["endpoint", "api_key", "deployment"],
+    "snowflake": ["account", "user", "password", "database", "schema", "warehouse", "role"],
+    "redis_memory": ["url"],
 }
 
 

devsper-2.7.1/devsper/evals/__init__.py

@@ -0,0 +1,19 @@
+"""
+devsper evals — eval harness and prompt optimization integration.
+"""
+
+from devsper.evals.types import EvalCase, EvalResult, EvalSummary, MetricFn
+from devsper.evals.dataset import EvalDataset
+from devsper.evals.metrics import get_metric, BUILTIN_METRICS
+from devsper.evals.runner import EvalRunner
+
+__all__ = [
+    "EvalCase",
+    "EvalResult",
+    "EvalSummary",
+    "EvalDataset",
+    "EvalRunner",
+    "MetricFn",
+    "get_metric",
+    "BUILTIN_METRICS",
+]