darwin-agents 0.7.0 → 0.7.1

package/CHANGELOG.md

@@ -2,6 +2,41 @@
 
 ## [Unreleased]
 
+## [0.7.1] — 2026-06-20
+
+**Documentation honesty + a reproducible evolution benchmark.** No library code
+changed — `src/` is byte-for-byte 0.7.0, **455 tests green**, `build` clean. This
+release makes every claim in the README survive scrutiny and ships a way to
+reproduce the one that matters.
+
+### Added
+
+- **`benchmark/`** — a reproducible evolution benchmark (`npm run benchmark`).
+  Ships the baseline `writer` prompt and the prompt Darwin's own optimizer
+  produced from it, a frozen held-out task set, and the exact critic-scoring
+  loop, so anyone can reproduce the baseline-vs-evolved delta on their own tasks.
+  `--quick` (1 task) and `--dry` (validate wiring, zero LLM calls) flags.
+
+### Changed
+
+- **README metrics are now real and dated.** Replaced the stale "300+ runs /
+  7.2–7.8" block with actual figures from 419 runs across 19 agents
+  (Mar–Jun 2026), including the measured v1→v2 evolution lift (writer +0.23,
+  marketing +0.28).
+- **Removed the "Darwin Pro — coming soon" tier.** PostgreSQL already ships free
+  in the open-source package; the old table implied a paywall and even listed
+  Postgres itself as a paid feature. Replaced with an honest "SQLite or
+  PostgreSQL — both free, both MIT" section + an open roadmap for the
+  genuinely-unbuilt features (pgvector semantic search, cross-agent learnings,
+  analytics, contradiction detection, data export).
+- **Clarified the LLM-as-judge mitigation.** Critics are multi-*dimension* by
+  default; the CLI spreads them across model families (GPT + Claude) only when
+  more than one provider key is present.
+- **Comparison table:** "MCP native" → "MCP-native memory bridge" — the specific,
+  defensible claim (other frameworks added MCP tool use during 2025).
+- Surfaced the production safety gate, online GEPA, and always-valid sequential
+  tests in a "why this isn't a toy" callout near the top of the README.
+
 ## [0.7.0] — 2026-06-20
 
 **Statistical rigor + coverage sampling.** Seven additive, opt-in upgrades that

package/README.md

@@ -79,6 +79,16 @@ Your agent got better.
 You did nothing.
 ```
 
+> **Why this isn't a toy.** The loop runs behind a real production **safety gate**
+> — regression rollback to last-known-good, data-quality guards that pause
+> evolution during a tool outage, and an alignment check on *every* mutation so a
+> rewrite can't quietly erode a safety constraint. It can drive that mutation with
+> a [GEPA](https://arxiv.org/abs/2507.19457) reflective optimizer running **online,
+> inside the gate** (not as an offline batch step), and the A/B gate supports
+> **always-valid sequential tests** (mSPRT / Hoeffding) so checking after every run
+> doesn't inflate false positives. Details: [reflective evolution](#reflective-evolution--gepa-online-v06-opt-in)
+> · [statistical rigor](#statistical-rigor--coverage-sampling-v07-opt-in).
+
 ## Quick Start
 
 ```bash
@@ -313,28 +323,31 @@ last run's lessons.
 
 <!-- REAL_METRICS_START -->
 
-### Real results from 300+ production runs
+### Real results from our own production use (Mar–Jun 2026)
 
-These are actual metrics from our development — not synthetic benchmarks.
+Actual numbers from **419 runs across 19 agents** in our internal `darwin_db` —
+not synthetic benchmarks. "Success" means the run completed and produced valid
+output (100% across 419 runs); "quality" is the critic's separate 1–10 score.
 
 ```
-Agents:      writer, researcher, marketing, blog-writer
-Total Runs:  300+
-Success Rate: 100%
-
-Writer:       7.2/10  (120 runs across tech, webdesign, market)
-Marketing:    7.8/10  (70 runs across LinkedIn, Instagram)
-Researcher:   7.6/10  (50+ runs, web research with citations)
+Agent          Runs   Avg quality
+writer          172   6.94 / 10
+marketing        70   7.74 / 10
+investigator     28   8.33 / 10
+blog-writer       5   8.20 / 10
 ```
 
-#### Multi-Model Critics in action
+**Evolution, measured.** When the safety gate adopted an evolved prompt, the
+critic score rose on the runs that followed — modest, but real and directional:
 
 ```
-platform-compliance  ████████░░  8/10
-scroll-stopping      ████████░░  8/10
-conversion-intent    ████████░░  8/10
+writer      v1  6.89 (126 runs)  →  v2  7.12 (42 runs)    +0.23
+marketing   v1  7.64 (45 runs)   →  v2  7.92 (25 runs)    +0.28
 ```
 
+Don't take our word for it — reproduce the v1-vs-evolved comparison on your own
+tasks with [`npm run benchmark`](benchmark/).
+
 <!-- REAL_METRICS_END -->
 
 ## Feature Comparison
@@ -346,7 +359,7 @@ conversion-intent    ████████░░  8/10
 | Safety gate + rollback | **Yes** | No | No | No | No |
 | TypeScript native | **Yes** | No (Python) | No (Python) | No (Python) | No (Python) |
 | Zero-config first agent | **Yes** | No | No | No | Partial |
-| MCP native | **Yes** | No | No | No | No |
+| MCP-native memory bridge | **Yes** | No | No | No | No |
 | File-based (no DB required) | **Yes** | No | No | No | No |
 | Built-in Critic agent | **Yes** | No | No | No | No |
 
@@ -374,8 +387,10 @@ Darwin uses SQLite by default — zero config, single file, no database to insta
     └── exp-researcher-2026-03-08-002.md
 ```
 
-**Want semantic search, cross-agent learnings, and analytics?**
-Upgrade to [Darwin Pro](#darwin-pro) for PostgreSQL + pgvector support.
+**Want concurrent multi-process writes and richer analytics?**
+PostgreSQL is supported out of the box, for free — set `DARWIN_POSTGRES_URL`.
+Semantic search (pgvector), cross-agent learnings and analytics are on the
+[roadmap](#storage-sqlite-or-postgresql--both-free-both-mit), not gated behind a paywall.
 
 ## CLI Reference
 
@@ -393,28 +408,32 @@ darwin evolve writer --reset       # Reset to v1
 darwin create my-agent             # Scaffold a new agent
 ```
 
-## Darwin Pro
-
-The free version uses SQLite — great for getting started, handles thousands of experiments.
+## Storage: SQLite or PostgreSQL — both free, both MIT
 
-For teams and production use, Darwin Pro adds:
+Darwin runs on SQLite by default (zero config, single file) and on PostgreSQL
+out of the box — just set `DARWIN_POSTGRES_URL`. Both backends ship in the
+open-source package. There is no paywall.
 
-| Feature | Free (SQLite) | Pro (PostgreSQL) |
+| Capability | SQLite | PostgreSQL |
 |---------|:---:|:---:|
 | Experiment tracking | ✓ | ✓ |
 | Prompt versioning | ✓ | ✓ |
-| A/B testing | ✓ | ✓ |
-| Safety gate | ✓ | ✓ |
-| Keyword search | ✓ | ✓ |
-| **Semantic search** (pgvector) | — | ✓ |
-| **Cross-agent learnings** | — | ✓ |
-| **Analytics & time series** | — | ✓ |
-| **Contradiction detection** | — | ✓ |
-| **Team support** (multi-user) | — | ✓ |
-| **Data export** (CSV/JSON) | — | ✓ |
-| **Learning decay** | — | ✓ |
-
-*Coming soon. Follow the repo for updates.*
+| A/B testing + safety gate | ✓ | ✓ |
+| Keyword search | ✓ (FTS5) | ✓ (GIN / `ts_rank`) |
+| Concurrent multi-process writes | — | ✓ |
+
+### Roadmap
+
+Not built yet — tracked in the open, PRs welcome:
+
+- Semantic search (pgvector embeddings)
+- Cross-agent learnings
+- Analytics & time series
+- Contradiction detection
+- Data export (CSV/JSON)
+
+The core stays MIT. If a hosted option ever ships, the self-host path keeps
+every feature.
 
 ## FAQ
 
@@ -438,7 +457,7 @@ The safety gate prevents regressions. If a new variant scores >20% lower, Darwin
 
 ## Known Limitations
 
-- **LLM-as-Judge bias**: Critics use LLMs to evaluate LLM outputs. We mitigate this with multi-model critics (GPT + Claude), but inherent self-preference bias exists. [Research context](https://openreview.net/forum?id=Ns8zGZ0lmM).
+- **LLM-as-Judge bias**: Critics use LLMs to evaluate LLM outputs. Each agent is scored by a **multi-dimension critic set** (several scoring rubrics per agent type, not a single number). When more than one provider key is present, the CLI also spreads those critics across model families — e.g. GPT for one, Claude for another — to blunt single-model self-preference; with one provider they all run on it. Inherent judge bias still exists. [Research context](https://openreview.net/forum?id=Ns8zGZ0lmM).
 - **Statistical simplicity (default)**: A/B tests use mean comparison with a 5% threshold by default, not formal significance tests. `computeDynamicMinRuns()` adjusts sample sizes based on variance. For rigor, v0.6 added an opt-in `requireConfidence` effect-size gate and **v0.7 ships proper always-valid sequential tests** — set `confidenceMethod: 'msprt'` (Mixture SPRT) or `'hoeffding'` (σ-free confidence sequence) on your `SafetyThresholds` to make the peeking-resistant gate statistically sound. The default path remains the simple threshold for zero-config use.
 - **No human-in-the-loop approval**: Prompt mutations go directly to A/B testing. Telegram notifications inform you, but there's no approval gate before testing starts.
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "darwin-agents",
-  "version": "0.7.0",
+  "version": "0.7.1",
   "description": "AI agents that improve themselves. Self-evolving prompts via A/B testing, multi-model critics, safety gates, and pattern detection.",
   "type": "module",
   "bin": {
@@ -52,6 +52,7 @@
     "build": "tsc",
     "dev": "tsc --watch",
     "darwin": "tsx src/cli/index.ts",
+    "benchmark": "tsx benchmark/evolution-benchmark.ts",
     "test": "tsx --test tests/*.test.ts",
     "typecheck:tests": "tsc -p tsconfig.test.json",
     "prepublishOnly": "npm run build && npm test"