factorforge-cds 3.4.5__tar.gz → 3.5.2__tar.gz

{factorforge_cds-3.4.5/src/factorforge_cds.egg-info → factorforge_cds-3.5.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: factorforge-cds
-Version: 3.4.5
+Version: 3.5.2
 Summary: FactorForge - open-source CDS design and pre-synthesis sequence review by Eijex.
 Author-email: Eijex <eijex.lab@gmail.com>
 License-Expression: AGPL-3.0-only
@@ -19,6 +19,9 @@ Requires-Dist: biopython>=1.81
 Requires-Dist: requests>=2.31
 Requires-Dist: click>=8.4.1
 Requires-Dist: pydantic>=2.0
+Provides-Extra: postgres
+Requires-Dist: psycopg2-binary>=2.9; extra == "postgres"
+Requires-Dist: sqlalchemy>=2.0; extra == "postgres"
 Provides-Extra: dev
 Requires-Dist: jsonschema>=4.0; extra == "dev"
 Requires-Dist: pandas>=2.0; extra == "dev"
@@ -29,6 +32,7 @@ Requires-Dist: pyyaml>=6.0; extra == "dev"
 Provides-Extra: ml
 Requires-Dist: transformers>=4.40; extra == "ml"
 Requires-Dist: torch>=2.0; extra == "ml"
+Requires-Dist: onnxruntime>=1.16; extra == "ml"
 Dynamic: license-file
 
 # FactorForge
@@ -42,10 +46,20 @@ Dynamic: license-file
 [![codecov](https://codecov.io/gh/eijex/factorforge-cds/branch/main/graph/badge.svg)](https://codecov.io/gh/eijex/factorforge-cds)
 [![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.20407330.svg)](https://doi.org/10.5281/zenodo.20407330)
 [![Web App](https://img.shields.io/badge/web-factorforge.eijex.com-brightgreen.svg)](https://factorforge.eijex.com)
-[![Sponsor](https://img.shields.io/badge/Sponsor-EA4AAA?style=flat&logo=github-sponsors&logoColor=white)](https://github.com/sponsors/eijex)
+[![Buy Me A Coffee](https://img.shields.io/badge/Buy%20Me%20a%20Coffee-Donate-yellow?style=flat&logo=buy-me-a-coffee)](https://www.buymeacoffee.com/eijex)
 
 
-FactorForge performs profile-guided CDS design with CAI/GC metrics, PolyA-signal screening, and Golden Gate/MoClo-aware checks. It is positioned as a pre-synthesis review harness: it helps teams generate reproducible CDS candidates, inspect assembly-relevant sequence constraints, and package design metadata before downstream synthesis, cloning, or experimental review. Primary support: *N. benthamiana* (agroinfiltration). Experimental host context: Tobacco BY-2 (`--host by2`).
+FactorForge performs deterministic CDS design with CAI/GC metrics, PolyA-signal screening, and Golden Gate/MoClo-aware checks. It is positioned as a pre-synthesis review harness: it helps teams generate reproducible CDS candidates, inspect assembly-relevant sequence constraints, and package design metadata before downstream synthesis, cloning, or experimental review. Primary support: *N. benthamiana* (agroinfiltration). Experimental host context: Tobacco BY-2 (`--host by2`).
+
+FactorForge v3.5.1 uses independently versioned engines:
+
+| Generation | Engine | Version | Availability |
+|---|---|---:|---|
+| Gen 1 | Rule/profile | 1.0.0 | Stable, public |
+| Gen 2 | DP v2 | 2.0.1 | Stable, public default feasibility path |
+| Gen 2 | DP v2.1.1 | 2.1.1 | Explicit local-guard path; computational evidence only |
+| Gen 3 | sLLM Hybrid | 0.2.0-preview.1 | Feature-gated constrained-generation research preview |
+| Rescue | Adaptive partial DP | 1.0.0 | Exact suffix rescue conditioned on a verified prefix |
 
 **→ [Full Documentation](https://eijex.github.io/factorforge-cds/)** · **[Roadmap](https://eijex.github.io/factorforge-cds/roadmap/)**
 
@@ -75,12 +89,30 @@ Or use the **[web app](https://factorforge.eijex.com)** — no installation requ
 
 ## Repository Structure
 
-The supported production engine is the deterministic profile engine under:
+The supported deterministic engines are the profile engine, stable DP v2, and
+the explicit DP v2.1.1 development candidate under:
 
 ```text
 src/factorforge/engines/profile/
+src/factorforge/engines/dp_v2.py
+src/factorforge/engines/dp_v2_1_1.py
+src/factorforge/engines/sllm/
+src/factorforge/discovery/
 ```
 
+DP v2.1.1 adds an exact active-layer 5′ GC guard and Aho-Corasick rejection of
+homopolymers of 6 nt or longer to the v2.1 initiation-aware objective. It emits
+local-composition metrics and, when ViennaRNA and sufficient transcript context
+are available, a separately evaluated 5′ MFE value. It is not the default; its
+single-target calibration does not establish holdout generalization or biological
+performance.
+
+The v3.5.1 discovery-slate surface generates versioned Top-K research candidates,
+applies a shared deterministic hard-constraint filter, and records generator and
+fallback lineage. The sLLM path is disabled by default. A partial-DP rescue solves
+an exact suffix conditioned on the retained prefix; it is not a claim of global
+optimality and does not establish biological performance.
+
 Historical implementation tracks are preserved under `archive/` for provenance
 and are not imported by the installed package or exposed as supported engines.
 
@@ -95,7 +127,7 @@ FactorForge outputs are **in-silico only** and have not been experimentally vali
 ## Citing
 
 ```
-FactorForge v3.4.5 (2026). Open-source constraint-based CDS design and sequence review.
+FactorForge v3.5.2 (2026). Open-source constraint-based CDS design and sequence review.
 Eijex. https://github.com/eijex/factorforge-cds
 ```
 
@@ -105,6 +137,12 @@ Eijex. https://github.com/eijex/factorforge-cds
 
 Mun-Kyu Kim ([@eijex](https://github.com/eijex))
 
+## Optional PostgreSQL support
+
+Install `pip install "factorforge-cds[postgres]"` only when using the PostgreSQL
+connector. Explicit local SQLite checkpoints do not require a PostgreSQL driver.
+Database integration tests require an explicitly configured `DATABASE_URL`.
+
 ## License
 
 GNU Affero General Public License v3.0 — see [LICENSE](LICENSE).
@@ -118,6 +156,12 @@ GNU Affero General Public License v3.0 — see [LICENSE](LICENSE).
 - **Sponsor** — Support our research via [GitHub Sponsors](https://github.com/sponsors/eijex) or [PayPal](https://paypal.me/munkyukim)
 - **Docs** — [eijex.github.io/factorforge-cds](https://eijex.github.io/factorforge-cds/)
 - **Wet-lab Feedback** — Public-safe feedback summaries are welcome via [Share Wet-lab Feedback (GitHub)](https://github.com/eijex/factorforge-cds/issues/new?template=wet_lab_result.yml). Do not submit raw sequences, confidential construct details, internal batch IDs, patient data, private contact information, exact process parameters, or confidential partner/customer data. Email `eijex.lab@gmail.com` for private or sensitive summaries. See [VALIDATION.md](VALIDATION.md) before submitting.
+- **Sequence Policy Profiles** — FactorForge applies the sequence-review policy
+  chosen by the user; it does not define or approve a laboratory SOP. The web
+  app provides an illustrative YAML example and active-profile export.
+  Sequence-free, non-confidential suggestions can be submitted through the
+  public policy-template form; use `eijex.lab@gmail.com` for private or
+  laboratory-specific profiles. See the [sequence policy boundary](docs/sequence-policy.md).
 - **GitHub Issues** — bugs, features: [github.com/eijex/factorforge-cds/issues](https://github.com/eijex/factorforge-cds/issues)
 - **Email** — eijex.lab@gmail.com
 - **FactorForge** — [factorforge.eijex.com](https://factorforge.eijex.com)

{factorforge_cds-3.4.5 → factorforge_cds-3.5.2}/README.md

@@ -9,10 +9,20 @@
 [![codecov](https://codecov.io/gh/eijex/factorforge-cds/branch/main/graph/badge.svg)](https://codecov.io/gh/eijex/factorforge-cds)
 [![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.20407330.svg)](https://doi.org/10.5281/zenodo.20407330)
 [![Web App](https://img.shields.io/badge/web-factorforge.eijex.com-brightgreen.svg)](https://factorforge.eijex.com)
-[![Sponsor](https://img.shields.io/badge/Sponsor-EA4AAA?style=flat&logo=github-sponsors&logoColor=white)](https://github.com/sponsors/eijex)
+[![Buy Me A Coffee](https://img.shields.io/badge/Buy%20Me%20a%20Coffee-Donate-yellow?style=flat&logo=buy-me-a-coffee)](https://www.buymeacoffee.com/eijex)
 
 
-FactorForge performs profile-guided CDS design with CAI/GC metrics, PolyA-signal screening, and Golden Gate/MoClo-aware checks. It is positioned as a pre-synthesis review harness: it helps teams generate reproducible CDS candidates, inspect assembly-relevant sequence constraints, and package design metadata before downstream synthesis, cloning, or experimental review. Primary support: *N. benthamiana* (agroinfiltration). Experimental host context: Tobacco BY-2 (`--host by2`).
+FactorForge performs deterministic CDS design with CAI/GC metrics, PolyA-signal screening, and Golden Gate/MoClo-aware checks. It is positioned as a pre-synthesis review harness: it helps teams generate reproducible CDS candidates, inspect assembly-relevant sequence constraints, and package design metadata before downstream synthesis, cloning, or experimental review. Primary support: *N. benthamiana* (agroinfiltration). Experimental host context: Tobacco BY-2 (`--host by2`).
+
+FactorForge v3.5.1 uses independently versioned engines:
+
+| Generation | Engine | Version | Availability |
+|---|---|---:|---|
+| Gen 1 | Rule/profile | 1.0.0 | Stable, public |
+| Gen 2 | DP v2 | 2.0.1 | Stable, public default feasibility path |
+| Gen 2 | DP v2.1.1 | 2.1.1 | Explicit local-guard path; computational evidence only |
+| Gen 3 | sLLM Hybrid | 0.2.0-preview.1 | Feature-gated constrained-generation research preview |
+| Rescue | Adaptive partial DP | 1.0.0 | Exact suffix rescue conditioned on a verified prefix |
 
 **→ [Full Documentation](https://eijex.github.io/factorforge-cds/)** · **[Roadmap](https://eijex.github.io/factorforge-cds/roadmap/)**
 
@@ -42,12 +52,30 @@ Or use the **[web app](https://factorforge.eijex.com)** — no installation requ
 
 ## Repository Structure
 
-The supported production engine is the deterministic profile engine under:
+The supported deterministic engines are the profile engine, stable DP v2, and
+the explicit DP v2.1.1 development candidate under:
 
 ```text
 src/factorforge/engines/profile/
+src/factorforge/engines/dp_v2.py
+src/factorforge/engines/dp_v2_1_1.py
+src/factorforge/engines/sllm/
+src/factorforge/discovery/
 ```
 
+DP v2.1.1 adds an exact active-layer 5′ GC guard and Aho-Corasick rejection of
+homopolymers of 6 nt or longer to the v2.1 initiation-aware objective. It emits
+local-composition metrics and, when ViennaRNA and sufficient transcript context
+are available, a separately evaluated 5′ MFE value. It is not the default; its
+single-target calibration does not establish holdout generalization or biological
+performance.
+
+The v3.5.1 discovery-slate surface generates versioned Top-K research candidates,
+applies a shared deterministic hard-constraint filter, and records generator and
+fallback lineage. The sLLM path is disabled by default. A partial-DP rescue solves
+an exact suffix conditioned on the retained prefix; it is not a claim of global
+optimality and does not establish biological performance.
+
 Historical implementation tracks are preserved under `archive/` for provenance
 and are not imported by the installed package or exposed as supported engines.
 
@@ -62,7 +90,7 @@ FactorForge outputs are **in-silico only** and have not been experimentally vali
 ## Citing
 
 ```
-FactorForge v3.4.5 (2026). Open-source constraint-based CDS design and sequence review.
+FactorForge v3.5.2 (2026). Open-source constraint-based CDS design and sequence review.
 Eijex. https://github.com/eijex/factorforge-cds
 ```
 
@@ -72,6 +100,12 @@ Eijex. https://github.com/eijex/factorforge-cds
 
 Mun-Kyu Kim ([@eijex](https://github.com/eijex))
 
+## Optional PostgreSQL support
+
+Install `pip install "factorforge-cds[postgres]"` only when using the PostgreSQL
+connector. Explicit local SQLite checkpoints do not require a PostgreSQL driver.
+Database integration tests require an explicitly configured `DATABASE_URL`.
+
 ## License
 
 GNU Affero General Public License v3.0 — see [LICENSE](LICENSE).
@@ -85,6 +119,12 @@ GNU Affero General Public License v3.0 — see [LICENSE](LICENSE).
 - **Sponsor** — Support our research via [GitHub Sponsors](https://github.com/sponsors/eijex) or [PayPal](https://paypal.me/munkyukim)
 - **Docs** — [eijex.github.io/factorforge-cds](https://eijex.github.io/factorforge-cds/)
 - **Wet-lab Feedback** — Public-safe feedback summaries are welcome via [Share Wet-lab Feedback (GitHub)](https://github.com/eijex/factorforge-cds/issues/new?template=wet_lab_result.yml). Do not submit raw sequences, confidential construct details, internal batch IDs, patient data, private contact information, exact process parameters, or confidential partner/customer data. Email `eijex.lab@gmail.com` for private or sensitive summaries. See [VALIDATION.md](VALIDATION.md) before submitting.
+- **Sequence Policy Profiles** — FactorForge applies the sequence-review policy
+  chosen by the user; it does not define or approve a laboratory SOP. The web
+  app provides an illustrative YAML example and active-profile export.
+  Sequence-free, non-confidential suggestions can be submitted through the
+  public policy-template form; use `eijex.lab@gmail.com` for private or
+  laboratory-specific profiles. See the [sequence policy boundary](docs/sequence-policy.md).
 - **GitHub Issues** — bugs, features: [github.com/eijex/factorforge-cds/issues](https://github.com/eijex/factorforge-cds/issues)
 - **Email** — eijex.lab@gmail.com
 - **FactorForge** — [factorforge.eijex.com](https://factorforge.eijex.com)

{factorforge_cds-3.4.5 → factorforge_cds-3.5.2}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "factorforge-cds"
-version = "3.4.5"
+version = "3.5.2"
 description = "FactorForge - open-source CDS design and pre-synthesis sequence review by Eijex."
 readme = "README.md"
 license = "AGPL-3.0-only"
@@ -27,6 +27,10 @@ dependencies = [
 ]
 
 [project.optional-dependencies]
+postgres = [
+    "psycopg2-binary>=2.9",
+    "sqlalchemy>=2.0",
+]
 dev = [
     "jsonschema>=4.0",
     "pandas>=2.0",
@@ -38,6 +42,7 @@ dev = [
 ml = [
     "transformers>=4.40",
     "torch>=2.0",
+    "onnxruntime>=1.16",
 ]
 
 [project.urls]
@@ -53,7 +58,7 @@ factorforge-validate = "factorforge.validation.cli:main"
 where = ["src"]
 
 [tool.setuptools.package-data]
-"factorforge" = ["data/*.json", "data/templates/*.json", "data/profiles/*.json", "data/reference/*.json", "schemas/*.json"]
+"factorforge" = ["data/*.json", "data/templates/*.json", "data/profiles/*.json", "data/reference/*.json", "schemas/*.json", "registry/*.json"]
 
 [tool.pytest.ini_options]
 testpaths = ["tests"]

{factorforge_cds-3.4.5 → factorforge_cds-3.5.2}/src/factorforge/__init__.py

@@ -4,7 +4,9 @@ FactorForge - Codon Optimization Platform
 profile: constraint-aware rule/profile engine
 """
 
-__version__ = "3.4.5"
+from factorforge.registry.versioning import product_version
+
+__version__ = product_version()
 __author__ = "Eijex"
 
 # Auto-register engines (safe when running from source tree)

{factorforge_cds-3.4.5 → factorforge_cds-3.5.2}/src/factorforge/analysis/metrics.py

@@ -434,3 +434,72 @@ def detect_invalid_codons(sequence: str) -> list[dict[str, Any]]:
         elif codon not in STANDARD_GENETIC_CODE:
             findings.append({"start": start, "end": start + 3, "codon": codon, "reason": "unknown"})
     return findings
+import math
+import collections
+
+def calculate_synonymous_entropy(sequence, codon_to_aa):
+    from factorforge.analysis.metrics import _codons
+    codons = _codons(sequence)
+    aa_to_codons = collections.defaultdict(list)
+    for c in codons:
+        aa = codon_to_aa.get(c, 'X')
+        aa_to_codons[aa].append(c)
+        
+    entropies = {}
+    for aa, used_codons in aa_to_codons.items():
+        if aa in ['M', 'W', 'X']:
+            entropies[aa] = 0.0
+            continue
+        total = len(used_codons)
+        counts = collections.Counter(used_codons)
+        h = 0.0
+        for c, count in counts.items():
+            p = count / total
+            h -= p * math.log2(p)
+        
+        # Max entropy for this AA
+        possible_codons = [k for k, v in codon_to_aa.items() if v == aa]
+        k = len(possible_codons)
+        h_max = math.log2(k) if k > 1 else 1.0
+        entropies[aa] = h / h_max if h_max > 0 else 0.0
+        
+    if not entropies:
+        return 0.0
+    return sum(entropies.values()) / len(entropies)
+
+def calculate_max_synonymous_share(sequence, codon_to_aa):
+    from factorforge.analysis.metrics import _codons
+    codons = _codons(sequence)
+    aa_to_codons = collections.defaultdict(list)
+    for c in codons:
+        aa = codon_to_aa.get(c, 'X')
+        aa_to_codons[aa].append(c)
+        
+    max_shares = {}
+    for aa, used_codons in aa_to_codons.items():
+        if aa in ['M', 'W', 'X']:
+            continue
+        total = len(used_codons)
+        counts = collections.Counter(used_codons)
+        if total == 0:
+            continue
+        max_share = max(counts.values()) / total
+        max_shares[aa] = max_share
+    return max_shares
+
+def calculate_reference_distance(sequence, reference_weights, codon_to_aa):
+    from factorforge.analysis.metrics import _codons
+    codons = _codons(sequence)
+    counts = collections.Counter(codons)
+    
+    distance = 0.0
+    for codon, ref_freq in reference_weights.items():
+        aa = codon_to_aa.get(codon, 'X')
+        # find sequence frequency for this codon among its synonymous peers
+        syn_codons = [c for c, a in codon_to_aa.items() if a == aa]
+        total_aa = sum(counts.get(c, 0) for c in syn_codons)
+        
+        seq_freq = counts.get(codon, 0) / total_aa if total_aa > 0 else 0.0
+        # simple absolute distance
+        distance += abs(seq_freq - ref_freq)
+    return distance

factorforge_cds-3.5.2/src/factorforge/analysis/sequence_risks.py

@@ -0,0 +1,97 @@
+"""Sequence risk scanners (PolyA, Repeats, Homopolymers, etc.)
+
+Extracted from legacy RuleEngine to serve as shared evaluation metrics.
+"""
+import re
+from typing import Dict, Any, List, Set, Tuple
+
+# Polyadenylation signals (true biological motifs)
+POLYA_PATTERNS = {
+    # Tier 1 (canonical, highest efficiency in plants/eukaryotes)
+    "AATAAA": "canonical",
+    "ATTAAA": "variant_1",
+    "AGTAAA": "variant_2",
+    # Tier 2 (lower-frequency but functional in plants)
+    "AATACA": "tier_2",
+    "AAGAAA": "tier_2",
+    "AATGAA": "tier_2",
+}
+
+UNSTABLE_MOTIFS = {
+    "ATTTA": "ARE (AU-rich element)", 
+    "WWWWWW": "W=A/T, 6+ in a row"
+}
+
+def scan_polya(seq: str) -> List[Dict[str, Any]]:
+    """Detect biological PolyA signal motifs (e.g. AATAAA)."""
+    seq_upper = seq.upper()
+    violations: List[Dict[str, Any]] = []
+    
+    for pattern, pattern_type in POLYA_PATTERNS.items():
+        pos = 0
+        while True:
+            idx = seq_upper.find(pattern, pos)
+            if idx == -1:
+                break
+            violations.append({
+                "type": "polya_signal",
+                "pattern": pattern,
+                "pattern_type": pattern_type,
+                "position": idx,
+                "context": seq_upper[max(0, idx - 10):min(len(seq_upper), idx + len(pattern) + 10)]
+            })
+            pos = idx + 1
+            
+    return violations
+
+def scan_homopolymers(seq: str, threshold: int = 6) -> List[Dict[str, Any]]:
+    """Find homopolymers (runs of any single nucleotide)."""
+    seq_upper = seq.upper()
+    violations: List[Dict[str, Any]] = []
+    for nt in "ATGC":
+        pattern = nt * threshold
+        pos = 0
+        while True:
+            idx = seq_upper.find(pattern, pos)
+            if idx == -1:
+                break
+            match = re.match(f"{nt}+", seq_upper[idx:])
+            length = match.end() if match else threshold
+            violations.append({
+                "type": "homopolymer",
+                "base": nt,
+                "length": length,
+                "position": idx,
+                "context": seq_upper[max(0, idx - 10):min(len(seq_upper), idx + length + 10)]
+            })
+            pos = idx + length
+    return violations
+
+def scan_repeats(seq: str, min_length: int = 20, max_distance: int = 1000) -> List[Dict[str, Any]]:
+    """Detect direct or inverted repeats causing assembly/stability issues.
+    
+    Direct repeats (exact k-mer matches) >= 20 bp are known to cause issues with 
+    DNA synthesis (e.g. Twist/IDT) and homologous recombination in host organisms.
+    """
+    seq_upper = seq.upper()
+    violations: List[Dict[str, Any]] = []
+    
+    # Naive k-mer search for direct repeats
+    # For a real implementation, a suffix tree or kmer hash map is better.
+    # We will use a fast localized search.
+    seen_kmers = {}
+    for i in range(len(seq_upper) - min_length + 1):
+        kmer = seq_upper[i:i + min_length]
+        if kmer in seen_kmers:
+            dist = i - seen_kmers[kmer]
+            if dist <= max_distance:
+                violations.append({
+                    "type": "direct_repeat",
+                    "pattern": kmer,
+                    "position1": seen_kmers[kmer],
+                    "position2": i,
+                    "distance": dist
+                })
+        seen_kmers[kmer] = i
+        
+    return violations

factorforge_cds-3.5.2/src/factorforge/benchmark/models.py

@@ -0,0 +1,62 @@
+from datetime import datetime
+from typing import Dict, Any, List, Optional
+from pydantic import BaseModel, Field
+
+from factorforge.evaluation.models import EvaluationResult
+
+
+class BenchmarkProvenance(BaseModel):
+    """Immutable provenance tracking for benchmark runs."""
+    input_sequence_hash: str
+    input_type: str = Field(description="protein or CDS")
+    terminal_stop_policy: str
+    codon_reference_id: str
+    constraint_profile_id: str
+    evaluator_version: str
+    method_version: str
+    git_commit: str
+    seed: Optional[int] = None
+    inference_mode: str
+    runtime_environment: str
+    timestamp: str = Field(default_factory=lambda: datetime.utcnow().isoformat())
+
+
+class BenchmarkTarget(BaseModel):
+    """A target sequence for benchmarking."""
+    target_id: str
+    protein_sequence: str
+    metadata: Dict[str, Any] = Field(default_factory=dict)
+
+
+class BenchmarkRunConfig(BaseModel):
+    """Configuration for a benchmark execution."""
+    suite_name: str = Field(..., description="e.g., Engineering Benchmark Suite")
+    host: str = "nbenthamiana"
+    target_gc_min_percent: float
+    target_gc_max_percent: float
+    forbidden_type_iis: List[str] = Field(default_factory=list)
+    terminal_stop_policy: str = "append"
+    seed: int = 42
+
+
+class EngineBenchmarkResult(BaseModel):
+    """Benchmark result for a single engine on a single target."""
+    engine_name: str
+    target_id: str
+    provenance: BenchmarkProvenance
+    optimized_sequence: str
+    evaluation: EvaluationResult
+    runtime_seconds: float
+
+
+class BenchmarkSuiteReport(BaseModel):
+    """Aggregated benchmark report across all targets and engines."""
+    suite_name: str
+    run_timestamp: str = Field(default_factory=lambda: datetime.utcnow().isoformat())
+    config: BenchmarkRunConfig
+    results: List[EngineBenchmarkResult]
+    
+    # Summary statistics mapped by engine
+    pass_counts: Dict[str, str] = Field(description="Raw pass counts e.g., '3/4'")
+    average_cai: Dict[str, Optional[float]]
+    average_gc: Dict[str, float]

factorforge_cds-3.5.2/src/factorforge/benchmark/runner.py

@@ -0,0 +1,144 @@
+from typing import List, Dict, Any, Type
+import time
+import hashlib
+
+from factorforge.engines import EngineRegistry, register_builtin_engines
+from factorforge.evaluation.evaluator import SharedEvaluator
+from factorforge.core.interfaces import OptimizerEngine
+from factorforge.benchmark.models import (
+    BenchmarkTarget,
+    BenchmarkRunConfig,
+    EngineBenchmarkResult,
+    BenchmarkSuiteReport,
+    BenchmarkProvenance
+)
+from factorforge.analysis.metrics import load_codon_usage_table
+
+# Initialize engines
+register_builtin_engines()
+
+class BenchmarkRunner:
+    """AgentOS Benchmark Harness for evaluating sequence generation methods."""
+
+    def __init__(self, config: BenchmarkRunConfig):
+        self.config = config
+        
+        # Load canonical host table to enable CAI in the SharedEvaluator
+        # Default internal path loads N. benthamiana by default, which is our benchmark host.
+        table = load_codon_usage_table() 
+        
+        self.evaluator = SharedEvaluator(
+            version="1.0.1", 
+            codon_weights=table.codon_weights
+        )
+
+    def run_suite(self, targets: List[BenchmarkTarget], engine_names: List[str]) -> BenchmarkSuiteReport:
+        """Execute the benchmark suite across requested engines and targets."""
+        
+        engines: Dict[str, OptimizerEngine] = {}
+        for name in engine_names:
+            engine_instance = EngineRegistry.get(name)
+            if not engine_instance:
+                raise ValueError(f"Engine '{name}' not found in registry.")
+            engines[name] = engine_instance
+
+        results: List[EngineBenchmarkResult] = []
+
+        for target in targets:
+            input_hash = hashlib.sha256(target.protein_sequence.encode()).hexdigest()
+
+            for engine_name, engine in engines.items():
+                
+                mode = getattr(engine, "inference_mode", "standard")
+                
+                provenance = BenchmarkProvenance(
+                    input_sequence_hash=input_hash,
+                    input_type="protein",
+                    terminal_stop_policy=self.config.terminal_stop_policy,
+                    codon_reference_id=f"host_{self.config.host}",
+                    constraint_profile_id="default_gc",
+                    evaluator_version=self.evaluator.version,
+                    method_version=engine.version,
+                    git_commit="current", # In real usage, this would be git rev-parse HEAD
+                    seed=self.config.seed,
+                    inference_mode=mode,
+                    runtime_environment="default"
+                )
+
+                start_time = time.time()
+                
+                # Execute generation
+                opt_result = engine.optimize(
+                    sequence=target.protein_sequence,
+                    host=self.config.host,
+                    target_gc_min=self.config.target_gc_min_percent / 100.0, # Some engines still expect fraction
+                    target_gc_max=self.config.target_gc_max_percent / 100.0,
+                    forbidden_type_iis=set(self.config.forbidden_type_iis),
+                    terminal_stop_policy=self.config.terminal_stop_policy,
+                    seed=self.config.seed
+                )
+                
+                runtime = time.time() - start_time
+                
+                # ONE SharedEvaluator for all generated candidates
+                eval_result_obj = self.evaluator.evaluate_candidate(
+                    candidate_dna=opt_result.sequence,
+                    expected_protein=target.protein_sequence,
+                    candidate_id=f"{engine_name}-{target.target_id}",
+                    target_gc_min_percent=self.config.target_gc_min_percent,
+                    target_gc_max_percent=self.config.target_gc_max_percent,
+                    forbidden_type_iis=set(self.config.forbidden_type_iis)
+                )
+
+                res = EngineBenchmarkResult(
+                    engine_name=engine_name,
+                    target_id=target.target_id,
+                    provenance=provenance,
+                    optimized_sequence=opt_result.sequence,
+                    evaluation=eval_result_obj,
+                    runtime_seconds=runtime
+                )
+                results.append(res)
+                
+        return self._aggregate_results(results)
+
+    def _aggregate_results(self, results: List[EngineBenchmarkResult]) -> BenchmarkSuiteReport:
+        """Aggregate results into a full BenchmarkSuiteReport."""
+        
+        pass_counts = {}
+        total_targets = len(set(r.target_id for r in results))
+        
+        cai_sums = {}
+        cai_vals = {}
+        gc_sums = {}
+        engine_counts = {}
+        
+        for r in results:
+            eng = r.engine_name
+            if eng not in pass_counts:
+                pass_counts[eng] = 0
+                cai_vals[eng] = []
+                gc_sums[eng] = 0.0
+                engine_counts[eng] = 0
+                
+            if r.evaluation.passed:
+                pass_counts[eng] += 1
+                
+            if r.evaluation.metrics.cai is not None:
+                cai_vals[eng].append(r.evaluation.metrics.cai)
+                
+            gc_sums[eng] += r.evaluation.metrics.gc_percent
+            engine_counts[eng] += 1
+            
+        pass_counts_str = {e: f"{count}/{total_targets}" for e, count in pass_counts.items()}
+        avg_cai = {e: (sum(cai_vals[e]) / len(cai_vals[e]) if cai_vals[e] else None) for e in pass_counts}
+        avg_gc = {e: (gc_sums[e] / engine_counts[e]) for e in pass_counts}
+        
+        return BenchmarkSuiteReport(
+            suite_name=self.config.suite_name,
+            config=self.config,
+            results=results,
+            pass_counts=pass_counts_str,
+            average_cai=avg_cai,
+            average_gc=avg_gc
+        )