substrai-evalforge 0.1.0__tar.gz

substrai_evalforge-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025 Gaurav Kumar Sinha (Substrai AI)
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

substrai_evalforge-0.1.0/PKG-INFO

@@ -0,0 +1,129 @@
+Metadata-Version: 2.4
+Name: substrai-evalforge
+Version: 0.1.0
+Summary: Automated LLM evaluation pipeline generator
+Author-email: Gaurav Kumar Sinha <gaurav@substrai.dev>
+License: MIT
+Project-URL: Homepage, https://github.com/substrai/evalforge
+Project-URL: Repository, https://github.com/substrai/evalforge
+Keywords: llm,evaluation,testing,mlops,genai,rag,metrics,pipeline,serverless,aws-lambda
+Classifier: Development Status :: 3 - Alpha
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: pyyaml>=6.0
+Provides-Extra: aws
+Requires-Dist: boto3>=1.28.0; extra == "aws"
+Provides-Extra: dev
+Requires-Dist: pytest>=7.0; extra == "dev"
+Requires-Dist: pytest-cov>=4.0; extra == "dev"
+Dynamic: license-file
+
+# EvalForge
+
+**Automated LLM evaluation pipeline generator.**
+
+> Built by [SubstrAI](https://github.com/substrai) — Open-source GenAI frameworks for serverless infrastructure.
+
+[![PyPI version](https://badge.fury.io/py/substrai-evalforge.svg)](https://pypi.org/project/substrai-evalforge/)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
+[![Python 3.9+](https://img.shields.io/badge/python-3.9+-blue.svg)](https://www.python.org/downloads/)
+
+## The Problem
+
+Every team deploying LLMs builds evaluation pipelines from scratch. RAGAS and DeepEval are libraries — they don't generate infrastructure, schedule runs, detect drift, or route to human reviewers.
+
+## The Solution
+
+Describe your use case → EvalForge generates the complete evaluation pipeline:
+
+```yaml
+# evalforge.yaml
+use_case:
+  type: rag
+  description: "Customer support chatbot"
+  model:
+    provider: bedrock
+    model_id: anthropic.claude-3-haiku-20240307-v1:0
+
+evaluation:
+  metrics: auto  # auto-selects: faithfulness, relevancy, precision, recall, toxicity
+```
+
+```bash
+evalforge run
+# Faithfulness:      0.91 ✓ (threshold: 0.85)
+# Answer Relevancy:  0.87 ✓ (threshold: 0.80)
+# Context Precision: 0.78 ✓ (threshold: 0.75)
+# Toxicity:          0.02 ✓ (threshold: 0.05)
+# Overall: PASS (4/4 metrics passing)
+```
+
+## Features
+
+- **Use-case-driven metric selection** — describe your app, get optimal metrics
+- **6 use case types** — RAG, summarization, classification, generation, chat, code
+- **16+ built-in metrics** — faithfulness, ROUGE, BLEU, toxicity, injection resistance, F1
+- **Synthetic test data generation** — adversarial, edge cases, domain-specific
+- **Drift detection** — alerts when quality degrades over time
+- **Human-in-the-loop** — route uncertain evaluations to reviewers
+- **Scheduled pipelines** — daily/weekly automated evaluation runs
+- **Benchmark registry** — compare against published benchmarks
+- **One-command deploy** — Step Functions + Lambda infrastructure
+
+## Installation
+
+```bash
+pip install substrai-evalforge
+```
+
+## Quick Start
+
+```bash
+# Scaffold project
+evalforge init my-eval --use-case rag
+
+# Run evaluation
+cd my-eval
+evalforge run
+
+# List available metrics
+evalforge metrics --use-case rag
+```
+
+## Python SDK
+
+```python
+from evalforge import EvalPipeline
+
+# Quick start for any use case
+pipeline = EvalPipeline.for_use_case("rag")
+results = pipeline.run()
+print(results.summary())
+print(f"All passing: {results.all_passing}")
+```
+
+## Supported Use Cases & Auto-Selected Metrics
+
+| Use Case | Auto-Selected Metrics |
+|---|---|
+| **rag** | faithfulness, answer_relevancy, context_precision, context_recall, toxicity |
+| **summarization** | rouge_l, bleu, coherence, conciseness, fluency |
+| **classification** | accuracy, precision, recall, f1_score |
+| **generation** | fluency, coherence, toxicity, bias_detection |
+| **chat** | coherence, toxicity, injection_resistance, fluency |
+| **code** | accuracy, coherence |
+
+## License
+
+MIT — see [LICENSE](LICENSE)
+
+## Author
+
+**Gaurav Kumar Sinha** — Founder, [SubstrAI](https://github.com/substrai)
+
+- Email: gaurav@substrai.dev
+- GitHub: [@substrai](https://github.com/substrai)

substrai_evalforge-0.1.0/README.md

@@ -0,0 +1,105 @@
+# EvalForge
+
+**Automated LLM evaluation pipeline generator.**
+
+> Built by [SubstrAI](https://github.com/substrai) — Open-source GenAI frameworks for serverless infrastructure.
+
+[![PyPI version](https://badge.fury.io/py/substrai-evalforge.svg)](https://pypi.org/project/substrai-evalforge/)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
+[![Python 3.9+](https://img.shields.io/badge/python-3.9+-blue.svg)](https://www.python.org/downloads/)
+
+## The Problem
+
+Every team deploying LLMs builds evaluation pipelines from scratch. RAGAS and DeepEval are libraries — they don't generate infrastructure, schedule runs, detect drift, or route to human reviewers.
+
+## The Solution
+
+Describe your use case → EvalForge generates the complete evaluation pipeline:
+
+```yaml
+# evalforge.yaml
+use_case:
+  type: rag
+  description: "Customer support chatbot"
+  model:
+    provider: bedrock
+    model_id: anthropic.claude-3-haiku-20240307-v1:0
+
+evaluation:
+  metrics: auto  # auto-selects: faithfulness, relevancy, precision, recall, toxicity
+```
+
+```bash
+evalforge run
+# Faithfulness:      0.91 ✓ (threshold: 0.85)
+# Answer Relevancy:  0.87 ✓ (threshold: 0.80)
+# Context Precision: 0.78 ✓ (threshold: 0.75)
+# Toxicity:          0.02 ✓ (threshold: 0.05)
+# Overall: PASS (4/4 metrics passing)
+```
+
+## Features
+
+- **Use-case-driven metric selection** — describe your app, get optimal metrics
+- **6 use case types** — RAG, summarization, classification, generation, chat, code
+- **16+ built-in metrics** — faithfulness, ROUGE, BLEU, toxicity, injection resistance, F1
+- **Synthetic test data generation** — adversarial, edge cases, domain-specific
+- **Drift detection** — alerts when quality degrades over time
+- **Human-in-the-loop** — route uncertain evaluations to reviewers
+- **Scheduled pipelines** — daily/weekly automated evaluation runs
+- **Benchmark registry** — compare against published benchmarks
+- **One-command deploy** — Step Functions + Lambda infrastructure
+
+## Installation
+
+```bash
+pip install substrai-evalforge
+```
+
+## Quick Start
+
+```bash
+# Scaffold project
+evalforge init my-eval --use-case rag
+
+# Run evaluation
+cd my-eval
+evalforge run
+
+# List available metrics
+evalforge metrics --use-case rag
+```
+
+## Python SDK
+
+```python
+from evalforge import EvalPipeline
+
+# Quick start for any use case
+pipeline = EvalPipeline.for_use_case("rag")
+results = pipeline.run()
+print(results.summary())
+print(f"All passing: {results.all_passing}")
+```
+
+## Supported Use Cases & Auto-Selected Metrics
+
+| Use Case | Auto-Selected Metrics |
+|---|---|
+| **rag** | faithfulness, answer_relevancy, context_precision, context_recall, toxicity |
+| **summarization** | rouge_l, bleu, coherence, conciseness, fluency |
+| **classification** | accuracy, precision, recall, f1_score |
+| **generation** | fluency, coherence, toxicity, bias_detection |
+| **chat** | coherence, toxicity, injection_resistance, fluency |
+| **code** | accuracy, coherence |
+
+## License
+
+MIT — see [LICENSE](LICENSE)
+
+## Author
+
+**Gaurav Kumar Sinha** — Founder, [SubstrAI](https://github.com/substrai)
+
+- Email: gaurav@substrai.dev
+- GitHub: [@substrai](https://github.com/substrai)

substrai_evalforge-0.1.0/pyproject.toml

@@ -0,0 +1,40 @@
+[build-system]
+requires = ["setuptools>=68.0", "wheel"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "substrai-evalforge"
+version = "0.1.0"
+description = "Automated LLM evaluation pipeline generator"
+readme = "README.md"
+license = {text = "MIT"}
+requires-python = ">=3.9"
+authors = [
+    {name = "Gaurav Kumar Sinha", email = "gaurav@substrai.dev"},
+]
+keywords = ["llm", "evaluation", "testing", "mlops", "genai", "rag", "metrics", "pipeline", "serverless", "aws-lambda"]
+classifiers = [
+    "Development Status :: 3 - Alpha",
+    "License :: OSI Approved :: MIT License",
+    "Programming Language :: Python :: 3",
+    "Topic :: Scientific/Engineering :: Artificial Intelligence",
+]
+dependencies = ["pyyaml>=6.0"]
+
+[project.optional-dependencies]
+aws = ["boto3>=1.28.0"]
+dev = ["pytest>=7.0", "pytest-cov>=4.0"]
+
+[project.scripts]
+evalforge = "evalforge.cli.main:main"
+
+[project.urls]
+Homepage = "https://github.com/substrai/evalforge"
+Repository = "https://github.com/substrai/evalforge"
+
+[tool.setuptools.packages.find]
+where = ["src"]
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]
+pythonpath = ["src"]

substrai_evalforge-0.1.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

substrai_evalforge-0.1.0/src/evalforge/__init__.py

@@ -0,0 +1,31 @@
+"""
+EvalForge - Automated LLM Evaluation Pipeline Generator
+
+Describe your GenAI use case in a config file, and EvalForge generates
+the complete evaluation infrastructure: metrics selection, test data,
+scheduled pipelines, drift detection, and human-in-the-loop review.
+
+Usage:
+    from evalforge import EvalPipeline, EvalConfig, EvalResult
+
+    pipeline = EvalPipeline.from_config("evalforge.yaml")
+    results = pipeline.run()
+    print(results.summary())
+"""
+
+__version__ = "0.1.0"
+
+from evalforge.core.config import EvalConfig, UseCaseType
+from evalforge.core.pipeline import EvalPipeline
+from evalforge.core.result import EvalResult, MetricScore
+from evalforge.metrics.registry import MetricRegistry, get_metrics_for_use_case
+
+__all__ = [
+    "EvalConfig",
+    "EvalPipeline",
+    "EvalResult",
+    "MetricScore",
+    "UseCaseType",
+    "MetricRegistry",
+    "get_metrics_for_use_case",
+]

substrai_evalforge-0.1.0/src/evalforge/cli/__init__.py

@@ -0,0 +1,2 @@
+"""EvalForge CLI."""
+from evalforge.cli.main import main

substrai_evalforge-0.1.0/src/evalforge/cli/main.py

@@ -0,0 +1,182 @@
+"""EvalForge CLI - command-line interface."""
+
+from __future__ import annotations
+
+import argparse
+import sys
+from pathlib import Path
+
+from evalforge.core.config import EvalConfig, UseCaseType
+from evalforge.core.pipeline import EvalPipeline
+
+
+def cmd_init(args):
+    """Initialize a new EvalForge project."""
+    project_name = args.name or "my-evaluation"
+    use_case = args.use_case or "rag"
+    project_dir = Path(project_name)
+
+    if project_dir.exists():
+        print(f"Error: Directory '{project_name}' already exists")
+        sys.exit(1)
+
+    # Create structure
+    dirs = [
+        project_dir / "metrics",
+        project_dir / "data" / "golden",
+        project_dir / "data" / "synthetic",
+        project_dir / "judges",
+        project_dir / "reports",
+        project_dir / "tests",
+    ]
+    for d in dirs:
+        d.mkdir(parents=True, exist_ok=True)
+
+    # Create evalforge.yaml
+    config = f"""project:
+  name: "{project_name}"
+  version: "1.0.0"
+
+use_case:
+  type: {use_case}
+  description: "Evaluation pipeline for {use_case} application"
+  model:
+    provider: bedrock
+    model_id: anthropic.claude-3-haiku-20240307-v1:0
+    region: us-east-1
+
+evaluation:
+  metrics: auto  # auto-selects based on use_case.type
+  thresholds: {{}}  # uses defaults for {use_case}
+
+test_data:
+  source: synthetic
+  count: 100
+  categories: [simple, complex, adversarial, edge_cases]
+
+schedule:
+  frequency: daily
+  time: "02:00"
+"""
+    (project_dir / "evalforge.yaml").write_text(config)
+    (project_dir / "metrics" / "__init__.py").write_text("")
+    (project_dir / "judges" / "__init__.py").write_text("")
+
+    # README
+    readme = f"""# {project_name}
+
+LLM evaluation pipeline managed by [EvalForge](https://github.com/substrai/evalforge).
+
+## Quick Start
+
+```bash
+evalforge validate
+evalforge run
+evalforge report --last 7d
+```
+"""
+    (project_dir / "README.md").write_text(readme)
+
+    print(f"✓ Created EvalForge project: {project_name}/")
+    print(f"  Use case: {use_case}")
+    print(f"  ├── evalforge.yaml")
+    print(f"  ├── metrics/")
+    print(f"  ├── data/golden/")
+    print(f"  ├── data/synthetic/")
+    print(f"  └── reports/")
+    print(f"\nNext steps:")
+    print(f"  cd {project_name}")
+    print(f"  evalforge validate")
+    print(f"  evalforge run")
+
+
+def cmd_validate(args):
+    """Validate configuration."""
+    config_path = Path(args.config or "evalforge.yaml")
+    if not config_path.exists():
+        print(f"Error: Config not found: {config_path}")
+        sys.exit(1)
+
+    try:
+        config = EvalConfig.from_file(config_path)
+        print(f"✓ Configuration valid")
+        print(config.summary())
+    except (ValueError, FileNotFoundError) as e:
+        print(f"✗ Configuration error: {e}")
+        sys.exit(1)
+
+
+def cmd_run(args):
+    """Run evaluation pipeline."""
+    config_path = Path(args.config or "evalforge.yaml")
+
+    if config_path.exists():
+        pipeline = EvalPipeline.from_config(config_path)
+    else:
+        use_case = args.use_case or "rag"
+        pipeline = EvalPipeline.for_use_case(use_case)
+
+    metrics_filter = args.metrics.split(",") if args.metrics else None
+    results = pipeline.run(metrics=metrics_filter)
+
+    print(results.summary())
+
+    if not results.all_passing:
+        sys.exit(1)
+
+
+def cmd_metrics(args):
+    """List available metrics."""
+    from evalforge.metrics.registry import MetricRegistry
+    registry = MetricRegistry()
+
+    if args.use_case:
+        metrics = registry.get_metrics_for(args.use_case)
+        print(f"Metrics for '{args.use_case}':")
+        for m in metrics:
+            print(f"  • {m.name} — {m.description}")
+    else:
+        print("All available metrics:")
+        for name in registry.list_metrics():
+            metric = registry.get(name)
+            print(f"  • {name} [{metric.category}] — {metric.description}")
+
+
+def main():
+    """Main CLI entry point."""
+    parser = argparse.ArgumentParser(
+        prog="evalforge",
+        description="EvalForge - Automated LLM Evaluation Pipeline Generator",
+    )
+    subparsers = parser.add_subparsers(dest="command")
+
+    # init
+    init_p = subparsers.add_parser("init", help="Initialize a new project")
+    init_p.add_argument("name", nargs="?", default=None)
+    init_p.add_argument("--use-case", default="rag", choices=["rag", "summarization", "classification", "generation", "chat", "code"])
+
+    # validate
+    val_p = subparsers.add_parser("validate", help="Validate configuration")
+    val_p.add_argument("--config", default=None)
+
+    # run
+    run_p = subparsers.add_parser("run", help="Run evaluation pipeline")
+    run_p.add_argument("--config", default=None)
+    run_p.add_argument("--metrics", default=None, help="Comma-separated metrics to run")
+    run_p.add_argument("--use-case", default=None)
+
+    # metrics
+    met_p = subparsers.add_parser("metrics", help="List available metrics")
+    met_p.add_argument("--use-case", default=None)
+
+    args = parser.parse_args()
+    commands = {"init": cmd_init, "validate": cmd_validate, "run": cmd_run, "metrics": cmd_metrics}
+
+    if args.command in commands:
+        commands[args.command](args)
+    else:
+        parser.print_help()
+
+
+if __name__ == "__main__":
+    main()

substrai_evalforge-0.1.0/src/evalforge/core/__init__.py

@@ -0,0 +1,4 @@
+"""Core modules for EvalForge."""
+from evalforge.core.config import EvalConfig, UseCaseType
+from evalforge.core.pipeline import EvalPipeline
+from evalforge.core.result import EvalResult, MetricScore