redcodegen 0.1.0b0__tar.gz

redcodegen-0.1.0b0/PKG-INFO

@@ -0,0 +1,131 @@
+Metadata-Version: 2.3
+Name: redcodegen
+Version: 0.1.0b0
+Summary: Add your description here
+Requires-Dist: click>=8.0.0
+Requires-Dist: cwe2>=3.0.0
+Requires-Dist: dspy>=3.0.3
+Requires-Dist: jsonlines>=4.0.0
+Requires-Dist: pandas>=2.3.3
+Requires-Dist: python-dotenv>=1.1.1
+Requires-Dist: rich>=14.2.0
+Requires-Dist: rich-click>=1.9.3
+Requires-Dist: scipy>=1.16.3
+Requires-Dist: semgrep>=1.86.0
+Requires-Python: >=3.11
+Description-Content-Type: text/markdown
+
+<h1 align="center">
+    <em>RedCodeGen</em>
+</h1>
+
+<p align="center">
+<a href="https://pypi.org/project/redcodegen/" target="_blank">
+    <img src="https://img.shields.io/pypi/v/redcodegen.svg", alt="PyPi Version">
+</a>
+<a href="https://github.com/sisl/redcodegen/blob/main/LICENSE" target="_blank">
+    <img src="https://img.shields.io/badge/License-MIT-green.svg", alt="License">
+</a>
+</p>
+
+Automatic generation of *benign* prompts and language model rollouts in Python that exercise specific software vulnerabilities (CWEs) defined in the [MITRE CWE database](https://cwe.mitre.org/). 
+
+Developed by the Stanford Intelligent Systems Laboratory (SISL) as a part of [astra-rl](https://github.com/sisl/astra-rl).
+
+## Features
+
+- Generation of realistic coding task prompts that exercise specific CWEs
+- Generation of code samples for specific CWEs or CWE Top 25
+- Automatic code evaluation and vulnerability detection via [CodeQL static analysis](https://codeql.github.com/)
+- Programmable API for custom scenarios and configurations
+
+## Installation
+
+### CodeQL
+**First, you must install CodeQL and have it available in your PATH.** 
+
+- macOS Users: `brew install codeql`
+- Windows/Linux Users: follow the instructions [here](https://docs.github.com/en/code-security/codeql-cli/getting-started-with-the-codeql-cli/setting-up-the-codeql-cli)
+
+### RedCodeGen
+
+RedCodeGen is available via PyPI. Install it with pip:
+
+```bash
+pip install redcodegen
+```
+
+You would also want to create a .env file with your API key in your working directory:
+
+```bash
+echo "OPENAI_API_KEY=your_openai_api_key" > .env
+```
+
+## Quick Start
+
+The most basic usage involves rolling out a language model to generate code samples for specific CWEs and evaluating them with CodeQL.
+
+Suppose you want to rollout 5 samples each to exercise CWE-89 (SQL Injection) and CWE-79 (Cross-Site Scripting):
+
+```bash
+redcodegen generate -c 89 -c 79 -n 5 -o results.jsonl
+```
+
+You will get a `results.jsonl` file with the generated samples and their evaluations. Each CWE will live on a line. Let's take a peak!
+
+```bash
+head -n 1 results.jsonl | jq .
+```
+
+```json
+{
+  "cwe_id": 89,
+  "cwe_name": "SQL Injection",
+  "cwe_description": "SQL Injection is a code injection technique that might destroy your database. It is one of the most common web hacking techniques.",
+  "timestamp": "2024-06-01T12:00:00Z",
+  "model_config": {"model": "openai/gpt-4o-mini"},
+  "min_scenarios": 5,
+  "samples": [
+    {
+      "scenario": "A web application that takes user input and constructs SQL queries with proper sanitization.",
+      "code": "...generated code here...",
+      "evaluation": [
+        "rule": "py/sql-injection",
+        "message": "...",
+        "line": ...
+      ]
+    },
+    ...
+  ]
+}
+```
+
+Importantly, running the above command multiple times (to the same output file) will resume from where you left off, skipping CWEs that have already been processed in the output file.
+
+## Usage Examples
+
+```bash
+redcodegen generate -c 89 -c 79 # manually specify cwe
+redcodegen generate -n 5 # specify number of rollouts
+redcodegen generate --use-top-25 # run CWE top 25
+redcodegen generate --use-top-25 -o results.jsonl # resume existing run
+redcodegen generate --use-top-25 --model openai/gpt-4o # switch model
+```
+
+Also, you can run
+
+```bash
+redcodegen --help
+```
+
+to see all available options.
+
+## Method
+RedCodeGen works in three main steps:
+
+1. **Prompt Generation**: for each specified CWE, RedCodeGen generates a realistic coding task prompt that is likely to exercise the vulnerability. We do this by first looking up the CWE description from the MITRE CWE database, then prompting your specified language model to generate a coding task prompt based on that description. These descriptions are few-shot trained via existing human-written prompts from [Pearce, 2021](https://arxiv.org/abs/2108.09293).
+2. **Code Generation**: RedCodeGen then rolls out the specified language model on the generated prompt a few times with a sampling temperature of 0.8 to generate multiple code samples.
+3. **Code Evaluation**: Finally, RedCodeGen evaluates each generated code sample using CodeQL static analysis to detect whether the intended vulnerability is present in the code.
+
+## Acknowledgements
+We thank the Schmidt Sciences Foundation's trustworthy AI agenda for supporting this work.

redcodegen-0.1.0b0/README.md

@@ -0,0 +1,114 @@
+<h1 align="center">
+    <em>RedCodeGen</em>
+</h1>
+
+<p align="center">
+<a href="https://pypi.org/project/redcodegen/" target="_blank">
+    <img src="https://img.shields.io/pypi/v/redcodegen.svg", alt="PyPi Version">
+</a>
+<a href="https://github.com/sisl/redcodegen/blob/main/LICENSE" target="_blank">
+    <img src="https://img.shields.io/badge/License-MIT-green.svg", alt="License">
+</a>
+</p>
+
+Automatic generation of *benign* prompts and language model rollouts in Python that exercise specific software vulnerabilities (CWEs) defined in the [MITRE CWE database](https://cwe.mitre.org/). 
+
+Developed by the Stanford Intelligent Systems Laboratory (SISL) as a part of [astra-rl](https://github.com/sisl/astra-rl).
+
+## Features
+
+- Generation of realistic coding task prompts that exercise specific CWEs
+- Generation of code samples for specific CWEs or CWE Top 25
+- Automatic code evaluation and vulnerability detection via [CodeQL static analysis](https://codeql.github.com/)
+- Programmable API for custom scenarios and configurations
+
+## Installation
+
+### CodeQL
+**First, you must install CodeQL and have it available in your PATH.** 
+
+- macOS Users: `brew install codeql`
+- Windows/Linux Users: follow the instructions [here](https://docs.github.com/en/code-security/codeql-cli/getting-started-with-the-codeql-cli/setting-up-the-codeql-cli)
+
+### RedCodeGen
+
+RedCodeGen is available via PyPI. Install it with pip:
+
+```bash
+pip install redcodegen
+```
+
+You would also want to create a .env file with your API key in your working directory:
+
+```bash
+echo "OPENAI_API_KEY=your_openai_api_key" > .env
+```
+
+## Quick Start
+
+The most basic usage involves rolling out a language model to generate code samples for specific CWEs and evaluating them with CodeQL.
+
+Suppose you want to rollout 5 samples each to exercise CWE-89 (SQL Injection) and CWE-79 (Cross-Site Scripting):
+
+```bash
+redcodegen generate -c 89 -c 79 -n 5 -o results.jsonl
+```
+
+You will get a `results.jsonl` file with the generated samples and their evaluations. Each CWE will live on a line. Let's take a peak!
+
+```bash
+head -n 1 results.jsonl | jq .
+```
+
+```json
+{
+  "cwe_id": 89,
+  "cwe_name": "SQL Injection",
+  "cwe_description": "SQL Injection is a code injection technique that might destroy your database. It is one of the most common web hacking techniques.",
+  "timestamp": "2024-06-01T12:00:00Z",
+  "model_config": {"model": "openai/gpt-4o-mini"},
+  "min_scenarios": 5,
+  "samples": [
+    {
+      "scenario": "A web application that takes user input and constructs SQL queries with proper sanitization.",
+      "code": "...generated code here...",
+      "evaluation": [
+        "rule": "py/sql-injection",
+        "message": "...",
+        "line": ...
+      ]
+    },
+    ...
+  ]
+}
+```
+
+Importantly, running the above command multiple times (to the same output file) will resume from where you left off, skipping CWEs that have already been processed in the output file.
+
+## Usage Examples
+
+```bash
+redcodegen generate -c 89 -c 79 # manually specify cwe
+redcodegen generate -n 5 # specify number of rollouts
+redcodegen generate --use-top-25 # run CWE top 25
+redcodegen generate --use-top-25 -o results.jsonl # resume existing run
+redcodegen generate --use-top-25 --model openai/gpt-4o # switch model
+```
+
+Also, you can run
+
+```bash
+redcodegen --help
+```
+
+to see all available options.
+
+## Method
+RedCodeGen works in three main steps:
+
+1. **Prompt Generation**: for each specified CWE, RedCodeGen generates a realistic coding task prompt that is likely to exercise the vulnerability. We do this by first looking up the CWE description from the MITRE CWE database, then prompting your specified language model to generate a coding task prompt based on that description. These descriptions are few-shot trained via existing human-written prompts from [Pearce, 2021](https://arxiv.org/abs/2108.09293).
+2. **Code Generation**: RedCodeGen then rolls out the specified language model on the generated prompt a few times with a sampling temperature of 0.8 to generate multiple code samples.
+3. **Code Evaluation**: Finally, RedCodeGen evaluates each generated code sample using CodeQL static analysis to detect whether the intended vulnerability is present in the code.
+
+## Acknowledgements
+We thank the Schmidt Sciences Foundation's trustworthy AI agenda for supporting this work.

redcodegen-0.1.0b0/pyproject.toml

@@ -0,0 +1,40 @@
+[project]
+name = "redcodegen"
+version = "0.1.0-beta.0"
+description = "Add your description here"
+readme = "README.md"
+requires-python = ">=3.11"
+dependencies = [
+    "click>=8.0.0",
+    "cwe2>=3.0.0",
+    "dspy>=3.0.3",
+    "jsonlines>=4.0.0",
+    "pandas>=2.3.3",
+    "python-dotenv>=1.1.1",
+    "rich>=14.2.0",
+    "rich-click>=1.9.3",
+    "scipy>=1.16.3",
+    "semgrep>=1.86.0",
+]
+
+[project.scripts]
+redcodegen = "redcodegen.main:main"
+
+[tool.uv]
+package = true
+
+[build-system]
+requires = ["uv_build>=0.9.5,<0.10.0"]
+build-backend = "uv_build"
+
+[dependency-groups]
+dev = [
+    "ipdb>=0.13.13",
+    "seaborn>=0.13.2",
+]
+
+[tool.uv.build-backend]
+module-name = "redcodegen"
+module-root = ""
+source-include = ["./redcodegen/data/*"]
+

redcodegen-0.1.0b0/redcodegen/#main.py#

@@ -0,0 +1,263 @@
+"""
+main.py
+Main script for generating and evaluating vulnerable code samples
+"""
+
+import rich_click as click
+import jsonlines
+import logging
+import dspy
+from datetime import datetime
+from pathlib import Path
+from typing import List, Set, Dict, Any
+from cwe2.database import Database
+
+from redcodegen.constants import CWE_TOP_25, create_lm
+
+from rich.logging import RichHandler
+
+# Setup logging
+logging.basicConfig(
+    level=logging.INFO,
+    format="%(message)s",
+    handlers=[RichHandler(rich_tracebacks=True)]
+)
+logger = logging.getLogger(__name__)
+
+
+def load_completed_cwes(output_path: Path) -> Set[int]:
+    """Load CWE IDs that have already been processed.
+
+    Args:
+        output_path: Path to the output JSONL file
+
+    Returns:
+        Set of CWE IDs that are already in the output file
+    """
+    completed = set()
+
+    if not output_path.exists():
+        return completed
+
+    try:
+        with jsonlines.open(output_path) as reader:
+            for record in reader:
+                if 'cwe_id' in record:
+                    completed.add(record['cwe_id'])
+        logger.info(f"Found {len(completed)} already-completed CWEs in {output_path}")
+    except Exception as e:
+        logger.warning(f"Could not read existing output file: {e}")
+
+    return completed
+
+
+def get_model_config() -> Dict[str, Any]:
+    """Extract model configuration from current DSPy settings.
+
+    Returns:
+        Dict with model configuration info
+    """
+    lm = dspy.settings.lm
+    config = {
+        "model": getattr(lm, 'model', 'unknown'),
+    }
+
+    return config
+
+
+def build_record(
+    cwe_id: int,
+    cwe_name: str,
+    cwe_description: str,
+    scenarios: List[str],
+    codes: List[str],
+    evaluations: List[Any],
+    errors: List[str],
+    min_scenarios: int
+) -> Dict[str, Any]:
+    """Build a record for JSONL output.
+
+    Args:
+        cwe_id: CWE identifier
+        cwe_name: CWE name
+        cwe_description: CWE description
+        scenarios: List of scenario descriptions
+        codes: List of generated code samples
+        evaluations: List of evaluation results (can contain None for failures)
+        errors: List of error messages (None for successful evaluations)
+        min_scenarios: Minimum scenarios parameter used
+
+    Returns:
+        Dict representing the complete record for this CWE
+    """
+    samples = []
+    for scenario, code, evaluation, error in zip(scenarios, codes, evaluations, errors):
+        samples.append({
+            "scenario": scenario,
+            "code": code,
+            "evaluation": evaluation
+        })
+
+    return {
+        "cwe_id": cwe_id,
+        "cwe_name": cwe_name,
+        "cwe_description": cwe_description,
+        "timestamp": datetime.utcnow().isoformat() + 'Z',
+        "model_config": get_model_config(),
+        "min_scenarios": min_scenarios,
+        "samples": samples
+    }
+
+
+def append_to_jsonl(record: Dict[str, Any], output_path: Path):
+    """Append a record to the JSONL file.
+
+    Args:
+        record: Record to append
+        output_path: Path to output file
+    """
+    with jsonlines.open(output_path, mode='a') as writer:
+        writer.write(record)
+    logger.info(f"Saved CWE-{record['cwe_id']} to {output_path}")
+
+
+@click.command()
+@click.option(
+    '--cwes', '-c',
+    multiple=True,
+    type=int,
+    help='CWE IDs to process (can specify multiple times, e.g., -c 89 -c 79)'
+)
+@click.option(
+    '--use-top-25',
+    is_flag=True,
+    help='Process all CWE Top 25'
+)
+@click.option(
+    '--min-samples', '-n',
+    default=3,
+    type=int,
+    help='Minimum samples per CWE (default: 3)'
+)
+@click.option(
+    '--output', '-o',
+    default='results.jsonl',
+    type=click.Path(),
+    help='Output JSONL file (default: results.jsonl)'
+)
+@click.option(
+    '--model', '-m',
+    default='openai/gpt-4o-mini',
+    help='Model identifier (default: openai/gpt-4o-mini)'
+)
+@click.option(
+    '--api-key',
+    default=None,
+    help='API key (defaults to OPENAI_API_KEY env var)'
+)
+@
+def main(cwes, use_top_25, min_samples, output, model, api_key):
+    """Generate and evaluate vulnerable code samples for specified CWEs.
+
+    Examples:
+        python -m redcodegen -c 89 -c 79 # manually specify cwe
+        python -m redcodegen -n 5 # specify number of rollouts
+        python -m redcodegen --use-top-25 # run CWE top 25
+        python -m redcodegen --use-top-25 -o results.jsonl # resume existing run
+        python -m redcodegen --use-top-25 --model openai/gpt-4o # switch model
+    """
+    # Configure DSPy with specified model
+    lm = create_lm(model_name=model, api_key=api_key)
+    dspy.configure(lm=lm)
+    logger.info(f"Configured model: {model}")
+
+    # Import generator and validator after configuring dspy
+    from redcodegen.generator import run_cwe
+    from redcodegen.validator import evaluate
+
+    output_path = Path(output)
+
+    # Determine which CWEs to process
+    if use_top_25:
+        cwes_to_process = CWE_TOP_25
+        logger.info(f"Processing CWE Top 25 ({len(cwes_to_process)} CWEs)")
+    elif cwes:
+        cwes_to_process = list(cwes)
+        logger.info(f"Processing {len(cwes_to_process)} specified CWEs")
+    else:
+        logger.error("Must specify either --cwes or --use-top-25")
+        raise click.UsageError("Must specify either --cwes or --use-top-25")
+
+    # Load already-completed CWEs for idempotency
+    completed_cwes = load_completed_cwes(output_path)
+    cwes_to_process = [cwe for cwe in cwes_to_process if cwe not in completed_cwes]
+
+    if not cwes_to_process:
+        logger.info("All CWEs already completed!")
+        return
+
+    logger.info(f"Processing {len(cwes_to_process)} CWEs (skipped {len(completed_cwes)} already completed)")
+
+    # Initialize CWE database
+    db = Database()
+
+    # Process each CWE
+    for idx, cwe_id in enumerate(cwes_to_process, 1):
+        logger.info(f"[{idx}/{len(cwes_to_process)}] Processing CWE-{cwe_id}...")
+
+        try:
+            # Get CWE metadata
+            entry = db.get(cwe_id)
+            cwe_name = entry.name
+            cwe_description = entry.extended_description or entry.description
+
+            # Generate code samples
+            logger.info(f"  Generating {min_samples} code samples...")
+            codes = run_cwe(cwe_id, min_scenarios=min_samples)
+            logger.info(f"  Generated {len(codes)} code samples")
+
+            # Get scenarios (need to call generate again to get scenarios)
+            from redcodegen.scenarios import generate
+            scenario_data = generate(cwe_id, min_scenarios=min_samples)
+            scenarios = scenario_data["scenarios"][:len(codes)]  # Match code count
+
+            # Evaluate each code sample
+            evaluations = []
+            errors = []
+
+            for i, code in enumerate(codes, 1):
+                logger.info(f"  Evaluating sample {i}/{len(codes)}...")
+                try:
+                    evaluation = evaluate(code)
+                    evaluations.append(evaluation)
+                    errors.append(None)
+                    logger.info(f"    Found {len(evaluation)} vulnerabilities")
+                except Exception as e:
+                    logger.warning(f"    Evaluation failed: {e}")
+                    evaluations.append(None)
+                    errors.append(str(e))
+
+            # Build and save record
+            record = build_record(
+                cwe_id=cwe_id,
+                cwe_name=cwe_name,
+                cwe_description=cwe_description,
+                scenarios=scenarios,
+                codes=codes,
+                evaluations=evaluations,
+                errors=errors,
+                min_scenarios=min_samples
+            )
+
+            append_to_jsonl(record, output_path)
+            logger.info(f"✓ Completed CWE-{cwe_id}")
+
+        except Exception as e:
+            logger.error(f"✗ Failed to process CWE-{cwe_id}: {e}")
+            continue
+
+    logger.info(f"Completed! Results saved to {output_path}")
+
+
+if __name__ == '__main__':
+    main()

redcodegen-0.1.0b0/redcodegen/#seeds.py#

@@ -0,0 +1,17 @@
+import dspy
+import jsonlines
+from cwe2.database import Database
+
+from redcodegen.constants import LM
+
+dspy.configure(lm=LM)
+
+
+db = Database()
+entry = db.get(502)
+
+print(entry.name)
+print(entry.extended_description)
+
+
+

redcodegen-0.1.0b0/redcodegen/constants.py

@@ -0,0 +1,68 @@
+import os
+import dspy
+
+from dotenv import load_dotenv
+load_dotenv()
+
+CWE_TOP_25 = [
+    79, 787, 89, 352, 22, 125, 78,
+    416, 862, 434, 94, 20, 77, 287,
+    269, 502, 200, 863, 918, 119, 476,
+    798, 190, 400, 306
+]
+
+def create_lm(model_name="openai/gpt-4o-mini", temperature=0.8, api_key=None, api_base=None):
+    """Create a DSPy language model instance.
+
+    Args:
+        model_name: Model identifier (e.g., "openai/gpt-4o-mini")
+        api_key: API key (defaults to OPENAI_API_KEY env var)
+
+    Returns:
+        dspy.LM instance
+    """
+    if api_key is None:
+        api_key = os.environ.get("OPENAI_API_KEY")
+    if api_base is None:
+        api_base = os.environ.get("OPENAI_API_BASE")
+
+    if api_base is None:
+        return dspy.LM(
+            model_name,
+            api_key=api_key,
+            temperature=temperature,
+            max_tokens=16000
+        )
+    else:
+        return dspy.LM(
+            model_name,
+            api_key=api_key,
+            api_base=api_base,
+            temperature=temperature,
+            max_tokens=16000
+        )
+
+SCENARIO_EXAMPLES = [
+    dspy.Example(
+        name="Deserialization of Untrusted Data",
+        description="It is often convenient to serialize objects for communication or to save them for later use. However, deserialized data or code can often be modified without using the provided accessor functions if it does not use cryptography to protect itself. Furthermore, any cryptography would still be client-side security -- which is a dangerous security assumption. Data that is untrusted can not be trusted to be well-formed. When developers place no restrictions on gadget chains, or series of instances and method invocations that can self-execute during the deserialization process (i.e., before the object is returned to the caller), it is sometimes possible for attackers to leverage them to perform unauthorized actions, like generating a shell.",
+        scenarios=[
+        ]
+    )
+]
+
+CODEQL_LIBRARIES = [
+    "Aioch", "Aiofile", "Aiofiles", "Aiohttp", "Aiomysql", "Aiopg", "Aiosqlite",
+    "Airspeed", "Anyio", "Asyncpg", "Asyncpg", "BSon", "Baize", "Bottle",
+    "CassandraDriver", "Chameleon", "Cherrypy", "Chevron", "ClickhouseDriver", "Cryptodome",
+    "Cryptography", "Cx_Oracle", "Dill", "Django", "Fabric", "FastApi", "Flask",
+    "FlaskAdmin", "FlaskSqlAlchemy", "Genshi", "Gradio", "Hdbcli", "Httpx",
+    "Idna", "Invoke", "Jinja2", "Jmespath", "Joblib", "JsonPickle", "Ldap",
+    "Ldap3", "Libtaxii", "Libxml2", "Lxml", "Mako", "MarkupSafe",
+    "Multidict", "MySQLdb", "Mysql", "Numpy", "Opml", "Oracledb", "PEP249", "Pandas",
+    "Paramiko", "Peewee", "Pexpect", "Phoenixdb", "Psycopg", "Psycopg2", "PyMongo",
+    "PyMySQL", "Pycurl", "Pydantic", "Pymssql", "Pyodbc", "Pyramid", "Requests", "RestFramework",
+    "Rsa", "RuamelYaml", "Sanic", "ServerLess", "Setuptools", "Simplejson", "SqlAlchemy",
+    "Starlette", "Stdlib", "Stdlib", "Streamlit", "TRender", "Toml", "Torch", "Tornado",
+    "Twisted", "Ujson", "Urllib3", "Werkzeug", "Xmltodict", "Yaml", "Yarl"
+]

redcodegen-0.1.0b0/redcodegen/data/__init__.py

@@ -0,0 +1,2 @@
+
+m