modelaudit 0.2.44__tar.gz → 0.2.45__tar.gz

modelaudit-0.2.45/.github/workflows/README.md

@@ -0,0 +1,14 @@
+# GitHub Actions Workflows
+
+| Workflow                | File                    | Triggers                                  |
+| ----------------------- | ----------------------- | ----------------------------------------- |
+| **Python CI**           | `test.yml`              | Python, workflow, and dependency changes  |
+| **Documentation Check** | `docs-check.yml`        | Markdown/text/RST/license changes         |
+| **Docker Image CI**     | `docker-image-test.yml` | Dockerfile or Python code changes         |
+| **Validate PR Title**   | `validate-pr-title.yml` | PR open/edit events                       |
+| **CodeQL**              | `codeql.yml`            | Pushes, PRs, weekly schedule, manual runs |
+| **Nightly CI**          | `nightly.yml`           | Nightly schedule and manual dispatch      |
+| **Release**             | `release-please.yml`    | Pushes to main and manual dispatch        |
+| **Docker Publish**      | `docker-publish.yml`    | Published releases and manual dispatch    |
+
+Python CI ignores documentation-only PRs, which are handled by the documentation check workflow. Code PRs run fast feedback on Python 3.12, root matrix coverage on Python 3.10 and 3.13, the NumPy compatibility lane on Python 3.10 and 3.11, Windows tests on Python 3.11, and the standalone pickle package matrix on Python 3.10-3.13. Pushes to `main` run the full root and NumPy matrices across Python 3.10-3.13.

{modelaudit-0.2.44 → modelaudit-0.2.45}/.release-please-manifest.json

@@ -1,4 +1,4 @@
 {
-  ".": "0.2.44",
+  ".": "0.2.45",
   "packages/modelaudit-picklescan": "0.1.5"
 }

{modelaudit-0.2.44 → modelaudit-0.2.45}/CHANGELOG.md

@@ -5,6 +5,12 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [0.2.45](https://github.com/promptfoo/modelaudit/compare/v0.2.44...v0.2.45) (2026-05-03)
+
+### Bug Fixes
+
+- remove checked-in benchmark files ([#1220](https://github.com/promptfoo/modelaudit/issues/1220)) ([1e258e4](https://github.com/promptfoo/modelaudit/commit/1e258e48f43522853ba167463a5fae8d5533612a))
+
 ## [0.2.44](https://github.com/promptfoo/modelaudit/compare/v0.2.43...v0.2.44) (2026-05-03)
 
 ### Bug Fixes
@@ -149,6 +155,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 - detect nested brace-format lookups that reach tracked `defaultdict` factories
 - avoid `str.format` picklescan false positives when a `ChainMap` shadows a `defaultdict`
+- remove the checked-in benchmark workflow and benchmark test files
 - block `statistics.quantiles` call-iterator consumption in picklescan call-graph analysis
 - block additional eager `statistics` consumers in picklescan call-graph analysis
 - avoid picklescan false positives for inert metadata under dangerous dotted globals

{modelaudit-0.2.44 → modelaudit-0.2.45}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: modelaudit
-Version: 0.2.44
+Version: 0.2.45
 Summary: Static scanning library for detecting malicious code, potential backdoor indicators, and other security risks in ML model files
 Project-URL: Repository, https://github.com/promptfoo/modelaudit
 Project-URL: Homepage, https://github.com/promptfoo/modelaudit

{modelaudit-0.2.44 → modelaudit-0.2.45}/docs/agents/performance-audit.md

@@ -15,9 +15,7 @@ Status: active audit started 2026-05-01
 - Telemetry disabled for benchmarks with `PROMPTFOO_DISABLE_TELEMETRY=1`
 - Cache disabled unless a benchmark explicitly measures cache behavior
 - Profiles collected with `cProfile`
-- Existing benchmark surfaces:
-  - `tests/benchmarks/test_scan_benchmarks.py`
-  - `tests/benchmarks/test_picklescan_benchmarks.py`
+- Existing performance surface:
   - `tests/test_performance_benchmarks.py`
 
 ## Initial Baseline
@@ -46,7 +44,6 @@ Benchmark artifacts:
   - `/tmp/modelaudit-profile-assets.{json,pstats}`
   - `/tmp/modelaudit-profile-dill.{json,pstats}`
   - `/tmp/modelaudit-profile-safe-large.{json,pstats}`
-- Existing `pytest-benchmark` suites currently skip on Python `3.12` because the global test allowlist in `tests/conftest.py` excludes the benchmark files on reduced-version lanes. Running with `uv run --with pytest-benchmark ...` installs the plugin, but the tests still skip under the repo's current lane policy.
 
 Slowest fixture files in the current corpus:
 
@@ -346,32 +343,30 @@ Priority 1:
 
 ### Measurement Infrastructure
 
-- [ ] Make the existing benchmark suites runnable in the intended benchmark lane.
-- [ ] Decide whether benchmark files belong in the reduced-Python allowlist or should run only in a dedicated full lane.
-- [ ] Add a checked-in benchmark command matrix for:
+- [ ] Decide what repeatable performance measurements should stay checked in versus live only in maintainer workflows.
+- [ ] Add a documented performance command matrix for:
   - [ ] single-file scans
   - [ ] directory scans
   - [ ] cache miss scans
   - [ ] cache hit scans
   - [ ] duplicate-heavy scans
   - [ ] many-small-file scans
-- [ ] Add stable JSON output generation for benchmark runs.
-- [ ] Add a small script that compares current benchmark JSON against a prior baseline.
-- [ ] Add explicit benchmark tags for:
+- [ ] Decide whether performance runs need stable JSON artifacts and a comparison script.
+- [ ] Add explicit performance tags for:
   - [ ] pickle-heavy
   - [ ] large-file
   - [ ] many-file
   - [ ] archive-heavy
   - [ ] metadata-heavy
 - [ ] Record host, Python, dependency set, cache state, and git revision with every benchmark artifact.
-- [ ] Add a benchmark corpus inventory file so future runs use the same representative inputs.
-- [ ] Add a benchmark for cold-process startup cost versus warm-process repeated scan cost.
-- [ ] Add a benchmark for first import cost of optional scanner families.
-- [ ] Add a benchmark for scan result serialization overhead.
-- [ ] Add a benchmark for progress-enabled versus progress-disabled directory scans.
-- [ ] Add a benchmark for aggregate result construction on very large directory scans.
-- [ ] Add a benchmark for SBOM generation when scan results are large.
-- [ ] Add a benchmark for telemetry-disabled and telemetry-enabled paths, only if telemetry behavior is intentionally in scope.
+- [ ] Add a corpus inventory for repeatable performance runs.
+- [ ] Add coverage for cold-process startup cost versus warm-process repeated scan cost.
+- [ ] Add coverage for first import cost of optional scanner families.
+- [ ] Add coverage for scan result serialization overhead.
+- [ ] Add coverage for progress-enabled versus progress-disabled directory scans.
+- [ ] Add coverage for aggregate result construction on very large directory scans.
+- [ ] Add coverage for SBOM generation when scan results are large.
+- [ ] Add coverage for telemetry-disabled and telemetry-enabled paths, only if telemetry behavior is intentionally in scope.
 
 ### Core Pipeline Instrumentation
 

{modelaudit-0.2.44 → modelaudit-0.2.45}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "modelaudit"
-version = "0.2.44"
+version = "0.2.45"
 description = "Static scanning library for detecting malicious code, potential backdoor indicators, and other security risks in ML model files"
 authors = [
     { name = "Ian Webster", email = "ian@promptfoo.dev" },

{modelaudit-0.2.44 → modelaudit-0.2.45}/scripts/README.md

@@ -16,9 +16,9 @@ Script to regenerate vendored TensorFlow protobuf stubs from `.proto` files.
 
 Utilities for fetching model metadata from HuggingFace for testing and validation.
 
-### `benchmark_report.py` / `profile_scan.py`
+### `profile_scan.py`
 
-Utilities for summarizing benchmark output and profiling scanner performance.
+Utility for profiling scanner performance.
 
 ### `large_pickle_corpus_qa.py`
 

{modelaudit-0.2.44 → modelaudit-0.2.45}/tests/conftest.py

@@ -178,7 +178,6 @@ def pytest_runtest_setup(item):
             "test_optimized_config.py",  # optimized cache config regression tests
             "test_large_file_handler.py",  # Large file handler regression tests
             "test_file_iterator.py",  # Streaming file iterator memory regression tests
-            "test_benchmark_report.py",  # benchmark CI summary and regression gate tests
             "test_large_pickle_corpus_qa.py",  # large PickleScan Rust corpus QA harness tests
             "test_call_graph_click.py",  # standalone picklescan Click editor call-graph RCE regressions
             "test_call_graph_execnet.py",  # standalone picklescan execnet call-graph RCE regressions

{modelaudit-0.2.44 → modelaudit-0.2.45}/uv.lock

@@ -2395,7 +2395,7 @@ wheels = [
 
 [[package]]
 name = "modelaudit"
-version = "0.2.44"
+version = "0.2.45"
 source = { editable = "." }
 dependencies = [
     { name = "click" },

modelaudit-0.2.44/.github/workflows/README.md

@@ -1,17 +0,0 @@
-# GitHub Actions Workflows
-
-| Workflow                   | File                    | Triggers                                  |
-| -------------------------- | ----------------------- | ----------------------------------------- |
-| **Python CI**              | `test.yml`              | Python, workflow, and dependency changes  |
-| **Performance Benchmarks** | `perf.yml`              | PRs, pushes to main, manual dispatch      |
-| **Documentation Check**    | `docs-check.yml`        | Markdown/text/RST/license changes         |
-| **Docker Image CI**        | `docker-image-test.yml` | Dockerfile or Python code changes         |
-| **Validate PR Title**      | `validate-pr-title.yml` | PR open/edit events                       |
-| **CodeQL**                 | `codeql.yml`            | Pushes, PRs, weekly schedule, manual runs |
-| **Nightly CI**             | `nightly.yml`           | Nightly schedule and manual dispatch      |
-| **Release**                | `release-please.yml`    | Pushes to main and manual dispatch        |
-| **Docker Publish**         | `docker-publish.yml`    | Published releases and manual dispatch    |
-
-Python CI ignores documentation-only PRs, which are handled by the documentation check workflow. Code PRs run fast feedback on Python 3.12, root matrix coverage on Python 3.10 and 3.13, the NumPy compatibility lane on Python 3.10 and 3.11, Windows tests on Python 3.11, and the standalone pickle package matrix on Python 3.10-3.13. Pushes to `main` run the full root and NumPy matrices across Python 3.10-3.13.
-
-The performance workflow posts a sticky benchmark summary comment on same-repo PRs, uploads benchmark JSON plus Markdown summaries as workflow artifacts, and reports regressions or missing benchmarks without blocking the PR.

modelaudit-0.2.44/.github/workflows/perf.yml

@@ -1,215 +0,0 @@
-name: Performance Benchmarks
-
-on:
-  pull_request:
-    paths:
-      - "modelaudit/**"
-      - "packages/modelaudit-picklescan/**"
-      - "tests/benchmarks/**"
-      - "tests/helpers/**"
-      - "tests/conftest.py"
-      - "tests/test_benchmark_report.py"
-      - "scripts/benchmark_report.py"
-      - "pyproject.toml"
-      - "uv.lock"
-      - ".github/workflows/perf.yml"
-  push:
-    branches:
-      - main
-    paths:
-      - "modelaudit/**"
-      - "packages/modelaudit-picklescan/**"
-      - "tests/benchmarks/**"
-      - "tests/helpers/**"
-      - "tests/conftest.py"
-      - "tests/test_benchmark_report.py"
-      - "scripts/benchmark_report.py"
-      - "pyproject.toml"
-      - "uv.lock"
-      - ".github/workflows/perf.yml"
-  workflow_dispatch:
-
-permissions:
-  contents: read
-  pull-requests: write
-
-concurrency:
-  group: ${{ github.workflow }}-${{ github.ref }}
-  cancel-in-progress: ${{ github.event_name == 'pull_request' }}
-
-jobs:
-  benchmarks:
-    name: Benchmarks (Python 3.11)
-    runs-on: ubuntu-latest
-    timeout-minutes: 25
-    steps:
-      - name: Checkout repo
-        uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
-        with:
-          fetch-depth: 0
-
-      - name: Install uv
-        uses: astral-sh/setup-uv@37802adc94f370d6bfd71619e3f0bf239e1f3b78 # v7
-        with:
-          enable-cache: true
-
-      - name: Pin Python version
-        run: |
-          uv python pin 3.11
-
-      - name: Install Rust toolchain
-        run: |
-          rustup toolchain install stable --profile minimal
-          rustup default stable
-
-      - name: Run standalone picklescan Rust tests
-        run: |
-          cargo test --manifest-path packages/modelaudit-picklescan/Cargo.toml
-
-      - name: Prepare benchmark temp directories
-        id: paths
-        run: |
-          artifact_dir="$RUNNER_TEMP/modelaudit-benchmarks"
-          base_worktree="$RUNNER_TEMP/modelaudit-base"
-          echo "BENCHMARK_ARTIFACT_DIR=$artifact_dir" >> "$GITHUB_ENV"
-          echo "BENCHMARK_BASE_WORKTREE=$base_worktree" >> "$GITHUB_ENV"
-          echo "artifact_dir=$artifact_dir" >> "$GITHUB_OUTPUT"
-          echo "base_worktree=$base_worktree" >> "$GITHUB_OUTPUT"
-          rm -rf "$artifact_dir" "$base_worktree"
-          mkdir -p "$artifact_dir"
-
-      - name: Benchmark base commit
-        if: github.event_name == 'pull_request'
-        env:
-          BASE_SHA: ${{ github.event.pull_request.base.sha }}
-        run: |
-          set -euo pipefail
-          git worktree add --detach "$BENCHMARK_BASE_WORKTREE" "$BASE_SHA"
-          if [ ! -f "$BENCHMARK_BASE_WORKTREE/tests/benchmarks/test_scan_benchmarks.py" ]; then
-            echo "Base branch does not include the benchmark suite yet; skipping baseline run."
-            exit 0
-          fi
-          benchmark_tests=(tests/benchmarks/test_scan_benchmarks.py)
-          if [ -f "$BENCHMARK_BASE_WORKTREE/tests/benchmarks/test_picklescan_benchmarks.py" ]; then
-            benchmark_tests+=(tests/benchmarks/test_picklescan_benchmarks.py)
-          fi
-          uv run --directory "$BENCHMARK_BASE_WORKTREE" --python 3.11 --locked --with pytest-benchmark pytest \
-            "${benchmark_tests[@]}" \
-            --benchmark-json="$BENCHMARK_ARTIFACT_DIR/benchmark-base.json" \
-            -q
-
-      - name: Benchmark current commit
-        run: |
-          # Keep this lane single-process; pytest-benchmark disables itself under xdist.
-          benchmark_tests=(tests/benchmarks/test_scan_benchmarks.py)
-          if [ -f tests/benchmarks/test_picklescan_benchmarks.py ]; then
-            benchmark_tests+=(tests/benchmarks/test_picklescan_benchmarks.py)
-          fi
-          uv run --locked --with pytest-benchmark pytest \
-            "${benchmark_tests[@]}" \
-            --benchmark-json="$BENCHMARK_ARTIFACT_DIR/benchmark-head.json" \
-            -q
-
-      - name: Compare against base
-        id: compare
-        if: github.event_name == 'pull_request'
-        run: |
-          {
-            echo "[Workflow run and artifacts](${GITHUB_SERVER_URL}/${GITHUB_REPOSITORY}/actions/runs/${GITHUB_RUN_ID})"
-            echo
-          } > "$BENCHMARK_ARTIFACT_DIR/benchmark-comment.md"
-          if [ -f "$BENCHMARK_ARTIFACT_DIR/benchmark-base.json" ]; then
-            uv run --locked python scripts/benchmark_report.py \
-              --current "$BENCHMARK_ARTIFACT_DIR/benchmark-head.json" \
-              --baseline "$BENCHMARK_ARTIFACT_DIR/benchmark-base.json" \
-              --threshold 0.15 \
-              --summary-file "$BENCHMARK_ARTIFACT_DIR/benchmark-compare.md"
-          else
-            {
-              echo "Base branch does not include the benchmark suite yet; showing current results only."
-              echo
-            } > "$BENCHMARK_ARTIFACT_DIR/benchmark-current.md"
-            uv run --locked python scripts/benchmark_report.py \
-              --current "$BENCHMARK_ARTIFACT_DIR/benchmark-head.json" \
-              --summary-file "$BENCHMARK_ARTIFACT_DIR/benchmark-current-report.md"
-            cat "$BENCHMARK_ARTIFACT_DIR/benchmark-current.md" >> "$BENCHMARK_ARTIFACT_DIR/benchmark-comment.md"
-            cat "$BENCHMARK_ARTIFACT_DIR/benchmark-current-report.md" >> "$BENCHMARK_ARTIFACT_DIR/benchmark-comment.md"
-          fi
-          if [ -f "$BENCHMARK_ARTIFACT_DIR/benchmark-compare.md" ]; then
-            cat "$BENCHMARK_ARTIFACT_DIR/benchmark-compare.md" >> "$BENCHMARK_ARTIFACT_DIR/benchmark-comment.md"
-          fi
-          cat "$BENCHMARK_ARTIFACT_DIR/benchmark-comment.md" >> "$GITHUB_STEP_SUMMARY"
-
-      - name: Summarize current results
-        if: github.event_name != 'pull_request'
-        run: |
-          {
-            echo "[Workflow run and artifacts](${GITHUB_SERVER_URL}/${GITHUB_REPOSITORY}/actions/runs/${GITHUB_RUN_ID})"
-            echo
-          } > "$BENCHMARK_ARTIFACT_DIR/benchmark-summary.md"
-          uv run --locked python scripts/benchmark_report.py \
-            --current "$BENCHMARK_ARTIFACT_DIR/benchmark-head.json" \
-            --summary-file "$BENCHMARK_ARTIFACT_DIR/benchmark-current.md"
-          cat "$BENCHMARK_ARTIFACT_DIR/benchmark-current.md" >> "$BENCHMARK_ARTIFACT_DIR/benchmark-summary.md"
-          cat "$BENCHMARK_ARTIFACT_DIR/benchmark-summary.md" >> "$GITHUB_STEP_SUMMARY"
-
-      - name: Comment benchmark summary on PR
-        if: >
-          always() &&
-          github.event_name == 'pull_request' &&
-          github.event.pull_request.head.repo.full_name == github.repository
-        uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9
-        env:
-          COMMENT_BODY_PATH: ${{ steps.paths.outputs.artifact_dir }}/benchmark-comment.md
-        with:
-          script: |
-            const fs = require("fs")
-            if (!fs.existsSync(process.env.COMMENT_BODY_PATH)) {
-              core.warning(`No benchmark comment body found at ${process.env.COMMENT_BODY_PATH}`)
-              return
-            }
-            const marker = "<!-- modelaudit-perf-benchmarks -->"
-            const body = `${marker}\n${fs.readFileSync(process.env.COMMENT_BODY_PATH, "utf8")}`
-            const { owner, repo } = context.repo
-            const issue_number = context.issue.number
-
-            const comments = await github.paginate(github.rest.issues.listComments, {
-              owner,
-              repo,
-              issue_number,
-              per_page: 100,
-            })
-
-            const existing = comments.find(
-              (comment) => comment.user?.type === "Bot" && comment.body?.includes(marker),
-            )
-
-            if (existing) {
-              await github.rest.issues.updateComment({
-                owner,
-                repo,
-                comment_id: existing.id,
-                body,
-              })
-            } else {
-              await github.rest.issues.createComment({
-                owner,
-                repo,
-                issue_number,
-                body,
-              })
-            }
-
-      - name: Upload benchmark artifacts
-        if: always()
-        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
-        with:
-          name: benchmark-results-python-3.11
-          path: ${{ steps.paths.outputs.artifact_dir }}
-          if-no-files-found: error
-          retention-days: 14
-
-      - name: Cleanup benchmark base worktree
-        if: always() && github.event_name == 'pull_request'
-        run: |
-          git worktree remove --force "$BENCHMARK_BASE_WORKTREE" || true