modelaudit 0.2.45__tar.gz → 0.2.47__tar.gz

modelaudit-0.2.47/.github/workflows/README.md

@@ -0,0 +1,22 @@
+# GitHub Actions Workflows
+
+| Workflow                   | File                    | Triggers                                  |
+| -------------------------- | ----------------------- | ----------------------------------------- |
+| **Python CI**              | `test.yml`              | Python, workflow, and dependency changes  |
+| **Performance Benchmarks** | `perf.yml`              | PRs, pushes to main, manual dispatch      |
+| **Documentation Check**    | `docs-check.yml`        | Markdown/text/RST/license changes         |
+| **Docker Image CI**        | `docker-image-test.yml` | Dockerfile or Python code changes         |
+| **Validate PR Title**      | `validate-pr-title.yml` | PR open/edit events                       |
+| **CodeQL**                 | `codeql.yml`            | Pushes, PRs, weekly schedule, manual runs |
+| **Nightly CI**             | `nightly.yml`           | Nightly schedule and manual dispatch      |
+| **Release**                | `release-please.yml`    | Pushes to main and manual dispatch        |
+| **Docker Publish**         | `docker-publish.yml`    | Published releases and manual dispatch    |
+
+Python CI ignores documentation-only PRs, which are handled by the documentation check workflow. Code PRs run fast feedback on Python 3.12, root matrix coverage on Python 3.10 and 3.13, the NumPy compatibility lane on Python 3.10 and 3.11, Windows tests on Python 3.11, and the standalone pickle package matrix on Python 3.10-3.13. Pushes to `main` run the full root and NumPy matrices across Python 3.10-3.13.
+
+The performance workflow compares workload-oriented benchmarks between the PR
+base and head, posts a sticky summary comment on same-repo PRs, uploads JSON and
+Markdown artifacts, and reports comparative regressions without blocking the
+PR. It separately runs the cache-disabled retained-memory stability guard from
+`tests/test_performance_benchmarks.py`, which fails the workflow if repeat scans
+retain excessive memory.

{modelaudit-0.2.45 → modelaudit-0.2.47}/.github/workflows/docker-image-test.yml

@@ -8,6 +8,7 @@ on:
       - "modelaudit/**"
       - "packages/modelaudit-picklescan/**"
       - "pyproject.toml"
+      - "requirements-tensorflow.txt"
       - "uv.lock"
       - ".github/workflows/docker-image-test.yml"
   push:
@@ -27,6 +28,7 @@ jobs:
     outputs:
       docker: ${{ steps.filter.outputs.docker }}
       full-image: ${{ steps.filter.outputs.full-image }}
+      tensorflow-image: ${{ steps.filter.outputs.tensorflow-image }}
     steps:
       - uses: actions/checkout@v6
       - uses: dorny/paths-filter@v4
@@ -45,6 +47,12 @@ jobs:
               - 'Dockerfile.full'
               - 'packages/modelaudit-picklescan/**'
               - '.github/workflows/docker-image-test.yml'
+            tensorflow-image:
+              - 'Dockerfile.tensorflow'
+              - 'requirements-tensorflow.txt'
+              - 'modelaudit/**'
+              - 'pyproject.toml'
+              - '.github/workflows/docker-image-test.yml'
 
   build-test-lightweight:
     name: Build and Test Lightweight Docker Image
@@ -177,9 +185,49 @@ jobs:
           # Scan the bind-mounted model as the non-root full image.
           docker run --rm -v $(pwd):/data modelaudit:full /data/test_numpy.pkl
 
+  build-test-tensorflow:
+    name: Build and Test TensorFlow Docker Image
+    needs: changes
+    runs-on: ubuntu-latest
+    timeout-minutes: 60
+    if: needs.changes.outputs.tensorflow-image == 'true'
+    steps:
+      - uses: actions/checkout@v6
+
+      - name: Set up Docker Buildx
+        uses: docker/setup-buildx-action@4d04d5d9486b7bd6fa91e7baf45bbb4f8b9deedd # v4
+
+      - name: Build TensorFlow image
+        uses: docker/build-push-action@bcafcacb16a39f128d818304e6c9c0c18556b85f # v7
+        with:
+          context: .
+          file: Dockerfile.tensorflow
+          tags: modelaudit:tensorflow
+          load: true
+          cache-from: type=gha,scope=tensorflow
+          cache-to: type=gha,mode=max,scope=tensorflow
+
+      - name: Test TensorFlow container help command
+        run: |
+          docker run --rm modelaudit:tensorflow --help
+
+      - name: Verify TensorFlow dependency and non-root runtime
+        run: |
+          docker run --rm --entrypoint python modelaudit:tensorflow -c "import os, tensorflow; print(tensorflow.__version__); assert os.getuid() == 10001"
+
+      - name: Scan TensorFlow image for vulnerabilities
+        uses: aquasecurity/trivy-action@57a97c7e7821a5776cebc9bb87c984fa69cba8f1 # 0.35.0
+        with:
+          version: v0.69.2
+          image-ref: modelaudit:tensorflow
+          format: table
+          exit-code: "1"
+          severity: CRITICAL,HIGH
+          ignore-unfixed: true
+
   docker-ci-success:
     name: Docker CI Success
-    needs: [build-test-lightweight, build-test-full]
+    needs: [build-test-lightweight, build-test-full, build-test-tensorflow]
     if: always()
     runs-on: ubuntu-latest
     steps:
@@ -187,13 +235,16 @@ jobs:
         run: |
           LIGHTWEIGHT_RESULT="${{ needs.build-test-lightweight.result }}"
           FULL_RESULT="${{ needs.build-test-full.result }}"
+          TENSORFLOW_RESULT="${{ needs.build-test-tensorflow.result }}"
           echo "Lightweight Docker build result: $LIGHTWEIGHT_RESULT"
           echo "Full Docker build result: $FULL_RESULT"
+          echo "TensorFlow Docker build result: $TENSORFLOW_RESULT"
 
           # Success or skipped are both acceptable
           # (skipped means the path filters determined the job wasn't needed)
           if [[ ("$LIGHTWEIGHT_RESULT" == "success" || "$LIGHTWEIGHT_RESULT" == "skipped") && \
-                ("$FULL_RESULT" == "success" || "$FULL_RESULT" == "skipped") ]]; then
+                ("$FULL_RESULT" == "success" || "$FULL_RESULT" == "skipped") && \
+                ("$TENSORFLOW_RESULT" == "success" || "$TENSORFLOW_RESULT" == "skipped") ]]; then
             echo "Docker CI checks passed (or were skipped due to path filters)!"
             exit 0
           else

{modelaudit-0.2.45 → modelaudit-0.2.47}/.github/workflows/nightly.yml

@@ -76,6 +76,6 @@ jobs:
         run: |
           cargo test --manifest-path packages/modelaudit-picklescan/Cargo.toml
 
-      - name: Run all tests
+      - name: Run all Windows tests except performance benchmarks
         run: |
-          uv run pytest -n auto --tb=short --durations=20
+          uv run pytest -n auto -m "not performance" --tb=short --durations=20

modelaudit-0.2.47/.github/workflows/perf.yml

@@ -0,0 +1,225 @@
+name: Performance Benchmarks
+
+on:
+  pull_request:
+    paths:
+      - "modelaudit/**"
+      - "packages/modelaudit-picklescan/**"
+      - "tests/benchmarks/**"
+      - "tests/helpers/**"
+      - "tests/conftest.py"
+      - "tests/test_benchmark_report.py"
+      - "tests/test_performance_benchmarks.py"
+      - "scripts/benchmark_report.py"
+      - "pyproject.toml"
+      - "uv.lock"
+      - ".github/workflows/perf.yml"
+  push:
+    branches:
+      - main
+    paths:
+      - "modelaudit/**"
+      - "packages/modelaudit-picklescan/**"
+      - "tests/benchmarks/**"
+      - "tests/helpers/**"
+      - "tests/conftest.py"
+      - "tests/test_benchmark_report.py"
+      - "tests/test_performance_benchmarks.py"
+      - "scripts/benchmark_report.py"
+      - "pyproject.toml"
+      - "uv.lock"
+      - ".github/workflows/perf.yml"
+  workflow_dispatch:
+
+permissions:
+  contents: read
+  pull-requests: write
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.ref }}
+  cancel-in-progress: ${{ github.event_name == 'pull_request' }}
+
+jobs:
+  benchmarks:
+    name: Benchmarks (Python 3.11)
+    runs-on: ubuntu-latest
+    timeout-minutes: 25
+    steps:
+      - name: Checkout repo
+        uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
+        with:
+          fetch-depth: 0
+
+      - name: Install uv
+        uses: astral-sh/setup-uv@37802adc94f370d6bfd71619e3f0bf239e1f3b78 # v7
+        with:
+          enable-cache: true
+
+      - name: Pin Python version
+        run: |
+          uv python pin 3.11
+
+      - name: Install Rust toolchain
+        run: |
+          rustup toolchain install stable --profile minimal
+          rustup default stable
+
+      - name: Run standalone picklescan Rust tests
+        run: |
+          cargo test --manifest-path packages/modelaudit-picklescan/Cargo.toml
+
+      - name: Prepare benchmark temp directories
+        id: paths
+        run: |
+          artifact_dir="$RUNNER_TEMP/modelaudit-benchmarks"
+          base_worktree="$RUNNER_TEMP/modelaudit-base"
+          echo "BENCHMARK_ARTIFACT_DIR=$artifact_dir" >> "$GITHUB_ENV"
+          echo "BENCHMARK_BASE_WORKTREE=$base_worktree" >> "$GITHUB_ENV"
+          echo "artifact_dir=$artifact_dir" >> "$GITHUB_OUTPUT"
+          echo "base_worktree=$base_worktree" >> "$GITHUB_OUTPUT"
+          rm -rf "$artifact_dir" "$base_worktree"
+          mkdir -p "$artifact_dir"
+
+      - name: Benchmark base commit
+        if: github.event_name == 'pull_request'
+        env:
+          BASE_SHA: ${{ github.event.pull_request.base.sha }}
+        run: |
+          set -euo pipefail
+          git worktree add --detach "$BENCHMARK_BASE_WORKTREE" "$BASE_SHA"
+          if [ ! -f "$BENCHMARK_BASE_WORKTREE/tests/benchmarks/test_scan_benchmarks.py" ]; then
+            echo "Base branch does not include the benchmark suite yet; skipping baseline run."
+            exit 0
+          fi
+          benchmark_tests=(tests/benchmarks/test_scan_benchmarks.py)
+          if [ -f "$BENCHMARK_BASE_WORKTREE/tests/benchmarks/test_picklescan_benchmarks.py" ]; then
+            benchmark_tests+=(tests/benchmarks/test_picklescan_benchmarks.py)
+          fi
+          uv run --directory "$BENCHMARK_BASE_WORKTREE" --python 3.11 --locked --with pytest-benchmark pytest \
+            "${benchmark_tests[@]}" \
+            --benchmark-json="$BENCHMARK_ARTIFACT_DIR/benchmark-base.json" \
+            -q
+
+      - name: Benchmark current commit
+        run: |
+          # Keep this lane single-process; pytest-benchmark disables itself under xdist.
+          benchmark_tests=(tests/benchmarks/test_scan_benchmarks.py)
+          if [ -f tests/benchmarks/test_picklescan_benchmarks.py ]; then
+            benchmark_tests+=(tests/benchmarks/test_picklescan_benchmarks.py)
+          fi
+          uv run --locked --with pytest-benchmark pytest \
+            "${benchmark_tests[@]}" \
+            --benchmark-json="$BENCHMARK_ARTIFACT_DIR/benchmark-head.json" \
+            -q
+
+      - name: Compare against base
+        id: compare
+        if: github.event_name == 'pull_request'
+        run: |
+          {
+            echo "[Workflow run and artifacts](${GITHUB_SERVER_URL}/${GITHUB_REPOSITORY}/actions/runs/${GITHUB_RUN_ID})"
+            echo
+          } > "$BENCHMARK_ARTIFACT_DIR/benchmark-comment.md"
+          if [ -f "$BENCHMARK_ARTIFACT_DIR/benchmark-base.json" ]; then
+            uv run --locked python scripts/benchmark_report.py \
+              --current "$BENCHMARK_ARTIFACT_DIR/benchmark-head.json" \
+              --baseline "$BENCHMARK_ARTIFACT_DIR/benchmark-base.json" \
+              --threshold 0.15 \
+              --summary-file "$BENCHMARK_ARTIFACT_DIR/benchmark-compare.md"
+          else
+            {
+              echo "Base branch does not include the benchmark suite yet; showing current results only."
+              echo
+            } > "$BENCHMARK_ARTIFACT_DIR/benchmark-current.md"
+            uv run --locked python scripts/benchmark_report.py \
+              --current "$BENCHMARK_ARTIFACT_DIR/benchmark-head.json" \
+              --summary-file "$BENCHMARK_ARTIFACT_DIR/benchmark-current-report.md"
+            cat "$BENCHMARK_ARTIFACT_DIR/benchmark-current.md" >> "$BENCHMARK_ARTIFACT_DIR/benchmark-comment.md"
+            cat "$BENCHMARK_ARTIFACT_DIR/benchmark-current-report.md" >> "$BENCHMARK_ARTIFACT_DIR/benchmark-comment.md"
+          fi
+          if [ -f "$BENCHMARK_ARTIFACT_DIR/benchmark-compare.md" ]; then
+            cat "$BENCHMARK_ARTIFACT_DIR/benchmark-compare.md" >> "$BENCHMARK_ARTIFACT_DIR/benchmark-comment.md"
+          fi
+          cat "$BENCHMARK_ARTIFACT_DIR/benchmark-comment.md" >> "$GITHUB_STEP_SUMMARY"
+
+      - name: Summarize current results
+        if: github.event_name != 'pull_request'
+        run: |
+          {
+            echo "[Workflow run and artifacts](${GITHUB_SERVER_URL}/${GITHUB_REPOSITORY}/actions/runs/${GITHUB_RUN_ID})"
+            echo
+          } > "$BENCHMARK_ARTIFACT_DIR/benchmark-summary.md"
+          uv run --locked python scripts/benchmark_report.py \
+            --current "$BENCHMARK_ARTIFACT_DIR/benchmark-head.json" \
+            --summary-file "$BENCHMARK_ARTIFACT_DIR/benchmark-current.md"
+          cat "$BENCHMARK_ARTIFACT_DIR/benchmark-current.md" >> "$BENCHMARK_ARTIFACT_DIR/benchmark-summary.md"
+          cat "$BENCHMARK_ARTIFACT_DIR/benchmark-summary.md" >> "$GITHUB_STEP_SUMMARY"
+
+      - name: Run retained-memory stability guard
+        env:
+          PROMPTFOO_DISABLE_TELEMETRY: "1"
+        run: |
+          uv run --locked --with psutil pytest \
+            tests/test_performance_benchmarks.py::TestPerformanceBenchmarks::test_memory_usage_stability \
+            -q
+
+      - name: Comment benchmark summary on PR
+        if: >
+          always() &&
+          github.event_name == 'pull_request' &&
+          github.event.pull_request.head.repo.full_name == github.repository
+        uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9
+        env:
+          COMMENT_BODY_PATH: ${{ steps.paths.outputs.artifact_dir }}/benchmark-comment.md
+        with:
+          script: |
+            const fs = require("fs")
+            if (!fs.existsSync(process.env.COMMENT_BODY_PATH)) {
+              core.warning(`No benchmark comment body found at ${process.env.COMMENT_BODY_PATH}`)
+              return
+            }
+            const marker = "<!-- modelaudit-perf-benchmarks -->"
+            const body = `${marker}\n${fs.readFileSync(process.env.COMMENT_BODY_PATH, "utf8")}`
+            const { owner, repo } = context.repo
+            const issue_number = context.issue.number
+
+            const comments = await github.paginate(github.rest.issues.listComments, {
+              owner,
+              repo,
+              issue_number,
+              per_page: 100,
+            })
+
+            const existing = comments.find(
+              (comment) => comment.user?.type === "Bot" && comment.body?.includes(marker),
+            )
+
+            if (existing) {
+              await github.rest.issues.updateComment({
+                owner,
+                repo,
+                comment_id: existing.id,
+                body,
+              })
+            } else {
+              await github.rest.issues.createComment({
+                owner,
+                repo,
+                issue_number,
+                body,
+              })
+            }
+
+      - name: Upload benchmark artifacts
+        if: always()
+        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
+        with:
+          name: benchmark-results-python-3.11
+          path: ${{ steps.paths.outputs.artifact_dir }}
+          if-no-files-found: error
+          retention-days: 14
+
+      - name: Cleanup benchmark base worktree
+        if: always() && github.event_name == 'pull_request'
+        run: |
+          git worktree remove --force "$BENCHMARK_BASE_WORKTREE" || true

{modelaudit-0.2.45 → modelaudit-0.2.47}/.github/workflows/test.yml

@@ -380,7 +380,9 @@ jobs:
     # Always run for fastest feedback
     if: github.ref == 'refs/heads/main' || needs.changes.outputs.python == 'true' || needs.changes.outputs.workflows == 'true'
     runs-on: ubuntu-latest
-    timeout-minutes: 30
+    # The full fast suite can approach 30 minutes on hosted runners when the
+    # bounded-replay security regressions land on the long-running worker.
+    timeout-minutes: 35
     steps:
       - name: Checkout repo
         uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
@@ -424,7 +426,7 @@ jobs:
     needs: changes
     if: github.ref == 'refs/heads/main' || needs.changes.outputs.python == 'true' || needs.changes.outputs.workflows == 'true'
     runs-on: windows-latest
-    timeout-minutes: 30
+    timeout-minutes: 40
     steps:
       - name: Checkout repo
         uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
@@ -455,7 +457,7 @@ jobs:
     # Only run if Python files changed
     if: github.ref == 'refs/heads/main' || needs.changes.outputs.python == 'true' || needs.changes.outputs.workflows == 'true'
     runs-on: ubuntu-latest
-    timeout-minutes: 30
+    timeout-minutes: 35
     strategy:
       matrix:
         # For PRs, only test min and max Python versions to save time

modelaudit-0.2.47/.release-please-manifest.json

@@ -0,0 +1,4 @@
+{
+  ".": "0.2.47",
+  "packages/modelaudit-picklescan": "0.1.6"
+}

{modelaudit-0.2.45 → modelaudit-0.2.47}/AGENTS.md

@@ -18,7 +18,7 @@ This repo publishes **two PyPI packages with independent versions**:
 | `modelaudit`            | `./` (root)                       | `pyproject.toml` + `uv.lock`    | `CHANGELOG.md`                                |
 | `modelaudit-picklescan` | `packages/modelaudit-picklescan/` | `pyproject.toml` + `Cargo.toml` | `packages/modelaudit-picklescan/CHANGELOG.md` |
 
-Root `modelaudit` hard-requires `modelaudit-picklescan>=0.1.0,<0.2.0` — when the sibling crosses `0.2.0`, bump the constraint in the same PR or the next `modelaudit` release is uninstallable. Both packages are driven by a single `release-please` workflow (`.github/workflows/release-please.yml`) with components defined in `release-please-config.json` and current versions in `.release-please-manifest.json`. Full publishing details — trusted publishing, manual `workflow_dispatch` recovery (`root_version` / `picklescan_version`), and yank procedure — are in [`docs/agents/release-process.md`](docs/agents/release-process.md). For work inside the picklescan package, start from [`packages/modelaudit-picklescan/AGENTS.md`](packages/modelaudit-picklescan/AGENTS.md).
+Root `modelaudit` hard-requires `modelaudit-picklescan>=0.1.4,<0.2.0` — when the sibling crosses `0.2.0`, bump the constraint in the same PR or the next `modelaudit` release is uninstallable. Both packages are driven by a single `release-please` workflow (`.github/workflows/release-please.yml`) with components defined in `release-please-config.json` and current versions in `.release-please-manifest.json`. Full publishing details — trusted publishing, manual `workflow_dispatch` recovery (`root_version` / `picklescan_version`), and yank procedure — are in [`docs/agents/release-process.md`](docs/agents/release-process.md). For work inside the picklescan package, start from [`packages/modelaudit-picklescan/AGENTS.md`](packages/modelaudit-picklescan/AGENTS.md).
 
 ## Mission & Principles