pytest-prairielearn-grader 0.2.0__tar.gz

pytest_prairielearn_grader-0.2.0/.github/copilot-instructions.md

@@ -0,0 +1,187 @@
+# Pytest PrairieLearn Grader - AI Coding Agent Instructions
+
+## Project Overview
+
+**pytest-prairielearn-grader** is a pytest plugin for autograding Python code, designed for integration with PrairieLearn. It executes student code in isolated sandbox environments and provides detailed feedback and scoring.
+
+**Key Architecture**: Test harnesses run in the main process, while student code executes in separate subprocesses via Unix sockets, enabling security isolation and timeout enforcement.
+
+## Core Components
+
+### 1. Sandboxed Execution (`StudentFixture`)
+
+- **Location**: [src/pytest_prairielearn_grader/fixture.py](src/pytest_prairielearn_grader/fixture.py)
+- **How it works**: Tests use the `sandbox` or `module_sandbox` fixtures to interact with student code
+- **Key methods**:
+  - `sandbox.query(var_name)` - retrieve variables from student code
+  - `sandbox.query_function(func_name, *args, **kwargs)` - execute functions in student code
+  - `sandbox.get_stdout()` - capture student code output
+- **Important**: Student code runs in subprocess via `_student_code_runner.py`; results are JSON-serialized through socket communication
+
+### 2. Student Code Runner Process
+
+- **Location**: [src/pytest_prairielearn_grader/\_student_code_runner.py](src/pytest_prairielearn_grader/_student_code_runner.py)
+- **Role**: Runs in isolated subprocess, executes student code, enforces security/timeouts
+- **Socket Protocol**: Receives JSON requests (setup, query, function calls), returns JSON responses
+- **Security Features**:
+  - Import whitelist/blacklist enforcement
+  - Builtin function restrictions
+  - Privilege dropping (Unix only)
+  - Timeout enforcement via asyncio
+
+### 3. Test Execution & Grading
+
+- **Location**: [src/pytest_prairielearn_grader/plugin.py](src/pytest_prairielearn_grader/plugin.py)
+- **Custom Pytest Plugin**: Registers fixtures and collects grading metadata
+- **Fixtures**:
+  - `sandbox` (function-scoped): Isolated per test, supports parameterization
+  - `module_sandbox` (module-scoped): Reused across module, single student code only
+  - `feedback` (function-scoped): Manages partial credit and messages
+  - `data_json` (module-scoped): Loads test parameters from `data.json`
+
+### 4. Test Scenario Structure
+
+- **Location**: [tests/scenario_root/](tests/scenario_root/)
+- **Pattern**: Each test scenario has a directory `test_*` containing:
+  - `scenario.py` - test code using autograder fixtures
+  - `data.json` - parameters passed to student code
+  - `setup_code.py` - initialization code for student sandbox
+  - `student_code_*.py` - various student code variants for testing
+  - `expected_outcome.json` - expected pytest results for validation
+
+## Critical Developer Workflows
+
+### Running Tests
+
+```bash
+# Run all scenario tests
+pytest tests/test_autograder_scenarios.py -v
+
+# Run specific scenario
+pytest tests/test_autograder_scenarios.py::test_autograder_scenario_with_pytester[test_quiz_2_1] -v
+
+# Run with output capture disabled (see prints)
+pytest tests/test_autograder_scenarios.py -s
+```
+
+### Running a Single Scenario Directly
+
+```bash
+# Copy scenario files, run: cd tests/scenario_root/test_quiz_2_1 && pytest test_quiz_2_1.py -v
+```
+
+### Linting & Type Checking
+
+```bash
+# Format with ruff
+ruff format src/ tests/
+
+# Check with ruff
+ruff check src/ tests/ --fix
+
+# Type check
+mypy src/
+```
+
+## Key Design Patterns & Conventions
+
+### 1. Grading Data Marks
+
+```python
+@pytest.mark.grading_data(name="Test Name", points=5, include_stdout_feedback=False)
+def test_something(sandbox: StudentFixture, feedback: FeedbackFixture) -> None:
+    # name: displayed to student
+    # points: max points for this test
+    # include_stdout_feedback: whether to capture student stdout
+```
+
+### 2. Partial Credit Pattern
+
+```python
+def test_multi_step(sandbox: StudentFixture, feedback: FeedbackFixture) -> None:
+    feedback.set_score(0.5)  # 50% credit if next assert fails
+    assert step_one_passes()  # if fails here, student gets 50%
+
+    feedback.set_score(0.8)   # 80% credit if next assert fails
+    assert step_two_passes()   # if fails here, student gets 80%
+
+    feedback.set_score(1.0)   # 100% if all pass
+```
+
+### 3. Function Query Pattern
+
+```python
+# In test:
+result = sandbox.query_function("student_func", arg1, arg2, timeout=5)
+
+# Query returns StudentFunctionResponse with:
+# - status: SUCCESS, EXCEPTION, TIMEOUT, NOT_FOUND
+# - value: JSON-deserialized return value
+# - stdout/stderr: captured output
+# - exception_name/message/traceback: if exception occurred
+```
+
+### 4. Data Configuration Pattern
+
+```python
+# data.json structure:
+{
+  "params": {
+    "sin_coefficient": 2.5,
+    "cos_coefficient": 1.3,
+    "names_for_user": [
+      {"name": "sin_coefficient", "type": "float", "description": "..."},
+      {"name": "array_data", "type": "ndarray", "description": "..."}
+    ],
+    "import_whitelist": ["numpy", "math"],
+    "builtin_whitelist": ["len", "range", "sum"]
+  }
+}
+
+# In setup_code.py, define variables; they're injected if in names_for_user
+```
+
+### 5. Serialization
+
+- Uses `dill` for complex Python objects (numpy arrays, pandas DataFrames, matplotlib plots)
+- Base64-encoded in JSON for transport
+- See [src/pytest_prairielearn_grader/json_utils.py](src/pytest_prairielearn_grader/json_utils.py) for custom serialization
+
+## Important Integration Points
+
+### PrairieLearn Integration
+
+- Test file must be named `tests/test_*.py`
+- Student code file is expected as `student_code.py` (configurable via `student_code_pattern` global variable)
+- Output is collected as JSON with test results, scores, and messages
+- Exit codes: 0 = success, non-zero = test execution failures
+
+### Command Line Options
+
+- `--worker-username`: Identifies which student submission is running (used for privilege dropping)
+- `--output-json`: Path where JSON results are written
+- Custom pytest options (e.g., `-v`, `-s`) work normally
+
+## Common Pitfalls & Solutions
+
+| Issue                                      | Solution                                                                              |
+| ------------------------------------------ | ------------------------------------------------------------------------------------- |
+| "Student code server process terminated"   | Check for unhandled exceptions in setup_code or timeout too short                     |
+| Function not found errors                  | Ensure function is defined in student_code.py, not just setup_code.py                 |
+| Timeout errors on network tests            | Increase timeout via `@pytest.mark.sandbox_timeout(N)`                                |
+| Serialization fails                        | Use `dill`-compatible types (check json_utils.py for supported types)                 |
+| Module sandbox with multiple student codes | Use regular `sandbox` fixture instead; module_sandbox requires single student_code.py |
+
+## Files to Reference When Extending
+
+- **Adding new fixture**: [src/pytest_prairielearn_grader/plugin.py#L195-L210](src/pytest_prairielearn_grader/plugin.py#L195-L210)
+- **New query type**: [src/pytest_prairielearn_grader/utils.py](src/pytest_prairielearn_grader/utils.py) (define TypedDict) + [src/pytest_prairielearn_grader/\_student_code_runner.py](src/pytest_prairielearn_grader/_student_code_runner.py) (handle message)
+- **Custom marks**: [src/pytest_prairielearn_grader/utils.py#L5-L20](src/pytest_prairielearn_grader/utils.py#L5-L20) (define enum) + [src/pytest_prairielearn_grader/plugin.py](src/pytest_prairielearn_grader/plugin.py) (use in marker parsing)
+- **Test examples**: [tests/scenario_root/](tests/scenario_root/) (all subdirectories follow same pattern)
+
+## Python Version & Dependencies
+
+- **Minimum Python**: 3.11
+- **Key dependencies**: pytest, dill, prettytable
+- **Dev dependencies**: numpy, pandas, mypy, ruff, matplotlib, sympy
+- **Code style**: Ruff with specific rules configured in pyproject.toml (140 char line length, specific linters enabled)

pytest_prairielearn_grader-0.2.0/.github/workflows/lint.yaml

@@ -0,0 +1,33 @@
+name: Lint
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+    branches: [main]
+
+jobs:
+  Lint:
+    strategy:
+      fail-fast: false
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          fetch-depth: 0
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.13"
+      - name: Install uv
+        uses: astral-sh/setup-uv@v5
+      - name: Check lockfile
+        run: uv lock --check
+      - name: Lint with ruff
+        continue-on-error: true
+        run: |
+          # Fail if codebase contains any of these issues
+          uv run ruff check .
+          uv run ruff format --check .
+      - name: Static Typechecking with mypy
+        run: |
+          uv run mypy --config-file pyproject.toml src tests

pytest_prairielearn_grader-0.2.0/.github/workflows/publish.yaml

@@ -0,0 +1,86 @@
+name: publish
+
+on:
+  push:
+    tags:
+      - "v*.*.*"
+  workflow_dispatch:
+    inputs:
+      tag:
+        description: "Git tag to build (e.g., v0.1.15)"
+        required: true
+        type: string
+
+jobs:
+  build:
+    if: github.event_name == 'push'
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          ref: ${{ github.event.inputs.tag || github.ref }}
+      - uses: astral-sh/setup-uv@v5
+      - name: Build and publish to pypi
+        run: |
+          uv build
+          uv publish --token ${{ secrets.PYPI_TOKEN }}
+
+  build-docker:
+    if: always()
+    needs: build
+    runs-on: ubuntu-latest
+    permissions:
+      contents: read
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          ref: ${{ github.event.inputs.tag || github.ref }}
+
+      - name: Set up Docker Buildx
+        uses: docker/setup-buildx-action@v3
+
+      - name: Log in to Docker Hub
+        uses: docker/login-action@v3
+        with:
+          username: ${{ secrets.DOCKER_USERNAME }}
+          password: ${{ secrets.DOCKER_PASSWORD }}
+
+      - name: Extract version from tag
+        id: version
+        run: |
+          if [ "${{ github.event_name }}" = "workflow_dispatch" ]; then\
+            VERSION="${{ github.event.inputs.tag }}"
+            echo "VERSION=${VERSION#v}" >> $GITHUB_OUTPUT
+          else
+            echo "VERSION=${GITHUB_REF#refs/tags/v}" >> $GITHUB_OUTPUT
+          fi
+
+      - name: Wait for PyPI package to be available
+        if: github.event_name != 'workflow_dispatch'
+        run: |
+          echo "Waiting 30 seconds for package to be available on PyPI..."
+          sleep 30
+
+          # Verify package is available
+          for i in {1..10}; do
+            if pip index versions pytest-prairielearn-grader 2>/dev/null | grep -q "${{ steps.version.outputs.VERSION }}"; then
+              echo "Package version ${{ steps.version.outputs.VERSION }} is available on PyPI"
+              exit 0
+            fi
+            if [ $i -lt 10 ]; then
+              echo "Attempt $i/10: Package not yet available, waiting 10 seconds..."
+              sleep 10
+            fi
+          done
+          echo "Warning: Could not verify package on PyPI, proceeding anyway"
+
+      - name: Build and push Docker image
+        uses: docker/build-push-action@v5
+        with:
+          context: ./docker
+          push: true
+          tags: |
+            ${{ secrets.DOCKER_USERNAME }}/grader-python-pytest:v${{ steps.version.outputs.VERSION }}
+            ${{ secrets.DOCKER_USERNAME }}/grader-python-pytest:latest
+          build-args: |
+            PACKAGE_VERSION=${{ steps.version.outputs.VERSION }}

pytest_prairielearn_grader-0.2.0/.github/workflows/test.yaml

@@ -0,0 +1,22 @@
+name: Tests
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+    branches: [main]
+
+jobs:
+  Tests:
+    strategy:
+      fail-fast: false
+      matrix:
+        python-version: ["3.11", "3.12", "3.13", "3.14"]
+        os: [ubuntu-latest]
+    runs-on: ${{ matrix.os }}
+    steps:
+      - uses: actions/checkout@v4
+      - name: Install uv
+        uses: astral-sh/setup-uv@v5
+      - name: Test with pytest
+        run: uv run pytest tests -p no:prairielearn-grader

pytest_prairielearn_grader-0.2.0/.gitignore

@@ -0,0 +1,80 @@
+*.py[cod]
+__pycache__
+
+# Temp files
+.*.sw[po]
+*~
+*.bak
+.DS_Store
+
+# C extensions
+*.so
+
+# Build and package files
+*.egg
+*.egg-info
+.bootstrap
+.build
+.cache
+.eggs
+.env
+.installed.cfg
+.ve
+bin
+build
+develop-eggs
+dist
+eggs
+lib
+lib64
+parts
+pip-wheel-metadata/
+pyvenv*/
+sdist
+var
+venv*/
+wheelhouse
+
+# Installer logs
+pip-log.txt
+
+# Unit test / coverage reports
+.benchmarks
+.coverage
+.coverage.*
+.pytest
+.pytest_cache/
+.tox
+coverage.xml
+htmlcov
+nosetests.xml
+
+# Translations
+*.mo
+
+# Buildout
+.mr.developer.cfg
+
+# IDE project files
+*.iml
+*.komodoproject
+.idea
+.project
+.pydevproject
+
+# Complexity
+output/*.html
+output/*/index.html
+
+# Sphinx
+docs/_build
+
+# Mypy Cache
+.mypy_cache/
+
+autograder_results.json
+**/autograder_output.json
+
+
+tests/test_demo.py
+tests/test_demo/**

pytest_prairielearn_grader-0.2.0/.vscode/extensions.json

@@ -0,0 +1,8 @@
+{
+  "recommendations": [
+      "charliermarsh.ruff",
+      "tamasfe.even-better-toml",
+      "ms-python.python",
+      "ms-python.vscode-pylance"
+  ]
+}

pytest_prairielearn_grader-0.2.0/.vscode/settings.json

@@ -0,0 +1,19 @@
+{
+  "python.testing.unittestEnabled": false,
+  "python.testing.pytestEnabled": true,
+  "[python]": {
+      "editor.formatOnSave": true,
+      "editor.defaultFormatter": "charliermarsh.ruff",
+      "editor.codeActionsOnSave": {
+          "source.organizeImports": "explicit",
+          "source.fixAll": "explicit"
+      }
+  },
+  "[yaml][toml][json]": {
+      "editor.formatOnSave": true
+  },
+  "files.trimTrailingWhitespace": true,
+  "files.insertFinalNewline": true,
+  "files.trimFinalNewlines": true,
+  "python.analysis.typeCheckingMode": "basic",
+}

pytest_prairielearn_grader-0.2.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025 Eliot W. Robson
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

pytest_prairielearn_grader-0.2.0/PKG-INFO

@@ -0,0 +1,43 @@
+Metadata-Version: 2.4
+Name: pytest-prairielearn-grader
+Version: 0.2.0
+Summary: A pytest plugin for autograding Python code. Designed for use with the PrairieLearn platform.
+Project-URL: repository, https://github.com/eliotwrobson/pl-python-autograder-v2
+Author-email: Eliot Robson <eliot.robson24@gmail.com>
+License: MIT License
+        
+        Copyright (c) 2025 Eliot W. Robson
+        
+        Permission is hereby granted, free of charge, to any person obtaining a copy
+        of this software and associated documentation files (the "Software"), to deal
+        in the Software without restriction, including without limitation the rights
+        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+        copies of the Software, and to permit persons to whom the Software is
+        furnished to do so, subject to the following conditions:
+        
+        The above copyright notice and this permission notice shall be included in all
+        copies or substantial portions of the Software.
+        
+        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+        SOFTWARE.
+License-File: LICENSE
+Classifier: Development Status :: 3 - Alpha
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
+Requires-Python: >=3.11
+Requires-Dist: dill>=0.4.0
+Requires-Dist: prettytable>=3.8.0
+Requires-Dist: pytest
+Description-Content-Type: text/markdown
+
+# pytest-prairielearn-grader
+
+A new version of the Python autograder for PrairieLearn, made using a pytest plugin.

pytest_prairielearn_grader-0.2.0/README.md

@@ -0,0 +1,3 @@
+# pytest-prairielearn-grader
+
+A new version of the Python autograder for PrairieLearn, made using a pytest plugin.

pytest_prairielearn_grader-0.2.0/docker/Dockerfile

@@ -0,0 +1,37 @@
+# https://docs.astral.sh/uv/guides/integration/docker/#available-images
+FROM ghcr.io/astral-sh/uv:alpine
+#ARG CACHEBUST=2025-07-15-14-16-18
+
+# Needed to properly handle UTF-8
+ENV PYTHONIOENCODING=UTF-8
+ENV LANG=en_US.UTF-8
+
+WORKDIR /
+
+RUN apk update \
+    && apk add --no-cache \
+    util-linux \
+    sudo \
+    build-base
+
+# NOTE all the crap below is needed to make pygraphviz work, but we probably don't need to include this
+# in the new grader image? We can add back if necessary but not sure what the use cases are.
+#    dos2unix \
+#    graphviz \
+#    graphviz-devel
+
+COPY requirements.txt /
+ARG PACKAGE_VERSION=latest
+RUN uv venv --seed && uv pip install --no-cache-dir -r requirements.txt && \
+    uv pip install --no-cache-dir pytest-prairielearn-grader==${PACKAGE_VERSION}
+
+# Anything that needs to be run post-install
+# TODO Not clear yet how to make the grader run subprocess as a less privileged user
+# RUN useradd ag
+COPY --chmod=755 run.sh /run.sh
+
+# Add serverFilesCourse to Python path
+# TODO do we need to copy anything here?
+ENV PYTHONPATH=/grade/serverFilesCourse/
+
+ENTRYPOINT [ "/run.sh" ]

pytest_prairielearn_grader-0.2.0/docker/requirements.txt

@@ -0,0 +1,29 @@
+
+beautifulsoup4==4.13.4
+bokeh==3.7.3
+colormath==3.0.0
+defusedxml==0.7.1
+Faker==37.5.3
+folium==0.20.0
+ipython==8.37.0
+matplotlib==3.10.3
+nbconvert==7.16.6
+nbformat==5.10.4
+networkx==3.4.2
+nltk==3.9.1
+numpy==2.3.2
+openpyxl==3.1.5
+pandas==2.3.1
+Pillow==11.3.0
+PuLP==3.2.2
+Pygments==2.19.2
+pyarrow==21.0.0
+pytest==8.4.1
+requests==2.32.4
+scikit-image==0.25.2
+scikit-learn==1.7.1
+scipy==1.15.3
+seaborn==0.13.2
+sympy==1.14.0
+text-unidecode==1.3
+typing-extensions==4.14.1

pytest_prairielearn_grader-0.2.0/docker/run.sh

@@ -0,0 +1,56 @@
+#! /bin/sh
+
+##########################
+# INIT
+##########################
+
+if [[ ! -d /grade ]]; then
+    echo "ERROR: /grade not found! Mounting may have failed."
+    exit 1
+fi
+
+# the parent directory containing everything about this grading job
+export JOB_DIR='/grade'
+# the job subdirectories
+STUDENT_DIR=$JOB_DIR'/student'
+TEST_DIR=$JOB_DIR'/tests'
+OUT_DIR=$JOB_DIR'/results'
+
+# where we will copy everything
+export MERGE_DIR=$JOB_DIR'/run'
+
+# now set up the stuff so that our run.sh can work
+mkdir $MERGE_DIR
+mkdir $MERGE_DIR/test_student
+mkdir $OUT_DIR
+
+mv $STUDENT_DIR/* $MERGE_DIR/test_student/
+# TODO this has some pretty hardcoded assumptions about file names. Discuss in meeting
+# what the best way to change this is
+mv $TEST_DIR/* $MERGE_DIR/test_student/
+mv $MERGE_DIR/test_student/test_student.py $MERGE_DIR/test_student.py
+
+# Do not allow ag user to modify, rename, or delete any existing files
+#chmod -R 755 "$MERGE_DIR"
+#chmod 1777 "$MERGE_DIR"
+
+##########################
+# RUN
+##########################
+
+echo "[run] starting autograder"
+
+# run the autograder as a limited user called ag
+# TODO pass in the ag user id as a CLI option
+#su -c "python3 $MERGE_DIR/pl_main.py" ag
+uv run pytest -p pl-grader --color=no "$MERGE_DIR"
+
+# TODO change the default output name
+mv "$MERGE_DIR/autograder_results.json" "$OUT_DIR/results.json"
+
+# if that didn't work, then print a last-ditch message
+if [ ! -s $OUT_DIR/results.json ]; then
+    echo '{"succeeded": false, "score": 0.0, "message": "Your code could not be processed by the autograder. Please contact course staff and have them check the logs for this submission."}' > $OUT_DIR/results.json
+fi
+
+echo "[run] autograder completed"