pl-e2e-test 0.1.2__tar.gz → 0.1.4__tar.gz

{pl_e2e_test-0.1.2/src/pl_e2e_test.egg-info → pl_e2e_test-0.1.4}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: pl-e2e-test
-Version: 0.1.2
+Version: 0.1.4
 Summary: Fuzz PrairieLearn question lifecycles against the real Python element runtime
 License-Expression: MIT
 Project-URL: Homepage, https://github.com/PrairieLearn/pl-e2e-test
@@ -18,6 +18,7 @@ Requires-Dist: platformdirs>=4.5
 Requires-Dist: prairielearn
 Requires-Dist: pyright>=1.1.414
 Requires-Dist: pytest>=9.1
+Requires-Dist: rich>=14
 Requires-Dist: ruff>=0.16
 Dynamic: license-file
 
@@ -34,6 +35,7 @@ The package installs the reusable `prairielearn_e2e` Python API and a single
 - `pl-e2e-test` (no command, or any pytest/harness options) runs the installed
   lifecycle pytest suite.
 - `pl-e2e-test diff` fuzzes questions selected from a Git diff.
+- `pl-e2e-test parallel` runs local shards and merges their results.
 - `pl-e2e-test add-seed` records a failing concrete variant seed.
 - `pl-e2e-test reports` merges parallel shard results into terminal and JUnit reports.
 - `pl-e2e-test version` prints the installed version.
@@ -96,6 +98,68 @@ uv run pl-e2e-test \
 variant set with `--fuzz-seed N`. Without either option, deterministic seeds start at
 zero. Question metadata with `singleVariant: true` always uses seed zero.
 
+### Local parallel runs
+
+Run multiple shards in one machine or CI job, with six concurrent processes by default:
+
+```sh
+uv run pl-e2e-test parallel --shard-count 6 run -- \
+  --question-path questions --fuzz-seeds
+
+uv run pl-e2e-test parallel --shard-count 6 --junitxml results.xml diff -- \
+  --base main --fuzz-seed 42
+```
+
+Launcher options go before the required `run` or `diff` selector; arguments after
+`--` are forwarded to that command. Normal runs remain deterministic. For
+`--fuzz-seeds` and diff runs, the launcher generates one shared master seed unless
+`--fuzz-seed` is supplied, and prints a command for replaying the run without sharding.
+
+Each shard runs in a separate Python process using the current environment and
+working directory. Terminals show Rich progress; redirected output prints completion
+lines. All shards finish before reports are merged, even when a test fails. Captured
+failure output and successful stderr are shown before the combined summary.
+Interrupting the launcher stops and reaps its children.
+
+Intermediate reports are temporary. Pass the launcher's `--junitxml` option to retain
+merged JUnit output. Forwarded `--shard`, `--shard-report`, `--junitxml`, and
+`--junit-xml` options are rejected because the launcher manages those outputs.
+Choose the course with the working directory, or `course_root` in the Python API.
+
+The same orchestration is available to Python callers:
+
+```python
+from prairielearn_e2e import run_parallel
+
+status = run_parallel(
+    ["--question-path", "questions", "--fuzz-seeds"],
+    command="run",
+    shard_count=6,
+    course_root="/path/to/course",
+    junitxml="results.xml",
+)
+```
+
+`run_parallel` returns an integer exit status, including 130 on interruption.
+Relative JUnit paths are resolved against the course root. With `junitxml=None`,
+merged output is temporary. Invalid launcher settings raise `ValueError`.
+
+Courses using `scripts/run_e2e_shards.py` can replace its invocation with
+`pl-e2e-test parallel`, move `--fuzz-seed` into the forwarded arguments, and explicitly
+pass `--fuzz-seeds` for full-suite runs to preserve the script's fuzzing behavior.
+For example:
+
+```make
+test-e2e-sharded:
+	uv run --active pl-e2e-test parallel --shard-count "$(E2E_SHARD_COUNT)" run -- \
+		$(E2E_TEST_ARGS) $(E2E_PRAIRIELEARN_ARG) --fuzz-seeds $(E2E_FUZZ_SEED_ARG) $(PYTEST_ARGS)
+
+test-e2e-fuzz-sharded:
+	uv run --active pl-e2e-test parallel --shard-count "$(E2E_SHARD_COUNT)" diff -- \
+		--base "$(E2E_DIFF_BASE)" --head "$(E2E_DIFF_HEAD)" \
+		--seed-count "$(E2E_FUZZ_SEED_COUNT)" $(E2E_FUZZ_SEED_ARG) $(E2E_PRAIRIELEARN_ARG) $(PYTEST_ARGS)
+```
+
 ### CI sharding
 
 Use `--shard INDEX/COUNT` to divide the selected question variants between parallel
@@ -191,6 +255,25 @@ element controllers are archived into the test run's temporary directory. Standa
 cache environment settings such as `XDG_CACHE_HOME` can relocate the managed cache;
 CI systems can persist that directory between runs if desired.
 
+### Harness configuration
+
+Selector settings are read automatically from the course's `pyproject.toml`:
+
+```toml
+[tool.pl-e2e-test]
+elementSolutionSelectors = ["pl-sketch"]
+nonGradableSelectors = ['pl-sketch[read-only="true"]', 'pl-sketch[read-only="1"]']
+```
+
+These are the defaults. `elementSolutionSelectors` identifies elements that provide
+solutions without a `correct_answers` entry. `nonGradableSelectors` excludes
+display-only answers from grading checks. Each supplied list replaces its default;
+omitted settings keep their defaults, and `[]` disables a list.
+
+Use `--config path/to/settings.toml` to select another TOML file with the same
+`[tool.pl-e2e-test]` table. This works with normal runs, `diff`, and arguments after
+`parallel run` or `parallel diff`. Explicit paths are relative to the working directory.
+
 ### Question-local configuration
 
 Store saved concrete seeds and narrow phase skips in `.question-e2e.json` beside the
@@ -267,6 +350,11 @@ from prairielearn_e2e import PrairieLearnBackend, QuestionCase, QuestionHarness
 increasingly complete portions of the lifecycle. `run_variant` remains an alias for
 the complete grade path.
 
+Pass `config=load_config(course_root=course_root)` to `QuestionHarness` to load the
+course's TOML settings, or `config=load_config("settings.toml")` for an explicit file.
+`load_config` and `HarnessConfig` are exported from `prairielearn_e2e`. Omitting
+`config` uses the built-in defaults.
+
 ## Development
 
 This project uses uv 0.9 or newer:

{pl_e2e_test-0.1.2 → pl_e2e_test-0.1.4}/README.md

@@ -11,6 +11,7 @@ The package installs the reusable `prairielearn_e2e` Python API and a single
 - `pl-e2e-test` (no command, or any pytest/harness options) runs the installed
   lifecycle pytest suite.
 - `pl-e2e-test diff` fuzzes questions selected from a Git diff.
+- `pl-e2e-test parallel` runs local shards and merges their results.
 - `pl-e2e-test add-seed` records a failing concrete variant seed.
 - `pl-e2e-test reports` merges parallel shard results into terminal and JUnit reports.
 - `pl-e2e-test version` prints the installed version.
@@ -73,6 +74,68 @@ uv run pl-e2e-test \
 variant set with `--fuzz-seed N`. Without either option, deterministic seeds start at
 zero. Question metadata with `singleVariant: true` always uses seed zero.
 
+### Local parallel runs
+
+Run multiple shards in one machine or CI job, with six concurrent processes by default:
+
+```sh
+uv run pl-e2e-test parallel --shard-count 6 run -- \
+  --question-path questions --fuzz-seeds
+
+uv run pl-e2e-test parallel --shard-count 6 --junitxml results.xml diff -- \
+  --base main --fuzz-seed 42
+```
+
+Launcher options go before the required `run` or `diff` selector; arguments after
+`--` are forwarded to that command. Normal runs remain deterministic. For
+`--fuzz-seeds` and diff runs, the launcher generates one shared master seed unless
+`--fuzz-seed` is supplied, and prints a command for replaying the run without sharding.
+
+Each shard runs in a separate Python process using the current environment and
+working directory. Terminals show Rich progress; redirected output prints completion
+lines. All shards finish before reports are merged, even when a test fails. Captured
+failure output and successful stderr are shown before the combined summary.
+Interrupting the launcher stops and reaps its children.
+
+Intermediate reports are temporary. Pass the launcher's `--junitxml` option to retain
+merged JUnit output. Forwarded `--shard`, `--shard-report`, `--junitxml`, and
+`--junit-xml` options are rejected because the launcher manages those outputs.
+Choose the course with the working directory, or `course_root` in the Python API.
+
+The same orchestration is available to Python callers:
+
+```python
+from prairielearn_e2e import run_parallel
+
+status = run_parallel(
+    ["--question-path", "questions", "--fuzz-seeds"],
+    command="run",
+    shard_count=6,
+    course_root="/path/to/course",
+    junitxml="results.xml",
+)
+```
+
+`run_parallel` returns an integer exit status, including 130 on interruption.
+Relative JUnit paths are resolved against the course root. With `junitxml=None`,
+merged output is temporary. Invalid launcher settings raise `ValueError`.
+
+Courses using `scripts/run_e2e_shards.py` can replace its invocation with
+`pl-e2e-test parallel`, move `--fuzz-seed` into the forwarded arguments, and explicitly
+pass `--fuzz-seeds` for full-suite runs to preserve the script's fuzzing behavior.
+For example:
+
+```make
+test-e2e-sharded:
+	uv run --active pl-e2e-test parallel --shard-count "$(E2E_SHARD_COUNT)" run -- \
+		$(E2E_TEST_ARGS) $(E2E_PRAIRIELEARN_ARG) --fuzz-seeds $(E2E_FUZZ_SEED_ARG) $(PYTEST_ARGS)
+
+test-e2e-fuzz-sharded:
+	uv run --active pl-e2e-test parallel --shard-count "$(E2E_SHARD_COUNT)" diff -- \
+		--base "$(E2E_DIFF_BASE)" --head "$(E2E_DIFF_HEAD)" \
+		--seed-count "$(E2E_FUZZ_SEED_COUNT)" $(E2E_FUZZ_SEED_ARG) $(E2E_PRAIRIELEARN_ARG) $(PYTEST_ARGS)
+```
+
 ### CI sharding
 
 Use `--shard INDEX/COUNT` to divide the selected question variants between parallel
@@ -168,6 +231,25 @@ element controllers are archived into the test run's temporary directory. Standa
 cache environment settings such as `XDG_CACHE_HOME` can relocate the managed cache;
 CI systems can persist that directory between runs if desired.
 
+### Harness configuration
+
+Selector settings are read automatically from the course's `pyproject.toml`:
+
+```toml
+[tool.pl-e2e-test]
+elementSolutionSelectors = ["pl-sketch"]
+nonGradableSelectors = ['pl-sketch[read-only="true"]', 'pl-sketch[read-only="1"]']
+```
+
+These are the defaults. `elementSolutionSelectors` identifies elements that provide
+solutions without a `correct_answers` entry. `nonGradableSelectors` excludes
+display-only answers from grading checks. Each supplied list replaces its default;
+omitted settings keep their defaults, and `[]` disables a list.
+
+Use `--config path/to/settings.toml` to select another TOML file with the same
+`[tool.pl-e2e-test]` table. This works with normal runs, `diff`, and arguments after
+`parallel run` or `parallel diff`. Explicit paths are relative to the working directory.
+
 ### Question-local configuration
 
 Store saved concrete seeds and narrow phase skips in `.question-e2e.json` beside the
@@ -244,6 +326,11 @@ from prairielearn_e2e import PrairieLearnBackend, QuestionCase, QuestionHarness
 increasingly complete portions of the lifecycle. `run_variant` remains an alias for
 the complete grade path.
 
+Pass `config=load_config(course_root=course_root)` to `QuestionHarness` to load the
+course's TOML settings, or `config=load_config("settings.toml")` for an explicit file.
+`load_config` and `HarnessConfig` are exported from `prairielearn_e2e`. Omitting
+`config` uses the built-in defaults.
+
 ## Development
 
 This project uses uv 0.9 or newer:

{pl_e2e_test-0.1.2 → pl_e2e_test-0.1.4}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "pl-e2e-test"
-version = "0.1.2"
+version = "0.1.4"
 description = "Fuzz PrairieLearn question lifecycles against the real Python element runtime"
 readme = "README.md"
 requires-python = ">=3.13"
@@ -19,6 +19,7 @@ dependencies = [
   "prairielearn",
   "pyright>=1.1.414",
   "pytest>=9.1",
+  "rich>=14",
   "ruff>=0.16",
 ]
 
@@ -36,9 +37,6 @@ pl-e2e-test = "prairielearn_e2e.cli:main"
 [tool.setuptools.packages.find]
 where = ["src"]
 
-[tool.setuptools.package-data]
-prairielearn_e2e = ["config.json"]
-
 [tool.pytest.ini_options]
 testpaths = ["tests"]
 

{pl_e2e_test-0.1.2 → pl_e2e_test-0.1.4/src/pl_e2e_test.egg-info}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: pl-e2e-test
-Version: 0.1.2
+Version: 0.1.4
 Summary: Fuzz PrairieLearn question lifecycles against the real Python element runtime
 License-Expression: MIT
 Project-URL: Homepage, https://github.com/PrairieLearn/pl-e2e-test
@@ -18,6 +18,7 @@ Requires-Dist: platformdirs>=4.5
 Requires-Dist: prairielearn
 Requires-Dist: pyright>=1.1.414
 Requires-Dist: pytest>=9.1
+Requires-Dist: rich>=14
 Requires-Dist: ruff>=0.16
 Dynamic: license-file
 
@@ -34,6 +35,7 @@ The package installs the reusable `prairielearn_e2e` Python API and a single
 - `pl-e2e-test` (no command, or any pytest/harness options) runs the installed
   lifecycle pytest suite.
 - `pl-e2e-test diff` fuzzes questions selected from a Git diff.
+- `pl-e2e-test parallel` runs local shards and merges their results.
 - `pl-e2e-test add-seed` records a failing concrete variant seed.
 - `pl-e2e-test reports` merges parallel shard results into terminal and JUnit reports.
 - `pl-e2e-test version` prints the installed version.
@@ -96,6 +98,68 @@ uv run pl-e2e-test \
 variant set with `--fuzz-seed N`. Without either option, deterministic seeds start at
 zero. Question metadata with `singleVariant: true` always uses seed zero.
 
+### Local parallel runs
+
+Run multiple shards in one machine or CI job, with six concurrent processes by default:
+
+```sh
+uv run pl-e2e-test parallel --shard-count 6 run -- \
+  --question-path questions --fuzz-seeds
+
+uv run pl-e2e-test parallel --shard-count 6 --junitxml results.xml diff -- \
+  --base main --fuzz-seed 42
+```
+
+Launcher options go before the required `run` or `diff` selector; arguments after
+`--` are forwarded to that command. Normal runs remain deterministic. For
+`--fuzz-seeds` and diff runs, the launcher generates one shared master seed unless
+`--fuzz-seed` is supplied, and prints a command for replaying the run without sharding.
+
+Each shard runs in a separate Python process using the current environment and
+working directory. Terminals show Rich progress; redirected output prints completion
+lines. All shards finish before reports are merged, even when a test fails. Captured
+failure output and successful stderr are shown before the combined summary.
+Interrupting the launcher stops and reaps its children.
+
+Intermediate reports are temporary. Pass the launcher's `--junitxml` option to retain
+merged JUnit output. Forwarded `--shard`, `--shard-report`, `--junitxml`, and
+`--junit-xml` options are rejected because the launcher manages those outputs.
+Choose the course with the working directory, or `course_root` in the Python API.
+
+The same orchestration is available to Python callers:
+
+```python
+from prairielearn_e2e import run_parallel
+
+status = run_parallel(
+    ["--question-path", "questions", "--fuzz-seeds"],
+    command="run",
+    shard_count=6,
+    course_root="/path/to/course",
+    junitxml="results.xml",
+)
+```
+
+`run_parallel` returns an integer exit status, including 130 on interruption.
+Relative JUnit paths are resolved against the course root. With `junitxml=None`,
+merged output is temporary. Invalid launcher settings raise `ValueError`.
+
+Courses using `scripts/run_e2e_shards.py` can replace its invocation with
+`pl-e2e-test parallel`, move `--fuzz-seed` into the forwarded arguments, and explicitly
+pass `--fuzz-seeds` for full-suite runs to preserve the script's fuzzing behavior.
+For example:
+
+```make
+test-e2e-sharded:
+	uv run --active pl-e2e-test parallel --shard-count "$(E2E_SHARD_COUNT)" run -- \
+		$(E2E_TEST_ARGS) $(E2E_PRAIRIELEARN_ARG) --fuzz-seeds $(E2E_FUZZ_SEED_ARG) $(PYTEST_ARGS)
+
+test-e2e-fuzz-sharded:
+	uv run --active pl-e2e-test parallel --shard-count "$(E2E_SHARD_COUNT)" diff -- \
+		--base "$(E2E_DIFF_BASE)" --head "$(E2E_DIFF_HEAD)" \
+		--seed-count "$(E2E_FUZZ_SEED_COUNT)" $(E2E_FUZZ_SEED_ARG) $(E2E_PRAIRIELEARN_ARG) $(PYTEST_ARGS)
+```
+
 ### CI sharding
 
 Use `--shard INDEX/COUNT` to divide the selected question variants between parallel
@@ -191,6 +255,25 @@ element controllers are archived into the test run's temporary directory. Standa
 cache environment settings such as `XDG_CACHE_HOME` can relocate the managed cache;
 CI systems can persist that directory between runs if desired.
 
+### Harness configuration
+
+Selector settings are read automatically from the course's `pyproject.toml`:
+
+```toml
+[tool.pl-e2e-test]
+elementSolutionSelectors = ["pl-sketch"]
+nonGradableSelectors = ['pl-sketch[read-only="true"]', 'pl-sketch[read-only="1"]']
+```
+
+These are the defaults. `elementSolutionSelectors` identifies elements that provide
+solutions without a `correct_answers` entry. `nonGradableSelectors` excludes
+display-only answers from grading checks. Each supplied list replaces its default;
+omitted settings keep their defaults, and `[]` disables a list.
+
+Use `--config path/to/settings.toml` to select another TOML file with the same
+`[tool.pl-e2e-test]` table. This works with normal runs, `diff`, and arguments after
+`parallel run` or `parallel diff`. Explicit paths are relative to the working directory.
+
 ### Question-local configuration
 
 Store saved concrete seeds and narrow phase skips in `.question-e2e.json` beside the
@@ -267,6 +350,11 @@ from prairielearn_e2e import PrairieLearnBackend, QuestionCase, QuestionHarness
 increasingly complete portions of the lifecycle. `run_variant` remains an alias for
 the complete grade path.
 
+Pass `config=load_config(course_root=course_root)` to `QuestionHarness` to load the
+course's TOML settings, or `config=load_config("settings.toml")` for an explicit file.
+`load_config` and `HarnessConfig` are exported from `prairielearn_e2e`. Omitting
+`config` uses the built-in defaults.
+
 ## Development
 
 This project uses uv 0.9 or newer:

{pl_e2e_test-0.1.2 → pl_e2e_test-0.1.4}/src/pl_e2e_test.egg-info/SOURCES.txt

@@ -13,7 +13,6 @@ src/prairielearn_e2e/__main__.py
 src/prairielearn_e2e/_git.py
 src/prairielearn_e2e/backend.py
 src/prairielearn_e2e/cli.py
-src/prairielearn_e2e/config.json
 src/prairielearn_e2e/config.py
 src/prairielearn_e2e/contract.py
 src/prairielearn_e2e/discovery.py
@@ -22,6 +21,7 @@ src/prairielearn_e2e/fuzz.py
 src/prairielearn_e2e/harness.py
 src/prairielearn_e2e/html_checks.py
 src/prairielearn_e2e/metadata.py
+src/prairielearn_e2e/parallel.py
 src/prairielearn_e2e/pytest_plugin.py
 src/prairielearn_e2e/question_tests.py
 src/prairielearn_e2e/regression.py
@@ -34,6 +34,7 @@ tests/__init__.py
 tests/conftest.py
 tests/stub_git.py
 tests/test_cli.py
+tests/test_config.py
 tests/test_contract.py
 tests/test_discovery.py
 tests/test_fuzz.py
@@ -41,6 +42,7 @@ tests/test_git.py
 tests/test_harness.py
 tests/test_html_checks.py
 tests/test_metadata.py
+tests/test_parallel.py
 tests/test_publish_version.py
 tests/test_pytest_plugin.py
 tests/test_regression.py

{pl_e2e_test-0.1.2 → pl_e2e_test-0.1.4}/src/pl_e2e_test.egg-info/requires.txt

@@ -7,4 +7,5 @@ platformdirs>=4.5
 prairielearn
 pyright>=1.1.414
 pytest>=9.1
+rich>=14
 ruff>=0.16

{pl_e2e_test-0.1.2 → pl_e2e_test-0.1.4}/src/prairielearn_e2e/__init__.py

@@ -3,16 +3,21 @@
 from importlib.metadata import version
 
 from .backend import PrairieLearnBackend
+from .config import HarnessConfig, load_config
 from .discovery import QuestionCase, discover_questions
 from .harness import LifecycleResult, QuestionHarness
+from .parallel import run_parallel
 
 __version__ = version("pl-e2e-test")
 
 __all__ = (
+    "HarnessConfig",
     "LifecycleResult",
     "PrairieLearnBackend",
     "QuestionCase",
     "QuestionHarness",
     "__version__",
     "discover_questions",
+    "load_config",
+    "run_parallel",
 )

{pl_e2e_test-0.1.2 → pl_e2e_test-0.1.4}/src/prairielearn_e2e/cli.py

@@ -6,10 +6,11 @@ import sys
 from collections.abc import Sequence
 from importlib.metadata import version
 
-from . import fuzz, regression, reporting, runner
+from . import fuzz, parallel, regression, reporting, runner
 
 _SUBCOMMANDS = {
     "diff": fuzz.main,
+    "parallel": parallel.main,
     "add-seed": regression.main,
     "reports": reporting.main,
 }
@@ -22,6 +23,7 @@ suite; any arguments (pytest options, --question-path, --seed-count, ...)
 are passed straight through to pytest.
 
 Commands:
+  parallel    run local shards and merge their results
   diff        fuzz questions changed between two Git refs
   add-seed    record a concrete regression seed for a question
   reports     merge parallel shard reports into one terminal/JUnit report

pl_e2e_test-0.1.4/src/prairielearn_e2e/config.py

@@ -0,0 +1,68 @@
+"""Load harness settings from the course's pyproject.toml."""
+
+from __future__ import annotations
+
+import tomllib
+from dataclasses import dataclass
+from pathlib import Path
+
+
+@dataclass(frozen=True, slots=True)
+class HarnessConfig:
+    element_solution_answer_selectors: tuple[str, ...] = ("pl-sketch",)
+    non_gradable_answer_selectors: tuple[str, ...] = (
+        'pl-sketch[read-only="true"]',
+        'pl-sketch[read-only="1"]',
+    )
+
+
+DEFAULT_CONFIG = HarnessConfig()
+
+
+def load_config(
+    path: str | Path | None = None, *, course_root: str | Path | None = None
+) -> HarnessConfig:
+    """Read [tool.pl-e2e-test], preserving defaults for omitted settings.
+
+    Without an explicit path, look for pyproject.toml in the course root.
+    A missing implicit file is allowed; a missing explicit file is an error.
+    """
+    config_path = (
+        Path(path)
+        if path is not None
+        else Path(course_root or Path.cwd()) / "pyproject.toml"
+    )
+    try:
+        with config_path.open("rb") as config_file:
+            document = tomllib.load(config_file)
+    except FileNotFoundError:
+        if path is not None:
+            raise
+        return DEFAULT_CONFIG
+    tool = document.get("tool", {})
+    if not isinstance(tool, dict):
+        raise TypeError("tool must be a table")
+    config = tool.get("pl-e2e-test", {})
+    if not isinstance(config, dict):
+        raise TypeError("tool.pl-e2e-test must be a table")
+
+    def selectors(key: str, default: tuple[str, ...]) -> tuple[str, ...]:
+        if key not in config:
+            return default
+        values = config[key]
+        if not isinstance(values, list) or any(
+            not isinstance(value, str) or not value.strip() for value in values
+        ):
+            raise TypeError(
+                f"tool.pl-e2e-test.{key} must be an array of non-empty strings"
+            )
+        return tuple(values)
+
+    return HarnessConfig(
+        element_solution_answer_selectors=selectors(
+            "elementSolutionSelectors", DEFAULT_CONFIG.element_solution_answer_selectors
+        ),
+        non_gradable_answer_selectors=selectors(
+            "nonGradableSelectors", DEFAULT_CONFIG.non_gradable_answer_selectors
+        ),
+    )

{pl_e2e_test-0.1.2 → pl_e2e_test-0.1.4}/src/prairielearn_e2e/fuzz.py

@@ -197,6 +197,9 @@ def positive_integer(value: str) -> int:
 def main(argv: Sequence[str] | None = None) -> int:
     parser = argparse.ArgumentParser(prog="pl-e2e-test diff", description=__doc__)
     parser.add_argument("--course-root", type=Path)
+    parser.add_argument(
+        "--config", type=Path, help="TOML config (default: COURSE_ROOT/pyproject.toml)"
+    )
     parser.add_argument("--base", default="main", help="base Git ref (default: main)")
     parser.add_argument("--head", default="HEAD", help="head Git ref (default: HEAD)")
     parser.add_argument("--seed-count", type=positive_integer, default=10)
@@ -276,6 +279,9 @@ def main(argv: Sequence[str] | None = None) -> int:
     print(f"Variant seeds: {', '.join(str(seed) for seed in seeds)}")
     replay_arguments = ["pl-e2e-test", "diff", "--fuzz-seed", str(fuzz_seed)]
     test_arguments: list[str] = []
+    if args.config is not None:
+        replay_arguments.extend(("--config", os.fspath(args.config)))
+        test_arguments.extend(("--config", os.fspath(args.config)))
     if args.prairielearn_path is not None:
         replay_arguments.extend(
             ("--prairielearn-path", os.fspath(args.prairielearn_path))

{pl_e2e_test-0.1.2 → pl_e2e_test-0.1.4}/src/prairielearn_e2e/harness.py

@@ -13,6 +13,7 @@ import lxml.html
 import numpy as np
 
 from .backend import PrairieLearnBackend
+from .config import DEFAULT_CONFIG, HarnessConfig
 from .contract import PHASE_FIELDS, Phase
 from .discovery import QuestionCase
 from .errors import HarnessError, UnsupportedQuestion
@@ -112,6 +113,7 @@ class QuestionHarness:
         backend: PrairieLearnBackend,
         *,
         course_root: str | Path | None = None,
+        config: HarnessConfig = DEFAULT_CONFIG,
     ) -> None:
         if isinstance(question, QuestionCase):
             self.question = question
@@ -124,6 +126,7 @@ class QuestionHarness:
                 directory = directory.parent
             self.question = QuestionCase(directory.resolve(), root)
         self.backend = backend
+        self.config = config
 
     def _options(self) -> dict[str, Any]:
         directory = self.question.directory
@@ -333,12 +336,12 @@ class QuestionHarness:
 
             show_correct = metadata.get("showCorrectAnswer", True)
             answer_names = validate_declared_answers(
-                prepared_html, base["correct_answers"], show_correct
+                prepared_html, base["correct_answers"], show_correct, self.config
             )
             tested_answer_names = tuple(
                 name
                 for name in answer_names
-                if name not in non_gradable_answer_names(prepared_html)
+                if name not in non_gradable_answer_names(prepared_html, self.config)
             )
             rendered_question = self._render(
                 source, base, server, panel="question", show_correct=show_correct

{pl_e2e_test-0.1.2 → pl_e2e_test-0.1.4}/src/prairielearn_e2e/html_checks.py

@@ -6,7 +6,7 @@ from typing import Any
 
 import lxml.html
 
-from .config import ELEMENT_SOLUTION_ANSWER_SELECTORS, NON_GRADABLE_ANSWER_SELECTORS
+from .config import DEFAULT_CONFIG, HarnessConfig
 from .errors import HarnessError
 
 _SIMPLE_CSS_SELECTOR = re.compile(
@@ -100,17 +100,23 @@ def _answer_names_matching_selectors(
     return frozenset(names)
 
 
-def element_solution_answer_names(html: str) -> frozenset[str]:
+def element_solution_answer_names(
+    html: str, config: HarnessConfig = DEFAULT_CONFIG
+) -> frozenset[str]:
     """Return answer names whose solution is owned by the element.
 
     These elements do not need an entry in ``correct_answers``, but they can
     still be graded and must remain part of the parse/grade lifecycle checks.
     """
 
-    return _answer_names_matching_selectors(html, ELEMENT_SOLUTION_ANSWER_SELECTORS)
+    return _answer_names_matching_selectors(
+        html, config.element_solution_answer_selectors
+    )
 
 
-def non_gradable_answer_names(html: str) -> frozenset[str]:
+def non_gradable_answer_names(
+    html: str, config: HarnessConfig = DEFAULT_CONFIG
+) -> frozenset[str]:
     """Return answer names intentionally excluded from server-side grading.
 
     Configured display-only elements still carry an ``answers-name`` for
@@ -118,15 +124,20 @@ def non_gradable_answer_names(html: str) -> frozenset[str]:
     partial score.
     """
 
-    return _answer_names_matching_selectors(html, NON_GRADABLE_ANSWER_SELECTORS)
+    return _answer_names_matching_selectors(html, config.non_gradable_answer_selectors)
 
 
 def validate_declared_answers(
-    html: str, correct_answers: dict[str, Any], show_correct_answer: bool
+    html: str,
+    correct_answers: dict[str, Any],
+    show_correct_answer: bool,
+    config: HarnessConfig = DEFAULT_CONFIG,
 ) -> tuple[str, ...]:
     names = declared_answer_names(html)
     if show_correct_answer:
-        ignored = non_gradable_answer_names(html) | element_solution_answer_names(html)
+        ignored = non_gradable_answer_names(
+            html, config
+        ) | element_solution_answer_names(html, config)
         require_correct_answers(
             tuple(name for name in names if name not in ignored), correct_answers
         )