pl-e2e-test 0.1.5__tar.gz → 0.1.7__tar.gz

{pl_e2e_test-0.1.5/src/pl_e2e_test.egg-info → pl_e2e_test-0.1.7}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: pl-e2e-test
-Version: 0.1.5
+Version: 0.1.7
 Summary: Fuzz PrairieLearn question lifecycles against the real Python element runtime
 License-Expression: MIT
 Project-URL: Homepage, https://github.com/PrairieLearn/pl-e2e-test
@@ -266,14 +266,90 @@ nonGradableSelectors = ['pl-sketch[read-only="true"]', 'pl-sketch[read-only="1"]
 ```
 
 These are the defaults. `elementSolutionSelectors` identifies elements that provide
-solutions without a `correct_answers` entry. `nonGradableSelectors` excludes
-display-only answers from grading checks. Each supplied list replaces its default;
-omitted settings keep their defaults, and `[]` disables a list.
+solutions without a `correct_answers` entry. It only relaxes that entry requirement;
+it does not generate a test submission. `nonGradableSelectors` excludes display-only
+answers from grading checks. Each supplied list replaces its default; omitted settings
+keep their defaults, and `[]` disables a list.
 
 Use `--config path/to/settings.toml` to select another TOML file with the same
 `[tool.pl-e2e-test]` table. This works with normal runs, `diff`, and arguments after
 `parallel run` or `parallel diff`. Explicit paths are relative to the working directory.
 
+### Lifecycle JSON boundary
+
+PrairieLearn crosses a JSON boundary after processing the elements in a lifecycle phase
+and again after calling the question's `server.py` hook. The harness mirrors those
+boundaries. It first checks the in-process result for legal field mutations, then
+serializes and deserializes `data`; the normalized result is what the next hook or phase
+receives.
+
+This accepts the same Python values as PrairieLearn's JSON encoder rather than requiring
+literal JSON-domain Python types. For example, tuples normalize to arrays (Python lists),
+and supported non-string object keys such as numeric drawing IDs normalize to strings.
+This conversion can be lossy: if two keys normalize to the same string, the last key in
+serialized order wins, matching PrairieLearn's boundary. Unsupported objects, non-finite
+floats, and integers outside JavaScript's safe range are rejected with the failing data
+path.
+
+### Correct submissions for custom elements
+
+The grade check uses PrairieLearn's `test` lifecycle to obtain a deterministic correct
+submission. PrairieLearn first calls `test(element_html, data)` on the elements in the
+question, then the harness calls `test(data)` in the question's `server.py`. A reusable
+course element should normally implement the element hook. The question hook can fill
+in a submission when the correct state is question-specific or when a custom element
+does not provide one itself.
+
+For every gradable `answers-name`, the correct test case must provide:
+
+- a non-null value in `data["raw_submitted_answers"]` under the answer name, or one or
+  more compound input keys prefixed with `answers-name-`; and
+- `data["partial_scores"][answers_name]["score"] = 1`.
+
+Raw values must survive PrairieLearn's JSON boundary and have the same browser transport
+shape that the element's `parse` hook expects. Structured widgets commonly serialize
+their state as a JSON string. The hook must derive its output only from deterministic
+variant state such as `params` and `correct_answers`, mutate `data` in place, and return
+`None`. The normal phase mutation checks and JSON-boundary normalization still apply.
+
+For example, a question can supply the correct state for a drag-and-drop element with:
+
+```python
+import json
+
+
+def test(data):
+    if data["test_type"] != "correct":
+        return
+
+    answers_name = "ordering"
+    state = {
+        "solution": data["params"]["correct_order"],
+        "starter": [],
+    }
+    data["raw_submitted_answers"][answers_name] = json.dumps(state)
+    data["partial_scores"][answers_name] = {"score": 1}
+```
+
+Compound elements keep the declared answer name as their grading identity while using
+namespaced child fields for browser inputs. For example, an element with
+`answers-name="match"` may test and parse `match-dropdown-0` and `match-dropdown-1`,
+then store its score in `partial_scores["match"]`; it does not need to invent a
+`submitted_answers["match"]` value. After parsing, every raw field associated with a
+declared gradable answer must either remain non-null under its child key or produce a
+non-null value under its declared parent answer. This also supports elements that
+normalize `answer-input` into `answer`. Unassociated transport fields from file-producing
+elements may instead be consumed into `submitted_answers["_files"]`; they are not answer
+parse-coverage obligations.
+
+This contract keeps element-specific synthesis with the element or question that owns
+the data. PrairieLearn's tuple-based `pl-matching` state normalizes to arrays at the JSON
+boundary, and its existing namespaced test fields require no harness adapter. A question
+using `pl-sketch` without an element-owned solution can supply its deterministic
+`answers-name-sketchresponse-submission` payload in `server.py`. A course-owned
+drag-and-drop element such as `pl-faded-parsons` can implement one reusable element
+test hook that serializes its prepared correct tray and blank state.
+
 ### Question-local configuration
 
 Store saved concrete seeds, narrow phase skips, and expected upstream errors in
@@ -407,7 +483,8 @@ environment. From a clean branch, run:
 make publish-version VERSION=0.2.0
 ```
 
-This runs the full checks, updates `pyproject.toml` and `uv.lock`, builds the
+This installs Python 3.13, performs the same locked dependency sync used in CI, runs
+the full checks, updates `pyproject.toml` and `uv.lock`, builds and smoke-tests the
 distributions, creates a release commit and annotated `v0.2.0` tag, and atomically
 pushes the branch and tag. The tag starts the workflow on `ubuntu-latest`; the workflow
 rechecks the tag/version match, tests, builds, generates attestations, and publishes

{pl_e2e_test-0.1.5 → pl_e2e_test-0.1.7}/README.md

@@ -242,14 +242,90 @@ nonGradableSelectors = ['pl-sketch[read-only="true"]', 'pl-sketch[read-only="1"]
 ```
 
 These are the defaults. `elementSolutionSelectors` identifies elements that provide
-solutions without a `correct_answers` entry. `nonGradableSelectors` excludes
-display-only answers from grading checks. Each supplied list replaces its default;
-omitted settings keep their defaults, and `[]` disables a list.
+solutions without a `correct_answers` entry. It only relaxes that entry requirement;
+it does not generate a test submission. `nonGradableSelectors` excludes display-only
+answers from grading checks. Each supplied list replaces its default; omitted settings
+keep their defaults, and `[]` disables a list.
 
 Use `--config path/to/settings.toml` to select another TOML file with the same
 `[tool.pl-e2e-test]` table. This works with normal runs, `diff`, and arguments after
 `parallel run` or `parallel diff`. Explicit paths are relative to the working directory.
 
+### Lifecycle JSON boundary
+
+PrairieLearn crosses a JSON boundary after processing the elements in a lifecycle phase
+and again after calling the question's `server.py` hook. The harness mirrors those
+boundaries. It first checks the in-process result for legal field mutations, then
+serializes and deserializes `data`; the normalized result is what the next hook or phase
+receives.
+
+This accepts the same Python values as PrairieLearn's JSON encoder rather than requiring
+literal JSON-domain Python types. For example, tuples normalize to arrays (Python lists),
+and supported non-string object keys such as numeric drawing IDs normalize to strings.
+This conversion can be lossy: if two keys normalize to the same string, the last key in
+serialized order wins, matching PrairieLearn's boundary. Unsupported objects, non-finite
+floats, and integers outside JavaScript's safe range are rejected with the failing data
+path.
+
+### Correct submissions for custom elements
+
+The grade check uses PrairieLearn's `test` lifecycle to obtain a deterministic correct
+submission. PrairieLearn first calls `test(element_html, data)` on the elements in the
+question, then the harness calls `test(data)` in the question's `server.py`. A reusable
+course element should normally implement the element hook. The question hook can fill
+in a submission when the correct state is question-specific or when a custom element
+does not provide one itself.
+
+For every gradable `answers-name`, the correct test case must provide:
+
+- a non-null value in `data["raw_submitted_answers"]` under the answer name, or one or
+  more compound input keys prefixed with `answers-name-`; and
+- `data["partial_scores"][answers_name]["score"] = 1`.
+
+Raw values must survive PrairieLearn's JSON boundary and have the same browser transport
+shape that the element's `parse` hook expects. Structured widgets commonly serialize
+their state as a JSON string. The hook must derive its output only from deterministic
+variant state such as `params` and `correct_answers`, mutate `data` in place, and return
+`None`. The normal phase mutation checks and JSON-boundary normalization still apply.
+
+For example, a question can supply the correct state for a drag-and-drop element with:
+
+```python
+import json
+
+
+def test(data):
+    if data["test_type"] != "correct":
+        return
+
+    answers_name = "ordering"
+    state = {
+        "solution": data["params"]["correct_order"],
+        "starter": [],
+    }
+    data["raw_submitted_answers"][answers_name] = json.dumps(state)
+    data["partial_scores"][answers_name] = {"score": 1}
+```
+
+Compound elements keep the declared answer name as their grading identity while using
+namespaced child fields for browser inputs. For example, an element with
+`answers-name="match"` may test and parse `match-dropdown-0` and `match-dropdown-1`,
+then store its score in `partial_scores["match"]`; it does not need to invent a
+`submitted_answers["match"]` value. After parsing, every raw field associated with a
+declared gradable answer must either remain non-null under its child key or produce a
+non-null value under its declared parent answer. This also supports elements that
+normalize `answer-input` into `answer`. Unassociated transport fields from file-producing
+elements may instead be consumed into `submitted_answers["_files"]`; they are not answer
+parse-coverage obligations.
+
+This contract keeps element-specific synthesis with the element or question that owns
+the data. PrairieLearn's tuple-based `pl-matching` state normalizes to arrays at the JSON
+boundary, and its existing namespaced test fields require no harness adapter. A question
+using `pl-sketch` without an element-owned solution can supply its deterministic
+`answers-name-sketchresponse-submission` payload in `server.py`. A course-owned
+drag-and-drop element such as `pl-faded-parsons` can implement one reusable element
+test hook that serializes its prepared correct tray and blank state.
+
 ### Question-local configuration
 
 Store saved concrete seeds, narrow phase skips, and expected upstream errors in
@@ -383,7 +459,8 @@ environment. From a clean branch, run:
 make publish-version VERSION=0.2.0
 ```
 
-This runs the full checks, updates `pyproject.toml` and `uv.lock`, builds the
+This installs Python 3.13, performs the same locked dependency sync used in CI, runs
+the full checks, updates `pyproject.toml` and `uv.lock`, builds and smoke-tests the
 distributions, creates a release commit and annotated `v0.2.0` tag, and atomically
 pushes the branch and tag. The tag starts the workflow on `ubuntu-latest`; the workflow
 rechecks the tag/version match, tests, builds, generates attestations, and publishes

{pl_e2e_test-0.1.5 → pl_e2e_test-0.1.7}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "pl-e2e-test"
-version = "0.1.5"
+version = "0.1.7"
 description = "Fuzz PrairieLearn question lifecycles against the real Python element runtime"
 readme = "README.md"
 requires-python = ">=3.13"

{pl_e2e_test-0.1.5 → pl_e2e_test-0.1.7/src/pl_e2e_test.egg-info}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: pl-e2e-test
-Version: 0.1.5
+Version: 0.1.7
 Summary: Fuzz PrairieLearn question lifecycles against the real Python element runtime
 License-Expression: MIT
 Project-URL: Homepage, https://github.com/PrairieLearn/pl-e2e-test
@@ -266,14 +266,90 @@ nonGradableSelectors = ['pl-sketch[read-only="true"]', 'pl-sketch[read-only="1"]
 ```
 
 These are the defaults. `elementSolutionSelectors` identifies elements that provide
-solutions without a `correct_answers` entry. `nonGradableSelectors` excludes
-display-only answers from grading checks. Each supplied list replaces its default;
-omitted settings keep their defaults, and `[]` disables a list.
+solutions without a `correct_answers` entry. It only relaxes that entry requirement;
+it does not generate a test submission. `nonGradableSelectors` excludes display-only
+answers from grading checks. Each supplied list replaces its default; omitted settings
+keep their defaults, and `[]` disables a list.
 
 Use `--config path/to/settings.toml` to select another TOML file with the same
 `[tool.pl-e2e-test]` table. This works with normal runs, `diff`, and arguments after
 `parallel run` or `parallel diff`. Explicit paths are relative to the working directory.
 
+### Lifecycle JSON boundary
+
+PrairieLearn crosses a JSON boundary after processing the elements in a lifecycle phase
+and again after calling the question's `server.py` hook. The harness mirrors those
+boundaries. It first checks the in-process result for legal field mutations, then
+serializes and deserializes `data`; the normalized result is what the next hook or phase
+receives.
+
+This accepts the same Python values as PrairieLearn's JSON encoder rather than requiring
+literal JSON-domain Python types. For example, tuples normalize to arrays (Python lists),
+and supported non-string object keys such as numeric drawing IDs normalize to strings.
+This conversion can be lossy: if two keys normalize to the same string, the last key in
+serialized order wins, matching PrairieLearn's boundary. Unsupported objects, non-finite
+floats, and integers outside JavaScript's safe range are rejected with the failing data
+path.
+
+### Correct submissions for custom elements
+
+The grade check uses PrairieLearn's `test` lifecycle to obtain a deterministic correct
+submission. PrairieLearn first calls `test(element_html, data)` on the elements in the
+question, then the harness calls `test(data)` in the question's `server.py`. A reusable
+course element should normally implement the element hook. The question hook can fill
+in a submission when the correct state is question-specific or when a custom element
+does not provide one itself.
+
+For every gradable `answers-name`, the correct test case must provide:
+
+- a non-null value in `data["raw_submitted_answers"]` under the answer name, or one or
+  more compound input keys prefixed with `answers-name-`; and
+- `data["partial_scores"][answers_name]["score"] = 1`.
+
+Raw values must survive PrairieLearn's JSON boundary and have the same browser transport
+shape that the element's `parse` hook expects. Structured widgets commonly serialize
+their state as a JSON string. The hook must derive its output only from deterministic
+variant state such as `params` and `correct_answers`, mutate `data` in place, and return
+`None`. The normal phase mutation checks and JSON-boundary normalization still apply.
+
+For example, a question can supply the correct state for a drag-and-drop element with:
+
+```python
+import json
+
+
+def test(data):
+    if data["test_type"] != "correct":
+        return
+
+    answers_name = "ordering"
+    state = {
+        "solution": data["params"]["correct_order"],
+        "starter": [],
+    }
+    data["raw_submitted_answers"][answers_name] = json.dumps(state)
+    data["partial_scores"][answers_name] = {"score": 1}
+```
+
+Compound elements keep the declared answer name as their grading identity while using
+namespaced child fields for browser inputs. For example, an element with
+`answers-name="match"` may test and parse `match-dropdown-0` and `match-dropdown-1`,
+then store its score in `partial_scores["match"]`; it does not need to invent a
+`submitted_answers["match"]` value. After parsing, every raw field associated with a
+declared gradable answer must either remain non-null under its child key or produce a
+non-null value under its declared parent answer. This also supports elements that
+normalize `answer-input` into `answer`. Unassociated transport fields from file-producing
+elements may instead be consumed into `submitted_answers["_files"]`; they are not answer
+parse-coverage obligations.
+
+This contract keeps element-specific synthesis with the element or question that owns
+the data. PrairieLearn's tuple-based `pl-matching` state normalizes to arrays at the JSON
+boundary, and its existing namespaced test fields require no harness adapter. A question
+using `pl-sketch` without an element-owned solution can supply its deterministic
+`answers-name-sketchresponse-submission` payload in `server.py`. A course-owned
+drag-and-drop element such as `pl-faded-parsons` can implement one reusable element
+test hook that serializes its prepared correct tray and blank state.
+
 ### Question-local configuration
 
 Store saved concrete seeds, narrow phase skips, and expected upstream errors in
@@ -407,7 +483,8 @@ environment. From a clean branch, run:
 make publish-version VERSION=0.2.0
 ```
 
-This runs the full checks, updates `pyproject.toml` and `uv.lock`, builds the
+This installs Python 3.13, performs the same locked dependency sync used in CI, runs
+the full checks, updates `pyproject.toml` and `uv.lock`, builds and smoke-tests the
 distributions, creates a release commit and annotated `v0.2.0` tag, and atomically
 pushes the branch and tag. The tag starts the workflow on `ubuntu-latest`; the workflow
 rechecks the tag/version match, tests, builds, generates attestations, and publishes

{pl_e2e_test-0.1.5 → pl_e2e_test-0.1.7}/src/prairielearn_e2e/contract.py

@@ -164,35 +164,117 @@ def validate_mutations(
             raise HarnessError(f"{phase}: data[{name!r}] was illegally modified")
 
 
-def validate_serializable(data: dict[str, Any], phase: Phase) -> None:
-    def check_json(value: Any, path: str) -> None:
-        if value is None or isinstance(value, (str, bool, int)):
-            return
-        if isinstance(value, float):
-            if not math.isfinite(value):
-                raise TypeError(f"{path} contains NaN or infinity")
-            return
-        if isinstance(value, list):
+_JSON_ERRORS = (TypeError, ValueError, OverflowError, RecursionError)
+
+
+def _validate_integer_limits(
+    value: Any, path: str, active_containers: set[int]
+) -> None:
+    """Apply PrairieLearn's integer limit to every JSON-supported container."""
+
+    if isinstance(value, int):
+        try:
+            assert_all_integers_within_limits(value)
+        except ValueError as exc:
+            raise ValueError(f"{path} contains oversized integer: {value}") from exc
+        return
+
+    if not isinstance(value, (dict, list, tuple)):
+        return
+
+    container_id = id(value)
+    if container_id in active_containers:
+        return
+    active_containers.add(container_id)
+    try:
+        if isinstance(value, dict):
+            for key, item in value.items():
+                if isinstance(key, int):
+                    _validate_integer_limits(
+                        key, f"{path} object key {key!r}", active_containers
+                    )
+                _validate_integer_limits(item, f"{path}[{key!r}]", active_containers)
+        else:
             for index, item in enumerate(value):
-                check_json(item, f"{path}[{index}]")
-            return
+                _validate_integer_limits(item, f"{path}[{index}]", active_containers)
+    finally:
+        active_containers.remove(container_id)
+
+
+def _json_fails(value: Any) -> bool:
+    try:
+        json.dumps(value, allow_nan=False)
+    except _JSON_ERRORS:
+        return True
+    return False
+
+
+def _json_failure_path(value: Any, path: str, active_containers: set[int]) -> str:
+    """Locate an encoder failure after json.dumps has already rejected the value."""
+
+    if not isinstance(value, (dict, list, tuple)):
+        return path
+
+    container_id = id(value)
+    if container_id in active_containers:
+        return path
+    active_containers.add(container_id)
+    try:
         if isinstance(value, dict):
             for key, item in value.items():
-                if not isinstance(key, str):
-                    raise TypeError(f"{path} has a non-string object key")
-                check_json(item, f"{path}[{key!r}]")
-            return
-        raise TypeError(f"{path} contains {type(value).__name__}, which is not JSON")
+                if _json_fails({key: None}):
+                    return f"{path} object key {key!r}"
+                if _json_fails(item):
+                    return _json_failure_path(
+                        item, f"{path}[{key!r}]", active_containers
+                    )
+        else:
+            for index, item in enumerate(value):
+                if _json_fails(item):
+                    return _json_failure_path(
+                        item, f"{path}[{index}]", active_containers
+                    )
+    finally:
+        active_containers.remove(container_id)
+    return path
+
+
+def validate_serializable(data: dict[str, Any], phase: Phase) -> None:
+    """Validate and normalize data at PrairieLearn's Python-to-JSON boundary."""
 
     try:
-        check_json(data, "data")
-        assert_all_integers_within_limits(data)
-        json.dumps(data, allow_nan=False)
-    except (TypeError, ValueError) as exc:
+        _validate_integer_limits(data, "data", set())
+    except ValueError as exc:
         raise HarnessError(
             f"{phase}: data is not PrairieLearn JSON serializable: {exc}"
         ) from exc
 
+    try:
+        encoded = json.dumps(data, allow_nan=False)
+    except _JSON_ERRORS as exc:
+        try:
+            path = _json_failure_path(data, "data", set())
+        except RecursionError:
+            path = "data"
+        raise HarnessError(
+            f"{phase}: data is not PrairieLearn JSON serializable: {path}: {exc}"
+        ) from exc
+
+    try:
+        normalized = json.loads(encoded)
+    except _JSON_ERRORS as exc:
+        raise HarnessError(
+            f"{phase}: data is not PrairieLearn JSON serializable: data: {exc}"
+        ) from exc
+    if not isinstance(normalized, dict):
+        raise HarnessError(
+            f"{phase}: data is not PrairieLearn JSON serializable: "
+            "data did not normalize to an object"
+        )
+
+    data.clear()
+    data.update(normalized)
+
 
 def snapshot(data: dict[str, Any], phase: Phase) -> dict[str, Any]:
     validate_shape(data, phase)

{pl_e2e_test-0.1.5 → pl_e2e_test-0.1.7}/src/prairielearn_e2e/harness.py

@@ -68,6 +68,52 @@ def _aggregate_score(data: dict[str, Any], partial_credit: bool) -> None:
         )
 
 
+def _associated_answer_name(raw_name: str, answer_names: tuple[str, ...]) -> str | None:
+    if raw_name in answer_names:
+        return raw_name
+    return max(
+        (
+            answer_name
+            for answer_name in answer_names
+            if raw_name.startswith(f"{answer_name}-")
+        ),
+        key=len,
+        default=None,
+    )
+
+
+def _assert_test_submission_coverage(
+    data: dict[str, Any], answer_names: tuple[str, ...]
+) -> None:
+    raw_submitted_answers = data["raw_submitted_answers"]
+    missing: list[str] = []
+    for name in answer_names:
+        has_raw_submission = any(
+            _associated_answer_name(raw_name, answer_names) == name
+            and raw_value is not None
+            for raw_name, raw_value in raw_submitted_answers.items()
+        )
+        partial_score = data["partial_scores"].get(name)
+        score = partial_score.get("score") if isinstance(partial_score, dict) else None
+        has_full_partial_score = (
+            isinstance(score, (int, float))
+            and not isinstance(score, bool)
+            and float(score) == 1
+        )
+        if not has_raw_submission or not has_full_partial_score:
+            missing.append(name)
+
+    if missing:
+        raise HarnessError(
+            "correct test phase did not provide complete submissions for: "
+            + ", ".join(missing)
+            + ". Implement test(element_html, data) in the owning element or "
+            "test(data) in question server.py; populate raw_submitted_answers "
+            "with the answer name or namespaced child keys and set "
+            "partial_scores[answer_name].score to 1"
+        )
+
+
 def _assert_full_credit(data: dict[str, Any], answer_names: tuple[str, ...]) -> None:
     if data["format_errors"]:
         raise HarnessError(
@@ -79,15 +125,20 @@ def _assert_full_credit(data: dict[str, Any], answer_names: tuple[str, ...]) ->
         raise HarnessError(
             f"correct submission received score {data['score']!r}, expected 1"
         )
-    unparsed = [
-        name
-        for name in answer_names
-        if name not in data["submitted_answers"]
-        or data["submitted_answers"][name] is None
-    ]
+    submitted_answers = data["submitted_answers"]
+    unparsed: list[str] = []
+    for raw_name in data["raw_submitted_answers"]:
+        parent_name = _associated_answer_name(raw_name, answer_names)
+        if parent_name is None:
+            continue
+        if (
+            submitted_answers.get(raw_name) is None
+            and submitted_answers.get(parent_name) is None
+        ):
+            unparsed.append(raw_name)
     if unparsed:
         raise HarnessError(
-            "declared answers did not parse successfully: " + ", ".join(unparsed)
+            "raw submission fields did not parse successfully: " + ", ".join(unparsed)
         )
     missing = [name for name in answer_names if name not in data["partial_scores"]]
     if missing:
@@ -359,6 +410,7 @@ class QuestionHarness:
             test = self._phase_data("test", base, test_type="correct")
             self._process("test", source, test, server)
             _aggregate_score(test, metadata.get("partialCredit", True))
+            _assert_test_submission_coverage(test, tested_answer_names)
 
             parse = self._phase_data(
                 "parse",