pl-e2e-test 0.1.8__tar.gz → 0.2.1__tar.gz

{pl_e2e_test-0.1.8 → pl_e2e_test-0.2.1}/MANIFEST.in

@@ -1,2 +1,3 @@
 graft tests
+include scripts/demo_cli.py
 global-exclude *.py[cod]

{pl_e2e_test-0.1.8/src/pl_e2e_test.egg-info → pl_e2e_test-0.2.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: pl-e2e-test
-Version: 0.1.8
+Version: 0.2.1
 Summary: Fuzz PrairieLearn question lifecycles against the real Python element runtime
 License-Expression: MIT
 Project-URL: Homepage, https://github.com/PrairieLearn/pl-e2e-test
@@ -10,6 +10,7 @@ Requires-Python: >=3.13
 Description-Content-Type: text/markdown
 License-File: LICENSE
 Requires-Dist: chevron==0.14.0
+Requires-Dist: cyclopts<6,>=5.3
 Requires-Dist: filelock>=3.20
 Requires-Dist: jsonschema>=4.26
 Requires-Dist: lxml>=6.1
@@ -20,6 +21,7 @@ Requires-Dist: pyright>=1.1.414
 Requires-Dist: pytest>=9.1
 Requires-Dist: rich>=14
 Requires-Dist: ruff>=0.16
+Requires-Dist: tomlkit>=0.15.2
 Dynamic: license-file
 
 # pl-e2e-test
@@ -32,19 +34,24 @@ instead of copying the harness and its support scripts.
 The package installs the reusable `prairielearn_e2e` Python API and a single
 `pl-e2e-test` command:
 
-- `pl-e2e-test` (no command, or any pytest/harness options) runs the installed
-  lifecycle pytest suite.
+- `pl-e2e-test run` runs the installed lifecycle pytest suite.
 - `pl-e2e-test diff` fuzzes questions selected from a Git diff.
-- `pl-e2e-test parallel` runs local shards and merges their results.
+- `pl-e2e-test parallel run` and `parallel diff` run local shards and merge them.
 - `pl-e2e-test add-seed` records a failing concrete variant seed.
 - `pl-e2e-test reports` merges parallel shard results into terminal and JUnit reports.
-- `pl-e2e-test version` prints the installed version.
-- `pl-e2e-test help [COMMAND]` prints top-level help, or one command's help.
+- `pl-e2e-test instances` shows configured, installed, cached, and local PrairieLearn
+  revisions.
+- `pl-e2e-test version [REVISION]` reads or updates the configured PrairieLearn revision.
+- `pl-e2e-test cache clean` clears the managed PrairieLearn instance cache.
+- `pl-e2e-test --version` prints the installed CLI version.
+- `pl-e2e-test help [COMMAND...]` prints top-level or nested command help. A bare
+  invocation also prints top-level help.
 
 ## Requirements and installation
 
 - Python 3.13 or newer
 - Git 2.30 or newer
+- uv 0.9 or newer
 
 The harness depends on pytest, Pyright, Ruff, Chevron, JSON Schema, lxml, NumPy,
 and PrairieLearn. PrairieLearn's Python package is currently consumed from its Git
@@ -80,20 +87,37 @@ Pass a question, a directory of questions, or an `info.json` file. Repeat
 `--question-path` to select multiple roots:
 
 ```sh
-uv run pl-e2e-test \
+uv run pl-e2e-test run \
   --question-path questions \
   --seed-count 3
 ```
 
 The installed suite exposes separate `generate`, `prepare`, `render`, and `grade`
-pytest markers. Any remaining arguments are passed to pytest:
+pytest markers. Put pytest selection and execution arguments after `--`:
 
 ```sh
-uv run pl-e2e-test \
+uv run pl-e2e-test -vv run \
   --question-path questions/chapter/example \
-  --fuzz-seeds -m render -x -vv
+  --fuzz-seeds -- -m render -x
 ```
 
+The native reporter owns progress, capture, tracebacks, summaries, and color. Use
+repeatable global `-q` or `-v` flags and `--color auto|always|never`; pytest
+presentation flags such as `-q`, `-v`, `--tb`, `-r`, `--color`, and `-s` are rejected
+after `--`. `NO_COLOR`, `FORCE_COLOR`, and `CLICOLOR_FORCE` are also honored.
+
+To preview the terminal UX without a PrairieLearn course or network access, run:
+
+```sh
+uv run python scripts/demo_cli.py
+uv run python scripts/demo_cli.py --color always mixed verbose cli-error
+uv run python scripts/demo_cli.py --list
+```
+
+The demo covers help and CLI errors plus passing, failing, skipped, xfailed/xpassed,
+warning, collection-error, empty-selection, quiet, verbose, and per-shard progress
+output.
+
 `--fuzz-seeds` creates and reports a random 64-bit master seed. Reproduce the same
 variant set with `--fuzz-seed N`. Without either option, deterministic seeds start at
 zero. Question metadata with `singleVariant: true` always uses seed zero.
@@ -103,22 +127,23 @@ zero. Question metadata with `singleVariant: true` always uses seed zero.
 Run multiple shards in one machine or CI job, with six concurrent processes by default:
 
 ```sh
-uv run pl-e2e-test parallel --shard-count 6 run -- \
+uv run pl-e2e-test parallel run --shard-count 6 \
   --question-path questions --fuzz-seeds
 
-uv run pl-e2e-test parallel --shard-count 6 --junitxml results.xml diff -- \
+uv run pl-e2e-test parallel diff --shard-count 6 --junitxml results.xml \
   --base main --fuzz-seed 42
 ```
 
-Launcher options go before the required `run` or `diff` selector; arguments after
-`--` are forwarded to that command. Normal runs remain deterministic. For
+Native options go after `parallel run` or `parallel diff` and before an optional `--`;
+only pytest arguments follow that boundary. Normal runs remain deterministic. For
 `--fuzz-seeds` and diff runs, the launcher generates one shared master seed unless
 `--fuzz-seed` is supplied, and prints a command for replaying the run without sharding.
 
 Each shard runs in a separate Python process using the current environment and
-working directory. Terminals show Rich progress; redirected output prints completion
-lines. All shards finish before reports are merged, even when a test fails. Captured
-failure output and successful stderr are shown before the combined summary.
+working directory. Terminals show aggregate progress; redirected output uses stable
+start and summary lines. All shards finish before reports are merged, even when a test
+fails. Raw child output is shown only when a launch or internal failure cannot produce
+a usable report.
 Interrupting the launcher stops and reaps its children.
 
 Intermediate reports are temporary. Pass the launcher's `--junitxml` option to retain
@@ -145,19 +170,19 @@ Relative JUnit paths are resolved against the course root. With `junitxml=None`,
 merged output is temporary. Invalid launcher settings raise `ValueError`.
 
 Courses using `scripts/run_e2e_shards.py` can replace its invocation with
-`pl-e2e-test parallel`, move `--fuzz-seed` into the forwarded arguments, and explicitly
+`pl-e2e-test parallel run`, move `--fuzz-seed` into the native arguments, and explicitly
 pass `--fuzz-seeds` for full-suite runs to preserve the script's fuzzing behavior.
 For example:
 
 ```make
 test-e2e-sharded:
-	uv run --active pl-e2e-test parallel --shard-count "$(E2E_SHARD_COUNT)" run -- \
-		$(E2E_TEST_ARGS) $(E2E_PRAIRIELEARN_ARG) --fuzz-seeds $(E2E_FUZZ_SEED_ARG) $(PYTEST_ARGS)
+	uv run --active pl-e2e-test parallel run --shard-count "$(E2E_SHARD_COUNT)" \
+		$(E2E_TEST_ARGS) $(E2E_PRAIRIELEARN_ARG) --fuzz-seeds $(E2E_FUZZ_SEED_ARG) -- $(PYTEST_ARGS)
 
 test-e2e-fuzz-sharded:
-	uv run --active pl-e2e-test parallel --shard-count "$(E2E_SHARD_COUNT)" diff -- \
+	uv run --active pl-e2e-test parallel diff --shard-count "$(E2E_SHARD_COUNT)" \
 		--base "$(E2E_DIFF_BASE)" --head "$(E2E_DIFF_HEAD)" \
-		--seed-count "$(E2E_FUZZ_SEED_COUNT)" $(E2E_FUZZ_SEED_ARG) $(E2E_PRAIRIELEARN_ARG) $(PYTEST_ARGS)
+		--seed-count "$(E2E_FUZZ_SEED_COUNT)" $(E2E_FUZZ_SEED_ARG) $(E2E_PRAIRIELEARN_ARG) -- $(PYTEST_ARGS)
 ```
 
 ### CI sharding
@@ -199,7 +224,7 @@ jobs:
       - run: uv python install && uv sync --locked
       - name: Run shard
         run: |
-          uv run pl-e2e-test \
+          uv run pl-e2e-test run \
             --question-path questions \
             --seed-count 10 \
             --fuzz-seed "${{ github.run_id }}" \
@@ -241,19 +266,62 @@ jobs:
 ```
 
 The course must provide `.prairielearn/schemas/infoQuestion.json`. The harness checks
-question metadata against that schema and uses the exact PrairieLearn revision
-installed in the uv environment. It stores a bare, partial PrairieLearn repository in
-the platform-standard user cache directory. The first use of a new Git source or
-revision requires network access; later runs reuse that exact revision offline. The
-uv lockfile, rather than the cache, decides which revision is tested.
+question metadata against that schema and resolves the configured PrairieLearn version
+before each run. The default, `latest`, resolves the Git source's remote `HEAD`; a full
+commit, branch, or unambiguous tag can be selected instead. Moving selectors require
+network access and never silently fall back to stale cache state.
+
+The selected revision's Python package and core elements are always used together. The
+harness keeps a blob-filtered bare repository in the platform-standard user cache and
+hydrates only `apps/prairielearn/python` and `apps/prairielearn/elements`. It launches
+the selected Python package through a frozen uv overlay, without changing the course's
+`pyproject.toml`, `uv.lock`, or virtual environment. A previously hydrated exact commit
+can be reused offline when its uv dependencies are also cached.
 
 To use an existing PrairieLearn worktree or bare repository instead, pass
 `--prairielearn-path /path/to/PrairieLearn`. The repository must already contain the
-installed revision. An explicit path is authoritative: the harness does not fetch into
-it or fall back to the managed cache. In either mode, only the pinned revision's Python
-element controllers are archived into the test run's temporary directory. Standard
-cache environment settings such as `XDG_CACHE_HOME` can relocate the managed cache;
-CI systems can persist that directory between runs if desired.
+selected revision. An explicit path is authoritative: the harness resolves selectors
+against its local refs, does not fetch into it, and does not fall back to the managed
+cache. For a local repository, `latest` means its current `HEAD`.
+
+Inspect available revisions without contacting remotes:
+
+```sh
+uv run pl-e2e-test instances
+```
+
+Add `--refresh` to resolve the configured moving selector and verify remote access.
+The table distinguishes fully hydrated runtime revisions from legacy element-only cache
+entries. Standard cache environment settings such as `XDG_CACHE_HOME` can relocate the
+managed cache; CI systems can persist that directory between runs if desired.
+Cache use refreshes each repository or hydrated runtime's last-use timestamp. Whenever
+the managed cache is used, unlocked items that have been idle for more than 30 days are
+removed automatically across all configured PrairieLearn sources.
+
+Read the effective configured selector without resolving it or contacting the network:
+
+```sh
+uv run pl-e2e-test version
+```
+
+Set `tool.pl-e2e-test.prairielearnVersion` in an existing `pyproject.toml` to a full
+commit or verified PrairieLearn release tag with:
+
+```sh
+uv run pl-e2e-test version 3.2.0
+```
+
+Release tags are verified against the configured PrairieLearn Git source before the
+file is changed. Use `--config PATH` to update another existing TOML file. Comments and
+unrelated formatting are preserved, and a missing file is not created.
+
+Remove all managed bare repositories and hydrated runtime revisions with:
+
+```sh
+uv run pl-e2e-test cache clean
+```
+
+The next test run fetches and hydrates its selected PrairieLearn revision again.
 
 ### Harness configuration
 
@@ -261,19 +329,25 @@ Selector settings are read automatically from the course's `pyproject.toml`:
 
 ```toml
 [tool.pl-e2e-test]
+prairielearnVersion = "latest"
 elementSolutionSelectors = ["pl-sketch"]
 nonGradableSelectors = ['pl-sketch[read-only="true"]', 'pl-sketch[read-only="1"]']
 ```
 
-These are the defaults. `elementSolutionSelectors` identifies elements that provide
+These are the defaults. Omitting `prairielearnVersion` is equivalent to `"latest"`.
+Use a full commit for reproducible or offline runs, or a branch/tag when a named moving
+target is intentional. `--prairielearn-version SELECTOR` overrides the TOML value for
+one invocation, and replay commands always print the resolved full commit.
+
+`elementSolutionSelectors` identifies elements that provide
 solutions without a `correct_answers` entry. It only relaxes that entry requirement;
 it does not generate a test submission. `nonGradableSelectors` excludes display-only
 answers from grading checks. Each supplied list replaces its default; omitted settings
 keep their defaults, and `[]` disables a list.
 
 Use `--config path/to/settings.toml` to select another TOML file with the same
-`[tool.pl-e2e-test]` table. This works with normal runs, `diff`, and arguments after
-`parallel run` or `parallel diff`. Explicit paths are relative to the working directory.
+`[tool.pl-e2e-test]` table. This works with `run`, `diff`, `parallel run`, and
+`parallel diff`. Explicit paths are relative to the working directory.
 
 ### Lifecycle JSON boundary
 
@@ -427,7 +501,7 @@ diffignore = [
 
 Rules use Git's native ignore syntax and are evaluated in order, so later rules may
 negate earlier ones with `!`. A missing `pyproject.toml`, tool table, or `diffignore`
-entry means that no paths are ignored. Additional arguments are forwarded to pytest.
+entry means that no paths are ignored. Additional pytest arguments must follow `--`.
 
 To migrate from `.question-e2e.diffignore`, copy its non-comment rules into the
 `diffignore` array. The former file and `--e2ediffignore` option are no longer used.
@@ -455,12 +529,16 @@ course's TOML settings, or `config=load_config("settings.toml")` for an explicit
 `load_config` and `HarnessConfig` are exported from `prairielearn_e2e`. Omitting
 `config` uses the built-in defaults.
 
+The command-line runner and high-level `run_parallel` API perform version resolution
+and uv bootstrapping. Manually constructed `PrairieLearnBackend` and `QuestionHarness`
+objects continue to use the PrairieLearn package already loaded by their interpreter.
+
 ## Development
 
 This project uses uv 0.9 or newer:
 
 ```sh
-uv sync
+make install
 make test
 make format
 make smoke-dist
@@ -472,44 +550,36 @@ into an isolated target, and imports it using the locked development dependencie
 
 ## Publishing
 
-There are three supported publishing paths. Choose one for a release; do not run the
-local and GitHub paths for the same version because PyPI versions are immutable.
-
-### GitHub-hosted trusted publishing
-
 Configure the PyPI project to trust the `publish.yml` workflow in the GitHub `pypi`
-environment. From a clean branch, run:
+environment. Releases must start from a clean `main` that is synchronized with
+`origin/main`, with `gh` authenticated and the publishing workflow available. To bump
+the current version, run one of:
 
 ```sh
-make publish-version VERSION=0.2.0
+make publish patch
+make publish minor
+make publish major
 ```
 
-This installs Python 3.13, performs the same locked dependency sync used in CI, runs
-the full checks, updates `pyproject.toml` and `uv.lock`, builds and smoke-tests the
-distributions, creates a release commit and annotated `v0.2.0` tag, and atomically
-pushes the branch and tag. The tag starts the workflow on `ubuntu-latest`; the workflow
-rechecks the tag/version match, tests, builds, generates attestations, and publishes
-with PyPI trusted-publishing OIDC credentials. Use `REMOTE=name` for a different Git
-remote.
-
-If an unpushed commit already contains the requested version, use the guarded bypass:
+If the synchronized commit already contains the intended, not-yet-tagged version, use:
 
 ```sh
-make publish-version VERSION=0.2.0 ALLOW_PREBUMPED_VERSION=1
+make publish current
 ```
 
-### Direct local publishing
-
-To publish the version already present in `pyproject.toml` directly from this machine,
-configure a uv-supported PyPI credential such as `UV_PUBLISH_TOKEN`, then run:
+The release command runs the shared `make release-check` gate, updates
+`pyproject.toml` and `uv.lock` when needed, creates a release commit and annotated tag,
+atomically pushes `main` and the tag, and watches the trusted-publishing workflow to
+completion. Preview the full process without changing files, refs, remotes, or workflows
+with:
 
 ```sh
-make publish-local
+make publish patch DRY_RUN=1
 ```
 
-This runs all checks, builds and smoke-tests the current wheel and source archive, and
-passes only those two version-matched files to `uv publish`. It does not create a Git
-commit or tag.
+If the release tag exists but trusted publishing needs to be dispatched again, run
+`make publish-pypi`. This validates that the tag matches the current version and is
+reachable from `origin/main`, then dispatches and watches a new workflow run.
 
 ### Local self-hosted GitHub runner
 
@@ -518,11 +588,11 @@ inside Docker. The container has no host mounts and does not receive the Docker
 Authenticate `gh` with repository administration permission, then run:
 
 ```sh
-make runner-start
-make publish-self-hosted RELEASE_TAG=v0.2.0
+make publish-self-hosted
 ```
 
-The tag must already exist. This dispatches `publish.yml` from `main` with the
-`self-hosted` runner choice. `make runner-ensure` starts Docker Desktop on macOS when
-needed and repairs the runner before dispatch. Other lifecycle commands are
-`runner-status`, `runner-stop`, `runner-remove`, `runner-logs`, and `clean-runner`.
+The current version's tag must already exist. This ensures the local runner is online,
+dispatches `publish.yml` from `main` with the `self-hosted` runner choice, and watches
+the run. `make runner-ensure` starts Docker Desktop on macOS when needed and repairs the
+runner before dispatch. Other lifecycle commands are `runner-status`, `runner-stop`,
+`runner-remove`, `runner-logs`, and `clean-runner`.