reeval-python 0.14.0__tar.gz → 0.16.1__tar.gz

{reeval_python-0.14.0 → reeval_python-0.16.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: reeval-python
-Version: 0.14.0
+Version: 0.16.1
 Summary: Python authoring SDK for reeval (*.eval.py eval files)
 Project-URL: Homepage, https://reeval.ai
 Project-URL: Repository, https://github.com/invertase/reeval
@@ -73,7 +73,7 @@ from reeval_python import SyncAttemptContext, evaluate, expect
 def lands(c: SyncAttemptContext) -> None:
     c.send("Create greet.py that prints hi")
     assert c.exec("test -f greet.py").ok
-    expect(c.judge("Does greet.py print hi?")).to_pass()
+    expect(c.judge("Does greet.py print hi?", contents=["greet.py"])).to_pass()
 ```
 
 Plain `assert` is first-class, as in pytest: each one in an eval file is a check in the run
@@ -83,9 +83,47 @@ as in Python, and is recorded with its message and both sides of a comparison as
 (`greet.py exists: assert 1 == 2`). An `AssertionError` the body catches isn't a failed check, so
 a retry loop can assert; one the escaping error carries (each failure in a `TaskGroup`'s
 `ExceptionGroup`, or one an error was raised from or while handling) is. Use typed results (`ExecResult.ok`, `Verdict.passed`, `Turn.text`, …) and `c.file(path)`
-for quick checks. Keep `expect` for reeval matchers (`to_have_file`, `to_succeed`, soft checks,
+for quick checks. Keep `expect` for reeval matchers (`to_have_file`, `to_succeed`, checks that carry on,
 labels) — Playwright-style.
 
+#### Checks carry on; `expect.require` is a gate
+
+A failed `expect` is recorded and the body carries on, so every later check and judge still runs;
+the attempt fails when the body ends. `expect.require` raises `ReevalAssertionError` instead, for a
+gate the rest depends on:
+
+```python
+c.send("Add a sign-up page")
+expect.require(c.exec("npm run build"), "builds").to_succeed()  # stops here if the build fails
+expect(c.exec("npm test"), "tests pass").to_succeed()  # recorded, carries on
+```
+
+reeval records a failed `expect.require` before the error reaches the body. One the body catches
+still fails the attempt when the body ends, and its error says the eval caught it. A plain
+`assert` the body catches records nothing, as in Python.
+
+| The body catches | Recorded | Fails the attempt |
+|---|---|---|
+| a failed `expect.require(…)` | a failed check | yes, when the body ends |
+| a failed `assert` | nothing | no |
+| neither (it escapes) | a failed check | yes |
+
+So retry with `expect.poll`, which retries the matcher and records only its final outcome, or with
+a plain `assert` or `if` in the loop, never `expect.require` in `try`/`except`:
+
+```python
+# Retries the command until it succeeds, for up to 10 seconds: one check.
+expect.poll(lambda: c.exec("curl -sf localhost:3000"), timeout=10).to_succeed()
+
+# A plain assert in a retry loop: only the try that held is recorded.
+for _ in range(3):
+    try:
+        assert c.exec("test -f ready.txt").ok
+        break
+    except AssertionError:
+        time.sleep(1)
+```
+
 Matchers that take a pattern (`to_have_file`, `to_have_run`, `to_have_searched`, `to_have_fetched`,
 `to_have_called_tool`, `to_have_edited`, `to_have_changed`) take a `str` or a compiled `re.Pattern`,
 `str` or `bytes`, with any flags. Python matches it with `pattern.search`, so anything `re` accepts
@@ -101,12 +139,19 @@ Use snake_case on the context (`read_file`, `write_file`, `file`) and on matcher
 `to_pass`, `to_equal`, `to_contain`, …). Matcher failures raise `ReevalAssertionError`.
 
 Options are typed (`EvalOptions`, and `JudgeOptions` for `c.judge(…)`), so pyright refuses a
-misspelt key such as `runz`. They take reeval's keys in snake_case at any depth (`min_lift`,
+misspelled key such as `runz`. They take reeval's keys in snake_case at any depth (`min_lift`,
 `delay_ms`); the keys of `env`, `headers` and the other maps of your own keep their spelling. Every
 duration is seconds or a `timedelta`: `{"timeout": 90}`, `docker(start_timeout=timedelta(minutes=2))`,
 `expect.poll(…, timeout=10)`. `docker()`, `fake_harness()`, `fake_judge()` and `plugin.*` return
 values to pass as options, not data to edit.
 
+`c.judge` takes the inputs the judge reads as keywords, and needs at least one: `diff=True`,
+`contents=["src/**"]`, `output=True` with `on=result`, and so on. With none it raises `TypeError`.
+
+```python
+expect(c.judge("Is it valid JSON?", on=result, output=True)).to_pass()
+```
+
 ### `def` and `async def` bodies
 
 A `def` body gets a blocking context: `c.exec(…)` runs the command and returns its `ExecResult`,

{reeval_python-0.14.0 → reeval_python-0.16.1}/README.md

@@ -49,7 +49,7 @@ from reeval_python import SyncAttemptContext, evaluate, expect
 def lands(c: SyncAttemptContext) -> None:
     c.send("Create greet.py that prints hi")
     assert c.exec("test -f greet.py").ok
-    expect(c.judge("Does greet.py print hi?")).to_pass()
+    expect(c.judge("Does greet.py print hi?", contents=["greet.py"])).to_pass()
 ```
 
 Plain `assert` is first-class, as in pytest: each one in an eval file is a check in the run
@@ -59,9 +59,47 @@ as in Python, and is recorded with its message and both sides of a comparison as
 (`greet.py exists: assert 1 == 2`). An `AssertionError` the body catches isn't a failed check, so
 a retry loop can assert; one the escaping error carries (each failure in a `TaskGroup`'s
 `ExceptionGroup`, or one an error was raised from or while handling) is. Use typed results (`ExecResult.ok`, `Verdict.passed`, `Turn.text`, …) and `c.file(path)`
-for quick checks. Keep `expect` for reeval matchers (`to_have_file`, `to_succeed`, soft checks,
+for quick checks. Keep `expect` for reeval matchers (`to_have_file`, `to_succeed`, checks that carry on,
 labels) — Playwright-style.
 
+#### Checks carry on; `expect.require` is a gate
+
+A failed `expect` is recorded and the body carries on, so every later check and judge still runs;
+the attempt fails when the body ends. `expect.require` raises `ReevalAssertionError` instead, for a
+gate the rest depends on:
+
+```python
+c.send("Add a sign-up page")
+expect.require(c.exec("npm run build"), "builds").to_succeed()  # stops here if the build fails
+expect(c.exec("npm test"), "tests pass").to_succeed()  # recorded, carries on
+```
+
+reeval records a failed `expect.require` before the error reaches the body. One the body catches
+still fails the attempt when the body ends, and its error says the eval caught it. A plain
+`assert` the body catches records nothing, as in Python.
+
+| The body catches | Recorded | Fails the attempt |
+|---|---|---|
+| a failed `expect.require(…)` | a failed check | yes, when the body ends |
+| a failed `assert` | nothing | no |
+| neither (it escapes) | a failed check | yes |
+
+So retry with `expect.poll`, which retries the matcher and records only its final outcome, or with
+a plain `assert` or `if` in the loop, never `expect.require` in `try`/`except`:
+
+```python
+# Retries the command until it succeeds, for up to 10 seconds: one check.
+expect.poll(lambda: c.exec("curl -sf localhost:3000"), timeout=10).to_succeed()
+
+# A plain assert in a retry loop: only the try that held is recorded.
+for _ in range(3):
+    try:
+        assert c.exec("test -f ready.txt").ok
+        break
+    except AssertionError:
+        time.sleep(1)
+```
+
 Matchers that take a pattern (`to_have_file`, `to_have_run`, `to_have_searched`, `to_have_fetched`,
 `to_have_called_tool`, `to_have_edited`, `to_have_changed`) take a `str` or a compiled `re.Pattern`,
 `str` or `bytes`, with any flags. Python matches it with `pattern.search`, so anything `re` accepts
@@ -77,12 +115,19 @@ Use snake_case on the context (`read_file`, `write_file`, `file`) and on matcher
 `to_pass`, `to_equal`, `to_contain`, …). Matcher failures raise `ReevalAssertionError`.
 
 Options are typed (`EvalOptions`, and `JudgeOptions` for `c.judge(…)`), so pyright refuses a
-misspelt key such as `runz`. They take reeval's keys in snake_case at any depth (`min_lift`,
+misspelled key such as `runz`. They take reeval's keys in snake_case at any depth (`min_lift`,
 `delay_ms`); the keys of `env`, `headers` and the other maps of your own keep their spelling. Every
 duration is seconds or a `timedelta`: `{"timeout": 90}`, `docker(start_timeout=timedelta(minutes=2))`,
 `expect.poll(…, timeout=10)`. `docker()`, `fake_harness()`, `fake_judge()` and `plugin.*` return
 values to pass as options, not data to edit.
 
+`c.judge` takes the inputs the judge reads as keywords, and needs at least one: `diff=True`,
+`contents=["src/**"]`, `output=True` with `on=result`, and so on. With none it raises `TypeError`.
+
+```python
+expect(c.judge("Is it valid JSON?", on=result, output=True)).to_pass()
+```
+
 ### `def` and `async def` bodies
 
 A `def` body gets a blocking context: `c.exec(…)` runs the command and returns its `ExecResult`,

{reeval_python-0.14.0 → reeval_python-0.16.1}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "reeval-python"
-version = "0.14.0"
+version = "0.16.1"
 description = "Python authoring SDK for reeval (*.eval.py eval files)"
 readme = "README.md"
 requires-python = ">=3.11"

{reeval_python-0.14.0 → reeval_python-0.16.1}/src/reeval_python/__init__.py

@@ -17,14 +17,15 @@ from reeval_python.environment import (
     plugin,
 )
 from reeval_python.errors import ReevalAssertionError, ReevalCallError
-from reeval_python.expect import Expectation, expect, soft
-from reeval_python.options import EvalOptions, JudgeDefaults, JudgeOptions
+from reeval_python.expect import Expectation, expect
+from reeval_python.options import BestOf, EvalOptions, JudgeDefaults, JudgeOptions
 from reeval_python.types import BackgroundHandle, Diff, ExecResult, FileChange, Turn, Verdict
 
 __all__ = [
     "DEFAULT_EGRESS_ALLOW",
     "AttemptContext",
     "BackgroundHandle",
+    "BestOf",
     "Diff",
     "DockerSandbox",
     "Environment",
@@ -52,6 +53,5 @@ __all__ = [
     "fake_judge",
     "group",
     "plugin",
-    "soft",
     "todo",
 ]