solver-support 0.4.1__tar.gz → 0.4.2__tar.gz

{solver_support-0.4.1/solver_support.egg-info → solver_support-0.4.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: solver-support
-Version: 0.4.1
+Version: 0.4.2
 Summary: Generic Conjure / Savile Row / Fast Downward / SymK / ENHSP / UPF solver pipeline
 Author: András Salamon
 License-Expression: BSD-3-Clause
@@ -37,6 +37,24 @@ subprocesses with timeouts and clean teardown, naming the per-run files, parsing
 solver output into a uniform timing / cost-layer model, and driving the horizon
 (increasing-difficulty) loop.
 
+Its main value is a **unified semantics for solver behaviour**. Each solver
+reports how a run ended in its own way, and the raw signals are misleading
+taken at face value:
+
+- Fast Downward's exit 12 can mean a proof, a time limit or a search that gave up.
+- ENHSP exits 0 after a proof and after failing to read its input alike.
+- Savile Row prints the same "no solution" whether the backend refuted the
+  instance or stopped at its own limit.
+- A SAT solver's 10 and 20 are answers, not failures.
+
+solver-support reads each solver's exit codes, logs and info files. It reports
+every run in one vocabulary: a `verdict` (`sat` / `unsat` / `unknown` / `error` /
+`killed`, with a reason) for each solve, and an `outcome` and `stop_reason` for
+the run. So `unsat` means a proof, whichever solver produced it. A run that
+merely found nothing, or crashed, never reads as one. The rules for each solver
+are checked against its source and its real output, and are documented in
+[`timing-model.md`](https://github.com/ott2/solver-support/blob/main/docs/timing-model.md).
+
 It carries **no** notion of any particular problem domain or objective. A
 consumer supplies:
 

{solver_support-0.4.1 → solver_support-0.4.2}/README.md

@@ -9,6 +9,24 @@ subprocesses with timeouts and clean teardown, naming the per-run files, parsing
 solver output into a uniform timing / cost-layer model, and driving the horizon
 (increasing-difficulty) loop.
 
+Its main value is a **unified semantics for solver behaviour**. Each solver
+reports how a run ended in its own way, and the raw signals are misleading
+taken at face value:
+
+- Fast Downward's exit 12 can mean a proof, a time limit or a search that gave up.
+- ENHSP exits 0 after a proof and after failing to read its input alike.
+- Savile Row prints the same "no solution" whether the backend refuted the
+  instance or stopped at its own limit.
+- A SAT solver's 10 and 20 are answers, not failures.
+
+solver-support reads each solver's exit codes, logs and info files. It reports
+every run in one vocabulary: a `verdict` (`sat` / `unsat` / `unknown` / `error` /
+`killed`, with a reason) for each solve, and an `outcome` and `stop_reason` for
+the run. So `unsat` means a proof, whichever solver produced it. A run that
+merely found nothing, or crashed, never reads as one. The rules for each solver
+are checked against its source and its real output, and are documented in
+[`timing-model.md`](https://github.com/ott2/solver-support/blob/main/docs/timing-model.md).
+
 It carries **no** notion of any particular problem domain or objective. A
 consumer supplies:
 

{solver_support-0.4.1 → solver_support-0.4.2}/docs/timing-model.md

@@ -296,9 +296,13 @@ where the code is ambiguous, into the same vocabulary as the Savile Row verdict.
 
 - `Time limit reached. Abort search.` (the search's `max_time=`) gives
   `search-limit`.
-- `Completely explored state space -- no solution!` gives `unsat`/`exhausted`,
-  or `bounded` if the `--search` string has a `bound=`. The log looks the same
-  either way, so the argv is read for it. The translator's
+- `Completely explored state space -- no solution!` gives `unsat`/`exhausted`.
+  It gives `bounded` instead if the search has a `bound=`, and
+  `unsafe-pruning` if an evaluator's dead ends are unreliable on this task
+  (below). The exhaustion line looks the same either way, so the search is read
+  from the driver's `search command line string`. That line has aliases
+  expanded and `--evaluator` predefinitions included. If the line is missing,
+  the search is read from the argv's `--search`. The translator's
   `Generating unsolvable task` lands here too, since the search then exhausts a
   trivially false goal.
 - SymK prints no exhaustion line. Its last `BOUND: L < U` line, with `L ≥ U`, is
@@ -311,14 +315,24 @@ where the code is ambiguous, into the same vocabulary as the Savile Row verdict.
   plan!`).
 
 **`exhausted` is this library's reading, not the planner's.** It is sound only
-if the search is complete and prunes safely. A relaxation heuristic prunes
-infinite-valued states even on a task with axioms, where infinity is not a
-dead end, so `astar(ff())` there "exhausts" without proving anything. The log
-cannot reveal this. From 26.6 the planner checks it (upstream issue1201): a
-safe exhaustion exits 11 and an unsafe one exits 12 with no exhaustion line, so
-`exhausted` only ever comes from older planners and SymK. Whether it becomes a
-run-level `unsat` is decided by the registry's `proves-unsat`, which a tune
-preset can switch off.
+if the search is complete and prunes safely. Before 26.6, a search prunes
+every state an evaluator calls a dead end, whether or not
+`dead_ends_are_reliable()`. So a relaxation heuristic on a task with axioms,
+where infinity is not a dead end, "exhausts" without proving anything. The log
+does show the pruning half, and `fd_verdict` checks it as 26.6 does:
+
+| evaluator in the logged search | dead ends unreliable when |
+|---|---|
+| `ff`, `add`, `hmax` | `Translator axioms:` is non-zero, or absent |
+| `cg`, `cea` | always |
+| `hm`, `lmcount` / `landmark_sum` | always: they are unreliable on tasks with conditional effects, which the log does not show |
+
+Any such evaluator gives `unknown`/`unsafe-pruning`. From 26.6 the planner
+checks it itself (upstream issue1201): a safe exhaustion exits 11 and an unsafe
+one exits 12 with no exhaustion line. So `exhausted` only ever comes from older
+planners and SymK. Completeness of the search algorithm itself (EHC, say) is
+left to the registry's `proves-unsat`, which a tune preset can switch off, and
+which decides whether `exhausted` becomes a run-level `unsat`.
 
 **The planner's own time limit is a timeout.** `search-limit` and
 `translate-limit` make `solve()` raise `TimeoutError`, exactly as a watchdog
@@ -355,6 +369,38 @@ verdict.
 - **Direct vs via-UPF**: the direct route passes `end_hint = duration -
   grounding_time`; the UPF route reuses the parser but passes no end hint.
 
+#### Verdicts: ENHSP exits 0 for almost everything
+
+ENHSP 0.21.0 exits 0 after a plan, after a proof, and after a run that failed to
+load its input. A missing domain file prints `Domain parsed`, then
+`Unsolvable Problem`, and exits 0. The exception goes to **stderr**. So
+`SolverStatsExtractor.enhsp_verdict` reads stdout and stderr, and
+`EnhspSolver` records the result on the `solving` phase, first match wins:
+
+| evidence | verdict | `verdict_reason` |
+|---|---|---|
+| `Problem Solved` | `sat` | |
+| `java.lang.OutOfMemoryError` (exit 1) | `unknown` | `memory` |
+| a logged exception (`SEVERE:`, `Exception in thread`) | `error` | `crash` |
+| `Some Syntax Error` (exit 255) | `error` | `input-error` |
+| any other non-zero exit | `error` | `crash` (signal) / `unexplained` |
+| `Problem unsolvable`, with `-anytime` / `-autoanytime` | `unknown` | `anytime` |
+| `Problem unsolvable`, with `-dl` | `unknown` | `bounded` |
+| `Problem unsolvable`, helpful actions (`sat-hmrph`, `sat-hmrphj`; or `-ha true` / `-ht true` with no `-planner`) | `unknown` | `incomplete` |
+| `Problem unsolvable`, otherwise | `unsat` | `exhausted` |
+| `Unsolvable Problem` (grounding returned nothing) | `unknown` | `grounding` |
+| none of these | `error` | `unexplained` |
+
+A `-planner` preset overrides `-ha` and `-ht`, as in ENHSP's own `setPlanner`.
+As with Fast Downward, `exhausted` becomes a run-level `unsat` only through
+`proves-unsat`.
+
+**Via UPF there is no verdict.** The UPF statuses are wrong in both directions:
+`up_enhsp` reports any exit 0 without a plan as `UNSOLVABLE_PROVEN`, and
+`up_fast_downward` reports any exit 12 as `UNSOLVABLE_INCOMPLETELY`. The UPF CLI
+is the consumer's, so if a consumer wants its UPF runs to conclude `unsat`, it
+must read the engine's log and record a verdict on the `solving` phase itself.
+
 ## invariants and conventions
 
 - **`duration` is ground truth; components are a subset.** Never assume the
@@ -409,8 +455,8 @@ proved. A status that stopped early (`timeout`, `hlim`, `stopped`) is never
 For a planner, being able to prove depends on the **search** as well as the model.
 `get_proves_unsat(model, tune=..., solver_name=...)` is the model's
 `proves-unsat` AND the effective tune's. A tune preset may carry
-`proves-unsat: false` (for example a relaxation heuristic that prunes unsafely
-on tasks with axioms), which takes the claim away. A tune can never grant it to
+`proves-unsat: false` (for example an incomplete search such as EHC, or an
+anytime portfolio), which takes the claim away. A tune can never grant it to
 a model that cannot prove. `finalize` keeps the value on the summary as
 `proves_unsat`, and a later `finalize` that passes None reuses it. So a
 re-finalize that folds in a late phase, as `horizon_utils` does, cannot drop it.
@@ -418,7 +464,7 @@ re-finalize that folds in a late phase, as `horizon_utils` does, cannot drop it.
 **A `solving` phase's verdict decides a completed run.** `failed` is the status
 of both a planner's proof and its crash, so `proves_unsat` alone would promote
 every crash of a complete model to `unsat`. Where the `solving` phase recorded a
-`verdict` (Fast Downward and SymK do, see above), it decides instead:
+`verdict` (Fast Downward, SymK and ENHSP do, see above), it decides:
 
 | verdict | `proves_unsat` | outcome |
 |---|---|---|
@@ -427,7 +473,15 @@ every crash of a complete model to `unsat`. Where the `solving` phase recorded a
 | `unknown` / `killed` | either | `unknown` |
 | `error` | either | `error` |
 
-A run with no verdict is read as before. Per-horizon verdicts
+| none | True | `unknown`: it may have been a proof, or a crash |
+| none | False | `error` |
+
+**No verdict, no `unsat`.** Without a verdict, a completed run's status is all
+there is to go on, and `failed` cannot tell a proof from a crash, so the run is
+`unknown` however complete the model. A UPF run is the usual case, and so is
+any consumer that drives a solver itself. To let such a run conclude `unsat`,
+record a verdict on its `solving` phase (`phase.solver_stats['verdict']`).
+Per-horizon verdicts
 (`solving-horizon-N`) are about one bound each and never decide the run.
 
 ## what the core owns
@@ -437,6 +491,7 @@ A run with no verdict is read as before. Per-horizon verdicts
 | per-solver stat parsers, cost-layer assembly, dispatch | `solver_stats.py` |
 | per-run Savile Row verdict (refuted vs gave up) | `solver_stats.py` (`savilerow_verdict`) |
 | per-run Fast Downward / SymK verdict (exit code + log) | `solver_stats.py` (`fd_verdict`) |
+| per-run ENHSP verdict (stdout + stderr) | `solver_stats.py` (`enhsp_verdict`) |
 | Savile Row `.info` split recording (per horizon) | `solver_horizon.py` (`_record_savilerow_times`) |
 | stats attached to a phase after each solve | `solver_runner.py`, `solver_horizon.py` |
 | the phase model (`RunSummary` / `PhaseResult`), outcome and success | `run_summary.py` |

{solver_support-0.4.1 → solver_support-0.4.2}/pyproject.toml

@@ -20,7 +20,7 @@ name = "solver-support"
 # escape hatch for a version that genuinely cannot be known without building.
 # Ours can. Carrying a `.devN` suffix between releases keeps the working tree
 # from claiming a version that is already tagged and published.
-version = "0.4.1"
+version = "0.4.2"
 description = "Generic Conjure / Savile Row / Fast Downward / SymK / ENHSP / UPF solver pipeline"
 readme = "README.md"
 requires-python = ">=3.9"

{solver_support-0.4.1 → solver_support-0.4.2}/solver_support/run_summary.py

@@ -245,13 +245,14 @@ class RunSummary:
         if final_status in _COMPLETED_STATUSES:
             verdict = self._solving_verdict()
             if verdict is None:
-                # No account of how the search ended, so read the status alone.
-                # The search finished and produced nothing. From a model that
-                # decides the problem that IS the answer; from any other model
-                # it means something went wrong, because a bounded model that
-                # finishes unsolved has not concluded anything -- it has merely
-                # stopped.
-                return 'unsat' if proves_unsat else 'error'
+                # No account of how the search ended, so the status is all
+                # there is, and it cannot prove anything: `failed` is equally
+                # what a proof, a rejected input and a crash look like. From a
+                # model that could prove, it may have been a proof, so the
+                # answer is unknown. From any other model, finishing unsolved
+                # means something went wrong. To conclude unsat, the solving
+                # phase must record a verdict.
+                return 'unknown' if proves_unsat else 'error'
             # The solver said how its search ended. Only a refutation from a
             # search that can prove one is a proof; an exhausted search that
             # cannot (an unsafe heuristic, a tune switched off) concluded

{solver_support-0.4.1 → solver_support-0.4.2}/solver_support/solver_enhsp.py

@@ -9,9 +9,10 @@ sas_plan, so the PDDLSolutionParser consumes the result unchanged.
 """
 
 from pathlib import Path
-from typing import Optional
+from typing import Any, Dict, Optional
 
 from .solver_base import Solver
+from .solver_stats import SolverStatsExtractor
 from .global_state import global_state
 
 
@@ -71,6 +72,17 @@ class EnhspSolver(Solver):
         # run_phase's expected_outputs, which would list it even on a failed parse.
         result = self.runner.run_phase("solving", cmd)
 
+        # ENHSP exits 0 for a proof and for a run that failed to load its input
+        # alike, so the summary's outcome is decided by this, not by `failed`.
+        verdict: Dict[str, Any] = {}
+        if not result.launch_failed:
+            verdict = SolverStatsExtractor.enhsp_verdict(
+                result.returncode, result.stdout, result.stderr, cmd,
+                timed_out=result.timed_out, interrupted=result.interrupted)
+            solving_phase = self.runner.summary.get_phase("solving")
+            if solving_phase is not None:
+                solving_phase.solver_stats.update(verdict)
+
         if result.timed_out:
             # No internal time limit outside anytime mode: the runner terminated
             # ENHSP mid-search, so no plan was written - a genuine timeout.
@@ -80,16 +92,19 @@ class EnhspSolver(Solver):
             self.logger.warning("ENHSP solving was interrupted")
             return self._finalise(solution_file)
 
-        if result.success:
-            return self._finalise(solution_file)
-
-        # ENHSP exits non-zero when it cannot parse the model (e.g. the axiom
-        # `pddl` domain, which it does not support) or fails to plan. A plan may
-        # still have been written on some paths, so salvage it if present.
-        salvaged = self._finalise(solution_file)
-        if salvaged is None:
-            self.logger.error(f"ENHSP solver failed: {result.stderr}")
-        return salvaged
+        # A plan may have been written even on a failing path, so take it if
+        # present. Without one, the verdict says whether anything went wrong:
+        # ENHSP exits 0 after a missing input file, and non-zero on a syntax
+        # error (e.g. the axioms it does not support) or out of memory.
+        plan = self._finalise(solution_file)
+        if plan is None:
+            failed = (verdict.get('verdict') == 'error' if verdict
+                      else not result.success)
+            if failed:
+                self.logger.error(f"ENHSP solver failed: {verdict} {result.stderr}")
+            else:
+                self.logger.info(f"ENHSP found no plan: {verdict}")
+        return plan
 
     def _finalise(self, solution_file: Path) -> Optional[Path]:
         """Register and return the plan file if ENHSP wrote one, else None."""

{solver_support-0.4.1 → solver_support-0.4.2}/solver_support/solver_stats.py

@@ -85,6 +85,20 @@ _ENHSP_LAYER_RE = re.compile(r'f\(n\)\s*=\s*([\d.]+)\s*\([^)]*Time:\s*([\d.]+)\)
 # success, neither on a timeout / unsolvable.
 _ENHSP_SOLVED_RE = re.compile(r'Problem Solved|Found Plan:')
 
+# ENHSP 0.21.0's verdict lines (see enhsp_verdict). It exits 0 for nearly every
+# ending, so these lines and stderr are all there is to go on.
+_ENHSP_SOLVED_LINE = 'Problem Solved'
+_ENHSP_EXHAUSTED_LINE = 'Problem unsolvable'      # the search ran out of states
+_ENHSP_REFUSED_LINE = 'Unsolvable Problem'        # parsing/grounding returned null
+_ENHSP_SYNTAX_LINE = 'Some Syntax Error'          # exits 255
+# A `catch (Exception)` logs through java.util.logging (to stderr) and carries
+# on; an uncaught one ends the JVM with exit 1.
+_ENHSP_EXCEPTION_RE = re.compile(r'^SEVERE: |^Exception in thread ', re.MULTILINE)
+_ENHSP_OOM = 'java.lang.OutOfMemoryError'
+# `-planner` presets that prune with helpful actions (ENHSP.setPlanner); with no
+# preset, `-ha true` / `-ht true` do the same. A preset overrides those flags.
+_ENHSP_HELPFUL_PLANNERS = frozenset({'sat-hmrph', 'sat-hmrphj'})
+
 
 # What a single Savile Row run concluded, derived from its ``.info`` file. The
 # two *decided* verdicts come first: they are the only ones that answer the
@@ -181,6 +195,18 @@ _FD_EXHAUSTED_LINE = 'Completely explored state space -- no solution!'   # befor
 _SYMK_BOUND_RE = re.compile(r'BOUND: (\d+) < (\d+)')
 _SYMK_INFINITY = 2147483647
 _FD_BOUND_ARG_RE = re.compile(r'\bbound\s*=')
+# The search the driver actually ran, aliases expanded and evaluators predefined
+# with --evaluator included. Read in preference to the argv the caller passes.
+_FD_SEARCH_LINE_RE = re.compile(r'search command line string: (.*)')
+_FD_AXIOMS_RE = re.compile(r'^Translator axioms: (\d+)', re.MULTILINE)
+# A pre-26.6 search prunes every state an evaluator calls a dead end, reliable or
+# not, so its exhaustion is a proof only if every evaluator's dead ends are. From
+# `dead_ends_are_reliable()` in the 26.6 sources: `cg` and `cea` never are; `hm`
+# is not on tasks with conditional effects, nor the landmark sum on most of them,
+# which the log does not show, so both are treated as never. The relaxation
+# heuristics are unreliable exactly when the task has axioms.
+_FD_UNRELIABLE_RE = re.compile(r'\b(?:cg|cea|hm|lmcount|landmark_sum)\(')
+_FD_UNRELIABLE_WITH_AXIOMS_RE = re.compile(r'\b(?:ff|add|hmax)\(')
 
 
 class SolverStatsExtractor:
@@ -621,17 +647,21 @@ class SolverStatsExtractor:
 
         ``unsat`` is about *the configured search*, not the task. With reason
         ``exhausted`` it is this function's own reading of a pre-26.6 planner
-        whose search ran out of states - sound only if that search is complete
-        and prunes safely, which the log cannot reveal (``astar(ff())`` on a
-        task with axioms prunes states it should not). Without a reason it is
-        the planner's own claim. Either way, turning it into a run-level outcome
-        is the caller's job, through the registry's ``get_proves_unsat``.
+        whose search ran out of states. That is sound only if the search is
+        complete and prunes safely. The pruning half is checked here, as 26.6
+        checks it itself: an evaluator whose dead ends are unreliable on this
+        task (``ff`` on a task with axioms, ``cg`` on any) gives ``unknown`` /
+        ``unsafe-pruning``. Completeness of the search algorithm is not
+        checked here. Without a reason, ``unsat`` is the planner's own claim.
+        Either way, turning it into a run-level outcome is the caller's job,
+        through the registry's ``get_proves_unsat``.
 
         Args:
             returncode: The planner driver's exit code.
             stdout: Its stdout, where the search logs.
             command: The argv that ran, read for a ``bound=`` in ``--search``
-                (a pre-26.6 bounded search logs exactly as an unbounded one).
+                (a pre-26.6 bounded search logs exactly as an unbounded one)
+                when the log lacks the driver's ``search command line string``.
             timed_out / interrupted: the runner killed it, so the planner's own
                 account of how the search ended was never written.
             anytime: it was a portfolio (``--alias``), whose 12 can mask its
@@ -667,13 +697,21 @@ class SolverStatsExtractor:
             return 'unknown', 'portfolio'
         if _FD_TIME_LIMIT_LINE in stdout:
             return 'unknown', 'search-limit'
-        search = ''
-        if '--search' in command:
+        logged = _FD_SEARCH_LINE_RE.findall(stdout)
+        if logged:
+            search = logged[-1]
+        elif '--search' in command:
             i = command.index('--search')
             search = command[i + 1] if i + 1 < len(command) else ''
+        else:
+            search = ''
         bounded = bool(_FD_BOUND_ARG_RE.search(search))
         if _FD_EXHAUSTED_LINE in stdout:
-            return ('unknown', 'bounded') if bounded else ('unsat', 'exhausted')
+            if bounded:
+                return 'unknown', 'bounded'
+            if SolverStatsExtractor._fd_pruned_unsafely(search, stdout):
+                return 'unknown', 'unsafe-pruning'
+            return 'unsat', 'exhausted'
         bounds = _SYMK_BOUND_RE.findall(stdout)
         if bounds:
             lower, upper = (int(b) for b in bounds[-1])
@@ -687,6 +725,88 @@ class SolverStatsExtractor:
         # (`Search terminated without finding a plan!`).
         return 'unknown', 'incomplete'
 
+    @staticmethod
+    def _fd_pruned_unsafely(search: str, stdout: str) -> bool:
+        """Whether a search that ran out of states may have pruned a live one."""
+        if _FD_UNRELIABLE_RE.search(search):
+            return True
+        if _FD_UNRELIABLE_WITH_AXIOMS_RE.search(search):
+            # No axiom count in the log reads as "may have axioms".
+            axioms = _FD_AXIOMS_RE.findall(stdout)
+            return not axioms or int(axioms[-1]) > 0
+        return False
+
+    @staticmethod
+    def enhsp_verdict(returncode: int, stdout: str, stderr: str = '',
+                      command: Optional[List[str]] = None,
+                      timed_out: bool = False,
+                      interrupted: bool = False) -> Dict[str, Any]:
+        """What an ENHSP run concluded, from its output: the twin of
+        :meth:`fd_verdict`, in the same vocabulary.
+
+        ENHSP 0.21.0 exits 0 for nearly every ending: a plan, a proof, a
+        search that pruned or was bounded, and a parse or grounding failure
+        that it logged and then reported as ``Unsolvable Problem``. So the exit
+        code says almost nothing, and stderr matters: a logged exception goes
+        there while the run carries on to exit 0.
+
+        ``unsat`` (reason ``exhausted``) means the configured search ran out of
+        states. Whether that search is complete is still the registry's
+        ``get_proves_unsat`` to say. Helpful-action pruning and a ``-dl`` cost
+        bound are read off the argv and give ``unknown``.
+        """
+        if timed_out or interrupted:
+            return {'verdict': 'killed'}
+        stdout = stdout or ''
+        stderr = stderr or ''
+        argv = command or []
+        if _ENHSP_SOLVED_LINE in stdout:
+            verdict, reason = 'sat', None
+        elif _ENHSP_OOM in stderr or _ENHSP_OOM in stdout:
+            verdict, reason = 'unknown', 'memory'
+        elif _ENHSP_EXCEPTION_RE.search(stderr):
+            verdict, reason = 'error', 'crash'
+        elif _ENHSP_SYNTAX_LINE in stdout:
+            verdict, reason = 'error', 'input-error'
+        elif returncode != 0:
+            verdict, reason = 'error', 'crash' if returncode < 0 else 'unexplained'
+        elif _ENHSP_EXHAUSTED_LINE in stdout:
+            verdict, reason = SolverStatsExtractor._enhsp_exhausted_verdict(argv)
+        elif _ENHSP_REFUSED_LINE in stdout:
+            # prepareForSearch returned false. That may be a sound reachability
+            # refutation, but the log does not say which, so it proves nothing.
+            verdict, reason = 'unknown', 'grounding'
+        else:
+            verdict, reason = 'error', 'unexplained'
+        stats: Dict[str, Any] = {'verdict': verdict}
+        if reason:
+            stats['verdict_reason'] = reason
+        return stats
+
+    @staticmethod
+    def _enhsp_exhausted_verdict(argv: List[str]):
+        """Read ``Problem unsolvable`` against the configuration that ran."""
+        def value(flag):
+            if flag in argv:
+                i = argv.index(flag)
+                return argv[i + 1] if i + 1 < len(argv) else ''
+            return None
+
+        if '-anytime' in argv or '-autoanytime' in argv:
+            # Each round is bounded by the last plan's cost, and its own
+            # -timeout ends the run the same way.
+            return 'unknown', 'anytime'
+        if value('-dl') is not None:
+            return 'unknown', 'bounded'
+        planner = value('-planner')
+        if planner is not None:
+            helpful = planner in _ENHSP_HELPFUL_PLANNERS
+        else:
+            helpful = value('-ha') == 'true' or value('-ht') == 'true'
+        if helpful:
+            return 'unknown', 'incomplete'
+        return 'unsat', 'exhausted'
+
     @staticmethod
     def extract_fast_downward_stats(stdout: str, stderr: str,
                                     duration: Optional[float] = None) -> Dict[str, Any]:

{solver_support-0.4.1 → solver_support-0.4.2}/solver_support/starter/models.yaml

@@ -51,9 +51,14 @@ models:
 
   # A PDDL planning model: `path` is the domain file, and the instance (problem)
   # file is given per run. Fast Downward runs it by default; SymK and ENHSP
-  # can run it too. `proves-unsat: true` says a run that finds no plan has
-  # *proved* there is none, which holds for a complete search. The tunes that
-  # are not complete opt out below, so it is safe to leave on here.
+  # can run it too. `proves-unsat: true` says this model can *prove* there is
+  # no plan, given a complete search. It is safe to leave on for these three
+  # planners, because each records a verdict read off its own log: a run is
+  # concluded unsat only when its search ran out of states without being
+  # bounded or pruned unsafely, never just because it found no plan. The one
+  # tune that cannot prove (the `soup` portfolio) opts out below. A solver
+  # whose backend records no verdict, such as a UPF planner, never concludes
+  # unsat, whatever this says.
   pddl:
     path: models/domain.pddl
     solver: fd
@@ -92,8 +97,8 @@ solvers:
       ff:
         search: "lazy_greedy([ff()])"
         # FF prunes states it scores as infinite, which is unsafe on tasks with
-        # axioms, so a search under it that runs out of states proves nothing.
-        proves-unsat: false
+        # axioms. The verdict sees that in the log and claims no proof there;
+        # on a task without axioms, running out of states is a proof.
         description: satisficing greedy best-first search with FF
       soup:
         alias: seq-sat-fdss-2018

{solver_support-0.4.1 → solver_support-0.4.2/solver_support.egg-info}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: solver-support
-Version: 0.4.1
+Version: 0.4.2
 Summary: Generic Conjure / Savile Row / Fast Downward / SymK / ENHSP / UPF solver pipeline
 Author: András Salamon
 License-Expression: BSD-3-Clause
@@ -37,6 +37,24 @@ subprocesses with timeouts and clean teardown, naming the per-run files, parsing
 solver output into a uniform timing / cost-layer model, and driving the horizon
 (increasing-difficulty) loop.
 
+Its main value is a **unified semantics for solver behaviour**. Each solver
+reports how a run ended in its own way, and the raw signals are misleading
+taken at face value:
+
+- Fast Downward's exit 12 can mean a proof, a time limit or a search that gave up.
+- ENHSP exits 0 after a proof and after failing to read its input alike.
+- Savile Row prints the same "no solution" whether the backend refuted the
+  instance or stopped at its own limit.
+- A SAT solver's 10 and 20 are answers, not failures.
+
+solver-support reads each solver's exit codes, logs and info files. It reports
+every run in one vocabulary: a `verdict` (`sat` / `unsat` / `unknown` / `error` /
+`killed`, with a reason) for each solve, and an `outcome` and `stop_reason` for
+the run. So `unsat` means a proof, whichever solver produced it. A run that
+merely found nothing, or crashed, never reads as one. The rules for each solver
+are checked against its source and its real output, and are documented in
+[`timing-model.md`](https://github.com/ott2/solver-support/blob/main/docs/timing-model.md).
+
 It carries **no** notion of any particular problem domain or objective. A
 consumer supplies:
 

{solver_support-0.4.1 → solver_support-0.4.2}/solver_support.egg-info/SOURCES.txt

@@ -30,6 +30,7 @@ solver_support.egg-info/top_level.txt
 solver_support/starter/models.yaml
 tests/conftest.py
 tests/test_command_builders.py
+tests/test_enhsp_verdict.py
 tests/test_fd_verdict.py
 tests/test_helpers.py
 tests/test_horizon_params.py

solver_support-0.4.2/tests/test_enhsp_verdict.py

@@ -0,0 +1,118 @@
+"""
+Tests for ``SolverStatsExtractor.enhsp_verdict`` (issue #6).
+
+ENHSP exits 0 for a plan, for a proof, and for a run that never loaded its
+input, so the verdict is read from the log. As with Fast Downward, the cases run
+against REAL planner output (tests/fixtures/enhsp, see its README).
+"""
+
+from pathlib import Path
+
+import pytest
+
+from solver_support.solver_stats import SAVILEROW_VERDICTS, SolverStatsExtractor
+
+FIXTURES = Path(__file__).parent / 'fixtures' / 'enhsp'
+
+
+def _read(name, stream):
+    path = FIXTURES / f'{name}.{stream}'
+    return path.read_text() if path.exists() else ''
+
+
+def _enhsp(*extra, planner='opt-blind'):
+    return ['enhsp', '-o', 'd.pddl', '-f', 'p.pddl', '-planner', planner,
+            '-sp', 'plan', *extra]
+
+
+def _verdict(name, returncode, command, **kwargs):
+    return SolverStatsExtractor.enhsp_verdict(
+        returncode, _read(name, 'stdout'), _read(name, 'stderr'), command,
+        **kwargs)
+
+
+@pytest.mark.parametrize('fixture, code, command, expected', [
+    ('enhsp-sat', 0, _enhsp(), {'verdict': 'sat'}),
+    ('enhsp-exhaust', 0, _enhsp(),
+     {'verdict': 'unsat', 'verdict_reason': 'exhausted'}),
+    # the same `Problem unsolvable`, from a search that pruned with helpful actions
+    ('enhsp-ha', 0, _enhsp(planner='sat-hmrph'),
+     {'verdict': 'unknown', 'verdict_reason': 'incomplete'}),
+    # and from a cost bound below the plan's (sat.pddl is solvable)
+    ('enhsp-dl', 0, _enhsp('-dl', '1'),
+     {'verdict': 'unknown', 'verdict_reason': 'bounded'}),
+    ('enhsp-syntax', 255, _enhsp(),
+     {'verdict': 'error', 'verdict_reason': 'input-error'}),
+    # exit 0 and `Unsolvable Problem`, after a FileNotFoundException on stderr
+    ('enhsp-nofile', 0, _enhsp(),
+     {'verdict': 'error', 'verdict_reason': 'crash'}),
+    ('enhsp-oom', 1, _enhsp(),
+     {'verdict': 'unknown', 'verdict_reason': 'memory'}),
+])
+def test_real_logs(fixture, code, command, expected):
+    assert _verdict(fixture, code, command) == expected
+
+
+def test_the_issue_6_hazard_is_not_a_proof():
+    """A missing domain file exits 0 with an `unsolvable` line on stdout. That
+    must never read as unsat, whatever the stderr check decides."""
+    assert _verdict('enhsp-nofile', 0, _enhsp())['verdict'] != 'unsat'
+
+
+def test_refused_without_an_exception_proves_nothing():
+    """`Unsolvable Problem` alone: grounding returned null, which may be a sound
+    reachability refutation, but the log does not say which."""
+    stdout = 'Domain parsed\nProblem parsed\nGrounding..\nUnsolvable Problem\n'
+    assert SolverStatsExtractor.enhsp_verdict(0, stdout, '', _enhsp()) == {
+        'verdict': 'unknown', 'verdict_reason': 'grounding'}
+
+
+def test_a_swallowed_search_exception_is_a_crash():
+    """ENHSP.planning() logs the exception at SEVERE and returns: no verdict
+    line, exit 0. No input here makes it throw, so this stderr is modelled on
+    enhsp-nofile.stderr."""
+    stdout = _read('enhsp-exhaust', 'stdout').split('Problem unsolvable')[0]
+    stderr = ('Oct 22, 2026 11:00:26 PM planners.ENHSP planning\n'
+              'SEVERE: null\njava.lang.NullPointerException\n')
+    assert SolverStatsExtractor.enhsp_verdict(0, stdout, stderr, _enhsp()) == {
+        'verdict': 'error', 'verdict_reason': 'crash'}
+
+
+@pytest.mark.parametrize('extra, planner, expected', [
+    (('-ha', 'true'), None, 'incomplete'),
+    (('-ht', 'true'), None, 'incomplete'),
+    (('-ha', 'false'), None, None),
+    # a preset overrides -ha (ENHSP.setPlanner resets it)
+    (('-ha', 'true'), 'opt-blind', None),
+    (('-anytime',), 'sat-hmrp', 'anytime'),
+    (('-autoanytime',), None, 'anytime'),
+])
+def test_configuration_flags(extra, planner, expected):
+    command = ['enhsp', '-o', 'd.pddl', '-f', 'p.pddl', *extra]
+    if planner:
+        command += ['-planner', planner]
+    stats = _verdict('enhsp-exhaust', 0, command)
+    if expected is None:
+        assert stats == {'verdict': 'unsat', 'verdict_reason': 'exhausted'}
+    else:
+        assert stats == {'verdict': 'unknown', 'verdict_reason': expected}
+
+
+@pytest.mark.parametrize('flag', ['timed_out', 'interrupted'])
+def test_killed(flag):
+    assert _verdict('enhsp-exhaust', -15, _enhsp(), **{flag: True}) == {
+        'verdict': 'killed'}
+
+
+def test_unexplained_exits():
+    assert SolverStatsExtractor.enhsp_verdict(0, '', '', _enhsp()) == {
+        'verdict': 'error', 'verdict_reason': 'unexplained'}
+    assert SolverStatsExtractor.enhsp_verdict(-9, '', '', _enhsp()) == {
+        'verdict': 'error', 'verdict_reason': 'crash'}
+
+
+def test_vocabulary_is_shared_with_savilerow():
+    names = ['enhsp-sat', 'enhsp-exhaust', 'enhsp-ha', 'enhsp-dl',
+             'enhsp-syntax', 'enhsp-nofile', 'enhsp-oom']
+    for name in names:
+        assert _verdict(name, 0, _enhsp())['verdict'] in SAVILEROW_VERDICTS

{solver_support-0.4.1 → solver_support-0.4.2}/tests/test_fd_verdict.py

@@ -7,6 +7,7 @@ from a search that gave up. The 12 cases therefore run against REAL planner
 logs (tests/fixtures/fd, see its README), not hand-written ones.
 """
 
+import re
 from pathlib import Path
 
 import pytest
@@ -26,6 +27,14 @@ def _fd(search):
             '--search', search]
 
 
+_EXHAUSTED = '[t=0.01s, 1 KB] Completely explored state space -- no solution!'
+
+
+def _swap_search(log, search):
+    """A real log with its logged search swapped, everything else kept."""
+    return re.sub(r"--search '[^']*'", lambda _: f"--search '{search}'", log)
+
+
 def _verdict(returncode, stdout='', command=None, **kwargs):
     return SolverStatsExtractor.fd_verdict(returncode, stdout, command, **kwargs)
 
@@ -57,11 +66,19 @@ class TestUnsolvedExit12:
     def test_real_logs(self, fixture, search, expected):
         assert _verdict(12, _log(fixture), _fd(search)) == expected
 
-    def test_bound_is_read_from_the_search_string(self):
-        """The log alone cannot see a bound: drop it from the argv and the same
-        output reads as a proof."""
+    def test_the_logged_search_outranks_the_argv(self):
+        """The driver logs the search it actually ran, so a bound is seen even
+        when the caller's argv does not show one (an alias, say)."""
         assert _verdict(12, _log('fd-bound'), _fd('astar(blind())')) == {
+            'verdict': 'unknown', 'verdict_reason': 'bounded'}
+
+    def test_bound_is_read_from_the_argv_without_the_log_line(self):
+        """Exhaustion alone reads as a proof; the argv's bound= takes it away."""
+        log = _EXHAUSTED + '\n'
+        assert _verdict(12, log, _fd('astar(blind())')) == {
             'verdict': 'unsat', 'verdict_reason': 'exhausted'}
+        assert _verdict(12, log, _fd('astar(blind(), bound=2)')) == {
+            'verdict': 'unknown', 'verdict_reason': 'bounded'}
 
     def test_portfolio_12_is_never_unsat(self):
         """A portfolio's 12 can mask its components' time-outs, and its
@@ -85,6 +102,46 @@ class TestUnsolvedExit12:
         assert _verdict(12, both, _fd('astar(blind())'))['verdict'] == 'unknown'
 
 
+class TestUnsafePruning:
+    """Issue #6: before 26.6, a search prunes every state an evaluator calls a
+    dead end, reliable or not, so running out of states proves nothing if one
+    of them was unreliable on this task."""
+
+    def test_real_ff_on_a_task_with_axioms(self):
+        assert _verdict(12, _log('fd-ff-ax'), None) == {
+            'verdict': 'unknown', 'verdict_reason': 'unsafe-pruning'}
+
+    @pytest.mark.parametrize('search', [
+        'lazy_greedy([ff()])', 'eager_greedy([add()])', 'astar(hmax())'])
+    def test_relaxation_heuristics_are_safe_without_axioms(self, search):
+        """`fd-exhaust` has `Translator axioms: 0`."""
+        assert _verdict(12, _swap_search(_log('fd-exhaust'), search), None) == {
+            'verdict': 'unsat', 'verdict_reason': 'exhausted'}
+
+    @pytest.mark.parametrize('search', [
+        'lazy_greedy([cg()])', 'lazy_greedy([cea()])', 'astar(hm(m=2))',
+        'lazy_greedy([lmcount(lm_rhw())])'])
+    def test_always_unreliable_evaluators(self, search):
+        assert _verdict(12, _swap_search(_log('fd-exhaust'), search), None) == {
+            'verdict': 'unknown', 'verdict_reason': 'unsafe-pruning'}
+
+    def test_a_predefined_evaluator_counts(self):
+        """`--evaluator h=cg()` is in the logged line, not in the search."""
+        log = _log('fd-exhaust').replace(
+            "--search 'astar(blind())'",
+            "--evaluator 'h=cg()' --search 'lazy_greedy([h])'")
+        assert _verdict(12, log, None)['verdict_reason'] == 'unsafe-pruning'
+
+    def test_blind_on_a_task_with_axioms_is_still_a_proof(self):
+        log = _swap_search(_log('fd-ff-ax'), 'astar(blind())')
+        assert _verdict(12, log, None) == {
+            'verdict': 'unsat', 'verdict_reason': 'exhausted'}
+
+    def test_no_axiom_count_reads_as_axioms(self):
+        assert _verdict(12, _EXHAUSTED, _fd('lazy_greedy([ff()])')) == {
+            'verdict': 'unknown', 'verdict_reason': 'unsafe-pruning'}
+
+
 class TestExitCodes:
     """Every code but 12 says what happened by itself."""
 

{solver_support-0.4.1 → solver_support-0.4.2}/tests/test_run_summary_success.py

@@ -155,7 +155,8 @@ def test_custom_status_without_a_solution_is_not_success():
 
 def test_planner_proven_unsat_is_success():
     """Exits non-zero, finalizes 'failed', and has answered the question."""
-    s = _summary(_phase("translation", True), _phase("solving", False))
+    s = _summary(_phase("translation", True),
+                 _phase("solving", False, verdict="unsat"))
     s.finalize("failed", proves_unsat=True)
     assert s.outcome == "unsat"
     assert s.success is True
@@ -193,12 +194,23 @@ def test_failed_run_is_read_through_its_verdict(verdict, proves, outcome):
     assert s.success is (outcome == "unsat")
 
 
-def test_failed_without_a_verdict_keeps_its_old_reading():
-    """Solvers that record no verdict (and summaries written before verdicts
-    existed) mean what they always meant."""
-    s = _summary(_phase("solving", False))
+def test_failed_without_a_verdict_is_never_unsat():
+    """Issue #6: with no account of how the search ended, `failed` is what a
+    proof, a rejected input and a crash all look like. A model that could have
+    proved gets `unknown`, and only a recorded verdict can make it `unsat`."""
+    s = _summary(_phase("solving", False, returncode=1))
     s.finalize("failed", proves_unsat=True)
-    assert s.outcome == "unsat"
+    assert s.outcome == "unknown"
+    assert s.success is False
+
+
+def test_issue_6_reproduction():
+    """The issue's own reproduction: an ENHSP run that recorded no verdict."""
+    s = RunSummary(instance='i', model='pddl', solver='enhsp')
+    s.add_phase(PhaseResult(name='solving', returncode=1,
+                            command=['enhsp', '-o', 'd.pddl', '-f', 'p.pddl']))
+    s.finalize('failed', proves_unsat=True)
+    assert (s.outcome, s.success) == ('unknown', False)
 
 
 def test_horizon_verdicts_never_decide_the_run():

{solver_support-0.4.1 → solver_support-0.4.2}/tests/test_solvers.py

@@ -609,6 +609,34 @@ class TestEnhspSolver(TempDirTestMixin):
 
         assert result is None
 
+    @pytest.mark.parametrize("fixture, expected, logged_as", [
+        ("enhsp-exhaust", {"verdict": "unsat", "verdict_reason": "exhausted"},
+         "info"),
+        # issue #6: exit 0 and an `unsolvable` line, after the domain failed to load
+        ("enhsp-nofile", {"verdict": "error", "verdict_reason": "crash"},
+         "error"),
+    ])
+    def test_enhsp_solver_records_its_verdict(self, fixture, expected, logged_as):
+        """The verdict goes on the solving phase, where the summary's outcome
+        reads it; only an `error` verdict is logged as a failure."""
+        enhsp_dir = Path(__file__).parent / "fixtures" / "enhsp"
+        stderr = enhsp_dir / f"{fixture}.stderr"
+        self.mock_runner.run_phase.return_value = SolverResult(
+            command=["enhsp"], returncode=0,
+            stdout=(enhsp_dir / f"{fixture}.stdout").read_text(),
+            stderr=stderr.read_text() if stderr.exists() else "", duration=0.5)
+        phase = PhaseResult(name="solving")
+        self.mock_summary.get_phase.return_value = phase
+
+        result = self.solver.solve(
+            param_file=self.temp_path / "problem.pddl", model_path="domain.pddl")
+
+        assert result is None
+        assert {k: phase.solver_stats[k] for k in expected} == expected
+        assert getattr(self.solver.logger, logged_as).called
+        if logged_as == "info":
+            assert not self.solver.logger.error.called
+
     def test_enhsp_solver_timeout(self):
         """A timed-out run (no plan) raises TimeoutError."""
         self.mock_runner.run_phase.return_value = SolverResult(

{solver_support-0.4.1 → solver_support-0.4.2}/tests/test_starter_config.py

@@ -120,12 +120,32 @@ def test_every_solver_has_a_default_model_that_exists(starter):
 
 def test_incomplete_tunes_do_not_claim_proofs(starter):
     assert starter.get_proves_unsat('pddl') is True
-    assert starter.get_proves_unsat('pddl', tune='ff') is False
-    assert starter.get_proves_unsat('pddl', tune='soup') is False
+    # complete; where it prunes unsafely (axioms), the verdict says so
+    assert starter.get_proves_unsat('pddl', tune='ff', solver_name='fd') is True
+    assert starter.get_proves_unsat('pddl', tune='soup', solver_name='fd') is False
     assert starter.get_proves_unsat('eprime') is False
     assert starter.get_proves_unsat('essence') is False
 
 
+# The solver types whose backend records a solving-phase verdict, the only
+# thing a run's outcome can conclude `unsat` from (issue #6).
+_VERDICT_SOLVER_TYPES = {'fd', 'enhsp'}
+
+
+def test_proofs_are_claimed_only_where_a_verdict_is_recorded(starter):
+    """For every solver that can run a model claiming proves-unsat, the backend
+    records a verdict, so the claim is never made off a bare `failed`."""
+    for model in starter.config['models']:
+        for solver in starter.list_solvers():
+            if not starter.is_solver_compatible_with_model(solver, model):
+                continue
+            tunes = [None, *starter.get_solver_config(solver).get('tune', {})]
+            if any(starter.get_proves_unsat(model, tune=t, solver_name=solver)
+                   for t in tunes):
+                assert starter.get_solver_type(solver) in _VERDICT_SOLVER_TYPES, (
+                    model, solver)
+
+
 def test_config_path_may_be_a_str(tmp_path):
     """The starter's own header wires it with a str. A str has no is_file(), and
     used to be silently skipped, so the registry raised as if nothing at all