truereward 0.1.0__tar.gz

truereward-0.1.0/LICENSE

@@ -0,0 +1,202 @@
+
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright [yyyy] [name of copyright owner]
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.

truereward-0.1.0/MANIFEST.in

@@ -0,0 +1,3 @@
+# The tests stay in the source tree: they spell the names the package must
+# not carry in order to refuse them, and those names are not published.
+prune tests

truereward-0.1.0/PKG-INFO

@@ -0,0 +1,51 @@
+Metadata-Version: 2.4
+Name: truereward
+Version: 0.1.0
+Summary: One case result, one fold, one grade document: the reward of a graded run, or the reason there is none.
+Author: truereward contributors
+License-Expression: Apache-2.0
+Requires-Python: >=3.11
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Provides-Extra: pytest
+Requires-Dist: pytest>=8; extra == "pytest"
+Requires-Dist: pytest-xdist>=3.6; extra == "pytest"
+Dynamic: license-file
+
+# truereward
+
+Turn the cases of a graded run into one reward, or into the reason there is
+none.
+
+A grader looks at some kind of evidence: a recorded end state, a test report,
+a list of criteria, a measured figure. Whatever it looked at, it answers with
+one record per case, `CaseResult`. This package is everything after that
+record, and it is the same for every kind of evidence.
+
+- **The fold.** `fold(rows)` applies `gated-binary/1`: the reward is 1.0 when
+  every pass-to-pass case holds and every fail-to-pass case passes, and 0.0
+  otherwise. `fix_rate` and `guards_hold` are reported beside it.
+- **Three outcomes.** A case passed, failed, or could not be graded. The third
+  carries a code from a closed list, each with who failed: the run, or the
+  grader. An ungradable run has no reward. It is never scored 0.
+- **The document.** `grade_document` builds `grade.json`; `write` publishes it
+  with `ctrf.json`, `reward.json` and `reward.txt` into a verifier's logs
+  directory, each file whole or not at all. An ungradable run publishes no
+  reward file.
+- **The pytest plugin.** One test per case, folded once at the end of the
+  session, in one process or under pytest-xdist:
+
+  ```python
+  @pytest.mark.parametrize("case", cases(CASES), ids=lambda c: c.id)
+  def test_case(case, grade):
+      grade(case, grade_case)
+  ```
+
+  Run it with `pytest -p truereward.plugin`. Install the `pytest` extra for
+  it: `pip install "truereward[pytest]"`. Folding and writing need the
+  standard library alone.
+
+A reader for one kind of evidence is a separate distribution. It may declare
+refusal codes of its own with `truereward.codes.declare`.
+
+Licensed under Apache-2.0.

truereward-0.1.0/README.md

@@ -0,0 +1,37 @@
+# truereward
+
+Turn the cases of a graded run into one reward, or into the reason there is
+none.
+
+A grader looks at some kind of evidence: a recorded end state, a test report,
+a list of criteria, a measured figure. Whatever it looked at, it answers with
+one record per case, `CaseResult`. This package is everything after that
+record, and it is the same for every kind of evidence.
+
+- **The fold.** `fold(rows)` applies `gated-binary/1`: the reward is 1.0 when
+  every pass-to-pass case holds and every fail-to-pass case passes, and 0.0
+  otherwise. `fix_rate` and `guards_hold` are reported beside it.
+- **Three outcomes.** A case passed, failed, or could not be graded. The third
+  carries a code from a closed list, each with who failed: the run, or the
+  grader. An ungradable run has no reward. It is never scored 0.
+- **The document.** `grade_document` builds `grade.json`; `write` publishes it
+  with `ctrf.json`, `reward.json` and `reward.txt` into a verifier's logs
+  directory, each file whole or not at all. An ungradable run publishes no
+  reward file.
+- **The pytest plugin.** One test per case, folded once at the end of the
+  session, in one process or under pytest-xdist:
+
+  ```python
+  @pytest.mark.parametrize("case", cases(CASES), ids=lambda c: c.id)
+  def test_case(case, grade):
+      grade(case, grade_case)
+  ```
+
+  Run it with `pytest -p truereward.plugin`. Install the `pytest` extra for
+  it: `pip install "truereward[pytest]"`. Folding and writing need the
+  standard library alone.
+
+A reader for one kind of evidence is a separate distribution. It may declare
+refusal codes of its own with `truereward.codes.declare`.
+
+Licensed under Apache-2.0.

truereward-0.1.0/pyproject.toml

@@ -0,0 +1,27 @@
+[build-system]
+requires = ["setuptools>=77"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "truereward"
+dynamic = ["version"]
+description = "One case result, one fold, one grade document: the reward of a graded run, or the reason there is none."
+readme = "README.md"
+requires-python = ">=3.11"
+license = "Apache-2.0"
+license-files = ["LICENSE"]
+authors = [{name = "truereward contributors"}]
+dependencies = []
+
+[project.optional-dependencies]
+# What plugin.py needs. Folding and writing need the standard library alone.
+pytest = ["pytest>=8", "pytest-xdist>=3.6"]
+
+[tool.setuptools]
+packages = ["truereward"]
+
+[tool.setuptools.dynamic]
+version = {attr = "truereward.__version__"}
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]

truereward-0.1.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

truereward-0.1.0/truereward/__init__.py

@@ -0,0 +1,39 @@
+"""Grading: one case result, one fold, one document, one writer.
+
+An environment answers what a run left behind and what it reported. Turning
+those answers into a reward is this package, and it is the same for every
+environment: a reader reduces its kind of evidence to a CaseResult per case
+(result.py), the fold turns the results and the run checks into one reward or
+a refusal with a code (fold.py, codes.py), and the writer publishes the grade
+document and the files a Harbor verifier is read through (document.py). The
+pytest plugin (plugin.py, over session.py) is how a task's tests reach them:
+one test per case, folded once, in parallel or not.
+
+Nothing here imports an environment, and nothing beyond the standard library
+is needed to fold and write. pytest is imported by plugin.py alone.
+"""
+
+from __future__ import annotations
+
+#: The version of this distribution, and the one place it is written down. It
+#: stands above the imports below because document.py reads it.
+__version__ = "0.1.0"
+
+from .codes import CODES, MEANING, SIDE, Ungradable
+from .document import GRADER, LOGS, ctrf, grade_document, logs, write
+from .fold import SCHEME, fold
+from .result import (
+    FAIL,
+    FAIL_TO_PASS,
+    KINDS,
+    PASS,
+    PASS_TO_PASS,
+    UNGRADABLE,
+    VERDICTS,
+    CaseResult,
+)
+
+__all__ = ["CODES", "MEANING", "SIDE", "Ungradable", "GRADER", "LOGS", "ctrf",
+           "grade_document", "logs", "write", "SCHEME", "fold", "FAIL",
+           "FAIL_TO_PASS", "KINDS", "PASS", "PASS_TO_PASS", "UNGRADABLE",
+           "VERDICTS", "CaseResult"]

truereward-0.1.0/truereward/codes.py

@@ -0,0 +1,152 @@
+"""Why a grade could not be given: a closed set of codes, each with a sentence.
+
+A grade has three outcomes and not two. A case passed, a case failed, or
+nothing was measured at all. The third is not a failure of the candidate and
+is never folded into one: SWE-bench once scored a suite that never ran as
+resolved, and Inspect AI types the same distinction as a reason on an unscored
+result rather than leaving it as prose.
+
+The reason is a CODE from this list and the sentence beside it is for a
+person. A consumer keys on ``code`` and never reads ``why``. Each code also
+says who failed (``SIDE``): ``run`` when the candidate's run could not be
+measured, ``grader`` when the grader itself could not do its work. A reader of
+many runs counts the two apart, because the first is a fact about the
+submission and the second is a defect to fix on this side.
+
+Adding a code is a change to the contract, so the list is short on purpose and
+every entry names a distinct thing a reader can act on. A code that only one
+kind of evidence can raise belongs with the reader of that evidence and is not
+listed here: the reader declares it (``declare``), as static data when it is
+imported, with its sentence and its side. A code nobody declared is refused
+wherever one is written down.
+"""
+
+from __future__ import annotations
+
+#: Who failed when a grade could not be given.
+RUN = "run"
+GRADER = "grader"
+SIDES = (RUN, GRADER)
+
+#: Every reason a case, or a whole run, can be ungradable. Closed.
+CODES = (
+    "CASE_INVALID",
+    "CASE_NOT_PREPARED",
+    "CASE_NOT_GRADED",
+    "NOTHING_GRADED",
+    "NO_FAIL_TO_PASS",
+    "RUN_NOT_DETERMINISTIC",
+)
+
+#: What each code means, in one sentence, for whoever has to fix it. Nothing
+#: parses these.
+MEANING = {
+    "CASE_INVALID":
+        "the case declaration does not satisfy the case schema, it names a "
+        "file the case does not ship, or two cases carry one id",
+    "CASE_NOT_PREPARED":
+        "the case could not be prepared for the run, so nothing was measured "
+        "for it and this run covers less than the case set holds",
+    "CASE_NOT_GRADED":
+        "the test of a case ended without a result: the reader raised "
+        "something other than a refusal, returned something other than a "
+        "result, or the process running it died, so nothing was measured for "
+        "that case",
+    "NOTHING_GRADED":
+        "no case of the case set was graded, or a case's evidence was "
+        "narrowed until it read nothing, and a case that reads nothing passes "
+        "forever while asserting nothing",
+    "NO_FAIL_TO_PASS":
+        "the case set classifies no fail-to-pass case, so nothing in it tests "
+        "the change that was asked for",
+    "RUN_NOT_DETERMINISTIC":
+        "the same submission run twice from the same start gave two different "
+        "results, so neither run is evidence about the submission",
+}
+
+#: Who failed, per code. The candidate's run could not be measured, or the
+#: grader could not do its work.
+SIDE = {
+    "CASE_INVALID": GRADER,
+    "CASE_NOT_PREPARED": GRADER,
+    "CASE_NOT_GRADED": GRADER,
+    "NOTHING_GRADED": GRADER,
+    "NO_FAIL_TO_PASS": GRADER,
+    "RUN_NOT_DETERMINISTIC": RUN,
+}
+
+
+#: The questions asked of the run rather than of a case, and the code each one
+#: answers with when it does not hold. A reader answers a check per case; the
+#: run holds it only when every case that answered it does.
+CHECKS = {"determinism": "RUN_NOT_DETERMINISTIC"}
+
+#: The codes a reader declared for its own kind of evidence, each
+#: ``{"meaning", "side"}``. Filled by ``declare`` and by nothing else.
+DECLARED: dict = {}
+
+
+def declare(codes: dict) -> None:
+    """Take a reader's own codes in: ``{code: {"meaning": the sentence,
+    "side": who failed}}``. A reader calls this once, where it is imported,
+    over a literal. Declaring the same code the same way again changes
+    nothing, so a module imported twice is no defect; a code of the list
+    above, a second meaning for a declared code, a missing sentence and a side
+    that is neither of the two are each refused."""
+    for code, said in codes.items():
+        if code in CODES:
+            raise ValueError(f"{code!r} is a code of the shared list, and a "
+                             f"reader declares only its own")
+        if set(said) != {"meaning", "side"} or not said["meaning"] \
+                or said["side"] not in SIDES:
+            raise ValueError(f"{code!r} is declared with a sentence and a "
+                             f"side of {', '.join(SIDES)}, got {said!r}")
+        if DECLARED.get(code, said) != said:
+            raise ValueError(f"{code!r} is already declared, and differently")
+    DECLARED.update({code: dict(said) for code, said in codes.items()})
+
+
+def known(code: str) -> bool:
+    """Whether ``code`` is a grading code: one of the shared list, or one a
+    reader declared."""
+    return code in CODES or code in DECLARED
+
+
+def _refuse_unknown(code: str) -> None:
+    if not known(code):
+        raise ValueError(f"{code!r} is no grading code; the set is "
+                         f"{', '.join(CODES + tuple(sorted(DECLARED)))}")
+
+
+def entry(code: str, why: str, case: str | None = None) -> dict:
+    """One refusal as the grade document carries it: the code, the sentence,
+    who failed, and the case where the refusal was about one."""
+    _refuse_unknown(code)
+    out = {"code": code, "why": why,
+           "side": SIDE.get(code) or DECLARED[code]["side"]}
+    if case:
+        out["case"] = case
+    return out
+
+
+class Ungradable(Exception):
+    """Raised where a grade cannot be given, carrying the code a consumer keys
+    on. Every raise site passes a ``why`` that names the case, the file or the
+    field, because the code says what kind of thing went wrong and the sentence
+    says which one."""
+
+    def __init__(self, code: str, why: str):
+        _refuse_unknown(code)
+        super().__init__(why)
+        self.code = code
+        self.why = why
+
+    def __reduce__(self):
+        """Both arguments, so a refusal raised in another process arrives as
+        the refusal it was. The default carries ``args`` alone, which is the
+        sentence and no code."""
+        return (type(self), (self.code, self.why))
+
+    def entry(self, case: str | None = None) -> dict:
+        """The refusal as the grade document carries it."""
+        return entry(self.code, self.why, case)

truereward-0.1.0/truereward/document.py

@@ -0,0 +1,152 @@
+"""The grade document, and the one writer of the files a Harbor verifier
+publishes.
+
+Harbor's rule is that the task produces its own reward: the verifier writes
+``reward.txt`` (one number) and ``reward.json`` (flat and numeric) into its
+logs directory and Harbor reads them. Beside them go ``grade.json``, the whole
+document, which is what every consumer on this side reads, and ``ctrf.json``,
+the per-case detail in CTRF for a person to open.
+
+An ungradable run publishes no reward file. Harbor then records a typed
+missing-reward error for the trial, which it never retries and never scores,
+where a written 0 would read as a fair grade of the submission. The document
+and the CTRF report are written all the same, so the run always leaves a
+well-formed record of why it could not be graded.
+
+Every file is written to a temporary name in the same directory and renamed
+into place, so a reader never opens half a document. ``reward.txt`` goes last:
+once it is there, everything beside it is complete.
+"""
+
+from __future__ import annotations
+
+import json
+import os
+import tempfile
+
+from . import __version__
+from .fold import fold
+from .result import FAIL, PASS
+
+#: Where Harbor reads a task's verdict from.
+LOGS = "/logs/verifier"
+
+#: The environment variable that names another logs directory.
+LOGS_ENV = "GRADE_LOGS"
+
+#: Who folded and wrote, as the document and the CTRF report name it: this
+#: distribution and its version.
+GRADER = {"name": "truereward", "version": __version__}
+
+#: The version of the document this module writes.
+SCHEMA_VERSION = 2
+
+#: The document, field for field. The list is the contract: a reader of it
+#: knows what arrives, and a key added to the fold reaches the document only by
+#: being named here.
+DOCUMENT_KEYS = ("grade", "schema_version", "grader", "gradable", "ungradable",
+                 "reward", "reward_scheme", "guards_hold", "fix_rate",
+                 "case_fraction", "cases", "checks", "detail")
+
+#: The files a gradable run publishes, in the order they are written, and the
+#: two of them an ungradable run must not leave behind.
+FILES = ("grade.json", "ctrf.json", "reward.json", "reward.txt")
+REWARD_FILES = ("reward.json", "reward.txt")
+
+
+def logs() -> str:
+    """The logs directory of this run: Harbor's, or the one ``GRADE_LOGS``
+    names. The writer publishes into it, and a reader that asks its
+    environment to keep the candidate's code out of the grader's files names
+    it among them."""
+    return os.environ.get(LOGS_ENV) or LOGS
+
+
+def grade_document(rows: list, checks: list | tuple = (),
+                   refusals: list | tuple = ()) -> dict:
+    """The grade document over one run.
+
+    ``rows`` is one CaseResult row per case, ungradable cases included.
+    ``checks`` is the run checks, each ``{"check", "code", "holds",
+    "answered", "why"}``. ``refusals`` is every refusal about the run that no
+    row carries. ``detail`` is the rows in id order, so two runs of one case
+    set give one document whatever order the cases finished in.
+    """
+    folded = fold(rows, checks, refusals)
+    doc = {
+        "grade": SCHEMA_VERSION,
+        "schema_version": SCHEMA_VERSION,
+        "grader": dict(GRADER),
+        "checks": [dict(c) for c in checks],
+        "detail": sorted((dict(r) for r in rows), key=lambda r: r["id"]),
+        **folded,
+    }
+    return {k: doc[k] for k in DOCUMENT_KEYS}
+
+
+def ctrf(doc: dict) -> dict:
+    """The per-case report in CTRF: a tool, a summary and one entry per case.
+    A case that could not be graded is neither passed nor failed, and CTRF has
+    the word for it, ``other``. A refusal about the run that no case carries
+    is one more ``other`` entry, named by its code. Each entry says what the
+    case is for and the code it was refused under in ``extra``."""
+    status = {PASS: "passed", FAIL: "failed"}
+    tests = [{"name": row["id"],
+              "status": status.get(row["verdict"], "other"),
+              "duration": int(row["duration_s"] * 1000),
+              "message": row["why"], "suite": row["kind"],
+              "extra": {"kind": row["kind"], "code": row["code"]}}
+             for row in doc["detail"]]
+    tests += [{"name": refusal["code"], "status": "other", "duration": 0,
+               "message": refusal["why"],
+               "extra": {"kind": "", "code": refusal["code"]}}
+              for refusal in doc["ungradable"] if "case" not in refusal]
+    count = [t["status"] for t in tests].count
+    return {"results": {
+        "tool": dict(doc["grader"]),
+        "summary": {"tests": len(tests), "passed": count("passed"),
+                    "failed": count("failed"), "pending": 0, "skipped": 0,
+                    "other": count("other"), "start": 0, "stop": 0},
+        "tests": tests}}
+
+
+def _write(logs: str, name: str, text: str) -> None:
+    """One file, whole or not at all: a temporary name beside it, then a
+    rename, which the file system does in one step."""
+    fd, tmp = tempfile.mkstemp(dir=logs, prefix=f".{name}.")
+    try:
+        with os.fdopen(fd, "w", encoding="utf-8") as fh:
+            fh.write(text)
+        os.replace(tmp, os.path.join(logs, name))
+    except BaseException:
+        os.unlink(tmp)
+        raise
+
+
+def write(doc: dict, logs: str = LOGS) -> list:
+    """Publish one run into ``logs`` and return the names written.
+
+    A gradable run writes all four files. An ungradable run writes the
+    document and the CTRF report, writes no reward file, and removes one a
+    previous run left, so a stale reward never stands beside a document that
+    says there is none. ``reward.json`` is flat and numeric, ``guards_hold`` as
+    1 or 0, because Harbor refuses anything else in it.
+    """
+    os.makedirs(logs, exist_ok=True)
+    texts = {
+        "grade.json": json.dumps(doc, indent=1, sort_keys=True) + "\n",
+        "ctrf.json": json.dumps(ctrf(doc), indent=1) + "\n",
+    }
+    if doc["gradable"]:
+        texts["reward.json"] = json.dumps(
+            {"reward": doc["reward"], "fix_rate": doc["fix_rate"],
+             "guards_hold": int(doc["guards_hold"])}, indent=1) + "\n"
+        texts["reward.txt"] = f"{doc['reward']}\n"
+    else:
+        for name in REWARD_FILES:
+            if os.path.exists(os.path.join(logs, name)):
+                os.unlink(os.path.join(logs, name))
+    written = [name for name in FILES if name in texts]
+    for name in written:
+        _write(logs, name, texts[name])
+    return written

truereward-0.1.0/truereward/fold.py

@@ -0,0 +1,86 @@
+"""The fold: many case results and a few run checks, one reward.
+
+A case set is the behaviour the change must produce (the fail-to-pass cases)
+and the behaviour it must not break (the pass-to-pass cases), which is
+SWE-bench's F2P and P2P. Under ``gated-binary/1`` the reward is 1.0 when every
+pass-to-pass case holds and every fail-to-pass case passes, and 0.0 otherwise.
+Nothing in between: an untouched codebase scores 0 because its fail-to-pass
+cases fail, and a fix that breaks a guard scores 0 because the guard is the
+contract. ``fix_rate`` and ``guards_hold`` are reported beside the reward for
+whoever wants a dense signal, and are never the grade.
+
+A case that could not be graded is never counted as a failed one. It is left
+out of the arithmetic and makes the whole run ungradable, as does a run check
+that does not hold, a run that graded no case, and a case set with no
+fail-to-pass case. An ungradable run has no reward: ``reward`` is None, never
+0, so an instrument failure cannot read as a fair grade. The checks and the
+cases are read in this one function, so a run that failed a check cannot
+publish 1.0 from its case rows.
+
+This is the only implementation of the rule. A consumer that has to recompute
+a reward from the rows of a document calls this fold.
+"""
+
+from __future__ import annotations
+
+from .codes import entry
+from .result import FAIL_TO_PASS, PASS, PASS_TO_PASS, UNGRADABLE
+
+#: How this reward is named wherever it is published.
+SCHEME = "gated-binary/1"
+
+
+def fold(rows: list, checks: list | tuple = (),
+         refusals: list | tuple = ()) -> dict:
+    """``rows`` is one CaseResult row per case, of which ``id``, ``kind``,
+    ``verdict``, ``code`` and ``why`` are read. ``checks`` is the run checks,
+    each ``{"check", "code", "holds", "why"}``. ``refusals`` is every refusal
+    about the run that no row carries, each as codes.entry builds it.
+
+    Returns ``gradable``, the full ``ungradable`` list, ``reward`` (None when
+    not gradable), ``reward_scheme``, ``guards_hold``, ``fix_rate``,
+    ``case_fraction``, the counts split by fail-to-pass and pass-to-pass, and
+    the ids that failed. The counts and the rates are over the cases that were
+    graded, whether or not the run is gradable.
+    """
+    graded = [r for r in rows if r["verdict"] != UNGRADABLE]
+    refused = [entry(r["code"], r["why"], r["id"])
+               for r in rows if r["verdict"] == UNGRADABLE]
+    refused += [entry(r["code"], r["why"], r.get("case")) for r in refusals]
+    refused += [entry(c["code"], c["why"]) for c in checks if not c["holds"]]
+    if not graded:
+        # Only when nothing has said why already: a run whose cases each
+        # refused has its reasons, and repeating the code over them adds no
+        # fact.
+        if not refused:
+            refused.append(entry("NOTHING_GRADED",
+                                 "no case of this case set was graded on "
+                                 "this run"))
+    elif not any(r["kind"] == FAIL_TO_PASS for r in rows):
+        refused.append(entry("NO_FAIL_TO_PASS",
+                             "this case set classifies no fail-to-pass case, "
+                             "so nothing in it tests the change that was "
+                             "asked for"))
+
+    fix = [r for r in graded if r["kind"] == FAIL_TO_PASS]
+    guards = [r for r in graded if r["kind"] == PASS_TO_PASS]
+    fix_pass = sum(1 for r in fix if r["verdict"] == PASS)
+    guard_pass = sum(1 for r in guards if r["verdict"] == PASS)
+    guards_hold = guard_pass == len(guards)
+    total = len(graded)
+    gradable = not refused
+    reward = 1.0 if guards_hold and fix_pass == len(fix) else 0.0
+    fix_rate = (fix_pass / len(fix)) if (fix and guards_hold) else 0.0
+    return {
+        "gradable": gradable,
+        "ungradable": refused,
+        "reward": reward if gradable else None,
+        "reward_scheme": SCHEME,
+        "guards_hold": guards_hold,
+        "fix_rate": round(fix_rate, 4),
+        "case_fraction": round((fix_pass + guard_pass) / total, 4) if total else 0.0,
+        "cases": {"total": total, "passed": fix_pass + guard_pass,
+                  "fail_to_pass": {"total": len(fix), "passed": fix_pass},
+                  "pass_to_pass": {"total": len(guards), "passed": guard_pass}},
+        "failed": [r["id"] for r in graded if r["verdict"] != PASS],
+    }

truereward-0.1.0/truereward/plugin.py

@@ -0,0 +1,134 @@
+"""The pytest plugin: one test per case, one fold per run, in parallel or not.
+
+A task's tests are one test per case, and a reward is one number over the
+whole case set, so something has to fold the cases after the last one has run.
+Under pytest-xdist the cases run in worker processes, and a ledger kept in
+memory would give every worker its own partial reward. So nothing is kept in
+the process that ran the case: the ``grade`` fixture attaches the case's row
+to the test report (``user_properties``, the channel pytest and xdist already
+carry between processes), and the controller, which is the only process that
+sees every report, collects the rows and folds and writes once at the end of
+the session. Without xdist the one process is its own controller and the path
+is the same. A worker never writes a file.
+
+The plugin is never active on its own: no entry point names it. A verifier
+names it on the command line, ``pytest -p <this module>``, which is read by
+the controller and by every worker. A ``pytest_plugins`` line in the task's
+test file is not enough: under xdist the controller never imports the test
+file, so it would collect no row and publish nothing. The task's test file
+stays a few lines::
+
+    @pytest.mark.parametrize("case", cases(CASES), ids=lambda c: c.id)
+    def test_case(case, grade):
+        grade(case, grade_case)
+
+where ``cases`` and ``grade_case(case) -> CaseResult`` are the reader's. The
+logs directory is Harbor's, or the one ``GRADE_LOGS`` names.
+
+A reader that declares codes of its own (codes.declare) is named on the
+command line too, for the same reason: the controller reads every row, a row
+is refused unless its code is declared in the process reading it, and the
+controller never imports the test file that imports the reader. Such a reader
+carries a ``pytest_plugins`` line naming this module, so the verifier names
+the reader and gets both. A row whose code the controller does not know is a
+refusal about that case, never a crash of the run.
+"""
+
+from __future__ import annotations
+
+import pytest
+
+from .document import logs, write
+from .result import PASS
+from .session import Session, grade_one
+
+#: The name a case's row travels under in a test report's user properties.
+ROW = "grade_row"
+
+#: The ways a session ends before its last test has run.
+STOPPED_EARLY = (pytest.ExitCode.INTERRUPTED, pytest.ExitCode.INTERNAL_ERROR)
+
+
+@pytest.fixture
+def grade(request):
+    """``grade(case, reader)``: grade one case, attach its row to this test's
+    report, and fail the test unless the case passed. A refusal is attached
+    like any other row and then re-raised, so the case's own test fails and
+    the run is ungradable rather than quietly short."""
+    def run(case, reader) -> None:
+        result, raised = grade_one(case, reader)
+        request.node.user_properties.append((ROW, result.to_row()))
+        if raised is not None:
+            raise raised
+        assert result.verdict == PASS, result.why or result.code
+    return run
+
+
+class Controller:
+    """The one process that sees every report. It is registered where there
+    is no ``workerinput``, which is how xdist marks a worker."""
+
+    def __init__(self, config):
+        self.config = config
+        self.session = Session()
+        self.collected: set = set()
+        self.seen: set = set()
+        self.ended_badly: dict = {}
+
+    def pytest_collection_finish(self, session) -> None:
+        """The case set of a run in one process: every test it collected."""
+        self.collected.update(item.nodeid for item in session.items)
+
+    @pytest.hookimpl(optionalhook=True)
+    def pytest_xdist_node_collection_finished(self, node, ids) -> None:
+        """The case set of a run under xdist, where the workers collect and
+        each tells the controller what it found."""
+        self.collected.update(ids)
+
+    def pytest_runtest_logreport(self, report) -> None:
+        """Take the row off a report. A test's set-up, call and tear-down
+        reports all carry its properties, so a test is read once. A test that
+        did not pass and carried no row is remembered, to be refused at the
+        end."""
+        for name, value in report.user_properties:
+            if name == ROW and report.nodeid not in self.seen:
+                self.seen.add(report.nodeid)
+                try:
+                    self.session.add(value)
+                except ValueError as error:
+                    self.session.lost(report.nodeid, f"its row was refused "
+                                                     f"here: {error}")
+        if not report.passed:
+            self.ended_badly.setdefault(
+                report.nodeid, f"the test {report.outcome} in {report.when} "
+                               f"and left no result")
+
+    def pytest_sessionfinish(self, session, exitstatus) -> None:
+        """Fold and write, once, after every case of this run has reported. A
+        run that was stopped before its last case (an interrupt, an internal
+        error, ``-x``) graded part of a case set, and says so."""
+        for nodeid, why in self.ended_badly.items():
+            if nodeid not in self.seen:
+                self.session.lost(nodeid, why)
+        for nodeid in sorted(self.collected - self.seen - set(self.ended_badly)):
+            self.session.lost(nodeid, "the test was collected and left no result")
+        chosen = [f"{flag} {value}" for flag, value in (
+            ("-k", self.config.getoption("keyword", "")),
+            ("-m", self.config.getoption("markexpr", "")),
+            ("--deselect", self.config.getoption("deselect", None))) if value]
+        if chosen:
+            self.session.lost(
+                "session", f"the run chose its cases ({', '.join(chosen)}), "
+                           f"and a reward is over the whole case set")
+        stopped = session.shouldfail or session.shouldstop
+        if stopped or exitstatus in STOPPED_EARLY:
+            self.session.lost(
+                "session", f"the run stopped early ({stopped or exitstatus!r}), "
+                           f"so cases of the set may never have run")
+        write(self.session.document(), logs())
+
+
+def pytest_configure(config) -> None:
+    """Register the controller in the controlling process alone."""
+    if not hasattr(config, "workerinput"):
+        config.pluginmanager.register(Controller(config), "grade-controller")

truereward-0.1.0/truereward/result.py

@@ -0,0 +1,119 @@
+"""The one record every kind of evidence reduces to: the result of one case.
+
+A case is graded by a reader that knows its kind of evidence: a recorded end
+state, a test report, a list of criteria, a measured figure. Whatever the
+reader looked at, it answers with this record, and everything after it (the
+fold, the grade document, the files a verifier publishes) reads this record
+and nothing else. A new kind of evidence is a new reader, never a new record.
+
+``to_row`` and ``from_row`` carry the record as a plain JSON dict, because a
+row crosses a process boundary whenever the cases run in parallel.
+"""
+
+from __future__ import annotations
+
+from dataclasses import dataclass, field
+
+from .codes import CHECKS, known
+
+#: What a case is for. A fail-to-pass case is behaviour the change must
+#: produce; a pass-to-pass case is behaviour it must not break.
+FAIL_TO_PASS = "fail_to_pass"
+PASS_TO_PASS = "pass_to_pass"
+KINDS = (FAIL_TO_PASS, PASS_TO_PASS)
+
+#: How a case came out. ``ungradable`` is neither of the other two: nothing
+#: was measured, and the code says why.
+PASS = "pass"
+FAIL = "fail"
+UNGRADABLE = "ungradable"
+VERDICTS = (PASS, FAIL, UNGRADABLE)
+
+#: One entry of ``observations``, field for field.
+OBSERVATION_KEYS = ("name", "verdict", "why")
+
+#: One entry of ``checks``, field for field: the run check this case answered,
+#: whether it held, and the sentence when it did not.
+CHECK_KEYS = ("check", "holds", "why")
+
+
+@dataclass(frozen=True)
+class CaseResult:
+    """One case, graded.
+
+    ``code`` is set exactly when the verdict is ``ungradable``, and is a code
+    of codes.CODES or one a reader declared. ``why`` is the sentence a person reads: the rule the case
+    failed on, or which thing could not be measured. ``observations`` is one
+    entry per thing the reader looked at, each ``{name, verdict, why}`` in the
+    reader's own vocabulary; ``measures`` is every figure it took, by name.
+    Neither enters the reward: the fold reads ``kind`` and ``verdict`` alone.
+    ``checks`` is this case's answer to each run check it could answer (one of
+    codes.CHECKS), each ``{check, holds, why}``. A check is a question about
+    the run and not about the case, so it moves no verdict here: the ledger
+    gathers the answers of every case and the fold reads them once.
+    ``containment`` is what the environment's report said about how this
+    case's run was kept from the grader's files, in the environment's own
+    words; None where the reader ran nothing or asked for none.
+    """
+
+    id: str
+    kind: str
+    verdict: str
+    code: str = ""
+    why: str = ""
+    observations: list = field(default_factory=list)
+    measures: dict = field(default_factory=dict)
+    checks: list = field(default_factory=list)
+    containment: dict | None = None
+    duration_s: float = 0.0
+
+    def __post_init__(self):
+        if not self.id or not isinstance(self.id, str):
+            raise ValueError(f"a case result needs an id, got {self.id!r}")
+        if self.kind not in KINDS:
+            raise ValueError(f"{self.id}: kind {self.kind!r} is not one of "
+                             f"{', '.join(KINDS)}")
+        if self.verdict not in VERDICTS:
+            raise ValueError(f"{self.id}: verdict {self.verdict!r} is not one "
+                             f"of {', '.join(VERDICTS)}")
+        if (self.verdict == UNGRADABLE) != bool(self.code):
+            raise ValueError(f"{self.id}: a code goes with the verdict "
+                             f"{UNGRADABLE!r} and with no other")
+        if self.code and not known(self.code):
+            raise ValueError(f"{self.id}: {self.code!r} is no grading code")
+        for seen in self.observations:
+            if not isinstance(seen, dict) or set(seen) != set(OBSERVATION_KEYS):
+                raise ValueError(f"{self.id}: an observation is "
+                                 f"{{name, verdict, why}}, got {seen!r}")
+        for answer in self.checks:
+            if not isinstance(answer, dict) or set(answer) != set(CHECK_KEYS) \
+                    or answer["check"] not in CHECKS \
+                    or not isinstance(answer["holds"], bool):
+                raise ValueError(f"{self.id}: a check is {{check, holds, "
+                                 f"why}} over one of {', '.join(CHECKS)}, "
+                                 f"got {answer!r}")
+        if self.containment is not None \
+                and not isinstance(self.containment, dict):
+            raise ValueError(f"{self.id}: containment is what a report said, "
+                             f"an object, got {self.containment!r}")
+        for name, figure in self.measures.items():
+            if isinstance(figure, bool) or not isinstance(figure, (int, float)):
+                raise ValueError(f"{self.id}: measure {name!r} is not a "
+                                 f"number: {figure!r}")
+
+    def to_row(self) -> dict:
+        """The record as a plain JSON dict."""
+        return {"id": self.id, "kind": self.kind, "verdict": self.verdict,
+                "code": self.code, "why": self.why,
+                "observations": [dict(seen) for seen in self.observations],
+                "measures": dict(self.measures),
+                "checks": [dict(answer) for answer in self.checks],
+                "containment": (None if self.containment is None
+                                else dict(self.containment)),
+                "duration_s": round(float(self.duration_s), 3)}
+
+    @classmethod
+    def from_row(cls, row: dict) -> CaseResult:
+        """The record a row carries. A row with a key this record does not
+        have is refused, so a field never rides along unread."""
+        return cls(**row)

truereward-0.1.0/truereward/session.py

@@ -0,0 +1,98 @@
+"""One graded run: the cases as they come in, and the one document they fold
+into.
+
+The reward is folded once, over the whole case set, and a test of one case
+cannot see the set. So each test grades its case into a row (``grade_one``),
+and one ledger per run (``Session``) takes the rows in, in whatever order and
+from whatever process they arrive, and builds the document at the end. Nothing
+here knows pytest; plugin.py is the part that does.
+
+A case that ends without a row is never dropped. The fold would otherwise run
+over the cases that are left and could publish 1.0 for a case set it only
+partly graded. A reader that raises, a reader that returns something else, and
+a test that died before its row arrived each become a refusal.
+"""
+
+from __future__ import annotations
+
+import dataclasses
+import time
+
+from .codes import CHECKS, Ungradable, entry
+from .document import grade_document
+from .result import UNGRADABLE, CaseResult
+
+
+def grade_one(case, reader) -> tuple[CaseResult, BaseException | None]:
+    """Grade one case through ``reader`` and time it.
+
+    ``case`` is whatever the reader discovers its cases as; it carries ``id``
+    and ``kind``. ``reader(case)`` returns a CaseResult, or raises Ungradable
+    where the case cannot be graded. Returns the result and whatever the
+    reader raised, so the caller records the first and re-raises the second.
+    A reader that raised anything else, or answered with no result for this
+    case, leaves an ungradable result under CASE_NOT_GRADED.
+    """
+    started = time.monotonic()
+    raised = None
+    try:
+        result = reader(case)
+        if not isinstance(result, CaseResult) or result.id != case.id:
+            raise TypeError(f"the reader answered {result!r}, which is no "
+                            f"result for case {case.id}")
+    except Ungradable as refusal:
+        raised = refusal
+        result = CaseResult(case.id, case.kind, UNGRADABLE, refusal.code,
+                            refusal.why)
+    except Exception as error:  # noqa: BLE001
+        raised = error
+        result = CaseResult(case.id, case.kind, UNGRADABLE, "CASE_NOT_GRADED",
+                            f"{case.id}: {type(error).__name__}: {error}")
+    return dataclasses.replace(
+        result, duration_s=time.monotonic() - started), raised
+
+
+class Session:
+    """The ledger of one run: every row that arrived, every refusal no row
+    carries, and the document they fold into."""
+
+    def __init__(self):
+        self.rows: dict = {}
+        self.refusals: list = []
+
+    def add(self, row: dict) -> None:
+        """Take one case's row in. A second row under one id is a defect of
+        the case set, and is refused rather than allowed to replace the
+        first."""
+        result = CaseResult.from_row(row)
+        if result.id in self.rows:
+            self.refusals.append(entry(
+                "CASE_INVALID", f"two cases carry the id {result.id}",
+                result.id))
+            return
+        self.rows[result.id] = result.to_row()
+
+    def lost(self, name: str, why: str) -> None:
+        """Record a test that ended badly with no row behind it."""
+        self.refusals.append(entry("CASE_NOT_GRADED", f"{name}: {why}", name))
+
+    def checks(self) -> list:
+        """Every run check, answered over the whole run: it holds when every
+        case that answered it says so. A check that no case answered is not a
+        failing check: the run that graded no case is refused by the fold on
+        its own terms, and borrowing this code for it would make the code mean
+        two things."""
+        out = []
+        for name, code in CHECKS.items():
+            answers = [answer for row in self.rows.values()
+                       for answer in row["checks"] if answer["check"] == name]
+            failed = [answer for answer in answers if not answer["holds"]]
+            out.append({"check": name, "code": code, "holds": not failed,
+                        "answered": len(answers),
+                        "why": " ".join(answer["why"] for answer in failed)})
+        return out
+
+    def document(self) -> dict:
+        """The grade document over everything that arrived, folded once."""
+        return grade_document(list(self.rows.values()), self.checks(),
+                              self.refusals)

truereward-0.1.0/truereward.egg-info/PKG-INFO

@@ -0,0 +1,51 @@
+Metadata-Version: 2.4
+Name: truereward
+Version: 0.1.0
+Summary: One case result, one fold, one grade document: the reward of a graded run, or the reason there is none.
+Author: truereward contributors
+License-Expression: Apache-2.0
+Requires-Python: >=3.11
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Provides-Extra: pytest
+Requires-Dist: pytest>=8; extra == "pytest"
+Requires-Dist: pytest-xdist>=3.6; extra == "pytest"
+Dynamic: license-file
+
+# truereward
+
+Turn the cases of a graded run into one reward, or into the reason there is
+none.
+
+A grader looks at some kind of evidence: a recorded end state, a test report,
+a list of criteria, a measured figure. Whatever it looked at, it answers with
+one record per case, `CaseResult`. This package is everything after that
+record, and it is the same for every kind of evidence.
+
+- **The fold.** `fold(rows)` applies `gated-binary/1`: the reward is 1.0 when
+  every pass-to-pass case holds and every fail-to-pass case passes, and 0.0
+  otherwise. `fix_rate` and `guards_hold` are reported beside it.
+- **Three outcomes.** A case passed, failed, or could not be graded. The third
+  carries a code from a closed list, each with who failed: the run, or the
+  grader. An ungradable run has no reward. It is never scored 0.
+- **The document.** `grade_document` builds `grade.json`; `write` publishes it
+  with `ctrf.json`, `reward.json` and `reward.txt` into a verifier's logs
+  directory, each file whole or not at all. An ungradable run publishes no
+  reward file.
+- **The pytest plugin.** One test per case, folded once at the end of the
+  session, in one process or under pytest-xdist:
+
+  ```python
+  @pytest.mark.parametrize("case", cases(CASES), ids=lambda c: c.id)
+  def test_case(case, grade):
+      grade(case, grade_case)
+  ```
+
+  Run it with `pytest -p truereward.plugin`. Install the `pytest` extra for
+  it: `pip install "truereward[pytest]"`. Folding and writing need the
+  standard library alone.
+
+A reader for one kind of evidence is a separate distribution. It may declare
+refusal codes of its own with `truereward.codes.declare`.
+
+Licensed under Apache-2.0.

truereward-0.1.0/truereward.egg-info/SOURCES.txt

@@ -0,0 +1,16 @@
+LICENSE
+MANIFEST.in
+README.md
+pyproject.toml
+truereward/__init__.py
+truereward/codes.py
+truereward/document.py
+truereward/fold.py
+truereward/plugin.py
+truereward/result.py
+truereward/session.py
+truereward.egg-info/PKG-INFO
+truereward.egg-info/SOURCES.txt
+truereward.egg-info/dependency_links.txt
+truereward.egg-info/requires.txt
+truereward.egg-info/top_level.txt

truereward-0.1.0/truereward.egg-info/dependency_links.txt

@@ -0,0 +1 @@
+

truereward-0.1.0/truereward.egg-info/requires.txt

@@ -0,0 +1,4 @@
+
+[pytest]
+pytest>=8
+pytest-xdist>=3.6

truereward-0.1.0/truereward.egg-info/top_level.txt

@@ -0,0 +1 @@
+truereward