entrylib-rubric 0.4.0__py3-none-any.whl

entrylib_rubric-0.4.0.dist-info/METADATA

@@ -0,0 +1,30 @@
+Metadata-Version: 2.4
+Name: entrylib-rubric
+Version: 0.4.0
+Summary: EntryLib's scoring rubric: reliability scores of claimsets, computed by public rules.
+License: Apache-2.0
+Project-URL: Homepage, https://github.com/entrylib/rubric
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Dynamic: license-file
+
+# EntryLib scoring rubric
+
+How [EntryLib](https://entrylib.org) scores the claimsets it publishes: a **reliability** score for every claim and for the paper, and a **novelty** label with a close-match check. The rules are public so that anyone can see why a claimset scored as it did, and what would raise it.
+
+- [RUBRIC.md](https://github.com/entrylib/rubric/blob/main/RUBRIC.md): the rubric, normative and versioned.
+- `src/rubric/`: the scorer that implements it (Python 3.10+, standard library only); the costs are in `src/rubric/rubric.json`.
+
+```python
+import json, claimset.validate, rubric
+doc = json.load(open("claimset.json"))                    # a published claimset file
+result = rubric.score(doc, warnings=claimset.validate.validate(doc, published=True).warnings)  # warnings optional
+result["paper"]["score"], result["claims"]["C1"]["dots"]
+```
+
+`rubric.report(result)` gives the scores as plain text: every issue that lowers a score and what would fix it. Before submitting, `claimset score paper.claimset.json` ([claimset](https://github.com/entrylib/claimset)) scores a claimset this way on your own machine, as it would be scored once published.
+
+Install: `pip install git+https://github.com/entrylib/rubric`. The portal installs it the same way and records the rubric version with every score.
+
+Planned: an AI reviewer ([#1](https://github.com/entrylib/rubric/issues/1)) and reproduction from artefacts ([#2](https://github.com/entrylib/rubric/issues/2)).

entrylib_rubric-0.4.0.dist-info/RECORD

@@ -0,0 +1,10 @@
+entrylib_rubric-0.4.0.dist-info/licenses/LICENSE,sha256=xx0jnfkXJvxRnG63LTGOxlggYnIysveWIZ6H3PNdCrQ,11357
+rubric/__init__.py,sha256=zzhQy_saVoRptJqv--h04Q7CU0pCfgKYTGx3qur_3Gs,1251
+rubric/novelty.py,sha256=eCC9L8793VUPZLjTfLKnjv1_WRot1Wr_2HmuYWs_PvQ,4794
+rubric/report.py,sha256=v_45iZZjKy5V-chsCmwNQKf1pg7idNMxRgYh8dhtUjs,4554
+rubric/rubric.json,sha256=8ETVWMfJOniY5EcP7ZlToL12a7nIE2Oe_cEoUVOF0q4,5249
+rubric/score.py,sha256=Or6SW2hA56bitgF9rPrcL57BxLZjlD988pkpc6EzFc4,19405
+entrylib_rubric-0.4.0.dist-info/METADATA,sha256=e50xvBloOqGGU77aM9Kk30vWtbBcLPXQhHcFGZW_A_M,1817
+entrylib_rubric-0.4.0.dist-info/WHEEL,sha256=mOdXR9C3wMVov7cSY8PY-XCzo3SmmWCHvGQr5csJoao,91
+entrylib_rubric-0.4.0.dist-info/top_level.txt,sha256=PB2Y0f8oFtUr8uA8PkbDIgEHmZsqJvCBG1zeQtm_4EI,7
+entrylib_rubric-0.4.0.dist-info/RECORD,,

entrylib_rubric-0.4.0.dist-info/WHEEL

@@ -0,0 +1,5 @@
+Wheel-Version: 1.0
+Generator: setuptools (84.2.1)
+Root-Is-Purelib: true
+Tag: py3-none-any
+

entrylib_rubric-0.4.0.dist-info/licenses/LICENSE

@@ -0,0 +1,201 @@
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright [yyyy] [name of copyright owner]
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.

entrylib_rubric-0.4.0.dist-info/top_level.txt

@@ -0,0 +1 @@
+rubric

rubric/__init__.py

@@ -0,0 +1,21 @@
+"""EntryLib's scoring rubric (RUBRIC.md): reliability scores of a claimset's claims and of the paper, computed by rules
+from the published claimset file, and the novelty close-match check.
+
+    score(doc, warnings=None)     -> the scores of a claimset: claims, paper, problems (see score.py); warnings
+                                     are the claimset validator's for the file (its validation report)
+    assess(doc, find=None, warnings=None)
+                                  -> score() with novelty, comparing principal claims with what find() returns
+    novelty(claim, matches, ...)  -> a claim's novelty label and close-match result (see novelty.py)
+    report(scores, biggest_n=0)   -> the scores as a plain-text report (see report.py)
+    similarity(a, b)              -> how closely two claim statements match, 0 to 1
+    stems(statement)              -> the stems similarity compares
+    VERSION                       -> the rubric version every score records
+
+Standard library only, Python 3.10+.
+"""
+
+from .novelty import assess, novelty, similarity, stems
+from .report import report
+from .score import RUBRIC, VERSION, band, score
+
+__all__ = ["RUBRIC", "assess", "VERSION", "band", "novelty", "report", "score", "similarity", "stems"]

rubric/novelty.py

@@ -0,0 +1,97 @@
+"""Novelty in this version: the authors' label and the close-match check (RUBRIC.md §6).
+
+similarity(a, b) -> 0..1, the overlap of the stems of two claim statements
+novelty(claim, matches, concept=None) -> {"label", "check", "matches"}
+
+`matches` are the claims of claimsets published on EntryLib before this one that the site found as candidates, each
+{"claim": "entrylib:2026.000014#C3", "statement": "..."}; the site finds candidates (by full-text search), this module
+decides which match closely and whether the claim acknowledges them.
+"""
+
+import re
+
+CLOSE = 0.6
+
+STOP = set("""
+a about above after again against all also an and any are as at be because been before being below between both but by
+can could did do does doing down during each either few for from further had has have having here how if in into is it
+its itself just less more most much must no nor not of off on once only or other our out over own same shall should so
+some such than that the their them then there these they this those through to too under until up upon very was we were
+what when where whether which while who whom why will with within without would yet
+""".split())
+
+# Endings removed, longest first, when at least three letters remain (RUBRIC.md §6).
+ENDINGS = ("ational", "ations", "ation", "ments", "ment", "ings", "ing", "ies", "ied", "ed", "es", "ly", "s")
+
+TERM = re.compile(r"\[\[[^\]:]+:([^\]]*)\]\]")
+MATH = re.compile(r"\$\$.+?\$\$|\$[^$]+\$|\\\(.+?\\\)", re.S)
+WORD = re.compile(r"[^\W_]+")
+
+
+def _stem(w):
+    for e in ENDINGS:
+        if w.endswith(e) and len(w) - len(e) >= 3:
+            return w[: -len(e)]
+    return w
+
+
+def stems(statement):
+    """The set of stems of a statement: terms as their text, mathematics removed, stop words dropped."""
+    text = MATH.sub(" ", TERM.sub(lambda m: m.group(1), statement if isinstance(statement, str) else ""))
+    return {_stem(w) for w in WORD.findall(text.lower()) if w not in STOP}
+
+
+def similarity(a, b):
+    """Overlap coefficient of the stems of two statements: shared stems over those of the shorter one."""
+    sa, sb = stems(a), stems(b)
+    if not sa or not sb:
+        return 0.0
+    return len(sa & sb) / min(len(sa), len(sb))
+
+
+def _concept(global_id):
+    """entrylib:2026.000014@v2#C3 -> entrylib:2026.000014"""
+    return re.split(r"[@#]", global_id, maxsplit=1)[0] if isinstance(global_id, str) else ""
+
+
+def novelty(claim, matches, concept=None):
+    """The claim's novelty label and the close-match check: check is "none" (no close match on EntryLib),
+    "acknowledged" (close matches, all related to by the claim) or "unacknowledged"; matches lists the close ones,
+    each with its similarity and whether it is acknowledged. Matches from the claim's own claimset (`concept`) are
+    ignored."""
+    label = claim.get("novelty")
+    targets = {r.get("target") for r in claim.get("relations") or [] if isinstance(r, dict)}
+    targets |= {_concept(t) for t in targets}
+    close = []
+    for m in matches or []:
+        gid = m.get("claim") if isinstance(m, dict) else None
+        if not isinstance(gid, str) or (concept and _concept(gid) == concept):
+            continue
+        s = similarity(claim.get("statement"), m.get("statement"))
+        if s >= CLOSE:
+            close.append({"claim": gid, "similarity": round(s, 2),
+                          "acknowledged": gid in targets or _concept(gid) in targets})
+    close.sort(key=lambda m: -m["similarity"])
+    check = ("none" if not close else "acknowledged" if all(m["acknowledged"] for m in close)
+             else "unacknowledged")
+    return {"label": label if label != "not_applicable" else None, "check": check, "matches": close}
+
+
+def assess(doc, find=None, warnings=None):
+    """score(doc) with `novelty`: {claim id: novelty() + `checked`} for every scored claim that has a novelty label.
+    Each principal claim is compared with the published claims that find(statement) returns ([{"claim", "statement"}],
+    supplied by the caller: the portal's search, or EntryLib's public search); `checked` says whether it was. Without
+    `find` nothing is compared and every `checked` is false. `warnings` are the claimset validator's, as for score()."""
+    from .score import score
+    result = score(doc, warnings)
+    concept = (doc.get("paper") or {}).get("id") if isinstance(doc.get("paper"), dict) else None
+    novelty_of = {}
+    for c in doc.get("claims") or []:
+        if not isinstance(c, dict) or c.get("id") not in result["claims"] or not c.get("novelty"):
+            continue
+        compared = find is not None and c.get("role") == "principal"
+        n = novelty(c, find(c.get("statement")) if compared else [], concept)
+        n["checked"] = compared
+        novelty_of[c["id"]] = n
+    result["novelty"] = novelty_of
+    return result

rubric/report.py

@@ -0,0 +1,84 @@
+"""The scores as a plain-text report (RUBRIC.md §4.3), for the authors and their agents: of a published claimset (the
+portal's issues.txt) or of one not yet submitted (`claimset score`).
+
+    report(scores, biggest_n=0) -> the report; `biggest_n` > 0 adds the problems that cost one claim the most
+    open_problems(scores)     -> [(id, problem, {claim: cost})] of the problems that lower a score, in rubric order
+    biggest(scores, n)        -> the n problems that cost one claim the most, most costly first
+
+`scores` is what score() or assess() returns; assess() adds `novelty`, and only claims it compared (`checked`) can be
+in question.
+"""
+
+
+def _sentence(words):
+    words = str(words)
+    return words if words.rstrip().endswith((".", "!", "?")) else words + "."
+
+
+def _lowered(scores, pid, cid, cost):
+    """"C5 (-10)", or "C5 (-10, from C3)" for a problem the claim inherits from what it rests on (§2)."""
+    source = ((scores.get("claims") or {}).get(cid) or {}).get("inherited", {}).get(pid)
+    return f"{cid} (-{cost}, from {', '.join(source)})" if source else f"{cid} (-{cost})"
+
+
+def open_problems(scores):
+    """[(id, problem, {claim id: cost})] of every problem that lowers some claim's score, in the order of score()
+    (those of the principal claims first, most costly first)."""
+    out = []
+    for pid, p in (scores.get("problems") or {}).items():
+        lowers = {cid: c["costs"][pid] for cid, c in (scores.get("claims") or {}).items() if pid in c["problems"]}
+        if lowers:
+            out.append((pid, p, lowers))
+    return out
+
+
+def biggest(scores, n=3):
+    """The n problems that cost one claim the most, most costly first (ties keep the rubric's order)."""
+    return sorted(open_problems(scores), key=lambda x: -max(x[2].values()))[:n]
+
+
+def report(scores, biggest_n=0, name="reliability"):
+    """The report of RUBRIC.md §4.3, as text ending in a newline. `name`: what the score is called, for a site that
+    shows it under another name (§4.3)."""
+    out = []
+    paper = scores.get("paper")
+    if paper and "lowest_score" in paper:                       # rubric 0.3: the mean, with the lowest shown
+        n = len(paper.get("principal") or [])
+        out.append(f"PAPER: {name} {paper['score']} ({paper['band']}), the mean of its {n} principal claim"
+                   f"{'s' if n != 1 else ''}; the lowest is {paper['lowest']} at {paper['lowest_score']}.")
+    elif paper:
+        out.append(f"PAPER: {name} {paper['score']} ({paper['band']}), the score of its lowest principal claim, "
+                   f"{paper['lowest']}. Raising the lowest principal claim raises the paper.")
+    else:
+        out.append("PAPER: no score (no principal claim is scored).")
+    out.append("")
+    top = biggest(scores, biggest_n) if biggest_n else []
+    if top:
+        out.append("BIGGEST ISSUES, most costly first:")
+        out += [f"  {pid} ({', '.join(f'{c} -{cost}' for c, cost in lowers.items())}): {_sentence(p['text'])} "
+                f"To fix: {p['fix']}" for pid, p, lowers in top]
+        out.append("")
+    flagged = {cid: n for cid, n in (scores.get("novelty") or {}).items()
+               if isinstance(n, dict) and n.get("check") == "unacknowledged" and n.get("checked", True)}
+    if flagged:
+        out.append("NOVELTY IN QUESTION: close matches on EntryLib that the claim does not relate to:")
+        out += [f"  {cid}: " + ", ".join(f"{m['claim']} (similarity {m['similarity']})" for m in n["matches"]
+                                        if not m["acknowledged"]) for cid, n in flagged.items()]
+        out.append("")
+    problems = open_problems(scores)
+    if not problems:
+        out.append("No issues: every scored claim keeps all its points.")
+        return "\n".join(out) + "\n"
+    out.append("ISSUES, those of the principal claims first, most costly first:")
+    for pid, p, lowers in problems:
+        out += ["", f"{pid}. {_sentence(p['text'])}",
+                f"  Check: {p['check']} (group: {p['group']}); costs up to {p['cost']} points.",
+                f"  Lowers: {', '.join(_lowered(scores, pid, c, cost) for c, cost in lowers.items())}."]
+        if p.get("objects"):
+            out.append(f"  Concerns: {', '.join(p['objects'])}.")
+        out.append(f"  To fix: {p['fix']}")
+    out += ["", "CLAIMS: score (band): issues"]
+    for cid, c in scores["claims"].items():
+        issues = ", ".join(f"{p} (-{c['costs'][p]})" for p in c["problems"]) or "none"
+        out.append(f"{cid}: {c['score']} ({c['band']}): {issues}")
+    return "\n".join(out) + "\n"

rubric/rubric.json

@@ -0,0 +1,101 @@
+{
+  "version": "0.4",
+  "bands": [
+    {"name": "high", "label": "High", "min": 80},
+    {"name": "medium", "label": "Medium", "min": 60},
+    {"name": "low", "label": "Low", "min": 0}
+  ],
+  "groups": [
+    {"name": "support", "label": "Support", "about": "what the claim rests on: evidence, proof, citation, argument or judgement"},
+    {"name": "plan", "label": "Plan", "about": "whether the test was planned before it was run, and how the plan was anchored"},
+    {"name": "checkable", "label": "Checkable", "about": "whether a reader can obtain what is needed to check the claim"},
+    {"name": "context", "label": "Context", "about": "where the claim holds, what earlier work it relates to, and which recorded results its numbers come from"},
+    {"name": "open", "label": "Open problems", "about": "limitations and assumptions that are still open"}
+  ],
+  "dots": {"green_max": 2, "amber_max": 10},
+  "checks": {
+    "support_kind": {
+      "group": "support", "count": "kind",
+      "costs": {"evidence": 0, "evidence_unspecified": 10, "formal_checked": 0, "formal_unchecked": 25,
+                "formal_disputed": 45, "cited": 20, "consensus": 25, "argument_from_claims": 10, "argument": 30,
+                "expert_judgement": 45, "unstated": 70},
+      "text": "Rests on {what}",
+      "fix": "Support the claim with evidence the claimset records, citing the result entries or table rows it reports, or have its proof checked."
+    },
+    "exploratory_inference": {
+      "group": "plan", "count": "kind",
+      "costs": {"with_public_record": 5, "without_public_record": 15},
+      "text": "{type} claim tested without a plan recorded beforehand{record}",
+      "fix": "Test it again under a lab-record plan or a registration, or publish the lab record that shows every run."
+    },
+    "exploratory_principal": {
+      "group": "plan", "count": "kind",
+      "costs": {"with_public_record": 3, "without_public_record": 5},
+      "text": "A principal claim reported without a plan recorded beforehand{record}",
+      "fix": "Plan the measurement in the lab record or a registration before running it, or publish the lab record that shows every run."
+    },
+    "plan_anchor": {
+      "group": "plan", "count": "kind",
+      "costs": {"git_remote": 2, "none": 10},
+      "text": "The plan's time is anchored {how}",
+      "fix": "Anchor plans with the platform, a timestamp authority (RFC 3161) or a registry."
+    },
+    "plan_deviations": {
+      "group": "plan", "count": "kind", "costs": {"deviations": 5},
+      "text": "The test deviated from its plan",
+      "fix": "Each deviation is a limitation: say how it bears on the result."
+    },
+    "plan_flags": {
+      "group": "plan", "count": "kind", "cost_each": 5,
+      "text": "The lab record flags the test: {flags}",
+      "fix": "Rerun the test as planned, or explain the flag in a limitation."
+    },
+    "verifiability": {
+      "group": "checkable", "count": "kind",
+      "costs": {"audit_only": 10, "unverifiable": 45},
+      "text": "Verifiability: {level}",
+      "fix": "Make the data, code and models the claim rests on available, or state why they cannot be."
+    },
+    "verifiability_flags": {
+      "group": "checkable", "count": "kind", "cost_each": 5,
+      "text": "Flags: {flags}",
+      "fix": "Publish what each flag says is missing."
+    },
+    "no_scope": {
+      "group": "context", "count": "kind", "costs": {"missing": 10},
+      "text": "No scope: where the claim holds is not stated",
+      "fix": "Give the claim a scope: population, setting, period, data, conditions, exclusions."
+    },
+    "no_prior_work": {
+      "group": "context", "count": "kind", "costs": {"missing": 5},
+      "text": "A principal claim related to no earlier work",
+      "fix": "Relate the claim to the earlier work it extends, addresses or departs from."
+    },
+    "numbers_without_results": {
+      "group": "context", "count": "kind", "costs": {"missing": 5},
+      "text": "Numbers in the statement match no recorded result",
+      "fix": "Record each number as a result or table cell of the evidence the claim cites, and list it in the support's results; or correct the number."
+    },
+    "limitation": {
+      "group": "open", "count": "each",
+      "costs": {
+        "could_reverse": {"none": 10, "partial": 5, "addressed": 0},
+        "could_bias_magnitude": {"none": 6, "partial": 3, "addressed": 0},
+        "reduces_precision": {"none": 3, "partial": 1, "addressed": 0},
+        "reduces_transferability": {"none": 1, "partial": 0, "addressed": 0}
+      },
+      "text": "Limitation {id} ({impact}, {mitigation}): {statement}",
+      "fix": "Mitigate it with evidence, or narrow the claim's scope so that it no longer applies."
+    },
+    "no_limitation": {
+      "group": "open", "count": "kind", "costs": {"missing": 10},
+      "text": "A principal claim that declares no limitation",
+      "fix": "Declare what could bias, reverse or limit the claim within its scope, each as a limitation that affects it."
+    },
+    "unchecked_assumption": {
+      "group": "open", "count": "each", "cost_each": 3,
+      "text": "Assumption {id} is not checked: {statement}",
+      "fix": "Test the assumption, or analyse how sensitive the claim is to it."
+    }
+  }
+}

rubric/score.py

@@ -0,0 +1,375 @@
+"""Reliability scores (RUBRIC.md §2–5).
+
+score(doc, warnings=None) -> {
+    "rubric": "0.4",
+    "claims": {"C1": {"score": 94, "band": "high", "dots": {"support": "green", ...}, "problems": ["P3", ...],
+                      "costs": {"P3": 6, ...}, "inherited": {"P5": ["C2"], ...}}, ...},
+    "assumptions": ["C24", ...],                 # not scored
+    "paper": {"score": 74, "band": "medium", "dots": {...}, "problems": [...], "principal": [...], "lowest": "C6",
+              "lowest_score": 60, "lowest_problems": [...]} or None when no principal claim is scored,
+    "problems": {"P1": {"check", "group", "cost", "claims", "arises_in", "objects", "text", "fix"}, ...},
+}
+
+`warnings` are the claimset validator's warnings for the file ([{rule, where, message}], its published validation
+report), when given: warning 17 decides `numbers_without_results`, and a limitation with warning 46 (it restates the
+declared competing interest) costs nothing.
+
+A problem in "problems" is one aggregated problem (§3.6) over the principal claims and what they rest on (then over the
+other claims); its "cost" is the most it costs any one claim, "claims" every claim it lowers and "arises_in" the claims
+it is found on. A claim lists the problems that reach it, with what each costs that claim under "costs", and under
+"inherited" the ones it has from what it rests on, with the claims they arise in. The paper's score is the mean of its
+principal claims' (§5); "problems" of the paper are every problem open in its principal claims, for the explanation.
+"""
+
+import json
+import re
+from importlib import resources
+
+RUBRIC = json.loads(resources.files(__package__).joinpath("rubric.json").read_text(encoding="utf-8"))
+VERSION = RUBRIC["version"]
+CHECKS = RUBRIC["checks"]
+GROUPS = [g["name"] for g in RUBRIC["groups"]]
+
+EVIDENCE_KINDS = {"empirical_quant", "empirical_qual", "empirical_obs", "simulation", "demonstration"}
+OBSERVATION_KINDS = {"empirical_quant", "empirical_obs", "simulation"}       # evidence whose observations are cited
+SPECIFIC = re.compile(r"\.R\d+$|\.T\d+\.r\d+$|#row=")                  # a result entry, a table row, data rows
+STUDIED_ROLES = {"subject", "baseline"}
+INFERENTIAL = {"associational", "causal", "mechanistic", "predictive"}
+SUPPORT_WORDS = {"evidence": "evidence",
+                 "evidence_unspecified": "evidence named without the observations the claim reports",
+                 "argument_from_claims": "an argument from other claims",
+                 "formal_checked": "a checked proof", "formal_unchecked": "an unchecked proof",
+                 "formal_disputed": "a disputed proof", "cited": "cited work only", "consensus": "consensus only",
+                 "argument": "argument only", "expert_judgement": "expert judgement only",
+                 "unstated": "no stated support"}
+TERM = re.compile(r"\[\[[^\]:]+:([^\]]*)\]\]")
+MATH = re.compile(r"\$\$.+?\$\$|\$[^$]+\$|\\\(.+?\\\)", re.S)
+
+
+def band(value):
+    """The band of a score: high, medium or low (§4.1)."""
+    if value is None:
+        return None
+    for b in RUBRIC["bands"]:
+        if value >= b["min"]:
+            return b["name"]
+    return RUBRIC["bands"][-1]["name"]
+
+
+def _dot(points):
+    d = RUBRIC["dots"]
+    return "green" if points <= d["green_max"] else "amber" if points <= d["amber_max"] else "red"
+
+
+def _list(x):
+    return x if isinstance(x, list) else []
+
+
+def _dict(x):
+    return x if isinstance(x, dict) else {}
+
+
+def is_assumption(c):
+    kinds = [s.get("kind") for s in _list(c.get("support")) if isinstance(s, dict)]
+    return bool(kinds) and all(k == "assumption" for k in kinds)
+
+
+def _public_record(doc):
+    rid = _dict(_dict(doc.get("paper")).get("exploration")).get("lab_record")
+    for a in _list(doc.get("artefacts")):
+        if isinstance(a, dict) and a.get("id") == rid:
+            return _dict(a.get("availability")).get("level") == "public"
+    return False
+
+
+def _cites_observations(s, evidence):
+    """Whether a support entry names the observations its claim reports (§3.1): a result entry, a table row or data
+    rows, or evidence that records no results or tables."""
+    if any(isinstance(r, str) and SPECIFIC.search(r) for r in _list(s.get("results"))):
+        return True
+    e = _dict(evidence.get(s.get("evidence")))
+    return bool(e) and not _list(e.get("results")) and not _list(e.get("tables"))
+
+
+def _support_cost(c, by_id, evidence):
+    costs = CHECKS["support_kind"]["costs"]
+    best = None
+    for s in _list(c.get("support")):
+        kind = s.get("kind") if isinstance(s, dict) else None
+        if kind in OBSERVATION_KINDS:
+            key = "evidence" if _cites_observations(s, evidence) else "evidence_unspecified"
+        elif kind in EVIDENCE_KINDS:
+            key = "evidence"
+        elif kind == "formal_proof":
+            v = c.get("verification")
+            key = ("formal_checked" if v in ("machine_checked", "agent_checked", "peer_checked")
+                   else "formal_disputed" if v == "disputed" else "formal_unchecked")
+        elif kind == "argument":
+            premises = [by_id.get(d) for d in _list(c.get("depends_on")) if isinstance(d, str)]
+            key = "argument_from_claims" if any(p and not is_assumption(p) for p in premises) else "argument"
+        elif kind in costs:
+            key = kind
+        else:
+            continue
+        if best is None or costs[key] < costs[best]:
+            best = key
+    return best
+
+
+def _api_model_studied(c, doc, evidence):
+    """Whether every API-only model in the evidence the claim's support names is what is studied there (§3.3)."""
+    weights = {m.get("id"): m.get("weights") for m in _list(doc.get("models")) if isinstance(m, dict)}
+    roles = [_dict(dm).get("role") for s in _list(c.get("support")) if isinstance(s, dict)
+             for dm in _list(_dict(evidence.get(s.get("evidence"))).get("models"))
+             if weights.get(_dict(dm).get("model")) == "api_only"]
+    return bool(roles) and all(r in STUDIED_ROLES for r in roles)
+
+
+NAME_WITH_DIGITS = re.compile(r"(?<![\w.])[A-Za-z0-9_]+(?:-[A-Za-z0-9_]+)*")
+
+
+def _has_numbers(statement):
+    text = MATH.sub(" ", TERM.sub(" ", statement if isinstance(statement, str) else ""))
+    text = re.sub(r"\b[A-Z]{1,2}\d+(?:\.[A-Za-z]*\d+)*\b", " ", text)       # object IDs: C4, E1.T1.r2
+    text = NAME_WITH_DIGITS.sub(lambda m: " " if re.search(r"[A-Za-z]", m.group(0)) else m.group(0), text)  # GPT-4o
+    return re.search(r"\d", text) is not None
+
+
+#: Planning values of claims planned before they were run (§3.2): tests, and estimates (claimset v0.42).
+PLANNED = ("confirmatory", "confirmatory_with_deviations", "planned_estimate")
+
+
+def _unplanned_cost(c, public_record):
+    """What the claim would cost for being exploratory (§3.2): the most its plan's problems may cost."""
+    record = "with_public_record" if public_record else "without_public_record"
+    if c.get("type") in INFERENTIAL:
+        return CHECKS["exploratory_inference"]["costs"][record]
+    if c.get("role") == "principal" and c.get("type") != "formal":
+        return CHECKS["exploratory_principal"]["costs"][record]
+    return 0
+
+
+def _capped(plan, limit):
+    """A plan's problems cut so that together they cost at most `limit`, the costliest kept first (§3.2)."""
+    out = []
+    for check, key, cost, claims, args in sorted(plan, key=lambda i: -i[2]):
+        cost = min(cost, limit)
+        limit -= cost
+        if cost > 0:
+            out.append((check, key, cost, claims, args))
+    return out
+
+
+def _instances(doc, c, by_id, public_record, evidence, warned):
+    """The problems of one claim itself: [(check, key, cost, objects, args)]. `warned` is the set of (rule, where) of
+    the validator's warnings, or None when they were not given."""
+    out = []
+    cid = c.get("id")
+    formal = c.get("type") == "formal"
+
+    key = _support_cost(c, by_id, evidence)
+    cost = CHECKS["support_kind"]["costs"].get(key, 0) if key else CHECKS["support_kind"]["costs"]["unstated"]
+    if cost:
+        out.append(("support_kind", cid, cost, [cid], {"what": SUPPORT_WORDS.get(key or "unstated")}))
+
+    planning = c.get("planning")
+    record = "; the lab record shows every run" if public_record else ""
+    if planning in (None, "exploratory"):
+        t = c.get("type")
+        cost = _unplanned_cost(c, public_record)
+        if t in INFERENTIAL:
+            out.append(("exploratory_inference", cid, cost, [cid],
+                        {"type": ("An " if t[0] in "aeiou" else "A ") + t, "record": record}))
+        elif cost:
+            out.append(("exploratory_principal", cid, cost, [cid], {"type": t or "", "record": record}))
+    plan = []
+    if planning in PLANNED:
+        anchor = c.get("planned_at_anchor")
+        costs = CHECKS["plan_anchor"]["costs"]
+        if anchor == "git_remote" or anchor in (None, "none"):
+            how = "by a git push only" if anchor == "git_remote" else "by nothing"
+            plan.append(("plan_anchor", cid, costs["git_remote" if anchor == "git_remote" else "none"], [cid],
+                         {"how": how}))
+    if planning == "confirmatory_with_deviations":
+        plan.append(("plan_deviations", cid, CHECKS["plan_deviations"]["costs"]["deviations"], [cid], {}))
+    flags = [f for f in _list(_dict(c.get("test")).get("flags")) + _list(_dict(c.get("estimate")).get("flags"))
+             if isinstance(f, str)]
+    if flags:
+        ch = CHECKS["plan_flags"]
+        plan.append(("plan_flags", cid, sum(repeated([ch["cost_each"]] * len(flags))), [cid],
+                     {"flags": ", ".join(f.replace("_", " ") for f in flags)}))
+    out += _capped(plan, _unplanned_cost(c, public_record)) if planning in PLANNED else plan
+
+    level = c.get("verifiability")
+    if level in CHECKS["verifiability"]["costs"]:
+        out.append(("verifiability", cid, CHECKS["verifiability"]["costs"][level], [cid],
+                    {"level": level.replace("_", " ")}))
+    vflags = [f for f in _list(c.get("verifiability_flags"))
+              if isinstance(f, str) and f not in ("unstated_support", "unanchored_plan")
+              and not (f == "api_model_dependency" and _api_model_studied(c, doc, evidence))]
+    if vflags:
+        ch = CHECKS["verifiability_flags"]
+        out.append(("verifiability_flags", cid, sum(repeated([ch["cost_each"]] * len(vflags))), [cid],
+                    {"flags": ", ".join(f.replace("_", " ") for f in vflags)}))
+
+    if not formal and not _dict(c.get("scope")):
+        out.append(("no_scope", cid, CHECKS["no_scope"]["costs"]["missing"], [cid], {}))
+    if c.get("role") == "principal" and c.get("novelty") != "not_applicable":
+        outside = any(isinstance(r, dict) and r.get("id_type") != "local" for r in _list(c.get("relations")))
+        cited = any(isinstance(s, dict) and s.get("kind") == "cited" for s in _list(c.get("support")))
+        if not (outside or cited):
+            out.append(("no_prior_work", cid, CHECKS["no_prior_work"]["costs"]["missing"], [cid], {}))
+    if warned is not None:
+        unsourced = not formal and ("W17", f"claims:{cid}.statement") in warned
+    else:
+        unsourced = not formal and _has_numbers(c.get("statement")) and not any(
+            isinstance(s, dict) and _list(s.get("results")) for s in _list(c.get("support")))
+    if unsourced:
+        out.append(("numbers_without_results", cid, CHECKS["numbers_without_results"]["costs"]["missing"], [cid], {}))
+
+    lim_costs = CHECKS["limitation"]["costs"]
+    for lim in _list(doc.get("limitations")):
+        if isinstance(lim, dict) and cid in _list(lim.get("affects")) and not _restated(lim, warned):
+            cost = _dict(lim_costs.get(lim.get("impact"))).get(lim.get("mitigation") or "none", 0)
+            if cost:
+                out.append(("limitation", lim.get("id"), cost, [cid],
+                            {"id": lim.get("id"), "impact": str(lim.get("impact", "")).replace("_", " "),
+                             "mitigation": "not mitigated" if (lim.get("mitigation") or "none") == "none"
+                             else f"mitigation {lim.get('mitigation')}", "statement": lim.get("statement", "")}))
+    for aid in _list(c.get("assumptions")):
+        a = by_id.get(aid)
+        if isinstance(a, dict) and a.get("check") == "unchecked":
+            out.append(("unchecked_assumption", aid, CHECKS["unchecked_assumption"]["cost_each"], [cid],
+                        {"id": aid, "statement": a.get("statement", "")}))
+    return out
+
+
+def _restated(lim, warned):
+    """A limitation that only restates the declared competing interest (warning 46; §3.5)."""
+    return warned is not None and ("W46", f"limitations:{lim.get('id')}.statement") in warned
+
+
+def _closure(cid, by_id):
+    """The claim and everything it rests on (depends_on and assumptions), followed to the end."""
+    seen, stack = [], [cid]
+    while stack:
+        x = stack.pop()
+        if x in seen or x not in by_id:
+            continue
+        seen.append(x)
+        c = by_id[x]
+        stack.extend(i for i in _list(c.get("depends_on")) + _list(c.get("assumptions")) if isinstance(i, str))
+    return seen
+
+
+def repeated(costs):
+    """The costs of repeated problems of one check on one claim (RUBRIC.md §3.6): sorted from the largest, the first
+    in full and each next one half the one before (rounded down, at least 1), so that every one removed or mitigated
+    lowers the total and the total stays below twice the largest."""
+    return [0 if c <= 0 else max(1, c // 2 ** i) for i, c in enumerate(sorted(costs, reverse=True))]
+
+
+def _aggregate(instances):
+    """Instances counted as RUBRIC.md §3.6: {(check, key): {cost, claims, objects, args}}; key is None for `kind`."""
+    out = {}
+    for check, key, cost, claims, args in instances:
+        k = (check, None) if CHECKS[check]["count"] == "kind" else (check, key)
+        p = out.setdefault(k, {"check": check, "cost": 0, "claims": [], "args": args})
+        if cost > p["cost"]:
+            p["cost"], p["args"] = cost, args
+        for c in claims:
+            if c not in p["claims"]:
+                p["claims"].append(c)
+    for check in {k[0] for k in out if CHECKS[k[0]]["count"] == "each"}:
+        items = sorted((p for k, p in out.items() if k[0] == check), key=lambda p: -p["cost"])
+        for p, cost in zip(items, repeated([p["cost"] for p in items])):
+            p["cost"] = cost
+    return out
+
+
+def _sort_claims(ids):
+    return sorted(ids, key=lambda i: (len(i), i))
+
+
+def _result(agg):
+    total = sum(p["cost"] for p in agg.values())
+    value = max(0, 100 - total)
+    lost = {g: 0 for g in GROUPS}
+    for p in agg.values():
+        lost[CHECKS[p["check"]]["group"]] += p["cost"]
+    return value, {g: _dot(v) for g, v in lost.items()}
+
+
+def score(doc, warnings=None):
+    """The reliability scores of a published claimset (RUBRIC.md §2–5), given the validator's warnings for it when
+    they are available (its validation report)."""
+    claims = [c for c in _list(doc.get("claims")) if isinstance(c, dict) and isinstance(c.get("id"), str)]
+    by_id = {c["id"]: c for c in claims}
+    evidence = {e.get("id"): e for e in _list(doc.get("evidence")) if isinstance(e, dict)}
+    warned = None if warnings is None else {(_dict(w).get("rule"), _dict(w).get("where")) for w in _list(warnings)}
+    public_record = _public_record(doc)
+    own = {c["id"]: _instances(doc, c, by_id, public_record, evidence, warned) for c in claims if not is_assumption(c)}
+    limited = {cid for lim in _list(doc.get("limitations")) if isinstance(lim, dict) and not _restated(lim, warned)
+               for cid in _list(lim.get("affects")) if isinstance(cid, str)}
+    for c in claims:                                            # no_limitation (§3.5): nothing it rests on declares one
+        if (c["id"] in own and c.get("role") == "principal" and c.get("type") != "formal"
+                and not limited & set(_closure(c["id"], by_id))):
+            own[c["id"]].append(("no_limitation", c["id"], CHECKS["no_limitation"]["costs"]["missing"], [c["id"]], {}))
+
+    def gathered(ids):
+        return [i for x in ids for cid in _closure(x, by_id) for i in own.get(cid, [])]
+
+    principal = [c["id"] for c in claims if c.get("role") == "principal" and c["id"] in own]
+    paper_agg = _aggregate(gathered(principal)) if principal else {}
+    everything = _aggregate(gathered(list(own)))
+    # Problem IDs: those of the paper first, by cost, then the rest; stable for a given file.
+    order = sorted(paper_agg, key=lambda k: (-paper_agg[k]["cost"], k[0], str(k[1])))
+    order += sorted((k for k in everything if k not in paper_agg), key=lambda k: (k[0], str(k[1])))
+    pid = {k: f"P{i + 1}" for i, k in enumerate(order)}
+
+    def describe(k, p):
+        ch = CHECKS[k[0]]
+        try:
+            text = ch["text"].format(**p["args"])
+        except (KeyError, IndexError):
+            text = ch["text"]
+        return {"check": k[0], "group": ch["group"], "cost": p["cost"], "claims": _sort_claims(p["claims"]),
+                "arises_in": _sort_claims(p["claims"]),
+                "objects": [k[1]] if k[1] is not None else [], "text": text, "fix": ch["fix"]}
+
+    problems = {pid[k]: describe(k, paper_agg.get(k) or everything[k]) for k in order}
+
+    out_claims = {}
+    for cid in own:
+        agg = _aggregate(gathered([cid]))
+        mine = set(_aggregate(own[cid]))
+        value, dots = _result(agg)
+        listed = sorted(agg, key=lambda k: (-agg[k]["cost"], pid[k]))
+        out_claims[cid] = {"score": value, "band": band(value), "dots": dots,
+                           "problems": [pid[k] for k in listed if agg[k]["cost"]],
+                           "costs": {pid[k]: agg[k]["cost"] for k in listed if agg[k]["cost"]},
+                           "inherited": {pid[k]: problems[pid[k]]["arises_in"] for k in listed
+                                         if agg[k]["cost"] and k not in mine}}
+
+    # A problem's cost is the most it costs any one claim (a repeated problem may cost one claim less than another);
+    # its claims are every claim it lowers, as the claims' own lists say (§2: inherited problems).
+    for k in order:
+        lowered = [cid for cid, c in out_claims.items() if pid[k] in c["costs"]]
+        if lowered:
+            problems[pid[k]]["cost"] = max(out_claims[c]["costs"][pid[k]] for c in lowered)
+            problems[pid[k]]["claims"] = _sort_claims(lowered)
+
+    paper = None
+    if principal:
+        # The paper's score is the mean of its principal claims' (§5), shown with the lowest; its dots, the worst of
+        # each group among them.
+        lowest = min(principal, key=lambda c: (out_claims[c]["score"], len(c), c))
+        value = int(sum(out_claims[c]["score"] for c in principal) / len(principal) + 0.5)
+        rank = {"green": 0, "amber": 1, "red": 2}
+        dots = {g: max((out_claims[c]["dots"][g] for c in principal), key=rank.get) for g in GROUPS}
+        listed = [k for k in order if k in paper_agg and paper_agg[k]["cost"]]
+        paper = {"score": value, "band": band(value), "dots": dots, "problems": [pid[k] for k in listed],
+                 "principal": principal, "lowest": lowest, "lowest_score": out_claims[lowest]["score"],
+                 "lowest_problems": out_claims[lowest]["problems"]}
+    return {"rubric": VERSION, "claims": out_claims,
+            "assumptions": [c["id"] for c in claims if is_assumption(c)], "paper": paper, "problems": problems}