dataeval-plots 0.0.13__tar.gz → 0.0.14__tar.gz

{dataeval_plots-0.0.13 → dataeval_plots-0.0.14}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: dataeval-plots
-Version: 0.0.13
+Version: 0.0.14
 Summary: DataEval companion package for plotting utilities
 Project-URL: Homepage, https://dataeval.ai/
 Project-URL: Repository, https://github.com/aria-ml/dataeval/
@@ -208,60 +208,64 @@ subtitle that says which models the band judges:
 
 The axis is split into three zones. Each zone classifies the requirements that fall inside it:
 
-- **Demonstrated** runs from the axis minimum up to the lower confidence bound of the best tested model. The band
-  reads "Requirements through *x*% are demonstrated".
+- **Demonstrated** runs from the axis minimum up to the lower confidence bound of the best tested model, which the
+  label under the band names, for example "95.7% lower confidence bound". The zone reads "demonstrated".
 - **Unknown** lies between the other two zones. The evidence neither demonstrates nor rules out these requirements.
-- **Ruled out** runs from the upper confidence bound of the lowest applicable ceiling to the top of the axis. It holds
-  only under the recorded assumptions, and the zone reads "ruled out under the recorded assumptions" wherever the
-  label fits.
+- **Ruled out** runs from the upper confidence bound of the lowest applicable ceiling to the top of the axis. The zone
+  reads "ruled out\*" wherever the label fits.
+
+The `*` marks a footnote: "ruled out" holds only under the recorded assumptions. Wherever "ruled out\*" appears, the
+caption under the same band ends with that footnote, "\* Ruled out under the recorded assumptions, at the stated
+confidence."
 
 Each of your requirements is marked on the axis, and the marker states the determination in words:
 
 | Marker | Meaning |
 |--------|---------|
-| "95% requirement: demonstrated ✓" | The requirement is at or below the lower confidence bound |
-| "not enough evidence ?" | The requirement falls in the unknown zone, and the evidence cannot decide it |
-| "ruled out under the recorded assumptions ✗" | The requirement is above the upper confidence bound of a ceiling |
-| "not ruled out ?" | The requirement also falls in the unknown zone: it is not above any ceiling's upper confidence bound, and no tested model reaches it |
+| "95%: demonstrated ✓" | The requirement is at or below the lower confidence bound |
+| "95%: insufficient evidence ?" | The requirement falls in the unknown zone, and the evidence cannot decide it |
+| "95%: ruled out\* ✗" | The requirement is above the upper confidence bound of a ceiling, under the recorded assumptions (see the footnote) |
+| "95%: not ruled out ?" | The requirement also falls in the unknown zone: it is not above any ceiling's upper confidence bound, and no tested model reaches it |
 
 The caption under the band reports the observed score next to the lower confidence bound it supports, for example
-"Your model scored 91.0% on the 2,000 evaluation images. Allowing for sampling uncertainty gives a lower
-confidence bound of 88.5%." The sentence about the observed score appears only when an observed score exists. The
-caption then states that the bounds concern new images drawn from the same population, and that confidence holds
-jointly across the whole evaluation (for example "Confidence is 95% jointly across this evaluation").
+"Your model scored 91.0% on 2,000 evaluation images; given sampling uncertainty, its lower confidence bound is
+88.5%." The sentence about the observed score appears only when an observed score exists. The caption then states
+that the bounds apply to new images from the same population, and that confidence holds jointly across the whole
+evaluation (for example "Bounds apply to new images from the same population, at 95% joint confidence.").
 
 Numbers read as DataEval prints them. Bounds are given to 0.1%, a lower bound rounded down and an upper bound rounded
 up, so rounding never overstates what the evidence shows. Requirements and confidence keep their own decimals, such as
-"95.5% requirement" or "Confidence is 97.5%".
+"95.5%" or "at 97.5% joint confidence".
 
-When a ceiling bounds the band, the caption adds "Under the recorded assumptions, requirements above *x*% are ruled out
-at the stated confidence." A group-based ceiling carries the extra note that its validity argument awaits lead-scientist
-sign-off.
+When a ceiling bounds the band, the caption ends with the footnote "\* Ruled out under the recorded assumptions, at
+the stated confidence." A group-based ceiling adds "(group-based ceiling; validity pending lead-scientist sign-off)".
 
 A ceiling that applies only to narrower models than the band judges never draws a zone. It appears as a note instead,
-for example that the group-based ceiling does not limit models that distinguish individual images.
+for example "Group-based ceiling (72.1%) applies only to models that give one answer per group."
 
 ### Other outcomes
 
-- **Ruled out.** A requirement above the lowest applicable ceiling's upper confidence bound is marked "ruled out
-  under the recorded assumptions ✗". The ruled-out zone covers everything above that bound, and the caption reads
-  "Under the recorded assumptions, requirements above *x*% are ruled out at the stated confidence."
+- **Ruled out.** A requirement above the lowest applicable ceiling's upper confidence bound is marked "ruled out\*
+  ✗". The ruled-out zone covers everything above that bound, which the label under the band names, for example
+  "93.0% upper confidence bound". The caption ends with the footnote "\* Ruled out under the recorded assumptions, at
+  the stated confidence."
 - **Evidence conflict.** When bounds from different sources contradict each other, the band draws no zones. A dashed
   warning outline reads "Evidence conflicts: check assumptions", both clashing bounds are marked, and the caption
-  says that the bounds disagree and to check the assumptions, including whether evaluation images entered training.
-  Even when the assumptions hold, the probability that any bound fails is at most the stated alpha (for example 5% at
-  95% confidence).
+  reads "Bounds disagree: check the assumptions, such as evaluation images entering training. Even when they hold,
+  the chance that any bound fails is at most 5%." The 5% is the stated alpha, at 95% confidence.
 - **Bounds at 0% or 100%.** A lower confidence bound of 0% demonstrates nothing and an upper confidence bound of 100%
   rules nothing out, so neither draws a zone.
 
 ### Showing the evidence
 
 `plot_evidence=True` replaces each band with one row per evidence source. Each row shows the observed score as a dot
-and the confidence bound as a tick, with a legend line explaining both: the dot is the score observed on the
-evaluation images, and the tick is the lower (or upper) confidence bound for new images from the same population.
-Evidence that does not apply to the judged models is drawn faint and labeled with the models it applies to. A source
-that produced no bound for the quantity is listed faint as "no bound available", and when no source has a bound the
-view says "No bound is available yet." instead of showing empty axes.
+and the confidence bound as a tick, with the legend "● observed · ┃ lower confidence bound" (and "· ┃ upper confidence
+bound" when a ceiling has one). The caption says what both mean: "Observed on the evaluation images; bounds apply to
+new images from the same population, at 95% joint confidence." In a conflict, the conflict caption follows it.
+Evidence that does not apply to the judged models is drawn faint, and its label names the models it applies to, for
+example "Group-based ceiling (one answer per group)", so this view has no narrower-scope notes. A source that produced
+no bound for the quantity is listed faint as "no bound available", and when no source has a bound the view says "No
+bound is available yet." instead of showing empty axes.
 
 ## Extending the Package
 

{dataeval_plots-0.0.13 → dataeval_plots-0.0.14}/README.md

@@ -162,60 +162,64 @@ subtitle that says which models the band judges:
 
 The axis is split into three zones. Each zone classifies the requirements that fall inside it:
 
-- **Demonstrated** runs from the axis minimum up to the lower confidence bound of the best tested model. The band
-  reads "Requirements through *x*% are demonstrated".
+- **Demonstrated** runs from the axis minimum up to the lower confidence bound of the best tested model, which the
+  label under the band names, for example "95.7% lower confidence bound". The zone reads "demonstrated".
 - **Unknown** lies between the other two zones. The evidence neither demonstrates nor rules out these requirements.
-- **Ruled out** runs from the upper confidence bound of the lowest applicable ceiling to the top of the axis. It holds
-  only under the recorded assumptions, and the zone reads "ruled out under the recorded assumptions" wherever the
-  label fits.
+- **Ruled out** runs from the upper confidence bound of the lowest applicable ceiling to the top of the axis. The zone
+  reads "ruled out\*" wherever the label fits.
+
+The `*` marks a footnote: "ruled out" holds only under the recorded assumptions. Wherever "ruled out\*" appears, the
+caption under the same band ends with that footnote, "\* Ruled out under the recorded assumptions, at the stated
+confidence."
 
 Each of your requirements is marked on the axis, and the marker states the determination in words:
 
 | Marker | Meaning |
 |--------|---------|
-| "95% requirement: demonstrated ✓" | The requirement is at or below the lower confidence bound |
-| "not enough evidence ?" | The requirement falls in the unknown zone, and the evidence cannot decide it |
-| "ruled out under the recorded assumptions ✗" | The requirement is above the upper confidence bound of a ceiling |
-| "not ruled out ?" | The requirement also falls in the unknown zone: it is not above any ceiling's upper confidence bound, and no tested model reaches it |
+| "95%: demonstrated ✓" | The requirement is at or below the lower confidence bound |
+| "95%: insufficient evidence ?" | The requirement falls in the unknown zone, and the evidence cannot decide it |
+| "95%: ruled out\* ✗" | The requirement is above the upper confidence bound of a ceiling, under the recorded assumptions (see the footnote) |
+| "95%: not ruled out ?" | The requirement also falls in the unknown zone: it is not above any ceiling's upper confidence bound, and no tested model reaches it |
 
 The caption under the band reports the observed score next to the lower confidence bound it supports, for example
-"Your model scored 91.0% on the 2,000 evaluation images. Allowing for sampling uncertainty gives a lower
-confidence bound of 88.5%." The sentence about the observed score appears only when an observed score exists. The
-caption then states that the bounds concern new images drawn from the same population, and that confidence holds
-jointly across the whole evaluation (for example "Confidence is 95% jointly across this evaluation").
+"Your model scored 91.0% on 2,000 evaluation images; given sampling uncertainty, its lower confidence bound is
+88.5%." The sentence about the observed score appears only when an observed score exists. The caption then states
+that the bounds apply to new images from the same population, and that confidence holds jointly across the whole
+evaluation (for example "Bounds apply to new images from the same population, at 95% joint confidence.").
 
 Numbers read as DataEval prints them. Bounds are given to 0.1%, a lower bound rounded down and an upper bound rounded
 up, so rounding never overstates what the evidence shows. Requirements and confidence keep their own decimals, such as
-"95.5% requirement" or "Confidence is 97.5%".
+"95.5%" or "at 97.5% joint confidence".
 
-When a ceiling bounds the band, the caption adds "Under the recorded assumptions, requirements above *x*% are ruled out
-at the stated confidence." A group-based ceiling carries the extra note that its validity argument awaits lead-scientist
-sign-off.
+When a ceiling bounds the band, the caption ends with the footnote "\* Ruled out under the recorded assumptions, at
+the stated confidence." A group-based ceiling adds "(group-based ceiling; validity pending lead-scientist sign-off)".
 
 A ceiling that applies only to narrower models than the band judges never draws a zone. It appears as a note instead,
-for example that the group-based ceiling does not limit models that distinguish individual images.
+for example "Group-based ceiling (72.1%) applies only to models that give one answer per group."
 
 ### Other outcomes
 
-- **Ruled out.** A requirement above the lowest applicable ceiling's upper confidence bound is marked "ruled out
-  under the recorded assumptions ✗". The ruled-out zone covers everything above that bound, and the caption reads
-  "Under the recorded assumptions, requirements above *x*% are ruled out at the stated confidence."
+- **Ruled out.** A requirement above the lowest applicable ceiling's upper confidence bound is marked "ruled out\*
+  ✗". The ruled-out zone covers everything above that bound, which the label under the band names, for example
+  "93.0% upper confidence bound". The caption ends with the footnote "\* Ruled out under the recorded assumptions, at
+  the stated confidence."
 - **Evidence conflict.** When bounds from different sources contradict each other, the band draws no zones. A dashed
   warning outline reads "Evidence conflicts: check assumptions", both clashing bounds are marked, and the caption
-  says that the bounds disagree and to check the assumptions, including whether evaluation images entered training.
-  Even when the assumptions hold, the probability that any bound fails is at most the stated alpha (for example 5% at
-  95% confidence).
+  reads "Bounds disagree: check the assumptions, such as evaluation images entering training. Even when they hold,
+  the chance that any bound fails is at most 5%." The 5% is the stated alpha, at 95% confidence.
 - **Bounds at 0% or 100%.** A lower confidence bound of 0% demonstrates nothing and an upper confidence bound of 100%
   rules nothing out, so neither draws a zone.
 
 ### Showing the evidence
 
 `plot_evidence=True` replaces each band with one row per evidence source. Each row shows the observed score as a dot
-and the confidence bound as a tick, with a legend line explaining both: the dot is the score observed on the
-evaluation images, and the tick is the lower (or upper) confidence bound for new images from the same population.
-Evidence that does not apply to the judged models is drawn faint and labeled with the models it applies to. A source
-that produced no bound for the quantity is listed faint as "no bound available", and when no source has a bound the
-view says "No bound is available yet." instead of showing empty axes.
+and the confidence bound as a tick, with the legend "● observed · ┃ lower confidence bound" (and "· ┃ upper confidence
+bound" when a ceiling has one). The caption says what both mean: "Observed on the evaluation images; bounds apply to
+new images from the same population, at 95% joint confidence." In a conflict, the conflict caption follows it.
+Evidence that does not apply to the judged models is drawn faint, and its label names the models it applies to, for
+example "Group-based ceiling (one answer per group)", so this view has no narrower-scope notes. A source that produced
+no bound for the quantity is listed faint as "no bound available", and when no source has a bound the view says "No
+bound is available yet." instead of showing empty axes.
 
 ## Extending the Package
 

{dataeval_plots-0.0.13 → dataeval_plots-0.0.14}/src/dataeval_plots/_version.py

@@ -18,7 +18,7 @@ version_tuple: tuple[int | str, ...]
 commit_id: str | None
 __commit_id__: str | None
 
-__version__ = version = '0.0.13'
-__version_tuple__ = version_tuple = (0, 0, 13)
+__version__ = version = '0.0.14'
+__version_tuple__ = version_tuple = (0, 0, 14)
 
 __commit_id__ = commit_id = None

{dataeval_plots-0.0.13 → dataeval_plots-0.0.14}/src/dataeval_plots/backends/_feasibility_layout.py

@@ -48,7 +48,8 @@ def text_width(text: str, size: float = LABEL_PT) -> float:
 
 def wrap(text: str, size: float, width: float) -> str:
     """Wrap to the figure's text width, never leaving a legend glyph at a line end."""
-    glued = text.replace("● ", "●\xa0").replace("┃ ", "┃\xa0")  # textwrap breaks only at ASCII whitespace
+    # textwrap breaks only at ASCII whitespace; a footnote's "*" stays with its first word
+    glued = text.replace("● ", "●\xa0").replace("┃ ", "┃\xa0").replace("* ", "*\xa0")
     return textwrap.fill(glued, int((width - 2 * MARGIN) * 72 / (size * 0.55))).replace("\xa0", " ")
 
 
@@ -114,7 +115,8 @@ def blocks(bands: list[FeasibilityBand], plot_evidence: bool, width: float) -> t
         ax_h = body + (max((p[3] for p in placed), default=-1) + 1) * LEVEL + 0.1
         # Below the axes: the evidence legend, the caption paragraph, then the notes, each wrapped
         texts = [(band.evidence_legend, 10.0, muted)] if plot_evidence else []
-        texts += [(" ".join(band.caption), CAPTION_PT, ink), *((note, 10.0, muted) for note in band.notes)]
+        caption, notes = (band.evidence_caption, []) if plot_evidence else (band.caption, band.notes)
+        texts += [(" ".join(caption), CAPTION_PT, ink), *((note, 10.0, muted) for note in notes)]
         wrapped = [(wrap(text, size, width), size, color) for text, size, color in texts]
         out.append((band, x_range, xlabel, ax_h, placed, wrapped))
         height += TITLE + ax_h + XAXIS + sum(text_height(text) for text, _, _ in wrapped) + GAP

{dataeval_plots-0.0.13 → dataeval_plots-0.0.14}/src/dataeval_plots/backends/_shared.py

@@ -684,13 +684,16 @@ _SCOPE_MODELS = {
     "fixed_representation": "models restricted to the embeddings",
     "partition_restricted": "models that give one answer per group",
 }
+_SCOPE_ROW = {"fixed_representation": "embeddings only", "partition_restricted": "one answer per group"}
+# "ruled out*" never stands alone: its "*" footnote in the same band's caption says it holds only under the recorded
+# assumptions, and an infeasible row always has a ceiling, which writes that footnote
 _MARKER_WORDS = {
     "demonstrated_feasible": "demonstrated ✓",
-    "unresolved": "not enough evidence ?",
-    "infeasible_under_recorded_assumptions": "ruled out under the recorded assumptions ✗",
+    "unresolved": "insufficient evidence ?",
+    "infeasible_under_recorded_assumptions": "ruled out* ✗",
     "not_ruled_out": "not ruled out ?",
 }
-_RULED_OUT = "ruled out under the recorded assumptions"  # never "ruled out" alone: it holds only under them
+_RULED_OUT = "ruled out*"
 _AGGREGATION_WORDS = {None: "", "sequence_mean": "per-sequence mean ", "frame_pooled": "frame-pooled "}
 _GROUP_CEILING = "group-based ceiling"
 _NO_BOUND = "no bound available"
@@ -750,6 +753,7 @@ class FeasibilityBand:
     evidence_markers: list[tuple[float, str]]  # requirement labels in the evidence view
     caption: list[str]
     notes: list[str]
+    evidence_caption: list[str]  # the evidence view's caption, in place of the caption and notes
     evidence: list[FeasibilityEvidenceRow]
     evidence_legend: str
     evidence_note: str | None  # drawn in the evidence view when no source has a bound for the band
@@ -788,7 +792,7 @@ def _feasibility_evidence(
         rank, judged = _SCOPE_RANK[info], _SCOPE_RANK[scope]
         restricted = rank < judged if kind == "upper" else rank > judged
         models = _SCOPE_MODELS[info].format(unit=unit)
-        label = _sentence_case(e["source"]) + (f" ({models})" if restricted else "")
+        label = _sentence_case(e["source"]) + (f" ({_SCOPE_ROW.get(info, models)})" if restricted else "")
         observed = None if bound is None else e["observed"]  # an observed score alone bounds nothing
         rows.append(
             FeasibilityEvidenceRow(
@@ -807,10 +811,9 @@ def _feasibility_evidence(
         if restricted and kind == "upper" and bound is not None:
             ceiling = _pct(bound, "upper")
             notes.append(
-                f"The group-based ceiling ({ceiling}) does not limit models that distinguish individual {unit}s."
+                f"Group-based ceiling ({ceiling}) applies only to models that give one answer per group."
                 if info == "partition_restricted"
-                else f"The {e['source']} ({ceiling}) applies only to {models}; "
-                "it does not limit the models judged here."
+                else f"{_sentence_case(e['source'])} ({ceiling}) applies only to {models}."
             )
     return rows, list(dict.fromkeys(notes))
 
@@ -880,37 +883,43 @@ def _feasibility_band(
     markers = []
     for r in sorted(rows, key=lambda r: r["threshold"]):
         words = _MARKER_WORDS.get(r["determination"])
-        markers.append((r["threshold"], f"{_percent(r['threshold'])} requirement" + (f": {words}" if words else "")))
+        markers.append((r["threshold"], _percent(r["threshold"]) + (f": {words}" if words else "")))
 
+    conflict_caption = (
+        f"Bounds disagree: check the assumptions, such as evaluation {unit}s entering training. "
+        f"Even when they hold, the chance that any bound fails is at most {_percent(alpha)}."
+    )
     if conflict:
-        caption = [
-            f"The bounds disagree. Check the assumptions, including whether evaluation {unit}s entered training. "
-            f"Even when assumptions hold, the probability that any bound fails is at most {_percent(alpha)}."
-        ]
+        caption = [conflict_caption]
     else:
         caption = []
         witness = None if best is None else _deciding(best, evidence)[0]
         if witness and witness["observed"] is not None and w is not None:
-            on = "" if witness["n"] is None else f"{witness['n']:,} "
+            on = "the " if witness["n"] is None else f"{witness['n']:,} "
             caption.append(
-                f"{_sentence_case(witness['source'])} scored {_pct(witness['observed'])} on the {on}evaluation "
-                f"{unit}s. Allowing for sampling uncertainty gives a lower confidence bound of {_pct(w, 'lower')}."
+                f"{_sentence_case(witness['source'])} scored {_pct(witness['observed'])} on {on}evaluation "
+                f"{unit}s; given sampling uncertainty, its lower confidence bound is {_pct(w, 'lower')}."
             )
+        caption.append(
+            f"Bounds apply to new {noun}s from the same population, at {_percent(1 - alpha)} joint confidence."
+        )
+        # The "*" footnote to every "ruled out*" goes last, as a footnote does
         if deciding is not None and c is not None:
             ceiling = _deciding(deciding, evidence)[1]
             sign_off = ceiling is not None and ceiling["source"] == _GROUP_CEILING
             caption.append(
-                f"Under the recorded assumptions, requirements above {_pct(c, 'upper')} are ruled out at the stated "
-                "confidence"
-                + (" (group-based ceiling; its validity argument awaits lead-scientist sign-off)" if sign_off else "")
+                "* Ruled out under the recorded assumptions, at the stated confidence"
+                + (" (group-based ceiling; validity pending lead-scientist sign-off)" if sign_off else "")
                 + "."
             )
-        caption.append(
-            f"Bounds concern new {noun}s drawn from the same population. "
-            f"Confidence is {_percent(1 - alpha)} jointly across this evaluation."
-        )
+    evidence_caption = [
+        f"Observed on the evaluation {unit}s; bounds apply to new {unit}s from the same population, "
+        f"at {_percent(1 - alpha)} joint confidence."
+    ]
+    if conflict:
+        evidence_caption.append(conflict_caption)
 
-    legend = f"● observed on the evaluation {unit}s · ┃ lower confidence bound for new {unit}s from the same population"
+    legend = "● observed · ┃ lower confidence bound"
     if any(r.bound_type == "upper" and r.bound is not None for r in ev_rows):
         legend += " · ┃ upper confidence bound"
 
@@ -926,6 +935,7 @@ def _feasibility_band(
         evidence_markers=[(t, f"{_percent(t)} required") for t, _ in markers],
         caption=caption,
         notes=notes,
+        evidence_caption=evidence_caption,
         evidence=ev_rows,
         evidence_legend=legend,
         evidence_note=None if any(r.bound is not None for r in ev_rows) else "No bound is available yet.",
@@ -1042,7 +1052,7 @@ def feasibility_zones(band: FeasibilityBand) -> list[tuple[str, float, float, st
     end = hi if band.ruled_out is None else band.ruled_out
     zones = [("unknown", start, end, "unknown")]
     if band.demonstrated is not None:
-        zones.insert(0, ("demonstrated", lo, start, f"Requirements through {_pct(start, 'lower')} are demonstrated"))
+        zones.insert(0, ("demonstrated", lo, start, "demonstrated"))
     if band.ruled_out is not None:
         zones.append(("ruled_out", end, hi, _RULED_OUT))
     return [zone for zone in zones if zone[2] > zone[1]]  # a zero-width zone has nothing to show