evalcore 2.5.0__tar.gz → 2.6.0__tar.gz

{evalcore-2.5.0 → evalcore-2.6.0}/CHANGELOG.md

@@ -6,6 +6,57 @@ All notable changes to this project are documented here. The format is based on
 
 ## [Unreleased]
 
+## [2.6.0] - 2026-09-12
+
+A metric says which way is good and what its values sit on.
+
+### Added
+- `MetricRange` and a `value_range` field on `Score` and `MetricValue`.
+  Nothing about a float says what it is on: `0.86` is 86% if the metric runs
+  0..1 and 4.3 out of 5 if it runs 1..5. A consumer holding only the number
+  guesses from magnitude, which is how a run whose costs happened to stay
+  under a dollar gets rendered as percentages, and how the same metric gets
+  classified differently depending on the window you look at.
+- Built-in graders declare their own ranges. Deterministic checks are 0..1;
+  `classification` separates its confusion-matrix metrics (0..1) from
+  `support_*` and `errors` (0..unbounded); judge metrics are 0..1, being
+  points over the scale, except `disagreement`, a spread in raw points,
+  which is 0..scale-1.
+- `range:` on a `numeric` field spec, as `{min, max}` or a two-element list.
+  That grader surfaces whatever the adapter put in the field, so it is the
+  one place the engine cannot know. It is a separate key from the `min`/`max`
+  bounds beside it, which are a pass/fail threshold rather than a domain: a
+  cost that must stay under a dollar can still cost five.
+- `maximum=None` is a positive statement - unbounded above, so not a
+  fraction of anything - and is different from carrying no range at all,
+  which says only that nobody declared one. Nothing is inferred from
+  observed values.
+- `direction` and `higher_is_better` on `MetricDelta`. A delta's sign and its
+  meaning are different questions - `f1` rising is an improvement, and
+  `false_negative_rate` rising is a regression - and until now the engine only
+  knew the difference for the single win metric, via `win_higher_is_better`.
+  Every other metric came out of `compare()` as a bare number, so anything
+  downstream that wanted to rank, colour or summarize deltas had to keep its
+  own list of which metrics are inverted, or get it wrong.
+- `thresholds.metrics`, an optional map declaring polarity per metric:
+  `lower_is_better`, `higher_is_better`, or `neutral` for one that moves
+  without either direction being a result.
+- **Polarity is inferred from the guardrails when it is not declared**, which
+  is what makes this useful without editing a single existing suite. A `max`
+  or `must_not_increase` rule is only ever written about a metric you want
+  low, and `min`/`must_not_decrease` about one you want high, so a gated
+  metric has already stated its direction. Precedence, strongest first: an
+  explicit `metrics:` entry, then `win_higher_is_better` for the win metric,
+  then the guardrails. A metric none of them mention defaults to
+  higher-is-better; a metric fenced in on both sides by a band resolves to
+  `neutral` rather than falling through to that default, since a band says
+  neither direction is the good one.
+
+### Changed
+- `_evaluate_win` derives the win verdict through the same call that fills in
+  each `MetricDelta.direction`, so `Comparison.win` and the win metric's own
+  delta row cannot disagree about the same number.
+
 ## [2.5.0] - 2026-08-11
 
 A suite can declare the modules it needs imported.
@@ -380,7 +431,8 @@ by semantic versioning: a breaking change to either means a 2.0.
   rating + ranking with judge agreement, Markdown/HTML reporters, JSON +
   column-store outbox, and content-hash provenance.
 
-[Unreleased]: https://github.com/scottpmiller/evalcore/compare/2.5.0...HEAD
+[Unreleased]: https://github.com/scottpmiller/evalcore/compare/2.6.0...HEAD
+[2.6.0]: https://github.com/scottpmiller/evalcore/compare/2.5.0...2.6.0
 [2.5.0]: https://github.com/scottpmiller/evalcore/compare/2.4.3...2.5.0
 [2.4.3]: https://github.com/scottpmiller/evalcore/compare/2.4.2...2.4.3
 [2.4.2]: https://github.com/scottpmiller/evalcore/compare/2.4.1...2.4.2

{evalcore-2.5.0 → evalcore-2.6.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: evalcore
-Version: 2.5.0
+Version: 2.6.0
 Summary: A generic, consumer-agnostic evaluation engine for prompt, model, and API outputs.
 Project-URL: Homepage, https://github.com/scottpmiller/evalcore
 Project-URL: Repository, https://github.com/scottpmiller/evalcore
@@ -429,6 +429,9 @@ thresholds:
   win_min_delta: 0.02              # dead band: |delta| <= this -> neutral
   on_regression: warn              # or 'fail' to hard-gate the win metric
   variants: {baseline: baseline, candidate: candidate}   # gate defaults
+  metrics:                         # optional: which way is good, per metric
+    generation_cost: lower_is_better
+    total_tool_calls: neutral      # moves, but neither way is a result
   guardrails:                      # hard constraints on the CANDIDATE
     - metric: false_negative_rate
       max: 0.10                    # absolute ceiling
@@ -441,7 +444,51 @@ thresholds:
 
 Guardrail rules compose: `max`, `min`, `must_not_increase`,
 `must_not_decrease`. A guardrail whose metric is missing on the candidate
-fails closed. Pick guardrails for the failures that must never ship, and
+fails closed.
+
+Every metric also carries a `value_range`, because nothing about a float says
+what it is on: `0.86` is 86% if the metric runs 0..1 and 4.3 out of 5 if it
+runs 1..5. The grader declares it - deterministic checks are 0..1,
+`classification` puts `support_*` and `errors` on 0..unbounded while its
+confusion-matrix metrics are 0..1, and judge metrics are 0..1 except
+`disagreement`, a spread in raw points, which is 0..scale-1. A custom grader
+passes `value_range=` on the `Score`s it returns.
+
+`numeric` is the exception, since it surfaces whatever the adapter put in the
+field, so declare it there:
+
+```yaml
+graders:
+  - type: numeric
+    fields:
+      - {ref: output.cost, range: {min: 0, max: null}}
+      - {ref: output.quality, range: [1, 5]}
+      # `range` and `max` are different things - see below
+      - {ref: output.latency, range: {min: 0}, max: 2.5}
+```
+
+That last field is the distinction worth knowing. `min`/`max` on a numeric
+field are a **pass/fail threshold** - "fail this case over 2.5 seconds" - and
+`range` is what the number could be at all. A cost that must stay under a
+dollar can still cost five, so the two keys stay separate.
+
+`maximum: null` is a statement, not a gap - unbounded above, so not a fraction
+of anything, render it as it is. A metric with no range at all is different
+again: nobody declared one. The engine never infers a range from the values it
+sees, because "everything stayed under 1, so it must be a 0..1 metric" is what
+turns a run whose costs happened to stay cheap into percentages.
+
+Every `MetricDelta` carries `direction` (`improved`/`regressed`/`neutral`)
+alongside `delta`, because the sign of a number does not say what it means: a
+rise in `f1` is an improvement and a rise in `false_negative_rate` is a
+regression. Direction comes from the first of these that speaks to the metric:
+an explicit `metrics:` entry, `win_higher_is_better` for the win metric, or the
+guardrails themselves - a `max` or `must_not_increase` rule means you want the
+metric low, a `min` or `must_not_decrease` rule means high. So a suite that
+gates a metric already declares its direction and needs no `metrics:` block at
+all. A metric nothing speaks to is assumed higher-is-better; one fenced in on
+both sides by a band, or declared `neutral`, reports `direction: neutral` and
+`higher_is_better: null` rather than guessing. Pick guardrails for the failures that must never ship, and
 one win metric for the improvement you're hunting; everything else is
 reported informationally.
 

{evalcore-2.5.0 → evalcore-2.6.0}/README.md

@@ -397,6 +397,9 @@ thresholds:
   win_min_delta: 0.02              # dead band: |delta| <= this -> neutral
   on_regression: warn              # or 'fail' to hard-gate the win metric
   variants: {baseline: baseline, candidate: candidate}   # gate defaults
+  metrics:                         # optional: which way is good, per metric
+    generation_cost: lower_is_better
+    total_tool_calls: neutral      # moves, but neither way is a result
   guardrails:                      # hard constraints on the CANDIDATE
     - metric: false_negative_rate
       max: 0.10                    # absolute ceiling
@@ -409,7 +412,51 @@ thresholds:
 
 Guardrail rules compose: `max`, `min`, `must_not_increase`,
 `must_not_decrease`. A guardrail whose metric is missing on the candidate
-fails closed. Pick guardrails for the failures that must never ship, and
+fails closed.
+
+Every metric also carries a `value_range`, because nothing about a float says
+what it is on: `0.86` is 86% if the metric runs 0..1 and 4.3 out of 5 if it
+runs 1..5. The grader declares it - deterministic checks are 0..1,
+`classification` puts `support_*` and `errors` on 0..unbounded while its
+confusion-matrix metrics are 0..1, and judge metrics are 0..1 except
+`disagreement`, a spread in raw points, which is 0..scale-1. A custom grader
+passes `value_range=` on the `Score`s it returns.
+
+`numeric` is the exception, since it surfaces whatever the adapter put in the
+field, so declare it there:
+
+```yaml
+graders:
+  - type: numeric
+    fields:
+      - {ref: output.cost, range: {min: 0, max: null}}
+      - {ref: output.quality, range: [1, 5]}
+      # `range` and `max` are different things - see below
+      - {ref: output.latency, range: {min: 0}, max: 2.5}
+```
+
+That last field is the distinction worth knowing. `min`/`max` on a numeric
+field are a **pass/fail threshold** - "fail this case over 2.5 seconds" - and
+`range` is what the number could be at all. A cost that must stay under a
+dollar can still cost five, so the two keys stay separate.
+
+`maximum: null` is a statement, not a gap - unbounded above, so not a fraction
+of anything, render it as it is. A metric with no range at all is different
+again: nobody declared one. The engine never infers a range from the values it
+sees, because "everything stayed under 1, so it must be a 0..1 metric" is what
+turns a run whose costs happened to stay cheap into percentages.
+
+Every `MetricDelta` carries `direction` (`improved`/`regressed`/`neutral`)
+alongside `delta`, because the sign of a number does not say what it means: a
+rise in `f1` is an improvement and a rise in `false_negative_rate` is a
+regression. Direction comes from the first of these that speaks to the metric:
+an explicit `metrics:` entry, `win_higher_is_better` for the win metric, or the
+guardrails themselves - a `max` or `must_not_increase` rule means you want the
+metric low, a `min` or `must_not_decrease` rule means high. So a suite that
+gates a metric already declares its direction and needs no `metrics:` block at
+all. A metric nothing speaks to is assumed higher-is-better; one fenced in on
+both sides by a band, or declared `neutral`, reports `direction: neutral` and
+`higher_is_better: null` rather than guessing. Pick guardrails for the failures that must never ship, and
 one win metric for the improvement you're hunting; everything else is
 reported informationally.
 

{evalcore-2.5.0 → evalcore-2.6.0}/examples/quickstart/graders.py

@@ -47,6 +47,10 @@ class AcknowledgesCustomer:
                 grader=self.name,
                 metric=self.name,
                 value=1.0 if ok else 0.0,
+                # 1/0 per case, so the run mean is a share of cases. A
+                # grader knows this about its own output; nothing
+                # downstream can work it out from the numbers alone.
+                value_range=models.MetricRange(minimum=0.0, maximum=1.0),
                 passed=ok,
                 detail=detail,
                 case_id=case.id,
@@ -83,6 +87,7 @@ class DistinctReplyRate:
                 grader=self.name,
                 metric=self.name,
                 value=rate,
+                value_range=models.MetricRange(minimum=0.0, maximum=1.0),
                 detail=f'{distinct} distinct across {len(by_case)} cases',
                 kind='aggregate',
             )

{evalcore-2.5.0 → evalcore-2.6.0}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "evalcore"
-version = "2.5.0"
+version = "2.6.0"
 description = "A generic, consumer-agnostic evaluation engine for prompt, model, and API outputs."
 readme = "README.md"
 requires-python = ">=3.11"

{evalcore-2.5.0 → evalcore-2.6.0}/src/evalcore/compare.py

@@ -118,6 +118,107 @@ def check_thresholds(
     )
 
 
+_POLARITY_WORDS = {
+    'higher_is_better': True,
+    'higher': True,
+    'up': True,
+    'lower_is_better': False,
+    'lower': False,
+    'down': False,
+    'neutral': None,
+    'none': None,
+}
+
+
+def _declared_polarity(thresholds: dict) -> dict[str, bool | None]:
+    """Polarity as the suite states it, under ``thresholds.metrics``.
+
+    A metric mapped to ``neutral`` is present here with a ``None`` value,
+    which is not the same as being absent: it says the suite considered the
+    metric and decided it has no good direction, and that beats inference.
+    An unrecognised word is ignored rather than guessed at.
+    """
+    declared: dict[str, bool | None] = {}
+    for metric, value in (thresholds.get('metrics') or {}).items():
+        if isinstance(value, bool):
+            declared[metric] = value
+        elif isinstance(value, str):
+            word = value.strip().lower()
+            if word in _POLARITY_WORDS:
+                declared[metric] = _POLARITY_WORDS[word]
+    return declared
+
+
+def _inferred_polarity(rules: list[dict]) -> dict[str, bool | None]:
+    """Polarity the guardrail rules already imply.
+
+    A ceiling (``max``, ``must_not_increase``) is only worth writing about a
+    metric you want low; a floor (``min``, ``must_not_decrease``) about one
+    you want high. So a suite that gates a metric has usually stated its
+    direction already without meaning to, and this reads it back - which is
+    what lets existing suites get correct directions with no edits.
+
+    A metric whose rules imply both - a band, ``min`` and ``max`` together -
+    resolves to ``None``, not to the default. Being fenced in on both sides
+    is a statement that neither direction is the good one, and letting it
+    fall through to higher-is-better would turn the one case we know is
+    ambiguous into a confident wrong answer.
+    """
+    votes: dict[str, set[bool]] = {}
+    for rule in rules:
+        metric = rule.get('metric')
+        if not metric:
+            continue
+        seen = votes.setdefault(metric, set())
+        if 'max' in rule or rule.get('must_not_increase'):
+            seen.add(False)
+        if 'min' in rule or rule.get('must_not_decrease'):
+            seen.add(True)
+    return {
+        metric: next(iter(seen)) if len(seen) == 1 else None
+        for metric, seen in votes.items()
+        if seen
+    }
+
+
+def _polarity(thresholds: dict) -> dict[str, bool | None]:
+    """Every metric the suite says something about, weakest source first.
+
+    Inference from the guardrails is the fallback; ``win_higher_is_better``
+    beats it for the win metric because it is a statement rather than a
+    reading; an explicit ``metrics:`` entry beats both. Metrics named by none
+    of the three are absent, and callers treat absent as higher-is-better.
+    """
+    resolved: dict[str, bool | None] = dict(
+        _inferred_polarity(thresholds.get('guardrails', []))
+    )
+    win_metric = thresholds.get('win_metric')
+    if win_metric:
+        resolved[win_metric] = bool(
+            thresholds.get('win_higher_is_better', True)
+        )
+    resolved.update(_declared_polarity(thresholds))
+    return resolved
+
+
+def _direction(
+    delta: float | None, higher_is_better: bool | None, min_delta: float = 0.0
+) -> str:
+    """Turn a signed delta into what it means for this metric.
+
+    ``min_delta`` is the dead band: a move smaller than it is noise, not a
+    result.
+    """
+    if delta is None or higher_is_better is None:
+        return 'neutral'
+    effective = delta if higher_is_better else -delta
+    if effective > min_delta:
+        return 'improved'
+    if effective < -min_delta:
+        return 'regressed'
+    return 'neutral'
+
+
 def _evaluate_win(
     thresholds: dict, baseline: models.Scorecard, candidate: models.Scorecard
 ) -> tuple[str | None, str]:
@@ -128,14 +229,13 @@ def _evaluate_win(
     cand = _metric(candidate, metric)
     if base is None or cand is None:
         return metric, 'neutral'
-    higher_better = thresholds.get('win_higher_is_better', True)
-    min_delta = thresholds.get('win_min_delta', 0.0)
-    delta = cand - base if higher_better else base - cand
-    if delta > min_delta:
-        return metric, 'improved'
-    if delta < -min_delta:
-        return metric, 'regressed'
-    return metric, 'neutral'
+    # Same call the metric's own MetricDelta makes, so the headline verdict
+    # and that row cannot disagree about the same number.
+    return metric, _direction(
+        cand - base,
+        _polarity(thresholds).get(metric, True),
+        thresholds.get('win_min_delta', 0.0),
+    )
 
 
 def compare(
@@ -146,14 +246,33 @@ def compare(
     """Compare two scorecards and produce a gate verdict."""
     thresholds = thresholds or {}
 
+    polarity = _polarity(thresholds)
+    win_metric = thresholds.get('win_metric')
+
     deltas: list[models.MetricDelta] = []
     for metric in sorted(set(baseline.metrics) | set(candidate.metrics)):
         base = _metric(baseline, metric)
         cand = _metric(candidate, metric)
         delta = cand - base if base is not None and cand is not None else None
+        # Absent from the map means nothing in the suite spoke to this
+        # metric, and higher-is-better is the convention almost every score
+        # follows. A declared 'neutral' is present with a None value and is
+        # a different answer: say nothing.
+        higher_is_better = polarity.get(metric, True)
         deltas.append(
             models.MetricDelta(
-                metric=metric, baseline=base, candidate=cand, delta=delta
+                metric=metric,
+                baseline=base,
+                candidate=cand,
+                delta=delta,
+                higher_is_better=higher_is_better,
+                direction=_direction(
+                    delta,
+                    higher_is_better,
+                    thresholds.get('win_min_delta', 0.0)
+                    if metric == win_metric
+                    else 0.0,
+                ),
             )
         )
 

{evalcore-2.5.0 → evalcore-2.6.0}/src/evalcore/graders/classification.py

@@ -89,11 +89,29 @@ class Classification:
         fpr = _safe_div(fp, fp + tn)
         accuracy = _safe_div(tp + tn, tp + tn + fp + fn)
 
-        def agg(metric: str, value: float) -> models.Score:
+        fraction = models.MetricRange(minimum=0.0, maximum=1.0)
+        # Unbounded above, and never negative: these count the rows behind
+        # the fractions rather than being one. maximum=None says that
+        # positively, which is what stops a consumer reading 0.9 support as
+        # 90% of something.
+        tally = models.MetricRange(minimum=0.0, maximum=None)
+
+        def agg(
+            metric: str,
+            value: float,
+            value_range: models.MetricRange = fraction,
+        ) -> models.Score:
             return models.Score(
-                grader=self.name, metric=metric, value=value, kind='aggregate'
+                grader=self.name,
+                metric=metric,
+                value=value,
+                value_range=value_range,
+                kind='aggregate',
             )
 
+        # One grader, two shapes of number, which is why the range sits on
+        # the score and not on the grader.
+        count = tally
         return [
             agg('precision', precision),
             agg('recall', recall),
@@ -101,7 +119,7 @@ class Classification:
             agg('false_negative_rate', fnr),
             agg('false_positive_rate', fpr),
             agg('accuracy', accuracy),
-            agg('support_positive', float(tp + fn)),
-            agg('support_negative', float(tn + fp)),
-            agg('errors', float(errors)),
+            agg('support_positive', float(tp + fn), count),
+            agg('support_negative', float(tn + fp), count),
+            agg('errors', float(errors), count),
         ]

{evalcore-2.5.0 → evalcore-2.6.0}/src/evalcore/graders/deterministic.py

@@ -21,10 +21,13 @@ def _context(case: models.Case, output: models.Output) -> dict:
 
 
 def _score(name: str, metric: str, case_id: str, ok: bool, detail: str):
+    # 1.0/0.0 per case, so the run-level mean is the share of cases that
+    # passed: 0..1 whatever the check itself was looking at.
     return models.Score(
         grader=name,
         metric=metric,
         value=1.0 if ok else 0.0,
+        value_range=models.MetricRange(minimum=0.0, maximum=1.0),
         passed=ok,
         detail=detail,
         case_id=case_id,

{evalcore-2.5.0 → evalcore-2.6.0}/src/evalcore/graders/judge.py

@@ -543,11 +543,18 @@ class RubricJudge:
             metric: str,
             value: float | None,
             judges: list[models.JudgeDetail] | None = None,
+            value_range: models.MetricRange | None = None,
         ) -> models.Score:
+            # A judge score is points / scale, so 0..1 by construction.
+            # `disagreement` is the exception and passes its own range: it
+            # is a spread in RAW points, so it runs 0..scale-1, and reading
+            # it as a fraction understates it by that factor.
             return models.Score(
                 grader=self.name,
                 metric=metric,
                 value=value,
+                value_range=value_range
+                or models.MetricRange(minimum=0.0, maximum=1.0),
                 detail=detail,
                 case_id=case.id,
                 kind='per_case',
@@ -634,7 +641,15 @@ class RubricJudge:
                 spreads.append(max(raws) - min(raws))
         mean_spread = sum(spreads) / len(spreads) if spreads else 0.0
         max_spread = max(spreads) if spreads else 0.0
-        out.append(score(f'{self.name}.disagreement', mean_spread))
+        out.append(
+            score(
+                f'{self.name}.disagreement',
+                mean_spread,
+                value_range=models.MetricRange(
+                    minimum=0.0, maximum=float(self.scale - 1)
+                ),
+            )
+        )
         out.append(
             score(
                 f'{self.name}.flagged',

{evalcore-2.5.0 → evalcore-2.6.0}/src/evalcore/graders/numeric.py

@@ -26,6 +26,24 @@ def _context(case: models.Case, output: models.Output) -> dict:
     }
 
 
+def _range(spec: dict) -> models.MetricRange | None:
+    """The metric's declared range, if the suite gave one.
+
+    Deliberately a separate key from the ``min``/``max`` bounds beside it.
+    Those are a pass/fail threshold - ``max: 1.0`` on a cost means "fail
+    over a dollar" - and the range is what the number could be at all. A
+    cost that must stay under a dollar can still cost five.
+    """
+    declared = spec.get('range')
+    if isinstance(declared, dict):
+        return models.MetricRange(
+            minimum=declared.get('min'), maximum=declared.get('max')
+        )
+    if isinstance(declared, (list, tuple)) and len(declared) == 2:
+        return models.MetricRange(minimum=declared[0], maximum=declared[1])
+    return None
+
+
 class _Field:
     """One resolved field spec: a ``$ref`` selector, a metric name, bounds."""
 
@@ -37,6 +55,12 @@ class _Field:
         self.maximum = spec.get('max')
         # Metric name defaults to the ref's leaf (``output.cost`` -> ``cost``).
         self.metric = spec.get('name') or self.ref.rsplit('.', 1)[-1]
+        # The one grader that cannot know its own range: the value is
+        # whatever the adapter put in the field, so a dollar cost, a token
+        # count and a rate all arrive as a bare float. The suite declares it
+        # or nobody does, and nobody is an honest answer, not a reason to
+        # guess.
+        self.value_range = _range(spec)
 
 
 def _as_float(value) -> float | None:
@@ -69,6 +93,7 @@ class Numeric:
                     grader=self.name,
                     metric=field.metric,
                     value=value,
+                    value_range=field.value_range,
                     passed=passed,
                     detail=detail,
                     case_id=case.id,

{evalcore-2.5.0 → evalcore-2.6.0}/src/evalcore/models.py

@@ -77,6 +77,36 @@ class JudgeDetail(pydantic.BaseModel):
     overall: float | None = None
 
 
+class MetricRange(pydantic.BaseModel):
+    """The values a metric can take, when the grader knows them.
+
+    Nothing about a float says what it is on. ``0.86`` is 86% if the metric
+    runs 0..1 and 4.3 out of 5 if it runs 1..5, and a consumer holding only
+    the number has to guess - usually from magnitude, which is how a run
+    whose costs happened to stay under a dollar gets rendered as
+    percentages, and how the same metric gets classified differently
+    depending on the window you look at.
+
+    ``maximum=None`` is a real answer rather than a missing one: the metric
+    is unbounded above - a count, a cost, an elapsed time - so it is not a
+    fraction of anything and should be shown as it is. That is different
+    from carrying no range at all, which says only that nobody declared one.
+    """
+
+    #: Frozen because a range is a value, not a record: two graders that
+    #: declare 0..1 for the same metric agree, and aggregation compares them
+    #: by equality in a set.
+    model_config = pydantic.ConfigDict(frozen=True)
+
+    minimum: float | None = None
+    maximum: float | None = None
+
+    @property
+    def is_fraction(self) -> bool:
+        """0..1, so the value can be rendered as a percentage."""
+        return self.minimum == 0.0 and self.maximum == 1.0
+
+
 class Score(pydantic.BaseModel):
     """A single metric emitted by a grader for one case (or aggregate).
 
@@ -88,6 +118,9 @@ class Score(pydantic.BaseModel):
     grader: str
     metric: str
     value: float | None = None
+    #: what the value is on; see :class:`MetricRange`. ``None`` means the
+    #: grader did not declare one, not that the metric is unbounded.
+    value_range: MetricRange | None = None
     passed: bool | None = None
     detail: str | None = None
     case_id: str | None = None
@@ -128,6 +161,10 @@ class MetricValue(pydantic.BaseModel):
     kind: typing.Literal['mean', 'aggregate']
     n: int
     stdev: float | None = None
+    #: carried up from the scores behind it, unchanged; see
+    #: :class:`MetricRange`. A mean sits on the same range its observations
+    #: did, so averaging does not move it.
+    value_range: MetricRange | None = None
 
 
 class Scorecard(pydantic.BaseModel):
@@ -446,12 +483,26 @@ class PairwiseAgreement(pydantic.BaseModel):
 
 
 class MetricDelta(pydantic.BaseModel):
-    """Per-metric baseline->candidate comparison."""
+    """Per-metric baseline->candidate comparison.
+
+    ``delta`` is the arithmetic; ``direction`` is what it means. They are not
+    the same thing and only the second one is safe to colour green: a rise in
+    ``f1`` is an improvement and a rise in ``false_negative_rate`` is a
+    regression, and nothing about the number says which.
+
+    ``higher_is_better`` is the fact the direction was derived from, kept so a
+    consumer storing these rows does not have to re-derive it. ``None`` means
+    the suite declared the metric ``neutral``, or the engine had nothing to
+    infer from and would rather say nothing than guess - in which case
+    ``direction`` is ``neutral`` whatever the delta was.
+    """
 
     metric: str
     baseline: float | None
     candidate: float | None
     delta: float | None
+    higher_is_better: bool | None = None
+    direction: typing.Literal['improved', 'regressed', 'neutral'] = 'neutral'
 
 
 class Comparison(pydantic.BaseModel):

{evalcore-2.5.0 → evalcore-2.6.0}/src/evalcore/runner.py

@@ -113,6 +113,28 @@ def _grader_info(*grader_groups) -> dict[str, models.GraderInfo]:
     }
 
 
+def _metric_range(scores: list[models.Score]) -> models.MetricRange | None:
+    """The range of a metric, carried up from the scores that produced it.
+
+    Declared or nothing. A mean sits on the same range its observations did,
+    so aggregation moves the value but never the range.
+
+    There is deliberately no inference from the values themselves. "Every
+    observation landed inside 0..1, so the metric runs 0..1" is what renders
+    a run whose costs happened to stay under a dollar as percentages, and it
+    answers differently depending on which window you ask about. A grader
+    that knows its range says so in one keyword; one that does not is not
+    improved by a guess that looks the same as knowledge downstream.
+
+    Two graders declaring different ranges for one metric is a bug in the
+    suite, not something to reconcile, so that comes back ``None`` too.
+    """
+    declared = {
+        score.value_range for score in scores if score.value_range is not None
+    }
+    return declared.pop() if len(declared) == 1 else None
+
+
 def _aggregate_metrics(
     results: list[models.CaseResult], agg_scores: list[models.Score]
 ) -> dict[str, models.MetricValue]:
@@ -120,8 +142,10 @@ def _aggregate_metrics(
     metrics: dict[str, models.MetricValue] = {}
 
     by_metric: dict[str, list[float]] = {}
+    scores_by_metric: dict[str, list[models.Score]] = {}
     for result in results:
         for score in result.scores:
+            scores_by_metric.setdefault(score.metric, []).append(score)
             if score.value is not None:
                 by_metric.setdefault(score.metric, []).append(score.value)
     for metric, values in by_metric.items():
@@ -131,6 +155,7 @@ def _aggregate_metrics(
             stdev=statistics.stdev(values) if len(values) > 1 else None,
             kind='mean',
             n=len(values),
+            value_range=_metric_range(scores_by_metric[metric]),
         )
 
     for score in agg_scores:
@@ -139,6 +164,7 @@ def _aggregate_metrics(
             value=score.value,
             kind='aggregate',
             n=len(results),
+            value_range=_metric_range([score]),
         )
     return metrics