structured-eval 0.1.0__tar.gz → 0.2.0__tar.gz

structured_eval-0.2.0/PKG-INFO

@@ -0,0 +1,426 @@
+Metadata-Version: 2.4
+Name: structured-eval
+Version: 0.2.0
+Summary: The LLM Structured Output Evaluation Framework
+License: Apache-2.0
+Project-URL: Homepage, https://github.com/kirillpechurin/structured-eval
+Project-URL: Repository, https://github.com/kirillpechurin/structured-eval
+Project-URL: Issues, https://github.com/kirillpechurin/structured-eval/issues
+Keywords: structured llm evaluation,structured eval,json eval,llm,structured output,pydantic
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: Apache Software License
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Topic :: Software Development :: Testing
+Classifier: Typing :: Typed
+Requires-Python: >=3.12
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: pydantic>=2.5.0
+Provides-Extra: yaml
+Requires-Dist: pyyaml>=6.0.1; extra == "yaml"
+Provides-Extra: fuzzy
+Requires-Dist: rapidfuzz>=3.0.0; extra == "fuzzy"
+Provides-Extra: jsonschema
+Requires-Dist: jsonschema>=4.20.0; extra == "jsonschema"
+Provides-Extra: rules
+Requires-Dist: jsonpath-ng>=1.6.0; extra == "rules"
+Provides-Extra: diff
+Requires-Dist: deepdiff>=7.0.0; extra == "diff"
+Provides-Extra: align
+Requires-Dist: scipy>=1.13.0; extra == "align"
+Provides-Extra: deepeval
+Requires-Dist: deepeval>=3.0.0; extra == "deepeval"
+Provides-Extra: langsmith
+Requires-Dist: langsmith>=0.8.0; extra == "langsmith"
+Provides-Extra: all
+Requires-Dist: structured-eval[align,diff,fuzzy,jsonschema,rules,yaml]; extra == "all"
+Dynamic: license-file
+
+# structured-eval
+
+[![CI](https://github.com/kirillpechurin/structured-eval/actions/workflows/ci.yml/badge.svg)](https://github.com/kirillpechurin/structured-eval/actions/workflows/ci.yml)
+[![codecov](https://codecov.io/gh/kirillpechurin/structured-eval/graph/badge.svg)](https://codecov.io/gh/kirillpechurin/structured-eval)
+[![PyPI](https://img.shields.io/pypi/v/structured-eval.svg)](https://pypi.org/project/structured-eval/)
+[![Python versions](https://img.shields.io/pypi/pyversions/structured-eval.svg)](https://pypi.org/project/structured-eval/)
+[![License](https://img.shields.io/pypi/l/structured-eval.svg)](https://github.com/kirillpechurin/structured-eval/blob/main/LICENSE)
+
+**The LLM Structured Output Evaluation Framework**
+
+When interacting with an LLM, returning structured data is a common task.
+And in any such integration, the quality of the returned data deserves attention —
+it should be evaluated with the appropriate tools.
+However, quality evaluation of structured data is often limited to the following:
+
+- Parses correctly
+- Data types are correct
+- Required fields are present and there are no extra fields
+
+That is not enough to be fully confident the data is correct, because these
+mechanisms target structural validation, not the values themselves.
+
+**structured-eval** focuses on evaluating the quality of an LLM's structured output,
+such as JSON or YAML. The library makes it possible to compare
+values, objects, and arrays at every level of nesting, which lets you check
+
+- Which fields matched or are close to the expected ones, and which are not
+- Whether values are grounded in a source
+- Whether fields are logically consistent with one another
+
+This kind of field-level evaluation gives you a systematic way to target improvements.
+A few examples of what you can learn using structured-eval:
+
+- How close an output value is to the expected one
+- Which scalars, objects, and arrays have problems most often
+- At dataset scale — how stable the LLM is
+
+## What it checks
+
+Correctness is a set of checks, where each check builds on the ones below it:
+
+| Level | Check                                     | How you can check it         |
+|-------|-------------------------------------------|------------------------------|
+| L0    | Parses correctly                          | structured-eval / jsonschema |
+| L1    | Data types are correct                    | structured-eval / jsonschema |
+| L2    | Required fields are present               | structured-eval / jsonschema |
+| L3    | No extra fields                           | structured-eval / jsonschema |
+| L4    | Values match or are close to the expected | structured-eval              |
+| L5    | Values are grounded in a source           | structured-eval              |
+| L6    | Fields are logically consistent           | structured-eval              |
+
+**The value of structured-eval is to provide a complete quality evaluation of
+structured data.**
+
+## Install
+
+```bash
+pip install structured-eval          # core depends only on Pydantic
+pip install "structured-eval[all]"   # + YAML, fuzzy, schema, rules, scipy alignment…
+```
+
+By default only `pydantic` is required, but to extend functionality there are
+[extras](docs/getting-started.md#install) — install only what you need.
+
+## Quick start
+
+The example uses a course record. The expected and the actual are structurally
+identical, but several values differ. A small evaluation config says *how* to
+judge each field, and the report shows where the data diverges:
+
+```python
+from structured_eval import evaluate
+from structured_eval.models import EvalConfig, FieldConfig
+from structured_eval.metrics import Numeric, TokenF1
+
+expected = {
+    "course_id": "COURSE-101",
+    "title": "Introduction to Python",
+    "published": True,
+    "duration_hours": 12,
+    "rating": 4.8,
+    "modules": [
+        {"name": "Basics", "lessons": 5},
+        {"name": "Functions", "lessons": 4},
+        {"name": "Classes", "lessons": 3},
+    ],
+}
+
+actual = {
+    "course_id": "COURSE-101",
+    "title": "Intro to Python",  # paraphrased
+    "published": True,
+    "duration_hours": 10,  # off by 2
+    "rating": 4.5,  # off by 0.3
+    "modules": [
+        {"name": "Basics", "lessons": 5},
+        {"name": "Functions", "lessons": 4},
+    ],  # the "Classes" module is missing
+}
+
+config = EvalConfig(fields={
+    "title": FieldConfig(metrics=[TokenF1()]),  # reward paraphrases
+    "duration_hours": FieldConfig(metrics=[Numeric(tolerance=2)]),
+    "rating": FieldConfig(metrics=[Numeric(tolerance=0.5)]),  # close enough is fine
+})
+
+report = evaluate(actual, expected, config)
+
+report.score  # 0.8889 — close, with the gaps pinpointed
+report.field_scores["title"].score  # 0.6667 — a paraphrase gets partial credit
+report.field_scores["duration_hours"].score  # 1.0 — within tolerance
+report.field_scores["modules"].score  # 0.6667 — 2 of 3 modules recovered
+report.field_scores["modules[0]"].score  # 1.0 — the first module is spot-on
+```
+
+You can assign several metrics to a single field and make the field's
+representative (`key_metric`) — the score that lands in the total — a separate
+aggregating metric over them. For example, `CompositeScore` blends metrics with
+given weights, while `MeanScore` takes their plain mean; metrics not included in
+the representative are still computed alongside, for detail:
+
+```python
+from structured_eval import evaluate
+from structured_eval.models import EvalConfig, FieldConfig
+from structured_eval.metrics import (
+    CompositeScore, ExactMatch, Fuzzy, MeanScore, Numeric, NumericCloseness, TokenF1,
+)
+
+config = EvalConfig(fields={
+    # the field's representative — a weighted blend of token_f1 and fuzzy (exact_match is for detail only)
+    "title": FieldConfig(
+        metrics=[ExactMatch(), TokenF1(), Fuzzy()],
+        key_metric=CompositeScore(weights={"exact_match": 0.1, "token_f1": 0.6, "fuzzy": 0.3}),
+    ),
+    # the field's representative — the plain mean of two numeric metrics
+    "rating": FieldConfig(
+        metrics=[Numeric(tolerance=0.5), NumericCloseness()],
+        key_metric=MeanScore(),
+    ),
+})
+
+report = evaluate(
+    {"title": "Intro to Python", "rating": 4.5},
+    {"title": "Introduction to Python", "rating": 4.8},
+    config,
+)
+
+report.field_scores["title"].score  # 0.6432  — CompositeScore: 0.1·exact_match + 0.6·token_f1 + 0.3·fuzzy
+report.field_scores["title"].metrics["exact_match"]  # 0.0 — computed alongside, for detail
+report.field_scores["rating"].score  # 0.96875 — MeanScore: mean of numeric and numeric_closeness
+```
+
+This is the key idea: *comparison is a metric*, not a separate "matcher" with a
+pre-computed similarity. Learn more —
+[comparison is a metric](docs/core-concepts/comparison-is-a-metric.md).
+
+Every field is scored — nested objects and array elements of any depth included —
+so you see not a single pass/fail, but exactly which fields match and which
+don't.
+
+### Sensible default metrics
+
+**structured-eval** ships a default metric for every node type, so you only
+configure the fields where the default isn't what you want. With no config at
+all, the same data is scored by these default metrics:
+
+```python
+report = evaluate(actual, expected)  # no config
+
+report.score  # 0.4444 — scored entirely by the defaults
+report.field_scores["title"].score  # 0.0  — exact match: "Intro to Python" ≠ "Introduction to Python"
+```
+
+Each node type gets a structural default, and every node's headline score (its
+*representative*) defaults to the mean of its own metrics:
+
+| Node                | Default metric   | What it does                                    |
+|---------------------|------------------|-------------------------------------------------|
+| scalar (leaf)       | `ExactMatch`     | the value must match exactly                    |
+| object              | `ObjectAccuracy` | mean correctness of its fields                  |
+| array               | `ArrayAccuracy`  | mean correctness of its aligned elements        |
+| any node (headline) | `MeanScore`      | the node's representative = mean of its metrics |
+
+Exact match is a strict baseline. Tuning metrics per field, as in the first
+example, is how you tell the evaluator what "close enough" means for *your* data.
+The defaults and the representative score are covered in
+[the evaluation model](docs/core-concepts/evaluation-model.md) and the
+[metric catalog](docs/metrics/index.md).
+
+## Explore — every level of correctness
+
+structured-eval covers the whole ladder, L0 through L6. Behind each level there
+is a tool and a concept page:
+
+### Structural checks
+
+Levels L0–L3 let you check successful parsing, data types, required fields, and
+the absence of extra fields.
+
+Validate against a Pydantic model or JSON Schema, with no ground-truth answer:
+
+```python
+from pydantic import BaseModel
+from structured_eval import evaluate
+from structured_eval.models import EvalConfig
+from structured_eval.metrics import SchemaValidity
+
+
+class Course(BaseModel):
+    title: str
+    duration_hours: int
+
+
+report = evaluate(
+    actual={"title": "ML", "duration_hours": "twelve"},
+    expected=None,
+    config=EvalConfig(key_metric=SchemaValidity(Course))
+)
+report.score  # 0.0
+report.metrics["schema_validity"].root().extra["schema_errors"]
+#   {'type_errors': ['duration_hours'], 'missing_required': [], 'extra_fields': []}
+```
+
+### Value correctness
+
+Level L4 lets you choose how flexibly to judge values,
+on your own criteria — for leaves as well as objects and arrays.
+
+**structured-eval** provides a large set of metrics, see more in the
+[metric catalog](docs/metrics/index.md).
+
+```python
+from structured_eval import evaluate
+from structured_eval.models import EvalConfig, FieldConfig
+from structured_eval.metrics import Numeric, TokenF1
+
+report = evaluate(
+    actual={"title": "Intro to Python", "duration_hours": 11},
+    expected={"title": "Introduction to Python", "duration_hours": 12},
+    config=EvalConfig(fields={
+        "title": FieldConfig(metrics=[TokenF1()]),  # token overlap
+        "duration_hours": FieldConfig(metrics=[Numeric(tolerance=2)]),
+    }),
+)
+report.field_scores["title"].score  # 0.6667 — partial credit for a paraphrase
+report.field_scores["duration_hours"].score  # 1.0 — within tolerance
+```
+
+Leaf scores can be rolled up into a metric on the object or array. The base
+metrics for such structures are precision, recall, F1 — see more in the full
+[metric catalog](docs/metrics/index.md).
+
+```python
+from structured_eval import evaluate
+from structured_eval.models import EvalConfig
+from structured_eval.metrics import ObjectF1
+
+report = evaluate(
+    actual={"a": 1, "b": 9},
+    expected={"a": 1, "b": 2, "c": 3},
+    config=EvalConfig(metrics=[ObjectF1()])
+)
+
+report.metrics["object_f1"].root()  # 0.4
+```
+
+For an array of objects the elements are first *aligned* (which actual element
+corresponds to which expected one), and then each pair is evaluated field by
+field. You set the strategy on `ArrayFieldConfig` — for example, `by_key` matches
+elements by a key, and array order stops mattering:
+
+```python
+from structured_eval import evaluate
+from structured_eval.models import ArrayFieldConfig, ArrayStrategy, EvalConfig
+from structured_eval.metrics import ArrayF1
+
+config = EvalConfig(fields={"items": ArrayFieldConfig(
+    strategy=ArrayStrategy.BY_KEY,  # match elements by the sku key
+    params={"key": "sku"},
+    metrics=[ArrayF1()],
+)})
+
+report = evaluate(
+    {"items": [{"sku": "B", "qty": 5}, {"sku": "A", "qty": 2}]},
+    {"items": [{"sku": "A", "qty": 2}, {"sku": "B", "qty": 3}]},
+    config,
+)
+
+report.array_matches["items"].matched  # [(0, 1), (1, 0)] — A↔A, B↔B despite the order
+report.field_scores["items"].score  # 0.5  — A matched (qty 2), B didn't (qty 5 ≠ 3)
+```
+
+Learn more about strategies in
+[array alignment](docs/core-concepts/array-alignment.md).
+
+### Source grounding
+
+Level L5 lets you catch hallucinations by checking each value against a source.
+Note that `expected` is not required for the computation.
+
+Learn more — [field faithfulness](docs/metrics/catalog/field_faithfulness.md).
+
+```python
+from structured_eval import evaluate
+from structured_eval.models import EvalConfig
+from structured_eval.metrics import FieldFaithfulness
+
+report = evaluate(
+    actual={"title": "Introduction to Python", "duration_hours": 40},
+    expected=None,
+    config=EvalConfig(metrics=[FieldFaithfulness()]),
+    source="Course: Introduction to Python. Duration: 12 hours.",
+)
+
+report.metrics["field_faithfulness"].by_path  # {'title': 1.0, 'duration_hours': 0.0 ← 40 ≠ 12}
+```
+
+### Logical consistency of values
+
+Level L6 offers an interface for describing cross-field business rules with a
+small DSL.
+
+Learn more — [rule pass rate](docs/metrics/catalog/rule-pass-rate.md).
+
+```python
+from structured_eval import evaluate
+from structured_eval.models import EvalConfig
+from structured_eval.metrics import Rule, RulePassRate
+
+report = evaluate(
+    actual={"subtotal": 100, "tax": 20, "total": 130},
+    expected=None,
+    config=EvalConfig(key_metric=RulePassRate([Rule("$.total").eq("$.subtotal + $.tax")]))
+)
+
+report.score  # 0.0  — 130 ≠ 120
+```
+
+## Scale it
+
+**structured-eval** offers ways to evaluate
+
+- A whole dataset
+- Prompt stability across repeated runs
+
+```python
+from structured_eval import evaluate_batch, evaluate_consistency
+from structured_eval.models import Sample
+
+# one report per sample + dataset aggregates
+batch = evaluate_batch([
+    Sample(actual={"x": 1}, expected={"x": 1}),
+    Sample(actual={"x": 1}, expected={"x": 2}),
+])
+batch.score  # 0.5
+batch.perfect_response_rate  # 0.5
+
+# repeated runs of the same prompt → which fields drift?
+runs = [
+    Sample(actual={"sentiment": "positive", "score": 0.9}, expected={"sentiment": "positive", "score": 0.9}),
+    Sample(actual={"sentiment": "positive", "score": 0.9}, expected={"sentiment": "positive", "score": 0.9}),
+    Sample(actual={"sentiment": "neutral", "score": 0.9}, expected={"sentiment": "positive", "score": 0.9}),
+]
+report = evaluate_consistency(runs, variance_threshold=0.05)
+report.stable_fields  # ['score']
+report.unstable_fields  # ['sentiment'] — flipped on one run
+```
+
+## Documentation
+
+- **[Introduction](docs/introduction.md)** — the L0–L6 ladder and why values matter.
+- **[Getting started](docs/getting-started.md)** — install → first evaluation →
+  reading and tuning the report.
+- **Core concepts** — [the evaluation model](docs/core-concepts/evaluation-model.md) ·
+  [comparison is a metric](docs/core-concepts/comparison-is-a-metric.md) ·
+  [array alignment](docs/core-concepts/array-alignment.md)
+- **[Evaluation functions](docs/evaluation/index.md)** — `evaluate`,
+  `evaluate_batch`, `evaluate_consistency`.
+- **[Metric catalog](docs/metrics/index.md)** — every metric, plus how to write your own.
+
+## License
+
+Apache-2.0 — see [LICENSE](LICENSE).