moju 2.3.2__tar.gz → 2.4.0__tar.gz

{moju-2.3.2 → moju-2.4.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: moju
-Version: 2.3.2
+Version: 2.4.0
 Summary: Physics admissibility auditing for ML surrogates and simulation data: scores how well predicted fields satisfy governing laws and constitutive relations, with implied-parameter checks and admissibility tiers, for JAX and PyTorch.
 Author-email: "Ifimo Lab (Ifimo Global Solutions LLC)" <lab@ifimoanalytics.com>
 Maintainer-email: "Ifimo Global Solutions LLC (dba Ifimo Lab)" <info@ifimoanalytics.com>
@@ -80,6 +80,32 @@ Moju is an open-source physics admissibility auditing library for scientific ML
 
 A tier is computed from an admissibility score in `[0, 1]`: High at `≥ 0.909`, Moderate at `≥ 0.667`, Low at `≥ 0.5`, and Non-Admissible below that (see `moju.monitor.auditor.admissibility_level`). A passing tier indicates consistency with the stated physics within the audited conditions: the laws, constitutive models, sample points, and scales you supplied. It is not a guarantee of correctness.
 
+### Evaluation coverage
+
+Coverage counts residual elements, including trailing vector/tensor components.
+An evaluation mask excludes all components at each excluded spatial point.
+Audit-scope monitor-log exports preserve the non-finite policy recorded by
+`audit`, including incomplete per-key verdicts and null scores in JSON series.
+
+NaN-aware reductions may score on the finite subset of a residual. Every scored key and the aggregate now carry an evaluation-coverage record (`n_total`, `n_masked`, `n_selected`, `n_finite`, `n_nonfinite`, `coverage_status`). Default `nonfinite_policy="flag"` keeps scores and tiers unchanged; set `"strict"` to treat incomplete or empty coverage as tier **Incomplete** (not assessable — distinct from **Not identifiable**). Under `"flag"`, a NaN residual is dropped from the score while an infinite residual enters the RMS and drives the score to 0; both are reported as `coverage_status = "incomplete"`. `"strict"` treats NaN and inf identically (null score, Incomplete).
+
+**Known limitation.** Coverage counts non-finite values in the residual array. A required input that is non-finite while the law's residual stays finite (for example, a law that applies `jnp.where` or `zeros_like` to that input) is currently reported as `coverage_status = "ok"`. Counting non-finite inputs per sample needs a mapping from input samples to residual samples that holds for scalars, parameters and broadcast shapes; this is planned for a follow-up release.
+
+```python
+import jax.numpy as jnp
+from moju.monitor import ResidualEngine, LawSpec, Scoring, audit
+
+eng = ResidualEngine(
+    laws=[LawSpec(name="unit_law", fn=lambda r: r, state_map={"r": "r"},
+                  scoring=Scoring("rms", dimensionless=True))],
+    law_implied_audits=False,
+)
+eng.compute_residuals({"r": jnp.array([0.0, float("nan")])}, run_mode="eval")
+report = audit(eng.log)  # default flag: High score, coverage_status="incomplete"
+print(report["coverage_status"])
+print(report["per_key"]["laws/unit_law"]["evaluation_coverage"])
+```
+
 ## What Moju Does
 
 - Builds residuals from composable `Laws`, `Groups`, and `Models`.

{moju-2.3.2 → moju-2.4.0}/README.md

@@ -31,6 +31,32 @@ Moju is an open-source physics admissibility auditing library for scientific ML
 
 A tier is computed from an admissibility score in `[0, 1]`: High at `≥ 0.909`, Moderate at `≥ 0.667`, Low at `≥ 0.5`, and Non-Admissible below that (see `moju.monitor.auditor.admissibility_level`). A passing tier indicates consistency with the stated physics within the audited conditions: the laws, constitutive models, sample points, and scales you supplied. It is not a guarantee of correctness.
 
+### Evaluation coverage
+
+Coverage counts residual elements, including trailing vector/tensor components.
+An evaluation mask excludes all components at each excluded spatial point.
+Audit-scope monitor-log exports preserve the non-finite policy recorded by
+`audit`, including incomplete per-key verdicts and null scores in JSON series.
+
+NaN-aware reductions may score on the finite subset of a residual. Every scored key and the aggregate now carry an evaluation-coverage record (`n_total`, `n_masked`, `n_selected`, `n_finite`, `n_nonfinite`, `coverage_status`). Default `nonfinite_policy="flag"` keeps scores and tiers unchanged; set `"strict"` to treat incomplete or empty coverage as tier **Incomplete** (not assessable — distinct from **Not identifiable**). Under `"flag"`, a NaN residual is dropped from the score while an infinite residual enters the RMS and drives the score to 0; both are reported as `coverage_status = "incomplete"`. `"strict"` treats NaN and inf identically (null score, Incomplete).
+
+**Known limitation.** Coverage counts non-finite values in the residual array. A required input that is non-finite while the law's residual stays finite (for example, a law that applies `jnp.where` or `zeros_like` to that input) is currently reported as `coverage_status = "ok"`. Counting non-finite inputs per sample needs a mapping from input samples to residual samples that holds for scalars, parameters and broadcast shapes; this is planned for a follow-up release.
+
+```python
+import jax.numpy as jnp
+from moju.monitor import ResidualEngine, LawSpec, Scoring, audit
+
+eng = ResidualEngine(
+    laws=[LawSpec(name="unit_law", fn=lambda r: r, state_map={"r": "r"},
+                  scoring=Scoring("rms", dimensionless=True))],
+    law_implied_audits=False,
+)
+eng.compute_residuals({"r": jnp.array([0.0, float("nan")])}, run_mode="eval")
+report = audit(eng.log)  # default flag: High score, coverage_status="incomplete"
+print(report["coverage_status"])
+print(report["per_key"]["laws/unit_law"]["evaluation_coverage"])
+```
+
 ## What Moju Does
 
 - Builds residuals from composable `Laws`, `Groups`, and `Models`.

{moju-2.3.2 → moju-2.4.0}/moju/__init__.py

@@ -1,3 +1,3 @@
 """Moju: Physics-AI supervision for engineering-grade simulations."""
 
-__version__ = "2.3.2"
+__version__ = "2.4.0"

{moju-2.3.2 → moju-2.4.0}/moju/monitor/__init__.py

@@ -45,6 +45,12 @@ from moju.monitor.implied_scoring import (
     resolve_implied_scoring_options,
     score_from_r_max,
 )
+from moju.monitor.evaluation_coverage import (
+    DEFAULT_NONFINITE_POLICY,
+    TIER_INCOMPLETE,
+    coverage_from_residual,
+    validate_nonfinite_policy,
+)
 from moju.monitor.regions import (
     DEFAULT_REGION_MIN_POINTS,
     normalize_regions,
@@ -101,9 +107,11 @@ from moju.monitor.visualize_labels import pretty_category_name, pretty_residual_
 from moju.monitor.tiers import (
     TIER_CUTOFFS,
     TIER_DEFINITION,
+    TIER_NON_SCORED,
     TIER_ORDER,
     tier_definition,
     tier_for_score,
+    tier_rank,
 )
 from moju.monitor.types import (
     REPORT_SCHEMA_VERSION,
@@ -154,6 +162,10 @@ __all__ = [
     "parse_implied_scoring_options",
     "resolve_implied_scoring_options",
     "score_from_r_max",
+    "DEFAULT_NONFINITE_POLICY",
+    "TIER_INCOMPLETE",
+    "coverage_from_residual",
+    "validate_nonfinite_policy",
     "DEFAULT_REGION_MIN_POINTS",
     "normalize_regions",
     "percentile_tail_adequacy",
@@ -176,9 +188,11 @@ __all__ = [
     "merge_implied_groups_first",
     "TIER_CUTOFFS",
     "TIER_DEFINITION",
+    "TIER_NON_SCORED",
     "TIER_ORDER",
     "tier_definition",
     "tier_for_score",
+    "tier_rank",
     "REPORT_SCHEMA_VERSION",
     "AuditReport",
     "BoundCheck",

{moju-2.3.2 → moju-2.4.0}/moju/monitor/auditor.py

@@ -381,7 +381,7 @@ def _r_eff_scalar(x: jnp.ndarray) -> jnp.ndarray:
     return rms_r * jnp.power(q, p)
 
 
-def admissibility_level(score: float) -> str:
+def admissibility_level(score: Optional[float]) -> str:
     """
     Map admissibility score in ``[0, 1]`` to a qualitative label (``Unknown`` if non-finite).
 
@@ -389,14 +389,25 @@ def admissibility_level(score: float) -> str:
     ``scale_k`` (:data:`ADM_HIGH_THRESHOLD`, :data:`ADM_MODERATE_THRESHOLD`,
     :data:`ADM_LOW_THRESHOLD`). High requires ``score >= ADM_HIGH_THRESHOLD`` (~90.9% at ±0.1%
     fractional RMS); Moderate and Low use inclusive lower bounds at ~66.7% and 50%.
+
+    ``None`` maps to ``Incomplete`` (not assessable under ``nonfinite_policy=\"strict\"``;
+    distinct from ``Not identifiable``).
     """
-    if not math.isfinite(score):
+    if score is None:
+        from moju.monitor.evaluation_coverage import TIER_INCOMPLETE
+
+        return TIER_INCOMPLETE
+    try:
+        s = float(score)
+    except (TypeError, ValueError):
         return "Unknown"
-    if score >= ADM_HIGH_THRESHOLD:
+    if not math.isfinite(s):
+        return "Unknown"
+    if s >= ADM_HIGH_THRESHOLD:
         return "High Admissibility"
-    if score >= ADM_MODERATE_THRESHOLD:
+    if s >= ADM_MODERATE_THRESHOLD:
         return "Moderate Admissibility"
-    if score >= ADM_LOW_THRESHOLD:
+    if s >= ADM_LOW_THRESHOLD:
         return "Low Admissibility"
     return "Non-Admissible"
 
@@ -982,6 +993,103 @@ def _apply_implied_scoring_to_entry(
         entry["implied_scoring"] = implied_scoring
 
 
+def _apply_evaluation_coverage_policy(
+    report: Dict[str, Any],
+    entry: Mapping[str, Any],
+    *,
+    policy: str,
+    last_residual_dict: Optional[Dict[str, Any]] = None,
+    eval_mask_label: Optional[str] = None,
+) -> None:
+    """Attach coverage records and apply ``nonfinite_policy`` to per-key / overall."""
+    from moju.monitor.evaluation_coverage import (
+        TIER_INCOMPLETE,
+        aggregate_coverage,
+        apply_nonfinite_policy_to_key,
+        coverage_map_from_flat,
+        is_incomplete_key,
+        validate_nonfinite_policy,
+    )
+
+    policy = validate_nonfinite_policy(policy)
+    cov_map: Dict[str, Any] = {}
+    raw = entry.get("evaluation_coverage")
+    if isinstance(raw, dict) and raw:
+        cov_map = {str(k): dict(v) for k, v in raw.items() if isinstance(v, dict)}
+    elif last_residual_dict is not None:
+        flat: Dict[str, Any] = {}
+        for cat, content in last_residual_dict.items():
+            if cat == "closure_debug" or not isinstance(content, dict):
+                continue
+            for name, arr in content.items():
+                flat[f"{cat}/{name}"] = arr
+        cov_map = coverage_map_from_flat(
+            flat,
+            eval_mask_label=eval_mask_label
+            or (entry.get("eval_mask") or {}).get("label"),
+            mask_meta=entry.get("eval_mask") if isinstance(entry.get("eval_mask"), dict) else None,
+        )
+    per_key = report.get("per_key") or {}
+    incomplete_keys: List[str] = []
+    for k, row in per_key.items():
+        if not isinstance(row, dict):
+            continue
+        cov = cov_map.get(k)
+        apply_nonfinite_policy_to_key(row, cov, policy=policy)
+        if is_incomplete_key(row):
+            incomplete_keys.append(str(k))
+    if cov_map:
+        report["evaluation_coverage"] = {
+            "per_key": cov_map,
+            "aggregate": aggregate_coverage(cov_map),
+        }
+        report["coverage_status"] = report["evaluation_coverage"]["aggregate"]["coverage_status"]
+    report["nonfinite_policy"] = policy
+    if policy == "strict" and incomplete_keys:
+        report["overall_admissibility_score"] = None
+        report["overall_admissibility_level"] = TIER_INCOMPLETE
+        report["incomplete_keys"] = incomplete_keys
+        # Drop incomplete keys from category numeric rollup already reflected via null scores;
+        # recompute categories excluding Incomplete / Not identifiable.
+        cats: Dict[str, float] = {}
+        buckets: Dict[str, List[float]] = {
+            "laws": [],
+            "constitutive": [],
+            "scaling": [],
+            "data": [],
+        }
+        for k, row in per_key.items():
+            if not isinstance(row, dict):
+                continue
+            if is_incomplete_key(row):
+                continue
+            if (
+                row.get("admissibility_level") == "Not identifiable"
+                or row.get("identifiability_status") == "not_identifiable"
+            ):
+                continue
+            if "/" not in k:
+                continue
+            prefix = k.split("/", 1)[0]
+            if prefix not in buckets:
+                continue
+            try:
+                vf = float(row.get("admissibility_score"))
+            except (TypeError, ValueError):
+                continue
+            if not math.isfinite(vf):
+                continue
+            buckets[prefix].append(vf)
+        for cat, vals in buckets.items():
+            if not vals:
+                continue
+            if cat == "constitutive":
+                cats[cat] = _min_admissibility(vals)
+            else:
+                cats[cat] = _geom_mean_admissibility(vals)
+        report["per_category"] = cats
+
+
 def audit(
     log: List[Dict[str, Any]],
     r_ref: Optional[Dict[str, float]] = None,
@@ -999,6 +1107,8 @@ def audit(
     regions: Any = None,
     region_names: Optional[Mapping[Any, str]] = None,
     region_min_points: Optional[int] = None,
+    nonfinite_policy: Optional[str] = None,
+    eval_mask_label: Optional[str] = None,
 ) -> Dict[str, Any]:
     """
     Physics admissibility from logged **R_eff** (field ``rms``) and scales.
@@ -1226,6 +1336,27 @@ def audit(
 
     report["audit_meta"] = audit_meta(log, r_ref=r_ref)
 
+    _apply_evaluation_coverage_policy(
+        report,
+        log[-1],
+        policy=nonfinite_policy,
+        last_residual_dict=last_residual_dict,
+        eval_mask_label=eval_mask_label,
+    )
+    # Keep log entry aligned with the returned report (strict may null scores).
+    if log:
+        log[-1]["overall_admissibility_score"] = report.get("overall_admissibility_score")
+        log[-1]["overall_admissibility_level"] = report.get("overall_admissibility_level")
+        if report.get("coverage_status") is not None:
+            log[-1]["coverage_status"] = report.get("coverage_status")
+        if report.get("nonfinite_policy") is not None:
+            log[-1]["nonfinite_policy"] = report.get("nonfinite_policy")
+        adm_map = log[-1].get("admissibility_score")
+        if isinstance(adm_map, dict):
+            for k, row in (report.get("per_key") or {}).items():
+                if isinstance(row, dict) and row.get("admissibility_level") == "Incomplete":
+                    adm_map[k] = None
+
     if export_dir:
         import zipfile
         from pathlib import Path
@@ -1424,11 +1555,12 @@ def _worst_keys_table_rows(
             continue
         level = str(rep.get("admissibility_level") or "")
         is_ni = level == "Not identifiable" or rep.get("identifiability_status") == "not_identifiable"
+        is_incomplete = level == "Incomplete" or rep.get("coverage_incomplete") is True
         try:
             rn_f = float(rep.get("r_norm"))
         except (TypeError, ValueError):
             rn_f = float("nan")
-        if not is_ni and not math.isfinite(rn_f):
+        if not is_ni and not is_incomplete and not math.isfinite(rn_f):
             continue
         try:
             r_eff = float(rep.get("rms"))
@@ -1445,7 +1577,8 @@ def _worst_keys_table_rows(
             else:
                 sk_f = float("nan")
         try:
-            adm_f = float(rep.get("admissibility_score"))
+            adm_raw = rep.get("admissibility_score")
+            adm_f = float(adm_raw) if adm_raw is not None else float("nan")
         except (TypeError, ValueError):
             adm_f = float("nan")
         prefix = k.split("/", 1)[0] if "/" in str(k) else "other"
@@ -1455,9 +1588,9 @@ def _worst_keys_table_rows(
             "r_eff": r_eff,
             "scale_k": sk_f,
             "r_norm": rn_f,
-            "admissibility": adm_f,
-            "admissibility_score": adm_f,
-            "admissibility_level": level or admissibility_level(adm_f),
+            "admissibility": adm_f if math.isfinite(adm_f) else None,
+            "admissibility_score": adm_f if math.isfinite(adm_f) else None,
+            "admissibility_level": level or admissibility_level(adm_f if math.isfinite(adm_f) else None),
             "category": prefix,
         }
         if rep.get("implied_metric") is not None:
@@ -1477,6 +1610,11 @@ def _worst_keys_table_rows(
             if rep.get("identifiability_reason"):
                 row["identifiability_reason"] = rep.get("identifiability_reason")
             not_ident_rows.append(row)
+        elif is_incomplete:
+            row["admissibility_level"] = "Incomplete"
+            row["admissibility"] = None
+            row["admissibility_score"] = None
+            rows.append(row)
         else:
             rows.append(row)
     rows.sort(key=lambda r: r["r_norm"], reverse=True)
@@ -1609,7 +1747,14 @@ def _build_visualize_bundle(
         "bar_display": bar_display,
         "bar_values": bar_values_arr,
         "bar_values_eff": bar_values_eff_arr,
-        "overall_adm": [float(metrics[i]["overall_admissibility_score"]) for i in range(n)],
+        "overall_adm": [
+            (
+                float(metrics[i]["overall_admissibility_score"])
+                if metrics[i].get("overall_admissibility_score") is not None
+                else float("nan")
+            )
+            for i in range(n)
+        ],
         "cats_fin": cats_fin,
         "cat_colors": cat_colors,
         "spatial": spatial_parsed,
@@ -2560,6 +2705,7 @@ class ResidualEngine:
         nondim_scales: Any = None,
         derivative_metadata: Optional[Dict[str, Any]] = None,
         eval_mask: Any = None,
+        eval_mask_label: Optional[str] = None,
         state_pred_alt: Optional[Dict[str, Any]] = None,
         estimate_derivative_error: Optional[bool] = None,
         derivative_error_threshold: Optional[float] = None,
@@ -2610,6 +2756,8 @@ class ResidualEngine:
           - ``derivative_metadata``: optional per-key metadata for consistency warnings.
           - ``eval_mask``: boolean array (True = keep). When set, RMS/max for this call use
             masked points only.
+          - ``eval_mask_label``: optional string recorded on evaluation-coverage records for
+            this mask (provenance only; does not change scores).
           - ``state_pred_alt`` / ``estimate_derivative_error``: opt-in derivative-error estimates.
           - ``point_weights`` / ``weighted_norms`` (Moju 2.0 default ``True``): area-weighted
             R_eff for scoring, with a percentile safeguard (see ``weighted_safeguard``). Use
@@ -2641,6 +2789,7 @@ class ResidualEngine:
             nondim_scales=nondim_scales,
             derivative_metadata=derivative_metadata,
             eval_mask=eval_mask,
+            eval_mask_label=eval_mask_label,
             state_pred_alt=state_pred_alt,
             estimate_derivative_error=estimate_derivative_error,
             derivative_error_threshold=derivative_error_threshold,
@@ -2681,6 +2830,7 @@ class ResidualEngine:
         nondim_scales: Any = None,
         derivative_metadata: Optional[Dict[str, Any]] = None,
         eval_mask: Any = None,
+        eval_mask_label: Optional[str] = None,
         state_pred_alt: Optional[Dict[str, Any]] = None,
         estimate_derivative_error: Optional[bool] = None,
         derivative_error_threshold: Optional[float] = None,
@@ -3224,12 +3374,22 @@ class ResidualEngine:
             }
 
         flat = _flatten_residual_dict(residuals)
+        unmasked_flat = flat
         mask_meta: Dict[str, Any] = {}
         if eval_mask is not None:
             from moju.monitor.eval_mask import mask_flat_residuals, mask_provenance
 
             flat = mask_flat_residuals(flat, eval_mask)
             mask_meta = mask_provenance(eval_mask)
+        from moju.monitor.evaluation_coverage import coverage_map_from_flat
+
+        evaluation_coverage = coverage_map_from_flat(
+            flat,
+            eval_mask=eval_mask,
+            eval_mask_label=eval_mask_label,
+            mask_meta=mask_meta or None,
+            unmasked_flat=unmasked_flat,
+        )
         rms_per_key = _rms_per_key(flat, to_python=log_to_python)
         r_max_per_key = _max_per_key(flat, to_python=log_to_python, scoring=self._scoring_by_key)
 
@@ -3649,6 +3809,8 @@ class ResidualEngine:
             entry["derivative_error"] = derivative_error_report
         if mask_meta:
             entry["eval_mask"] = mask_meta
+        if evaluation_coverage:
+            entry["evaluation_coverage"] = evaluation_coverage
         if rms_weighted is not None:
             entry["rms_weighted"] = rms_weighted
             entry["weighted_norms"] = {
@@ -3857,6 +4019,8 @@ def evaluate(
     region_names: Optional[Mapping[Any, str]] = None,
     region_min_points: Optional[int] = None,
     evaluate_debug_loop: bool = False,
+    nonfinite_policy: Optional[str] = None,
+    eval_mask_label: Optional[str] = None,
     **compute_kwargs: Any,
 ) -> List[Dict[str, Any]]:
     """
@@ -3953,6 +4117,8 @@ def evaluate(
             implied_percentile=implied_percentile,
             implied_noise_normalization=implied_noise_normalization,
             require_batch=require_batch,
+            nonfinite_policy=nonfinite_policy,
+            eval_mask_label=eval_mask_label,
         )
         if batched is not None:
             for report in batched:
@@ -3973,7 +4139,13 @@ def evaluate(
             **compute_kwargs,
         )
         entry["index"] = i
-        report = audit([entry], r_ref=r_ref)
+        report = audit(
+            [entry],
+            r_ref=r_ref,
+            last_residual_dict=residuals if return_residuals else residuals,
+            nonfinite_policy=nonfinite_policy,
+            eval_mask_label=eval_mask_label,
+        )
         if return_residuals:
             report["residuals"] = residuals
         report["execution"] = "loop"

{moju-2.3.2 → moju-2.4.0}/moju/monitor/evaluate_batch.py

@@ -43,6 +43,8 @@ def evaluate_batched(
     implied_percentile: Optional[float] = None,
     implied_noise_normalization: Optional[bool] = None,
     require_batch: bool = False,
+    nonfinite_policy: Optional[str] = None,
+    eval_mask_label: Optional[str] = None,
 ) -> Optional[List[Dict[str, Any]]]:
     """
     Score every candidate in one compiled call.
@@ -294,8 +296,24 @@ def evaluate_batched(
                     compact[sk] = jnp.asarray(arr)[i]
                 id_map[str(bn)] = materialize_identifiability_meta(compact)
             entry["implied_identifiability"] = id_map
+        if "coverage_nums" in out:
+            from moju.monitor.evaluation_coverage import materialize_coverage_counts
+
+            cov_map: Dict[str, Any] = {}
+            for k, nums in out["coverage_nums"].items():
+                host_nums = {sk: jnp.asarray(arr)[i] for sk, arr in nums.items()}
+                cov_map[str(k)] = materialize_coverage_counts(
+                    host_nums, eval_mask_label=eval_mask_label
+                )
+            entry["evaluation_coverage"] = cov_map
         entry["monitor_settings"] = ms
-        report = audit([entry], r_ref=r_ref)
+        report = audit(
+            [entry],
+            r_ref=r_ref,
+            last_residual_dict=residuals,
+            nonfinite_policy=nonfinite_policy,
+            eval_mask_label=eval_mask_label,
+        )
         if return_residuals:
             report["residuals"] = residuals
         reports.append(report)
@@ -558,6 +576,11 @@ def _score_one(
     flat = _flat(residuals)
     rms = {k: r_eff(v) for k, v in flat.items()}
     rmax = {k: r_max(v) for k, v in flat.items()}
+    from moju.monitor.evaluation_coverage import coverage_counts_arrays
+
+    coverage_nums: Dict[str, Dict[str, Any]] = {
+        k: coverage_counts_arrays(v) for k, v in flat.items()
+    }
     dynamic_scale: Dict[str, Any] = {}
     recipe_ok: Dict[str, Any] = {}
     for key, recipe, spec, _is_user in plan.auto_scale:
@@ -582,6 +605,8 @@ def _score_one(
         "scale": dynamic_scale,
         "recipe_ok": recipe_ok,
     }
+    if coverage_nums:
+        out["coverage_nums"] = coverage_nums
     if implied_id:
         out["implied_id"] = implied_id