fieldkit 0.4.0__tar.gz → 0.4.2__tar.gz

{fieldkit-0.4.0 → fieldkit-0.4.2}/CHANGELOG.md

@@ -6,6 +6,49 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and
 
 ## [Unreleased]
 
+## [0.4.2] — 2026-05-15
+
+Patch release. Two card-rendering polish lifts on `fieldkit.publish` driven by the 2026-05-15 cyber-vertical cycle (`Orionfold/SecurityLLM-GGUF`, the third vertical card on this surface — zero fieldkit source changes between Saul / cyber, the v0.4.1 publishing surface generalized exactly as designed). Both lifts are additive (one new `ModelCard` field already shipped on `main` in `ff1b92f`; one new `ArtifactManifest` field added here). No new modules, no new public classes, no breaking changes — purely a tightening pass.
+
+### Added — `fieldkit.publish` card-rendering polish
+
+- **`ModelCard.llama_cpp_example_prompt: Optional[str]`** — new field. Threads through `publish_quant(..., llama_cpp_example_prompt=...)` and from a duck-typed report's `.llama_cpp_example_prompt` attribute. The default `## How to run` body's `llama-cpp-python` snippet now uses this string for the user-message; when omitted it falls back to a neutral `"Summarize the key idea in one paragraph."` placeholder instead of the previously-hardcoded `"Explain working capital."` (which leaked into the legal + cyber vertical cards on first push). Multi-line MCQ-shaped prompts are JSON-escaped (`\n`) so the snippet stays single-line + valid Python — caller passes the raw prompt, the renderer handles escaping.
+- **Side fix:** the previous renderer rendered the hardcoded finance prompt on every vertical card; the cyber + legal cards on HF were patched out-of-band on 2026-05-15 (commits `365dfe2`, `0824439`). Going forward, every `publish_quant` call should pass `llama_cpp_example_prompt=...` matching the article's "Using this release" section, per `[[feedback_customer_link_audit]]`.
+- **`ArtifactManifest.recommended_variant: Optional[str]`** — new field. Was already on `ModelCard` (so the README's How-to-run snippets template against the article's pick) but did NOT flow into the `<slug>.yaml` manifest, so the destination catalog couldn't see the article's narrative choice and ran its own rank-avg picker instead. `publish_quant` now threads `recommended_variant` into both surfaces — the HF README badge and the destination "Sweet spot" badge stay in sync from one kwarg. Mac added the matching `recommended_variant: z.string().optional()` to its artifacts schema in PR #6 (`mac-sweep/2026-05-15-cyber-vertical`) and pinned cyber's catalog `Q4_K_M` manually; source `src/content.config.ts` now mirrors that field for forward-compat. Motivated by cyber-vertical (2026-05-15): `Q4_K_M` topped CyberMetric at 40% but its worst-in-class perplexity dragged its rank-avg down, so without the override the picker selected `Q5_K_M`.
+
+### Test suite
+
+**+3 new tests:** `test_artifact_manifest_carries_recommended_variant_when_set` + `test_artifact_manifest_omits_recommended_variant_when_unset` (round-trip + elision on the new manifest field) and `test_publish_quant_threads_recommended_variant_into_card_and_manifest` (kwarg threads to both surfaces via `publish_quant`). Total: **378 passed, 3 skipped** offline (`pytest -q`). The 3 skips are the two `--spark`-gated live-integration tests + the `torch`-import skip in `test_training.py` (CPU-only venv).
+
+### Articles in this release
+
+- [`becoming-a-cyber-curator-on-spark`](https://ainative.business/field-notes/becoming-a-cyber-curator-on-spark/) — third Orionfold quant card. Drives both lifts: surfaces the `llama_cpp_example_prompt` leak (cyber's MCQ prompt would have shipped as "Explain working capital." otherwise) and motivates `ArtifactManifest.recommended_variant` (the destination's rank-avg picker would have surfaced `Q5_K_M` instead of `Q4_K_M`).
+
+### Verified on Spark
+
+- **Live HF push:** `Orionfold/SecurityLLM-GGUF` (5 GGUF variants + README, ~26 GB) shipped 2026-05-15 via the same `publish_quant(dry_run=False)` path as Saul and finance-chat. Zero source changes in `fieldkit.publish` between Saul (v0.4.1) and cyber (the cycle that drove this v0.4.2 patch) — the surface generalized as designed across three verticals.
+
+## [0.4.1] — 2026-05-14
+
+Patch release. The `fieldkit.eval.VerticalBench` overlay introduced in v0.4.0 needed two kwargs to score FinanceBench correctly (open-book context-prepend) and to bound a JSONL slice (subset filter on `question_type`). Both lifts came out of the 2026-05-13 V1 attempt on `AdaptLLM/finance-chat` (0/50 closed-book vs. 14–18%/50 open-book on the same JSONL) and the 2026-05-14 legal-curator scoring run on `Equall/Saul-7B-Instruct-v1`. The two scripts under `scripts/g3_*` that carried duplicated loaders now call into the package surface. No new modules, no new public classes — additive kwargs only.
+
+### Added — `fieldkit.eval.VerticalBench` open-book mode
+
+- **`VerticalBench.from_jsonl(..., open_book=...)`** — new kwarg. When `True`, FinanceBench rows have their `evidence[*].evidence_text` prepended to the question (templated as "Context from <doc>: …\n\nQuestion: …\n\nAnswer with just the numeric value.") so the model sees the 10-K excerpt the gold answer was derived from. Default `None` auto-resolves to `True` for `financebench` and `False` for `legalbench` / `generic` — the right defaults per benchmark convention. Lifts inline `_load_finbench_open_book` helpers from `scripts/g3_preflight_bench.py` and `scripts/g3_measure_variants.py` into the package surface; both scripts now call `VerticalBench.from_jsonl(open_book=True, subset=…)` instead of carrying duplicated loaders. The 2026-05-13 V1 attempt on AdaptLLM/finance-chat scored 0/50 closed-book and 14–18%/50 open-book on the same JSONL — open-book is the load-bearing flag for FinanceBench scoring.
+- **`VerticalBench.from_jsonl(..., subset=...)`** — new kwarg. FinanceBench-only convenience filter on the `question_type` column. Drops non-matching rows before the loader hits the `limit` cap, so callers can score the `metrics-generated` subset with `limit=50` and get 50 metrics-generated questions (not 50 mixed rows of which N are metrics-generated).
+
+### Test suite
+
+**+8 new tests** on `TestOpenBook` in `tests/test_vertical_bench.py` covering: auto-default for financebench, explicit `False` keeps closed-book, missing-evidence falls back to closed-book, legalbench / generic are no-ops, list-of-strings evidence shape, subset filter, subset × limit composition. Total: **375 passed, 3 skipped** offline (`pytest -q`). The 3 skips are the two `--spark`-gated live-integration tests + the `torch`-import skip in `test_training.py` (CPU-only venv).
+
+### Articles in this release
+
+- [`becoming-a-legal-curator-on-spark`](https://ainative.business/field-notes/becoming-a-legal-curator-on-spark/) — second Orionfold quant card, swaps FinanceBench for a curated 5-task LegalBench subset. Drives the `subset` kwarg's first non-finance use (LegalBench tasks via `legalbench` format) and validates that the `open_book` default-off branch is correct for LegalBench JSONLs.
+
+### Verified on Spark
+
+- **Live HF push:** `Orionfold/Saul-7B-Instruct-v1-GGUF` (5 GGUF variants + README, ~37 GB) shipped 2026-05-14 via the same `publish_quant(dry_run=False)` path the finance-chat card used a week earlier. Zero source changes in `fieldkit.publish` between the two pushes — the v0.4.0 surface generalized as designed.
+
 ## [0.4.0] — 2026-05-14
 
 Fourth public release. Two new top-level modules (`fieldkit.publish` + `fieldkit.quant`) for the G3 GGUF / Quantization Publisher pick (MTBM Pick #1 per `ideas/mtbm-use-cases.md` §6), the v0.4.x **vertical-curator overlay** on `fieldkit.eval` (`VerticalBench`), and post-dry-run card-rendering fixes that landed the first live HF push (`Orionfold/finance-chat-GGUF`). The two new modules together unlock most of Cluster G; this cut implements the GGUF critical path and stubs the other quant formats with named entry points pointing at the v0.5+ roadmap.

{fieldkit-0.4.0 → fieldkit-0.4.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: fieldkit
-Version: 0.4.0
+Version: 0.4.2
 Summary: Verified-on-Spark patterns lifted from the ai-field-notes blog into one importable Python package.
 Project-URL: Homepage, https://ainative.business/fieldkit/
 Project-URL: Source, https://github.com/manavsehgal/ai-field-notes/tree/main/fieldkit

{fieldkit-0.4.0 → fieldkit-0.4.2}/docs/api/cli.md

@@ -2,7 +2,7 @@
 module: cli
 title: fieldkit (CLI)
 summary: A thin Typer wrapper over the modules. Quick checks and smoke benchmarks without writing Python.
-order: 7
+order: 9
 ---
 
 ## What it is
@@ -17,7 +17,7 @@ Print the installed package version.
 
 ```bash
 $ fieldkit version
-0.2.0
+0.4.0
 ```
 
 ### `fieldkit envelope <size>`

{fieldkit-0.4.0 → fieldkit-0.4.2}/docs/api/eval.md

@@ -11,7 +11,7 @@ The eval harnesses the project keeps reinventing: a per-call latency benchmarker
 
 **v0.4.x additions** (vertical-curator surface for the G3 GGUF publisher pipeline):
 
-- `VerticalBench` — Spark-overlay scorer for FinanceBench / LegalBench / SemEval-style JSONL test sets. Wraps `Bench`, so latency aggregates alongside accuracy and refusal. Network access lives in the caller (`llama-cli`, NIM, vLLM) — the bench itself is offline-only and unit-testable.
+- `VerticalBench` — Spark-overlay scorer for FinanceBench / LegalBench / SemEval-style JSONL test sets. Wraps `Bench`, so latency aggregates alongside accuracy and refusal. Network access lives in the caller (`llama-cli`, NIM, vLLM) — the bench itself is offline-only and unit-testable. **v0.4.1 lift:** `from_jsonl(..., open_book=…, subset=…)` — open-book mode prepends FinanceBench evidence text to the question (default-on for `financebench`, default-off elsewhere); `subset` filters FinanceBench rows by `question_type` before the `limit` cap.
 - `VerticalQA` — one test case (qid + question + expected + tags) lifted from a vertical-eval JSONL.
 - `exact_match` / `contains` / `numeric_match` — the three built-in scorers. `numeric_match` is the FinanceBench default (first-number ±1% rel-tol); `exact_match` is the LegalBench default; `contains` is the right pick when the model answers in prose around a key fact.
 
@@ -241,6 +241,8 @@ vb = VerticalBench.from_jsonl(
     "financebench.jsonl",
     scorer=numeric_match,         # FinanceBench → first-number ±1%
     limit=50,
+    subset="metrics-generated",   # v0.4.1 — filter question_type before limit
+    open_book=True,               # v0.4.1 — prepend evidence_text to the question
 )
 
 def model_fn(prompt: str) -> str:
@@ -250,7 +252,10 @@ bench = vb.run(model_fn, extra_tags={"variant": "Q4_K_M"})
 print(bench.report())             # accuracy + refusal_rate + latency
 ```
 
-`VerticalBench.from_jsonl(path, *, format="auto", limit=None, scorer=None, scorer_kwargs=None)` auto-sniffs FinanceBench / LegalBench / generic schemas from the first JSON row. Rows missing the question or expected field are silently dropped (the row-count delta vs the JSONL is the diagnostic). The default scorer is `numeric_match` for FinanceBench and `exact_match` everywhere else; pass `scorer=` to override.
+`VerticalBench.from_jsonl(path, *, format="auto", limit=None, scorer=None, scorer_kwargs=None, open_book=None, subset=None)` auto-sniffs FinanceBench / LegalBench / generic schemas from the first JSON row. Rows missing the question or expected field are silently dropped (the row-count delta vs the JSONL is the diagnostic). The default scorer is `numeric_match` for FinanceBench and `exact_match` everywhere else; pass `scorer=` to override.
+
+- **`open_book=` *(v0.4.1)*** — when `True`, FinanceBench rows have their `evidence[*].evidence_text` prepended to the question (templated as `Context from <doc>: …\n\nQuestion: …\n\nAnswer with just the numeric value.`) so the model sees the 10-K excerpt the gold answer was derived from. Default `None` auto-resolves to `True` for `financebench` and `False` for `legalbench` / `generic` — the right defaults per benchmark convention. The 2026-05-13 V1 attempt on `AdaptLLM/finance-chat` scored 0/50 closed-book and 14–18%/50 open-book on the same JSONL; open-book is the load-bearing flag for FinanceBench scoring. Lifted from inline helpers in `scripts/g3_preflight_bench.py` and `scripts/g3_measure_variants.py` into the package surface.
+- **`subset=` *(v0.4.1)*** — FinanceBench-only convenience filter on the `question_type` column. Drops non-matching rows *before* the loader hits the `limit` cap, so callers can score the `metrics-generated` subset with `limit=50` and get 50 metrics-generated questions (not 50 mixed rows of which N happen to be metrics-generated). No-op on `legalbench` / `generic` formats.
 
 `VerticalBench.run(model_fn, *, limit=None, on_error="record", extra_tags=None)` returns the underlying `Bench` so callers route through the existing `.summary()` / `.report()` / `.dump()` pipeline. Each `BenchCall` carries `accuracy` (0.0/1.0 from the scorer) and `refusal` (0.0/1.0 from `is_refusal`) metrics; per-row metadata (company, doc_period, question_type) flows through to `BenchCall.tags` for downstream slice-by aggregation.
 

{fieldkit-0.4.0 → fieldkit-0.4.2}/docs/api/publish.md

@@ -70,6 +70,7 @@ ModelCard(
     hf_repo="Orionfold/finance-chat-GGUF",                    # drives default `## How to run` body
     chat_format="llama-2",                                     # → llama_cpp.Llama(chat_format=...)
     recommended_variant="Q5_K_M",                              # featured in default snippets
+    llama_cpp_example_prompt="Explain working capital.",       # user-message in the default `llama-cpp-python` snippet; falls back to a neutral placeholder when omitted
     ollama_pull_handle=None,                                   # opt-in override; default body wins otherwise
     transformers_snippet=None,
     lineage_prompt=None,                                       # injected by publish_quant if a LineageStore is supplied
@@ -98,6 +99,7 @@ m = ArtifactManifest(
     sustained_load_minutes=2.18,
     vertical_eval={"Q4_K_M": 0.14, ...},
     vertical_eval_name="FinanceBench (n=50, numeric_match)",
+    recommended_variant="Q5_K_M",           # article-narrative pick; destination pins the "Sweet spot" badge to this variant
     lineage_run_id=None,
     license_tier="free",                    # Orionfold commercial tier (free / pro)
     license_commercial_tier=None,
@@ -112,6 +114,8 @@ print(m.to_yaml())
 
 The `license_tier` / `license_commercial_tier` fields live alongside `model_license` under a nested `license:` block in YAML output. Mac destination's Zod schema mirrors this shape.
 
+`recommended_variant` (v0.4.2+) lets the article's narrative pick — the variant the writeup recommends — override the destination's rank-avg picker. Cyber's `Q4_K_M` topped CyberMetric but its worst-in-class perplexity dragged its rank-avg down, so without this field the catalog page would pin `Q5_K_M` as the "Sweet spot" instead. Same value flows into both `ModelCard` (HF README's How-to-run snippets) and `ArtifactManifest` (destination catalog) so the badge and the snippet stay in sync.
+
 ### `write_artifact_manifest(manifest, *, artifacts_dir)`
 
 Writes the manifest to `<artifacts_dir>/<slug>.yaml`. Creates the directory if missing. Returns the absolute path of the written file — callers can stage it alongside the article for the next git commit.
@@ -134,7 +138,7 @@ Token resolution order: explicit `token=` arg → `HF_TOKEN` env → `HUGGING_FA
 
 ### `publish_quant(*, quant_report, base_model, repo_name, staging_dir, ...) → PublishResult`
 
-The one-line orchestrator. Reads the duck-typed `quant_report` fields (`.format`, `.variants`, `.perplexity`, `.tokens_per_sec`, `.sustained_load_minutes`, `.variant_files`, `.vertical_eval`, `.vertical_eval_name`, `.model_license`, `.chat_format`, `.recommended_variant`), builds a `ModelCard`, stages the README + variant files, writes the `ArtifactManifest` (if `artifacts_dir` supplied), and invokes `HFHubAdapter.push_folder()`. Explicit kwargs override duck-typed report attrs.
+The one-line orchestrator. Reads the duck-typed `quant_report` fields (`.format`, `.variants`, `.perplexity`, `.tokens_per_sec`, `.sustained_load_minutes`, `.variant_files`, `.vertical_eval`, `.vertical_eval_name`, `.model_license`, `.chat_format`, `.recommended_variant`, `.llama_cpp_example_prompt`), builds a `ModelCard`, stages the README + variant files, writes the `ArtifactManifest` (if `artifacts_dir` supplied), and invokes `HFHubAdapter.push_folder()`. Explicit kwargs override duck-typed report attrs.
 
 ```python
 result = publish_quant(
@@ -150,6 +154,7 @@ result = publish_quant(
     model_license="llama2",            # critical — never default silently to apache-2.0
     chat_format="llama-2",
     recommended_variant="Q5_K_M",
+    llama_cpp_example_prompt="Explain working capital.",  # mirror the article's example user-message
     lineage_store=store,                # optional; injects ## Lineage block
     dry_run=True,                       # flip to False for the actual push
 )

{fieldkit-0.4.0 → fieldkit-0.4.2}/src/fieldkit/_version.py

@@ -6,4 +6,4 @@
 build time, so bumping it here is enough to bump the wheel version too.
 """
 
-__version__ = "0.4.0"
+__version__ = "0.4.2"

{fieldkit-0.4.0 → fieldkit-0.4.2}/src/fieldkit/eval/vertical.py

@@ -29,6 +29,15 @@ for custom scoring. ``exact_match`` and ``contains`` are deterministic;
 ``numeric_match`` extracts the first number from the prediction and compares
 to the reference under a relative tolerance — the right default for
 FinanceBench's quantitative questions.
+
+**Open-book mode (v0.4.1+).** FinanceBench is an *open-book* benchmark — the
+right answer is in the 10-K excerpt cited under ``evidence[*].evidence_text``.
+``VerticalBench.from_jsonl(..., open_book=True)`` rewrites the
+``VerticalQA.question`` to include the evidence text + a numeric-answer prompt
+before the model sees it. Default is auto: ``True`` for `financebench`,
+``False`` for everything else (LegalBench/generic). The 2026-05-13 V1 attempt
+on AdaptLLM/finance-chat scored 0/50 closed-book and 14–18%/50 open-book on
+the same JSONL — open-book is the load-bearing flag for FinanceBench scoring.
 """
 
 from __future__ import annotations
@@ -143,8 +152,20 @@ def _detect_format(row: dict[str, Any]) -> str:
     return "generic"
 
 
-def _row_to_qa(row: dict[str, Any], fmt: str, fallback_idx: int) -> VerticalQA | None:
-    """Map a JSONL row to `VerticalQA`. Returns None if required fields missing."""
+def _row_to_qa(
+    row: dict[str, Any],
+    fmt: str,
+    fallback_idx: int,
+    *,
+    open_book: bool = False,
+) -> VerticalQA | None:
+    """Map a JSONL row to `VerticalQA`. Returns None if required fields missing.
+
+    When `open_book=True` and `fmt == "financebench"`, prepends the row's
+    ``evidence[*].evidence_text`` to the question so the model sees the
+    10-K excerpt the gold answer was derived from. No-op for other formats —
+    LegalBench / generic JSONLs don't have a standard evidence field.
+    """
     if fmt == "financebench":
         qid = str(row.get("financebench_id") or f"fb-{fallback_idx}")
         question = row.get("question") or ""
@@ -159,6 +180,16 @@ def _row_to_qa(row: dict[str, Any], fmt: str, fallback_idx: int) -> VerticalQA |
             for k in ("company", "doc_period", "doc_type", "question_type")
             if k in row
         }
+        if open_book and question:
+            evidence_text = _extract_evidence_text(row)
+            if evidence_text:
+                doc_name = row.get("doc_name") or "the filing"
+                question = (
+                    f"Context from {doc_name}:\n\n"
+                    f"{evidence_text}\n\n"
+                    f"Question: {question}\n\n"
+                    f"Answer with just the numeric value."
+                )
     elif fmt == "legalbench":
         qid = str(row.get("id") or row.get("index") or f"lb-{fallback_idx}")
         question = row.get("text") or row.get("input") or row.get("question") or ""
@@ -180,6 +211,23 @@ def _row_to_qa(row: dict[str, Any], fmt: str, fallback_idx: int) -> VerticalQA |
     return VerticalQA(qid=qid, question=str(question), expected=str(expected), tags=tags)
 
 
+def _extract_evidence_text(row: dict[str, Any]) -> str:
+    """Flatten FinanceBench's `evidence: [{evidence_text: ...}, ...]` into a
+    blank-line-joined string. Accepts either list-of-dicts (canonical shape)
+    or list-of-strings (some pre-flattened dumps). Returns ``""`` when no
+    evidence field is present — caller falls back to closed-book.
+    """
+    chunks: list[str] = []
+    for e in row.get("evidence") or []:
+        if isinstance(e, dict):
+            txt = e.get("evidence_text") or ""
+            if txt:
+                chunks.append(str(txt))
+        elif isinstance(e, str):
+            chunks.append(e)
+    return "\n\n".join(chunks)
+
+
 # --- VerticalBench -------------------------------------------------------
 
 
@@ -224,6 +272,8 @@ class VerticalBench:
         limit: int | None = None,
         scorer: Callable[..., float] | None = None,
         scorer_kwargs: dict[str, Any] | None = None,
+        open_book: bool | None = None,
+        subset: str | None = None,
     ) -> VerticalBench:
         """Load a JSONL test set from disk and return a configured bench.
 
@@ -232,6 +282,18 @@ class VerticalBench:
         first row, so a partially-corrupt JSONL still triggers
         format-specific behavior. Rows missing question or expected are
         silently dropped (they show up as a row-count delta vs the JSONL).
+
+        `open_book` controls whether per-row evidence text is prepended to the
+        question. Default ``None`` resolves to ``True`` for `financebench`
+        (where the gold answer lives in the cited 10-K excerpt) and ``False``
+        for everything else. Pass ``True`` / ``False`` to override. Currently
+        only `financebench` rows have a standard evidence field; the flag is
+        a no-op for the other formats.
+
+        `subset` is a FinanceBench-only convenience filter that drops rows
+        whose ``question_type`` doesn't match. Useful for scoring only the
+        ``metrics-generated`` subset (quantitative questions) without
+        pre-filtering the JSONL.
         """
         p = Path(path)
         questions: list[VerticalQA] = []
@@ -247,11 +309,19 @@ class VerticalBench:
                     continue
                 if fmt is None:
                     fmt = _detect_format(row)
-                qa = _row_to_qa(row, fmt, fallback_idx=i)
+                # Resolve open_book on the first row once fmt is known.
+                if open_book is None:
+                    open_book = fmt == "financebench"
+                if subset is not None and row.get("question_type") != subset:
+                    continue
+                qa = _row_to_qa(row, fmt, fallback_idx=i, open_book=open_book)
                 if qa is not None:
                     questions.append(qa)
                 if limit is not None and len(questions) >= limit:
                     break
+        # If the file was empty, open_book may still be None — collapse to False.
+        if open_book is None:
+            open_book = False
         # Pick a sensible default scorer per format if caller didn't override.
         if scorer is None:
             scorer = numeric_match if fmt == "financebench" else exact_match

{fieldkit-0.4.0 → fieldkit-0.4.2}/src/fieldkit/publish/__init__.py

@@ -201,6 +201,12 @@ class ModelCard:
     """Variant to feature in the default How-to-run snippets (e.g. `Q5_K_M`).
     Falls back to `Q5_K_M` if present in `variants`, else the first listed
     variant."""
+    llama_cpp_example_prompt: Optional[str] = None
+    """User-message string for the templated `llama-cpp-python` snippet.
+    Mirrors what the article's "Using this release" section asks the model —
+    keep article + HF card in lockstep. When omitted, falls back to a
+    neutral placeholder. Multi-line strings (e.g. MCQ-shaped prompts) render
+    via JSON-escaped `\\n` so the snippet stays single-line + valid Python."""
     ollama_pull_handle: Optional[str] = None
     transformers_snippet: Optional[str] = None
     lineage_prompt: Optional[str] = None
@@ -329,7 +335,12 @@ def _render_how_to_run(card: ModelCard) -> list[str]:
             out.append(f"    n_ctx=4096, n_gpu_layers=99{chat_format_kw},")
             out.append(")")
             out.append("out = llm.create_chat_completion(")
-            out.append('    messages=[{"role": "user", "content": "Explain working capital."}],')
+            example_prompt = (
+                card.llama_cpp_example_prompt
+                or "Summarize the key idea in one paragraph."
+            )
+            escaped_prompt = json.dumps(example_prompt, ensure_ascii=False)
+            out.append(f'    messages=[{{"role": "user", "content": {escaped_prompt}}}],')
             out.append("    temperature=0.0,")
             out.append(")")
             out.append('print(out["choices"][0]["message"]["content"])')
@@ -525,6 +536,14 @@ class ArtifactManifest:
     vertical_eval_name: Optional[str] = None
     """Display name for the vertical eval (e.g.,
     "FinanceBench (n=50, numeric_match)")."""
+    recommended_variant: Optional[str] = None
+    """Article-narrative pick for the "Sweet spot" — the variant the catalog
+    page should pin under `recommended_variant:` instead of letting the
+    destination's rank-avg picker choose. Mirrors `ModelCard.recommended_variant`
+    so the HF README badge and the destination catalog page stay in sync.
+    Added in v0.4.2 after cyber-vertical: Q4_K_M topped the bench but its
+    worst-in-class perplexity dragged its rank-avg down, so the picker selected
+    Q5_K_M. This field lets the article's narrative judgment win."""
     lineage_run_id: Optional[str] = None
     license_tier: str = "free"
     license_commercial_tier: Optional[str] = None
@@ -564,6 +583,8 @@ class ArtifactManifest:
             d["vertical_eval"] = dict(self.vertical_eval)
         if self.vertical_eval_name:
             d["vertical_eval_name"] = self.vertical_eval_name
+        if self.recommended_variant:
+            d["recommended_variant"] = self.recommended_variant
         if self.lineage_run_id:
             d["lineage_run_id"] = self.lineage_run_id
         d["license"] = {"tier": self.license_tier}
@@ -813,6 +834,7 @@ def publish_quant(
     model_license: Optional[str] = None,
     chat_format: Optional[str] = None,
     recommended_variant: Optional[str] = None,
+    llama_cpp_example_prompt: Optional[str] = None,
 ) -> PublishResult:
     """Orchestrate model-card render + manifest write + HF push.
 
@@ -860,6 +882,8 @@ def publish_quant(
         chat_format = getattr(quant_report, "chat_format", None)
     if recommended_variant is None:
         recommended_variant = getattr(quant_report, "recommended_variant", None)
+    if llama_cpp_example_prompt is None:
+        llama_cpp_example_prompt = getattr(quant_report, "llama_cpp_example_prompt", None)
 
     # Build the card.
     tag_set: list[str] = [
@@ -916,6 +940,7 @@ def publish_quant(
         hf_repo=hf_repo,
         chat_format=chat_format,
         recommended_variant=recommended_variant,
+        llama_cpp_example_prompt=llama_cpp_example_prompt,
         ollama_pull_handle=ollama_pull_handle,
         transformers_snippet=transformers_snippet,
         lineage_prompt=lineage_prompt,
@@ -951,6 +976,7 @@ def publish_quant(
             sustained_load_minutes=sustained,
             vertical_eval=vertical_eval,
             vertical_eval_name=vertical_eval_name,
+            recommended_variant=recommended_variant,
             lineage_run_id=lineage_run_id,
             model_license=model_license,
             article=f"articles/{article_slug}/" if article_slug else None,

{fieldkit-0.4.0 → fieldkit-0.4.2}/tests/test_publish.py

@@ -457,6 +457,32 @@ def test_artifact_manifest_carries_vertical_eval_when_set() -> None:
     assert "vertical_eval_name:" in yaml_text
 
 
+def test_artifact_manifest_carries_recommended_variant_when_set() -> None:
+    m = ArtifactManifest(
+        slug="securityllm-gguf",
+        kind="quant",
+        artifact_class="gguf",
+        base_model="ZySec-AI/SecurityLLM",
+        hf_repo="Orionfold/SecurityLLM-GGUF",
+        variants=("Q4_K_M", "Q5_K_M"),
+        recommended_variant="Q4_K_M",
+    )
+    d = m.to_dict()
+    assert d["recommended_variant"] == "Q4_K_M"
+    yaml_text = m.to_yaml()
+    assert "recommended_variant: Q4_K_M" in yaml_text
+
+
+def test_artifact_manifest_omits_recommended_variant_when_unset() -> None:
+    m = ArtifactManifest(
+        slug="s", kind="quant", artifact_class="gguf",
+        base_model="b", hf_repo="Orionfold/x",
+    )
+    d = m.to_dict()
+    assert "recommended_variant" not in d
+    assert "recommended_variant" not in m.to_yaml()
+
+
 def test_artifact_manifest_yaml_is_parseable_round_trip() -> None:
     yaml_text = ArtifactManifest(
         slug="s",
@@ -653,6 +679,29 @@ def test_publish_quant_reads_vertical_eval_from_quant_report_duck_typed(
     assert "55.0%" in card and "60.0%" in card
 
 
+def test_publish_quant_threads_recommended_variant_into_card_and_manifest(
+    tmp_path: Path,
+) -> None:
+    """`recommended_variant` kwarg flows to the README's How-to-run snippets
+    AND to the manifest YAML so the destination catalog pins the same pick."""
+    qr = _stub_quant_report(tmp_path / "source")
+    result = publish_quant(
+        quant_report=qr,
+        base_model="ZySec-AI/SecurityLLM",
+        repo_name="SecurityLLM-GGUF",
+        staging_dir=tmp_path / "stage",
+        artifacts_dir=tmp_path / "content" / "artifacts",
+        recommended_variant="Q4_K_M",
+        dry_run=True,
+    )
+    card = result.card_path.read_text()
+    # The default How-to-run snippet templates against the recommended variant
+    assert "Q4_K_M" in card
+    # Manifest YAML carries the recommended_variant field
+    manifest = result.manifest_path.read_text()
+    assert "recommended_variant: Q4_K_M" in manifest
+
+
 def test_publish_quant_threads_model_license_into_card_and_manifest(
     tmp_path: Path,
 ) -> None:

{fieldkit-0.4.0 → fieldkit-0.4.2}/tests/test_vertical_bench.py

@@ -217,6 +217,114 @@ class TestFromJsonl:
         assert vb.name == "financebench-mini"
 
 
+# --- Open-book mode (v0.4.1+) -----------------------------------------------
+
+
+def _fb_row(qid: str, question: str, gold: str, evidence_text: str, **extra) -> dict:
+    return {
+        "financebench_id": qid,
+        "question": question,
+        "gold_standard": gold,
+        "doc_name": extra.get("doc_name", "ACME_2024_10-K"),
+        "company": extra.get("company", "ACME"),
+        "doc_period": extra.get("doc_period", "FY2024"),
+        "question_type": extra.get("question_type", "metrics-generated"),
+        "evidence": [{"evidence_text": evidence_text}] if evidence_text else [],
+    }
+
+
+class TestOpenBook:
+    def test_financebench_auto_open_book(self, tmp_path: Path) -> None:
+        # No explicit open_book — default for financebench is True.
+        rows = [_fb_row("fb-1", "What was revenue?", "1234", "Revenue: $1,234M")]
+        p = _write_jsonl(tmp_path, "fb.jsonl", rows)
+        vb = VerticalBench.from_jsonl(p)
+        assert len(vb.questions) == 1
+        q = vb.questions[0]
+        # Evidence is now in the question prompt.
+        assert "Revenue: $1,234M" in q.question
+        assert "Question: What was revenue?" in q.question
+        assert "ACME_2024_10-K" in q.question
+        # Expected is unchanged.
+        assert q.expected == "1234"
+
+    def test_explicit_open_book_false_keeps_closed(self, tmp_path: Path) -> None:
+        rows = [_fb_row("fb-1", "What was revenue?", "1234", "Revenue: $1,234M")]
+        p = _write_jsonl(tmp_path, "fb.jsonl", rows)
+        vb = VerticalBench.from_jsonl(p, open_book=False)
+        assert vb.questions[0].question == "What was revenue?"
+
+    def test_open_book_no_evidence_falls_back(self, tmp_path: Path) -> None:
+        # A financebench row missing evidence — open_book=True is a no-op.
+        rows = [_fb_row("fb-1", "What was revenue?", "1234", "")]
+        p = _write_jsonl(tmp_path, "fb.jsonl", rows)
+        vb = VerticalBench.from_jsonl(p)
+        assert vb.questions[0].question == "What was revenue?"
+
+    def test_legalbench_open_book_is_noop(self, tmp_path: Path) -> None:
+        # No standard evidence field — open_book=True doesn't change the question.
+        rows = [{"id": "lb-1", "text": "Is X enforceable?", "answer": "yes"}]
+        p = _write_jsonl(tmp_path, "lb.jsonl", rows)
+        vb = VerticalBench.from_jsonl(p, open_book=True)
+        assert vb.questions[0].question == "Is X enforceable?"
+
+    def test_open_book_default_off_for_legalbench(self, tmp_path: Path) -> None:
+        # Auto-default for non-financebench is False, even if a stray evidence
+        # field is present.
+        rows = [
+            {
+                "id": "lb-1",
+                "text": "Is X enforceable?",
+                "answer": "yes",
+                "evidence": [{"evidence_text": "Section 3.1 prohibits X."}],
+            }
+        ]
+        p = _write_jsonl(tmp_path, "lb.jsonl", rows)
+        vb = VerticalBench.from_jsonl(p)
+        # No prepending — only financebench triggers open-book by default.
+        assert vb.questions[0].question == "Is X enforceable?"
+
+    def test_evidence_text_as_strings(self, tmp_path: Path) -> None:
+        # Some pre-flattened dumps put evidence as list-of-strings.
+        rows = [
+            {
+                "financebench_id": "fb-1",
+                "question": "q",
+                "gold_standard": "g",
+                "evidence": ["chunk-1", "chunk-2"],
+            }
+        ]
+        p = _write_jsonl(tmp_path, "fb.jsonl", rows)
+        vb = VerticalBench.from_jsonl(p)
+        assert "chunk-1" in vb.questions[0].question
+        assert "chunk-2" in vb.questions[0].question
+
+    def test_subset_filter_financebench(self, tmp_path: Path) -> None:
+        rows = [
+            _fb_row("fb-1", "q1", "1", "ev-1", question_type="metrics-generated"),
+            _fb_row("fb-2", "q2", "2", "ev-2", question_type="domain-relevant"),
+            _fb_row("fb-3", "q3", "3", "ev-3", question_type="metrics-generated"),
+        ]
+        p = _write_jsonl(tmp_path, "fb.jsonl", rows)
+        vb = VerticalBench.from_jsonl(p, subset="metrics-generated")
+        assert len(vb.questions) == 2
+        assert {q.qid for q in vb.questions} == {"fb-1", "fb-3"}
+
+    def test_subset_limit_compose(self, tmp_path: Path) -> None:
+        # subset filters before limit applies.
+        rows = [
+            _fb_row(f"fb-{i}", f"q{i}", str(i), f"ev-{i}", question_type="metrics-generated")
+            for i in range(5)
+        ] + [
+            _fb_row(f"fbx-{i}", f"qx{i}", str(i), f"ev-{i}", question_type="other")
+            for i in range(5)
+        ]
+        p = _write_jsonl(tmp_path, "fb.jsonl", rows)
+        vb = VerticalBench.from_jsonl(p, subset="metrics-generated", limit=3)
+        assert len(vb.questions) == 3
+        assert all(q.qid.startswith("fb-") for q in vb.questions)
+
+
 # --- VerticalBench.run -------------------------------------------------------