fieldkit 0.4.1__tar.gz → 0.4.2__tar.gz

{fieldkit-0.4.1 → fieldkit-0.4.2}/CHANGELOG.md

@@ -6,6 +6,28 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and
 
 ## [Unreleased]
 
+## [0.4.2] — 2026-05-15
+
+Patch release. Two card-rendering polish lifts on `fieldkit.publish` driven by the 2026-05-15 cyber-vertical cycle (`Orionfold/SecurityLLM-GGUF`, the third vertical card on this surface — zero fieldkit source changes between Saul / cyber, the v0.4.1 publishing surface generalized exactly as designed). Both lifts are additive (one new `ModelCard` field already shipped on `main` in `ff1b92f`; one new `ArtifactManifest` field added here). No new modules, no new public classes, no breaking changes — purely a tightening pass.
+
+### Added — `fieldkit.publish` card-rendering polish
+
+- **`ModelCard.llama_cpp_example_prompt: Optional[str]`** — new field. Threads through `publish_quant(..., llama_cpp_example_prompt=...)` and from a duck-typed report's `.llama_cpp_example_prompt` attribute. The default `## How to run` body's `llama-cpp-python` snippet now uses this string for the user-message; when omitted it falls back to a neutral `"Summarize the key idea in one paragraph."` placeholder instead of the previously-hardcoded `"Explain working capital."` (which leaked into the legal + cyber vertical cards on first push). Multi-line MCQ-shaped prompts are JSON-escaped (`\n`) so the snippet stays single-line + valid Python — caller passes the raw prompt, the renderer handles escaping.
+- **Side fix:** the previous renderer rendered the hardcoded finance prompt on every vertical card; the cyber + legal cards on HF were patched out-of-band on 2026-05-15 (commits `365dfe2`, `0824439`). Going forward, every `publish_quant` call should pass `llama_cpp_example_prompt=...` matching the article's "Using this release" section, per `[[feedback_customer_link_audit]]`.
+- **`ArtifactManifest.recommended_variant: Optional[str]`** — new field. Was already on `ModelCard` (so the README's How-to-run snippets template against the article's pick) but did NOT flow into the `<slug>.yaml` manifest, so the destination catalog couldn't see the article's narrative choice and ran its own rank-avg picker instead. `publish_quant` now threads `recommended_variant` into both surfaces — the HF README badge and the destination "Sweet spot" badge stay in sync from one kwarg. Mac added the matching `recommended_variant: z.string().optional()` to its artifacts schema in PR #6 (`mac-sweep/2026-05-15-cyber-vertical`) and pinned cyber's catalog `Q4_K_M` manually; source `src/content.config.ts` now mirrors that field for forward-compat. Motivated by cyber-vertical (2026-05-15): `Q4_K_M` topped CyberMetric at 40% but its worst-in-class perplexity dragged its rank-avg down, so without the override the picker selected `Q5_K_M`.
+
+### Test suite
+
+**+3 new tests:** `test_artifact_manifest_carries_recommended_variant_when_set` + `test_artifact_manifest_omits_recommended_variant_when_unset` (round-trip + elision on the new manifest field) and `test_publish_quant_threads_recommended_variant_into_card_and_manifest` (kwarg threads to both surfaces via `publish_quant`). Total: **378 passed, 3 skipped** offline (`pytest -q`). The 3 skips are the two `--spark`-gated live-integration tests + the `torch`-import skip in `test_training.py` (CPU-only venv).
+
+### Articles in this release
+
+- [`becoming-a-cyber-curator-on-spark`](https://ainative.business/field-notes/becoming-a-cyber-curator-on-spark/) — third Orionfold quant card. Drives both lifts: surfaces the `llama_cpp_example_prompt` leak (cyber's MCQ prompt would have shipped as "Explain working capital." otherwise) and motivates `ArtifactManifest.recommended_variant` (the destination's rank-avg picker would have surfaced `Q5_K_M` instead of `Q4_K_M`).
+
+### Verified on Spark
+
+- **Live HF push:** `Orionfold/SecurityLLM-GGUF` (5 GGUF variants + README, ~26 GB) shipped 2026-05-15 via the same `publish_quant(dry_run=False)` path as Saul and finance-chat. Zero source changes in `fieldkit.publish` between Saul (v0.4.1) and cyber (the cycle that drove this v0.4.2 patch) — the surface generalized as designed across three verticals.
+
 ## [0.4.1] — 2026-05-14
 
 Patch release. The `fieldkit.eval.VerticalBench` overlay introduced in v0.4.0 needed two kwargs to score FinanceBench correctly (open-book context-prepend) and to bound a JSONL slice (subset filter on `question_type`). Both lifts came out of the 2026-05-13 V1 attempt on `AdaptLLM/finance-chat` (0/50 closed-book vs. 14–18%/50 open-book on the same JSONL) and the 2026-05-14 legal-curator scoring run on `Equall/Saul-7B-Instruct-v1`. The two scripts under `scripts/g3_*` that carried duplicated loaders now call into the package surface. No new modules, no new public classes — additive kwargs only.

{fieldkit-0.4.1 → fieldkit-0.4.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: fieldkit
-Version: 0.4.1
+Version: 0.4.2
 Summary: Verified-on-Spark patterns lifted from the ai-field-notes blog into one importable Python package.
 Project-URL: Homepage, https://ainative.business/fieldkit/
 Project-URL: Source, https://github.com/manavsehgal/ai-field-notes/tree/main/fieldkit

{fieldkit-0.4.1 → fieldkit-0.4.2}/docs/api/eval.md

@@ -11,7 +11,7 @@ The eval harnesses the project keeps reinventing: a per-call latency benchmarker
 
 **v0.4.x additions** (vertical-curator surface for the G3 GGUF publisher pipeline):
 
-- `VerticalBench` — Spark-overlay scorer for FinanceBench / LegalBench / SemEval-style JSONL test sets. Wraps `Bench`, so latency aggregates alongside accuracy and refusal. Network access lives in the caller (`llama-cli`, NIM, vLLM) — the bench itself is offline-only and unit-testable.
+- `VerticalBench` — Spark-overlay scorer for FinanceBench / LegalBench / SemEval-style JSONL test sets. Wraps `Bench`, so latency aggregates alongside accuracy and refusal. Network access lives in the caller (`llama-cli`, NIM, vLLM) — the bench itself is offline-only and unit-testable. **v0.4.1 lift:** `from_jsonl(..., open_book=…, subset=…)` — open-book mode prepends FinanceBench evidence text to the question (default-on for `financebench`, default-off elsewhere); `subset` filters FinanceBench rows by `question_type` before the `limit` cap.
 - `VerticalQA` — one test case (qid + question + expected + tags) lifted from a vertical-eval JSONL.
 - `exact_match` / `contains` / `numeric_match` — the three built-in scorers. `numeric_match` is the FinanceBench default (first-number ±1% rel-tol); `exact_match` is the LegalBench default; `contains` is the right pick when the model answers in prose around a key fact.
 
@@ -241,6 +241,8 @@ vb = VerticalBench.from_jsonl(
     "financebench.jsonl",
     scorer=numeric_match,         # FinanceBench → first-number ±1%
     limit=50,
+    subset="metrics-generated",   # v0.4.1 — filter question_type before limit
+    open_book=True,               # v0.4.1 — prepend evidence_text to the question
 )
 
 def model_fn(prompt: str) -> str:
@@ -250,7 +252,10 @@ bench = vb.run(model_fn, extra_tags={"variant": "Q4_K_M"})
 print(bench.report())             # accuracy + refusal_rate + latency
 ```
 
-`VerticalBench.from_jsonl(path, *, format="auto", limit=None, scorer=None, scorer_kwargs=None)` auto-sniffs FinanceBench / LegalBench / generic schemas from the first JSON row. Rows missing the question or expected field are silently dropped (the row-count delta vs the JSONL is the diagnostic). The default scorer is `numeric_match` for FinanceBench and `exact_match` everywhere else; pass `scorer=` to override.
+`VerticalBench.from_jsonl(path, *, format="auto", limit=None, scorer=None, scorer_kwargs=None, open_book=None, subset=None)` auto-sniffs FinanceBench / LegalBench / generic schemas from the first JSON row. Rows missing the question or expected field are silently dropped (the row-count delta vs the JSONL is the diagnostic). The default scorer is `numeric_match` for FinanceBench and `exact_match` everywhere else; pass `scorer=` to override.
+
+- **`open_book=` *(v0.4.1)*** — when `True`, FinanceBench rows have their `evidence[*].evidence_text` prepended to the question (templated as `Context from <doc>: …\n\nQuestion: …\n\nAnswer with just the numeric value.`) so the model sees the 10-K excerpt the gold answer was derived from. Default `None` auto-resolves to `True` for `financebench` and `False` for `legalbench` / `generic` — the right defaults per benchmark convention. The 2026-05-13 V1 attempt on `AdaptLLM/finance-chat` scored 0/50 closed-book and 14–18%/50 open-book on the same JSONL; open-book is the load-bearing flag for FinanceBench scoring. Lifted from inline helpers in `scripts/g3_preflight_bench.py` and `scripts/g3_measure_variants.py` into the package surface.
+- **`subset=` *(v0.4.1)*** — FinanceBench-only convenience filter on the `question_type` column. Drops non-matching rows *before* the loader hits the `limit` cap, so callers can score the `metrics-generated` subset with `limit=50` and get 50 metrics-generated questions (not 50 mixed rows of which N happen to be metrics-generated). No-op on `legalbench` / `generic` formats.
 
 `VerticalBench.run(model_fn, *, limit=None, on_error="record", extra_tags=None)` returns the underlying `Bench` so callers route through the existing `.summary()` / `.report()` / `.dump()` pipeline. Each `BenchCall` carries `accuracy` (0.0/1.0 from the scorer) and `refusal` (0.0/1.0 from `is_refusal`) metrics; per-row metadata (company, doc_period, question_type) flows through to `BenchCall.tags` for downstream slice-by aggregation.
 

{fieldkit-0.4.1 → fieldkit-0.4.2}/docs/api/publish.md

@@ -70,6 +70,7 @@ ModelCard(
     hf_repo="Orionfold/finance-chat-GGUF",                    # drives default `## How to run` body
     chat_format="llama-2",                                     # → llama_cpp.Llama(chat_format=...)
     recommended_variant="Q5_K_M",                              # featured in default snippets
+    llama_cpp_example_prompt="Explain working capital.",       # user-message in the default `llama-cpp-python` snippet; falls back to a neutral placeholder when omitted
     ollama_pull_handle=None,                                   # opt-in override; default body wins otherwise
     transformers_snippet=None,
     lineage_prompt=None,                                       # injected by publish_quant if a LineageStore is supplied
@@ -98,6 +99,7 @@ m = ArtifactManifest(
     sustained_load_minutes=2.18,
     vertical_eval={"Q4_K_M": 0.14, ...},
     vertical_eval_name="FinanceBench (n=50, numeric_match)",
+    recommended_variant="Q5_K_M",           # article-narrative pick; destination pins the "Sweet spot" badge to this variant
     lineage_run_id=None,
     license_tier="free",                    # Orionfold commercial tier (free / pro)
     license_commercial_tier=None,
@@ -112,6 +114,8 @@ print(m.to_yaml())
 
 The `license_tier` / `license_commercial_tier` fields live alongside `model_license` under a nested `license:` block in YAML output. Mac destination's Zod schema mirrors this shape.
 
+`recommended_variant` (v0.4.2+) lets the article's narrative pick — the variant the writeup recommends — override the destination's rank-avg picker. Cyber's `Q4_K_M` topped CyberMetric but its worst-in-class perplexity dragged its rank-avg down, so without this field the catalog page would pin `Q5_K_M` as the "Sweet spot" instead. Same value flows into both `ModelCard` (HF README's How-to-run snippets) and `ArtifactManifest` (destination catalog) so the badge and the snippet stay in sync.
+
 ### `write_artifact_manifest(manifest, *, artifacts_dir)`
 
 Writes the manifest to `<artifacts_dir>/<slug>.yaml`. Creates the directory if missing. Returns the absolute path of the written file — callers can stage it alongside the article for the next git commit.
@@ -134,7 +138,7 @@ Token resolution order: explicit `token=` arg → `HF_TOKEN` env → `HUGGING_FA
 
 ### `publish_quant(*, quant_report, base_model, repo_name, staging_dir, ...) → PublishResult`
 
-The one-line orchestrator. Reads the duck-typed `quant_report` fields (`.format`, `.variants`, `.perplexity`, `.tokens_per_sec`, `.sustained_load_minutes`, `.variant_files`, `.vertical_eval`, `.vertical_eval_name`, `.model_license`, `.chat_format`, `.recommended_variant`), builds a `ModelCard`, stages the README + variant files, writes the `ArtifactManifest` (if `artifacts_dir` supplied), and invokes `HFHubAdapter.push_folder()`. Explicit kwargs override duck-typed report attrs.
+The one-line orchestrator. Reads the duck-typed `quant_report` fields (`.format`, `.variants`, `.perplexity`, `.tokens_per_sec`, `.sustained_load_minutes`, `.variant_files`, `.vertical_eval`, `.vertical_eval_name`, `.model_license`, `.chat_format`, `.recommended_variant`, `.llama_cpp_example_prompt`), builds a `ModelCard`, stages the README + variant files, writes the `ArtifactManifest` (if `artifacts_dir` supplied), and invokes `HFHubAdapter.push_folder()`. Explicit kwargs override duck-typed report attrs.
 
 ```python
 result = publish_quant(
@@ -150,6 +154,7 @@ result = publish_quant(
     model_license="llama2",            # critical — never default silently to apache-2.0
     chat_format="llama-2",
     recommended_variant="Q5_K_M",
+    llama_cpp_example_prompt="Explain working capital.",  # mirror the article's example user-message
     lineage_store=store,                # optional; injects ## Lineage block
     dry_run=True,                       # flip to False for the actual push
 )

{fieldkit-0.4.1 → fieldkit-0.4.2}/src/fieldkit/_version.py

@@ -6,4 +6,4 @@
 build time, so bumping it here is enough to bump the wheel version too.
 """
 
-__version__ = "0.4.1"
+__version__ = "0.4.2"

{fieldkit-0.4.1 → fieldkit-0.4.2}/src/fieldkit/publish/__init__.py

@@ -201,6 +201,12 @@ class ModelCard:
     """Variant to feature in the default How-to-run snippets (e.g. `Q5_K_M`).
     Falls back to `Q5_K_M` if present in `variants`, else the first listed
     variant."""
+    llama_cpp_example_prompt: Optional[str] = None
+    """User-message string for the templated `llama-cpp-python` snippet.
+    Mirrors what the article's "Using this release" section asks the model —
+    keep article + HF card in lockstep. When omitted, falls back to a
+    neutral placeholder. Multi-line strings (e.g. MCQ-shaped prompts) render
+    via JSON-escaped `\\n` so the snippet stays single-line + valid Python."""
     ollama_pull_handle: Optional[str] = None
     transformers_snippet: Optional[str] = None
     lineage_prompt: Optional[str] = None
@@ -329,7 +335,12 @@ def _render_how_to_run(card: ModelCard) -> list[str]:
             out.append(f"    n_ctx=4096, n_gpu_layers=99{chat_format_kw},")
             out.append(")")
             out.append("out = llm.create_chat_completion(")
-            out.append('    messages=[{"role": "user", "content": "Explain working capital."}],')
+            example_prompt = (
+                card.llama_cpp_example_prompt
+                or "Summarize the key idea in one paragraph."
+            )
+            escaped_prompt = json.dumps(example_prompt, ensure_ascii=False)
+            out.append(f'    messages=[{{"role": "user", "content": {escaped_prompt}}}],')
             out.append("    temperature=0.0,")
             out.append(")")
             out.append('print(out["choices"][0]["message"]["content"])')
@@ -525,6 +536,14 @@ class ArtifactManifest:
     vertical_eval_name: Optional[str] = None
     """Display name for the vertical eval (e.g.,
     "FinanceBench (n=50, numeric_match)")."""
+    recommended_variant: Optional[str] = None
+    """Article-narrative pick for the "Sweet spot" — the variant the catalog
+    page should pin under `recommended_variant:` instead of letting the
+    destination's rank-avg picker choose. Mirrors `ModelCard.recommended_variant`
+    so the HF README badge and the destination catalog page stay in sync.
+    Added in v0.4.2 after cyber-vertical: Q4_K_M topped the bench but its
+    worst-in-class perplexity dragged its rank-avg down, so the picker selected
+    Q5_K_M. This field lets the article's narrative judgment win."""
     lineage_run_id: Optional[str] = None
     license_tier: str = "free"
     license_commercial_tier: Optional[str] = None
@@ -564,6 +583,8 @@ class ArtifactManifest:
             d["vertical_eval"] = dict(self.vertical_eval)
         if self.vertical_eval_name:
             d["vertical_eval_name"] = self.vertical_eval_name
+        if self.recommended_variant:
+            d["recommended_variant"] = self.recommended_variant
         if self.lineage_run_id:
             d["lineage_run_id"] = self.lineage_run_id
         d["license"] = {"tier": self.license_tier}
@@ -813,6 +834,7 @@ def publish_quant(
     model_license: Optional[str] = None,
     chat_format: Optional[str] = None,
     recommended_variant: Optional[str] = None,
+    llama_cpp_example_prompt: Optional[str] = None,
 ) -> PublishResult:
     """Orchestrate model-card render + manifest write + HF push.
 
@@ -860,6 +882,8 @@ def publish_quant(
         chat_format = getattr(quant_report, "chat_format", None)
     if recommended_variant is None:
         recommended_variant = getattr(quant_report, "recommended_variant", None)
+    if llama_cpp_example_prompt is None:
+        llama_cpp_example_prompt = getattr(quant_report, "llama_cpp_example_prompt", None)
 
     # Build the card.
     tag_set: list[str] = [
@@ -916,6 +940,7 @@ def publish_quant(
         hf_repo=hf_repo,
         chat_format=chat_format,
         recommended_variant=recommended_variant,
+        llama_cpp_example_prompt=llama_cpp_example_prompt,
         ollama_pull_handle=ollama_pull_handle,
         transformers_snippet=transformers_snippet,
         lineage_prompt=lineage_prompt,
@@ -951,6 +976,7 @@ def publish_quant(
             sustained_load_minutes=sustained,
             vertical_eval=vertical_eval,
             vertical_eval_name=vertical_eval_name,
+            recommended_variant=recommended_variant,
             lineage_run_id=lineage_run_id,
             model_license=model_license,
             article=f"articles/{article_slug}/" if article_slug else None,

{fieldkit-0.4.1 → fieldkit-0.4.2}/tests/test_publish.py

@@ -457,6 +457,32 @@ def test_artifact_manifest_carries_vertical_eval_when_set() -> None:
     assert "vertical_eval_name:" in yaml_text
 
 
+def test_artifact_manifest_carries_recommended_variant_when_set() -> None:
+    m = ArtifactManifest(
+        slug="securityllm-gguf",
+        kind="quant",
+        artifact_class="gguf",
+        base_model="ZySec-AI/SecurityLLM",
+        hf_repo="Orionfold/SecurityLLM-GGUF",
+        variants=("Q4_K_M", "Q5_K_M"),
+        recommended_variant="Q4_K_M",
+    )
+    d = m.to_dict()
+    assert d["recommended_variant"] == "Q4_K_M"
+    yaml_text = m.to_yaml()
+    assert "recommended_variant: Q4_K_M" in yaml_text
+
+
+def test_artifact_manifest_omits_recommended_variant_when_unset() -> None:
+    m = ArtifactManifest(
+        slug="s", kind="quant", artifact_class="gguf",
+        base_model="b", hf_repo="Orionfold/x",
+    )
+    d = m.to_dict()
+    assert "recommended_variant" not in d
+    assert "recommended_variant" not in m.to_yaml()
+
+
 def test_artifact_manifest_yaml_is_parseable_round_trip() -> None:
     yaml_text = ArtifactManifest(
         slug="s",
@@ -653,6 +679,29 @@ def test_publish_quant_reads_vertical_eval_from_quant_report_duck_typed(
     assert "55.0%" in card and "60.0%" in card
 
 
+def test_publish_quant_threads_recommended_variant_into_card_and_manifest(
+    tmp_path: Path,
+) -> None:
+    """`recommended_variant` kwarg flows to the README's How-to-run snippets
+    AND to the manifest YAML so the destination catalog pins the same pick."""
+    qr = _stub_quant_report(tmp_path / "source")
+    result = publish_quant(
+        quant_report=qr,
+        base_model="ZySec-AI/SecurityLLM",
+        repo_name="SecurityLLM-GGUF",
+        staging_dir=tmp_path / "stage",
+        artifacts_dir=tmp_path / "content" / "artifacts",
+        recommended_variant="Q4_K_M",
+        dry_run=True,
+    )
+    card = result.card_path.read_text()
+    # The default How-to-run snippet templates against the recommended variant
+    assert "Q4_K_M" in card
+    # Manifest YAML carries the recommended_variant field
+    manifest = result.manifest_path.read_text()
+    assert "recommended_variant: Q4_K_M" in manifest
+
+
 def test_publish_quant_threads_model_license_into_card_and_manifest(
     tmp_path: Path,
 ) -> None: