@jini-ai/agent-plugins 0.6.17 → 0.6.19

package/jev/skills/jev/SKILL.md

@@ -0,0 +1,308 @@
+---
+name: jev
+description: Design and write code that calls Jev, TypeSafe's System One model — a fast, non-generative model that turns state plus a typed Choice/Score/Noul question into a typed answer with a calibrated probability distribution. Use this for classification, routing, scoring, ranking, extraction-by-selection, and confidence-gated automation; not for generating prose. Covers the three primitives' exact request/response shapes, confidence, the documented patterns, known jev-1.13 limitations, and the HTTP API / JavaScript SDK. This plugin declares no MCP server — nothing here lets the assistant call Jev directly in chat.
+---
+
+# Jev (TypeSafe System One)
+
+## The one thing to say before anything else
+
+**This plugin has no MCP server.** `mcp.json` declares `"mcpServers": {}` on purpose. Jev has no
+official MCP server, and TypeSafe's docs index lists no MCP integration. There
+is no `mcp__jev__*` tool, and enabling this plugin does not let the assistant call the Jev API
+during a chat turn. What this skill is for: **helping design and write code** — a host product feature, a
+script, a separate service — that calls Jev itself, over plain HTTP or the JavaScript SDK. If an
+operator asks "can you check with Jev right now," the honest answer is no, not yet; what you can do
+is write the code that will.
+
+**The API key goes in `TYPESAFE_API_KEY`, never in chat.** That's the env var TypeSafe's SDKs read
+by default. The JS SDK also accepts an explicit `new TypeSafeClient({ apiKey })`, and that value takes
+precedence over the env var, so a key loaded from a secret store works too. Whoever sets up the
+integration puts the key in the environment or a secret store, the same way any other API credential
+belongs in this codebase. Never ask for it in chat, never paste one into code you write, and never
+echo one back if a user pastes it anyway. Tell them to treat it as compromised and rotate it at
+TypeSafe's dashboard (`console.typesafe.ai/keys`).
+
+---
+
+## What Jev is
+
+Jev is TypeSafe's flagship model and the first "System One" model: instead of generating text, it
+takes a **state** (the content to reason about — a string, JSON object, or array of text) and one or
+more typed **questions**, and returns a typed **answer** per question with a calibrated probability
+distribution. All questions in one request are evaluated in parallel, independently, against the
+same state — asking more questions barely changes response time. It is served at one HTTP endpoint,
+`POST https://api.typesafe.ai/v1/systemone`, and there are official Python and JavaScript SDKs.
+Jev currently accepts **text input only** — no images, audio, or video — and its primary training
+language is English; other languages (including CJK scripts) are accepted but currently less
+accurate.
+
+## When Jev fits
+
+Jev is for judgments with a **bounded, structured answer space** over content you hand it directly:
+
+- **Classification / routing** — which team, which intent, which handler should this go to (Choice).
+- **Scoring / rubrics** — how severe, how frustrated, how relevant, on a rubric you define (Score).
+- **Yes/no gating** — is this urgent, does this need a human, does this contain X (Noul).
+- **Ranking / re-ranking** — score each candidate independently, sort by the returned probability.
+- **Extraction by selection** — find candidates with code (regex, retrieval), have Jev pick which one
+  answers the question, then copy that value verbatim — never have it generate the value.
+- **Confidence-gated automation** — act automatically above a threshold, ask for confirmation in the
+  middle, and route to a human below it; see [Confidence](#confidence) below.
+- **A cheap feature layer in front of classical ML** — turn free text into structured Score/Noul
+  columns for a downstream model, rather than hand-writing feature extraction.
+
+## When Jev does not fit
+
+- **Generating prose, summaries, or free text.** Jev is not trained to generate text. "You can force
+  it to by chaining choices, [but] this will not work well and will be very slow" (jaggedness #9).
+  Use a generative LLM for that half of the workflow, and Jev for the structured judgments around it.
+- **Exact arithmetic, counting, or date/time math.** "Jev is not a calculator." It does not count
+  reliably (characters, occurrences, list length), performs worse on numeric/hex/RGB/binary
+  representations than semantic ones, and reads dates as text, not as ordered quantities. Do the math
+  in code; ask Jev only for the judgment calls code cannot make (see
+  [references/jaggedness.md](references/jaggedness.md), items 2–3).
+- **Deep multi-hop or "property of a property" reasoning.** Instructions with double negatives or
+  several hops of indirection are answered less reliably. Decompose into direct, atomic questions and
+  combine them in code instead (jaggedness #4).
+- **Anything needing images, audio, or video.** Text only, today.
+- **A decision where getting fooled by adversarial input is unacceptable, without hardening it
+  first.** "State is data, and jev-1.13 does not treat it as hostile by default" — content written to
+  steer the model can move the answer. Be explicit in criteria and test before trusting it at scale
+  (jaggedness #6).
+- **Non-English content where accuracy is critical**, without testing first — accuracy is currently
+  lower outside English.
+
+When in doubt, read [references/jaggedness.md](references/jaggedness.md) before designing a new
+question — it is the single most load-bearing reference here for "will this actually work well."
+
+---
+
+## The three primitives
+
+Every question has an **id** (your key — never sent to the model; write the full question in
+`instructions` even if the id looks self-explanatory), a **`type`**, `instructions`, and `criteria`.
+Full field-by-field reference, including the "structured `instructions`/`criteria`" mechanism (every
+one of these fields accepts a string, object, array, or `null`, not just plain text) is in
+[references/primitives-reference.md](references/primitives-reference.md).
+
+| Type | Answers | Request `criteria` | Answer fields |
+|---|---|---|---|
+| **Choice** | Which of these options? | Required map: option name → description (`string\|object\|array\|null`). Max **255** options. | `choice` (winning option), `probabilities` (map, sums to 1), `confidence` |
+| **Score** | Where on this rubric? | Required ordered array of level descriptions, low → high. Min 2, max **10** levels. | `score` (probability-weighted mean of level index — can land between levels), `legend`, `probabilities`, `confidence` |
+| **Noul** | Is this true? | Optional `{true, false}` clarification. | `noul` only — probability the answer is yes, 0 to 1. **No `confidence` field.** |
+
+One combined example (request and response both real, quoted from the docs — a Choice, a Score, and
+a Noul batched into one call):
+
+```json
+// POST https://api.typesafe.ai/v1/systemone
+// Authorization: Bearer <TYPESAFE_API_KEY>
+// Content-Type: application/json
+{
+  "state": "Hi, I've been trying to connect my Stripe account for 3 days and the integration keeps failing. I'm losing sales. Please help ASAP.",
+  "model": "jev-latest",
+  "questions": {
+    "department": {
+      "type": "choice",
+      "instructions": "Which team should handle this",
+      "criteria": { "billing": "Payment or subscription issues", "technical": "Bugs or integration problems", "sales": "Pricing or account questions" }
+    },
+    "frustration": {
+      "type": "score",
+      "instructions": "How frustrated the customer appears",
+      "criteria": ["Calm, just stating facts", "Frustrated but civil", "Very angry, strong language"]
+    },
+    "is_urgent": { "type": "noul", "instructions": "The message conveys urgency or time-sensitivity" }
+  }
+}
+```
+
+```json
+// response
+{
+  "model": "jev-1.13.0",
+  "answers": {
+    "department": { "type": "choice", "choice": "technical", "confidence": 0.78, "probabilities": { "technical": 0.85, "sales": 0.0, "billing": 0.15 } },
+    "frustration": { "type": "score", "score": 1.0, "confidence": 1.0, "legend": { "0": "Calm, just stating facts", "1": "Frustrated but civil", "2": "Very angry, strong language" }, "probabilities": { "0": 0.0, "1": 1.0, "2": 0.0 } },
+    "is_urgent": { "type": "noul", "noul": 1.0 }
+  },
+  "usage": { "input_tokens": 392, "output_tokens": 65 }
+}
+```
+
+Note the request sent `model: "jev-latest"` (an alias) and the response reports the concrete
+resolved version, `"jev-1.13.0"`. See [Cost, latency, and limits](#cost-latency-and-limits).
+
+**Batch, don't chain.** Put every question a workflow might need — including ones that only matter
+for some inputs — into one request (the "speculative fan-out" pattern below); evaluating them is
+parallel, so extra questions cost little latency and only the tokens for the extra text. Only split
+into two sequential requests when code genuinely cannot build the second request until it has the
+first answer (see [references/patterns.md](references/patterns.md)).
+
+## Confidence
+
+`confidence` (0 to 1) is on every Choice and Score answer, derived from the shape of `probabilities`
+— **Noul has no `confidence`**, since a yes/no distribution is already fully described by the single
+`noul` value. TypeSafe does not publish the exact formula ("a convenient measure that fits most
+use-cases... you are never locked into our definition" — use `probabilities` directly for a custom
+statistic). Use it as a second axis alongside the answer itself, in three bands:
+
+- **High** → act automatically.
+- **Medium** → proceed with caution — confirm, flag, or gather more information.
+- **Low** → do not act — route to a human, ask a clarifying question, or fall back.
+
+There is no universal threshold; the docs' own worked examples use different cutoffs per action
+based on the cost of being wrong (a low-stakes balance check acts at 0.5+, a high-stakes transfer
+approval waits for >0.9). Start conservative, test against your own data, and put the threshold
+constants in one place in your code so they're easy to review — see
+[references/patterns.md](references/patterns.md) for the confidence-gated-routing pattern with a
+worked example.
+
+If you only need the single best option, read the Choice answer's `choice` field, which is already
+the highest-probability option. Don't compare it against a fixed threshold. A threshold decides
+*whether to act*, not which option wins. For your own statistic, use `probabilities`.
+
+## Patterns
+
+Four named patterns, each with a worked example, are documented in
+[references/patterns.md](references/patterns.md):
+
+| Pattern | What it does | Why |
+|---|---|---|
+| **Speculative fan-out** | Send every question a workflow might need in one call; let code decide what's relevant after. | Parallel evaluation makes extra questions nearly free. |
+| **Confidence-gated routing** | Use `confidence` as a second axis, with a different floor per action based on its stakes. | Reliability, safety. |
+| **Composite scoring** | Split one complex judgment into several independent Score questions, normalize, combine with code-owned weights. | Auditability — you can see exactly how the final number was built, and reuse the same raw scores with different weights for different downstream decisions. |
+| **Intent routing** | Use Jev as a cheap up-front classifier that sends each request to deterministic code, a specialist LLM, or a human. | Cost, speed — most requests never need an expensive model call. |
+
+## Known limitations (jev-1.13 jaggedness)
+
+TypeSafe publishes a dated list of known failure modes for the current model version, each with a
+"do this instead." Full detail, including worked examples, is in
+[references/jaggedness.md](references/jaggedness.md); the headline ones, worth internalizing before
+writing any question:
+
+1. **Literal reading** — answers the question you wrote, not the one you meant. Be exact; put
+   boundary cases in `criteria`.
+2. **Not a calculator** — counting, numeric/hex/RGB comparisons, and interpolating an exact number
+   from a Score are all unreliable. Do math in code.
+3. **Dates are read as text**, not ordered quantities — extract the parts with a bounded Choice, do
+   date arithmetic in code.
+4. **Indirection costs accuracy** — double negatives and multi-hop "property of a property" questions
+   degrade. Ask directly.
+5. **Large, mostly-irrelevant state hurts accuracy** — "Jev suffers from context rot" when `state`
+   carries content the question doesn't need. (This is a different claim from "more questions don't
+   cost accuracy," which is about batching questions, not about bloating shared state — both are true,
+   they are not in tension.) Filter in code before sending, or use a Noul as a relevance pre-filter.
+6. **State is not treated as adversarial by default** — content written to steer the model can move
+   the answer. Be explicit in criteria; test before trusting it at scale.
+7. **Contradictory instructions and criteria degrade the answer** — keep them aligned.
+8. **No guaranteed structural invariants** — a Noul and an equivalent Choice can disagree; a Noul and
+   its logical negation are not guaranteed to sum to 1. Don't assume arithmetic identities across
+   separate questions, and don't reuse a confidence threshold tuned on one question type for another.
+9. **Not built to generate text** — use Choice over a bounded option set instead of forcing free-text
+   generation.
+
+## Cost, latency, and limits
+
+Everything here is stated by TypeSafe's docs directly, not estimated (see
+[references/primitives-reference.md](references/primitives-reference.md) for the full Models-page
+table):
+
+- **Current model:** `jev-1.13.0`. Aliases: `jev-latest` (most recent *stable* release — the SDK
+  default) and `jev-preview` (most recent release, stable or not; currently identical to
+  `jev-latest`). The response's `model` field always reports the resolved versioned id — pin that id
+  instead of an alias once you've tuned confidence thresholds against it, since an alias can move
+  answers out from under you when TypeSafe ships a new release.
+- **Pricing:** $42 per billion input tokens ($0.042 / million). **Output tokens are free**. `usage`
+  still reports `output_tokens`, but only input tokens are billed.
+- **Latency:** the only figure TypeSafe states directly is "most queries complete in about 100 ms."
+  TypeSafe's own cookbooks measured 111–114 ms for a multi-question batched call, consistent with
+  that figure.
+- **Context limits:** 64k tokens per request (`state` + every question combined); 32k tokens for
+  `state` plus the single longest question.
+- **Rate limits:** 250,000 tokens/second, 1,200 requests/minute — TypeSafe states these are adjusting
+  dynamically as demand grows and may change without notice; a `429` means back off and retry with
+  exponential backoff (SDKs do this by default).
+- **Errors:** `401` bad/missing key, `422` invalid request body, `429` rate-limited, `529`
+  TypeSafe-side overload.
+
+## Calling Jev: HTTP or the JavaScript SDK
+
+**HTTP**, directly:
+
+```http
+POST https://api.typesafe.ai/v1/systemone
+Authorization: Bearer <TYPESAFE_API_KEY>
+Content-Type: application/json
+```
+
+Body: `{ "state": ..., "model": "jev-latest", "questions": { "<id>": { "type": ..., "instructions": ..., "criteria": ... } } }`.
+Response: `{ "model": ..., "answers": { "<id>": {...} }, "usage": { "input_tokens": ..., "output_tokens": ... } }`.
+Full field reference and the error table are in
+[references/primitives-reference.md](references/primitives-reference.md).
+
+**JavaScript SDK** (`@typesafe-ai/sdk`, requires Node 20+):
+
+```sh
+npm install @typesafe-ai/sdk
+```
+
+```ts
+import { choice, score, noul, TypeSafeClient } from "@typesafe-ai/sdk";
+
+const client = new TypeSafeClient(); // reads TYPESAFE_API_KEY from the environment
+
+const response = await client.systemOne({
+  state: { document: "I was charged twice. Please fix this ASAP." },
+  questions: {
+    category: choice("What is this ticket about?", { billing: null, technical: null, other: null }),
+    frustration: score("How frustrated the customer appears", [
+      "Calm, just stating facts",
+      "Frustrated but civil",
+      "Very angry, strong language",
+    ]),
+    billing: noul("Is this about billing?"),
+  },
+});
+
+console.log(response.answers.category.choice, response.answers.frustration.score, response.answers.billing.noul);
+```
+
+The helpers are documented on the SDK reference pages as `choice(instructions, criteria)`,
+`score(instructions, criteria)` (an ordered array, at least two levels) and
+`noul(instructions?, criteria?)` (optional `{ true, false }`). `model` is optional on `systemOne()`.
+It falls back to the client's default model, which is `jev-latest` unless configured. Retries are
+on by default. Config options, retry defaults, error classes, and the one browser-safety flag are in
+[references/javascript-sdk.md](references/javascript-sdk.md). Check it before you invent an option
+name.
+
+## Cookbooks
+
+TypeSafe publishes worked, benchmarked cookbooks for common shapes — self-consistency, re-ranking,
+semantic search, structure recovery, function calling, skill suggestion, entity alignment, RAG
+passage classification, citation verification, LLM guardrails, extraction cascades, date extraction,
+value extraction, hierarchical classification, feature discovery, and confidence-based classification.
+Short digests of all 18, with their measured numbers and the one generalizable lesson from each, are
+in [references/cookbooks.md](references/cookbooks.md) — read the relevant one before designing a
+workflow that looks similar; it often shows a better decomposition than a first-pass design would.
+
+## References
+
+- [references/primitives-reference.md](references/primitives-reference.md) — full field-by-field
+  reference for Choice/Score/Noul/structured questions, the HTTP API, and the Models/pricing/limits
+  table.
+- [references/patterns.md](references/patterns.md) — the four named patterns with worked examples.
+- [references/jaggedness.md](references/jaggedness.md) — all 9 known jev-1.13 failure modes with
+  mitigations. Read this before designing a new question.
+- [references/javascript-sdk.md](references/javascript-sdk.md): JS SDK client config and env vars,
+  question helpers, retry defaults, and error classes, each checked against the SDK reference pages.
+- [references/cookbooks.md](references/cookbooks.md) — digests of all 18 published cookbooks.
+- TypeSafe's own docs are the live source of truth and move faster than this skill:
+  [docs.typesafe.ai](https://docs.typesafe.ai) (append `.md` to any page path for raw Markdown;
+  index at [docs.typesafe.ai/llms.txt](https://docs.typesafe.ai/llms.txt)). Some structure in this
+  skill (the doc-index walking convention, the task table under "find the useful shape") was informed
+  by TypeSafe's own MIT-licensed agent skill,
+  [github.com/typesafe-ai/skills](https://github.com/typesafe-ai/skills) — read live docs over this
+  file when the two disagree; TypeSafe ships new model versions and jaggedness notes faster than a
+  bundled plugin can track them.

package/jev/skills/jev/references/cookbooks.md

@@ -0,0 +1,197 @@
+# Cookbook digests
+
+Short digests of all 18 cookbooks TypeSafe publishes at `docs.typesafe.ai/cookbooks/*` (crawled
+2026-09-21). Every cookbook's own code uses the Python SDK (`client.system_one(...)`,
+`Choice`/`Score`/`Noul` classes) — the call shape differs cosmetically from the JavaScript SDK's
+`client.systemOne(...)` / lowercase `choice(...)` helper (see `javascript-sdk.md`), but the
+primitives, request/response shapes, and lessons are identical across languages. Read the full page
+for a shape close to what you're building — these are worth more than a first-pass design from
+scratch.
+
+## Self-consistency: nouls
+
+`cookbooks/consistency_noul_cookbook.md` — Runs a 14-question Noul rubric over one insurance claim,
+15 times each, across non-reasoning LLMs, reasoning LLMs, and Jev. Jev's mean per-question
+probability std dev (0.0102) beats every LLM condition; a 14-question Jev call runs ~111ms /
+$0.000043, vs. the cheapest LLM condition at 1113ms / $0.000950 (10x slower, 22x costlier) up to a
+reasoning-model condition at 13886ms / $0.034275 (125x slower, 805x costlier). **Lesson:** repeated
+calls still wobble even for Jev — pair a raw probability with an "uncertain band" (e.g. 0.30–0.70)
+that routes borderline cases to human review, rather than trusting one call's label outright.
+
+## Self-consistency: choices
+
+`cookbooks/consistency_choice_cookbook.md` — Same test, an 8-question moderation Choice rubric, 15
+repeats. Jev's plurality-label agreement (90.8%) is lower than some LLM conditions (87.5–100%) on
+raw repeats, but gating on `confidence ≥ 0.6` raises Jev's agreement to 99.2% with 74.2%
+auto-labeled. Cost/speed: ~114ms / $0.000046 per call vs. 826ms–12978ms / $0.000936–$0.041255 for LLM
+conditions. **Lesson:** confidence-gating turns Jev's imperfect self-consistency into a reliable
+automated majority, at a fraction of LLM cost/latency.
+
+## Parallel questions
+
+`cookbooks/parallel_questions.md` — Batches a 13-question regulatory briefing (8 Noul + 2 Choice + 3
+Score) over the ~54k-character GDPR Wikipedia article, one request vs. 13 separate requests, 5
+repeats. Batching: **12.2x cheaper, 10.0x faster** ($0.000497/0.27s vs. $0.006090/2.71s), with
+near-identical answers across repeats regardless of batching. **Lesson:** batching multiple
+independent questions about the same state collapses the document-resend cost — the larger the
+shared state relative to the questions, the closer the saving is to a full Nx.
+
+## Re-ranking
+
+`cookbooks/rerank_typesafe.md` — Two-stage legal search: BM25 builds a 30-candidate shortlist per
+query (3,565 CLERC passages, 40 queries), then one Jev Noul question per (query, candidate) pair
+re-ranks by the returned probability. Top-1 accuracy 5%→18%, top-5 15%→35%, top-10 38%→62%.
+**Lesson:** cheap retrieval is good at shortlisting, bad at picking the single best match; a per-pair
+Noul re-ranker meaningfully improves precision at the top of the list. The cookbook asks one
+question per pair for clarity. It says a real application would ask several questions about the
+same pair in one call.
+
+## Line-by-line search
+
+`cookbooks/semantic_find.md` — Tags every line of GitHub's ToS with an id, then one request asks a
+Choice question ranking all 218 line ids by relevance plus a Noul checking whether the document
+actually contains an answer at all. A ranking-only approach can look confident (top-ranked line at
+0.86 relevance) even when the real answer isn't in the document (`exists` probability only 0.14).
+**Lesson:** pair a ranking Choice with a separate existence-check Noul so you can distinguish "found
+it," "not in the document," and "partially answered," instead of returning the top rank uncritically.
+
+## Structure recovery
+
+`cookbooks/autoformat.md` — Reconstructs lost Markdown structure from hard-wrapped plain text in two
+sequential requests: Pass 1 asks one Noul per adjacent line pair ("did this break split a
+sentence?"); Pass 2 asks one Choice per merged block (heading/list/code/callout/paragraph) plus
+companion questions read only when relevant. The model never generates text — every output character
+still traces to the input. Example: 2 round trips, 10,211 tokens, 0.8s, $0.0015. **Lesson:** keep
+anything code can already determine (blank lines, explicit markers) out of the model call entirely —
+send only the judgment calls code genuinely can't make, and let code do all the rendering.
+
+## Function calling
+
+`cookbooks/function_calling.md` — Maps natural-language trading requests to ten typed Python
+functions: each closed-set/`Literal` argument becomes a Choice question, one request answers every
+argument for the call, and a confidence-aware `Dispatcher` returns the function name plus arguments.
+Example: `rolling_correlation(symbol='AMD', benchmark='NVDA')` at confidence 0.82 overall, with
+per-argument probabilities (0.87, 0.78) and omitted arguments falling back to function defaults
+(p=0.96/0.99). **Lesson:** a compound call's confidence should be the *minimum* across its parts
+(the weakest judgment), not a product — a product understates confidence as argument count grows.
+An explicit "was this stated?" question lets an argument fall back to its default instead of being
+forced into a confident-sounding guess.
+
+## Skill suggestion
+
+`cookbooks/skill_suggestion.md` — Picks at most one skill per agent turn from a 182-skill roster in
+two sequential requests: request 1 ranks all 182 skills cheaply and decides whether the turn needs
+one at all; request 2 re-reads only the top 3 in full and can reject all of them. Across 488 turns:
+agent alone with the roster — 16.8% wrong-skill loads, 9.8% loads-when-nothing-fits; agent +
+suggestion — 7.3% / 4.0% (more than half reduction); agent handed the right answer directly — 2.5% /
+1.2% (the floor isn't zero even with a perfect hint). **Lesson:** for large rosters, don't truncate
+descriptions into the prompt — cheap rank-all-candidates, then read-and-verify only the top few, with
+the ability to reject all of them.
+
+## Knowledge graph entity alignment
+
+`cookbooks/entity_alignment.md` — Decides which of 450 candidate pairs across two beer-catalogue
+sources describe the same product, using one Score question with exactly 3 ordered levels (different
+/ related-but-maybe-not-same / same → merge), plus 3 riding Noul questions telling a curator which
+field two records disagree on. Of 450 pairs: 40 (8.9%) auto-merged, 50 (11.1%) queued for a curator,
+360 (80.0%) left unlinked. **Lesson:** use Score (not a Noul threshold or Choice) when outcome levels
+are genuinely ordered and each needs its own criteria — it avoids inventing an arbitrary probability
+cutoff, since the levels map directly onto the outcomes.
+
+## Classifying RAG passages
+
+`cookbooks/classifying_rag_passages.md` — Inserts a classification stage between retrieval and
+generation: for each retrieved passage, one request carries 4 Noul questions (relevant? states
+usable evidence? contradicts the query's premise? tries to instruct the model?), and ordinary
+threshold code decides accept / flag-conflict / drop. Catches a planted prompt-injection passage and
+flags queries with false premises against real evidence. **Lesson:** keep the accept/flag/drop policy
+as explicit code-level thresholds over structured probabilities. The jaggedness page links this
+cookbook as the worked example for #5 (irrelevant state). It also shows one way to screen out
+prompt injection (#6), though the jaggedness page doesn't cite it for that.
+
+## Double-checking citations
+
+`cookbooks/citation_check.md` — Verifies an LLM's cited quotes: a plain string match first checks
+whether the quote exists verbatim in the source (free, catches fabrication outright), then a Choice
+question classifies how the quote's context relates to the claim (supports / contradicts /
+says-nothing), gated at `confidence ≥ 0.8` before accepting the verdict without human review.
+8 citations tested: 4 accurate ones verified at confidence ≥0.93; all 4 planted failures caught.
+**Lesson:** don't send every check to the model — a free code-level pre-filter (string match) should
+catch what it can before spending a model call on the judgment it actually requires.
+
+## Guardrails for LLMs
+
+`cookbooks/llm_guardrails.md` — Screens inbound and outbound LLM messages with one request per
+message: a battery of Noul questions gives P(hazard) per named hazard (jailbreak, policy-breaking
+reply, harm/crime, medical dosage, self-harm), plus one Score question rating how much harm
+complying would do; app code thresholds the results into pass/review/block/route. **Lesson:** put
+safety policy in externally-readable, editable code-level thresholds over structured probabilities,
+not buried in a system prompt or a second general-purpose "judge" LLM call — and run the same guard
+on both input and output, since a safe-looking prompt can still produce an unsafe reply.
+
+## SDE cascade
+
+`cookbooks/sde_cascade.md` — A 2-stage structured-data-extraction cascade: extract with a cheap
+small model in plain-text mode, verify each extracted field with a per-field Jev Noul question
+("is this value absent from the source?", framed so `true` = escalate), escalate only flagged items
+to an expensive reasoning model. Stage pricing: cheap extractor $0.75/$4.50 per 1M tokens (in/out);
+reasoning escalation $5.00/$30.00 (~7x); Jev verifier $0.042/$0.00. The swept-threshold cascade's
+Pareto frontier beats every single model tested on quality-per-dollar. **Lesson:** a good verifier
+question must be narrow and grounded (one checkable yes/no per field against the source), framed so
+the *escalate* case is `true`, and aggregated per-field with `max` (any single flag escalates) rather
+than averaged — averaging silences one confident red flag among many quiet fields.
+
+## Date extraction
+
+`cookbooks/date_extraction_cookbook.md` — Extracts absolute and relative dates by asking Choice
+questions about which date parts (month/day/year/weekday, date kind) the text actually states, then
+resolving those parts to a real date entirely in code. Example: 6 questions across 4 documents — 5
+auto-accepted (confidence 0.91–0.97), 1 sent to review at confidence 0.46 (correctly flagged as
+incomplete rather than the model inventing a date). Review threshold used: confidence < 0.60.
+**Lesson:** let code do all date arithmetic; the model only reads what the text literally names —
+this directly implements jaggedness #3's mitigation.
+
+## Pre-parsed value extraction
+
+`cookbooks/pre_parsed_value_extraction_cookbook.md` — A regex finds candidate spans (emails, phone
+numbers, dollar amounts); Jev's Choice then picks which candidate answers a specific question, and
+code copies that span verbatim and normalizes it — the returned value is always literally one of the
+regex's own matches, never invented. Two stated hard limits: Choice allows at most 255 options (past
+that, narrow in two stages); and result quality depends on having a good candidate-finder (regex,
+NER, or another cheap proposer) — names in particular have no simple regex. **Lesson:** when a value
+must come verbatim from the source with zero risk of alteration, never let the model output free
+text — have code enumerate exhaustive candidates and let Choice only ever select among them.
+
+## Hierarchical classification
+
+`cookbooks/hierarchical_classification.md` — Classifies documents down deep taxonomies (patents,
+product categories, biomedical subjects, a codebase file tree) by walking node-to-node with Choice
+questions, comparing greedy search (always take the top child) against beam search (keep top-K
+paths, score by length-normalized geometric-mean edge probability, explored via parallel requests).
+Beam search (K=3) matched 4/4 expected leaves across 4 hierarchies; greedy matched 2/4 (an early
+wrong turn couldn't be recovered). **Lesson:** for deep/branching taxonomies, prefer beam search over
+greedy — parallel requests make exploring several candidate paths cheap, and it recovers from an
+ambiguous early decision that greedy can't undo. This directly implements the "walking a taxonomy"
+technique in `primitives-reference.md`.
+
+## Autoresearch feature discovery
+
+`cookbooks/autoresearch_feature_discovery.md` — An LLM proposes candidate numeric features (as
+natural-language questions) for a gradient-boosting model predicting a wine score from tasting notes;
+each round, Jev answers all proposed questions for every row (intensity → Score, presence → Noul),
+k-fold CV judges whether to keep/revise/drop each question. Over 5 rounds (1,200 dev / 800 held-out
+rows), held-out RMSE improved 0.097 points; ended with 38 kept questions. **Lesson:** Jev questions
+can serve as an LLM-designed, numerically-scored feature layer in front of an ordinary ML model — the
+LLM proposes what to ask, Jev answers it cheaply and consistently at scale, and a held-out set that
+the loop never reads is what actually validates whether new questions help.
+
+## Classification using confidence
+
+`cookbooks/classification_using_confidence.md` — Classifies SEC filings into 1 of 75 industry groups
+with a single Choice question, then uses that answer's own `confidence` to decide whether to report
+the specific group or fall back to its broader parent division. Across 60 filings, a confidence cutoff
+of 0.9 splits them roughly in half: the confident half is correct 90% of the time at the specific
+level, the unsure half only 40% — but reporting the unsure half at the broader division level raises
+it to 70% correct. **Lesson:** a Choice answer's own confidence is a free built-in signal for
+"trust the specific label, or hedge" — when labels form a hierarchy, hedging to the parent category is
+a nearly-free way to trade specificity for reliability exactly where it's needed.

package/jev/skills/jev/references/jaggedness.md

@@ -0,0 +1,135 @@
+# jev-1.13 jaggedness — known limitations
+
+Source: `docs.typesafe.ai/model-jaggedness/jev-1.13.md`, TypeSafe's own dated list ("Last reviewed
+2026-09-17" at crawl time, 2026-09-21) of known failure modes for the current model version, each
+with a stated mitigation. TypeSafe expects some of these to be fixed in later versions — check the
+live page before assuming a mitigation is still required for whichever `jev-*` version you're
+actually calling.
+
+This is the single most load-bearing reference for "will this question actually work well" — read
+it before designing a new question, not after it misbehaves in production.
+
+## 1. Literal reading
+
+Jev "answers the question you wrote, not the one you meant." Scoping words, negations, and implied
+conditions are read at face value, not interpreted the way a person reading between the lines would.
+
+**Do instead:** state the exact condition you mean; be specific rather than relying on implication;
+put boundary cases directly into `criteria`; if a question unavoidably needs interpretation, split it
+into two literal sub-questions and combine the results in code rather than trusting one question to
+carry the nuance.
+
+## 2. Math and numbers
+
+**"Jev is not a calculator. We strongly recommend implementing any mathematical logic in code."**
+
+- **Counting** does not work reliably — characters in a word, occurrences of a term, items in a long
+  list. Error grows with the size of the thing being counted. **Do instead:** count in code. For a
+  *filtered* count (count items matching some semantic condition), ask one Noul per candidate item
+  and sum the true/false results in code, rather than asking Jev to count directly.
+- **Numeric representations** (hex, RGB, binary, low-level/assembly forms) perform worse than
+  semantic or high-level representations of the same information — Jev "cannot reliably judge
+  whether two [RGB/hex] values are near each other." **Do instead:** convert to semantic form or
+  named buckets in code before asking.
+- **Using `score` for exact magnitude:** don't use a Score answer's probability-weighted `score` (or
+  the underlying `probabilities`) to compute the *exact* numeric value between two levels of a
+  criterion — score can be thresholded or compared, but treating `1.43` as a precise interpolated
+  measurement is not supported.
+
+## 3. Date and time comparison
+
+Jev "reads dates as text, not as ordered quantities." Ordering, distance, and window-membership
+judgments over dates are unreliable, and get worse with mixed formats, relative references ("next
+Thursday"), and domain-specific boundaries.
+
+**Do instead:** extract date *components* as a bounded Choice (e.g. one of 12 months, one of 31
+days, a bounded year range) — including an explicit "not stated" option for text that doesn't
+actually name a date — then do all ordering, duration, and offset/weekday arithmetic in code. See
+`references/cookbooks.md`'s "Date extraction" entry for a full worked example, including a case where
+this approach correctly flagged a date the source text never actually stated, instead of the model
+inventing one.
+
+## 4. Indirection
+
+"Instructions carrying double negatives or complex indirection are answered less reliably. A
+question about a property of a property, or something that requires multiple hops of reasoning,
+costs accuracy."
+
+**Do instead:** write instructions directly; name the relevant part of `state` explicitly rather
+than making the model infer which part you mean.
+
+## 5. Large state full of irrelevant detail
+
+"Accuracy falls as the state grows with content unrelated to the decision." Jev has a bounded
+context window (see `primitives-reference.md`'s Models table for the exact 64k/32k token limits) —
+but the accuracy cost of irrelevant content in `state` is a separate, independent problem from the
+token limit itself.
+
+**Do instead:** filter in code first and send only the fields a given question actually needs. When
+code-side filtering isn't feasible (e.g. you don't know in advance which chunk of a document is
+relevant), use a Noul question as a relevance pre-filter before the real question — see
+`references/cookbooks.md`'s "Classifying RAG passages" entry.
+
+**Note on an apparent tension:** the introduction docs say that batching *more questions* into one
+request "does not create context-rot" — that claim is true and is about parallel, independent
+question evaluation. This jaggedness item is a different claim: irrelevant material *in the shared
+`state`* does cost accuracy. Both are true at once; don't conflate "add more questions" (safe) with
+"pad the state with content the question doesn't need" (costly).
+
+## 6. Adversarial content
+
+"State is data, and jev-1.13 does not treat it as hostile by default." Content written to
+adversarially steer the model (prompt injection embedded in retrieved or user-supplied text) can
+move the answer. TypeSafe states it expects to improve this in future versions.
+
+**Do instead:** be explicit in `criteria` about what would and wouldn't count; test thoroughly with
+adversarial inputs before deploying a question over untrusted content to many users. See
+`references/cookbooks.md`'s "Classifying RAG passages" entry for a worked example that specifically
+screens for and drops a planted prompt-injection passage using a dedicated Noul question.
+
+## 7. Contradictory instructions and criteria
+
+A mismatch between `instructions` and `criteria` (e.g., a Noul where the `criteria.true` description
+actually describes what most people would call "no") degrades performance.
+
+**Do instead:** treat `criteria` as an extension of the instruction, not a separate spec — keep both
+aligned in clear, plain language, and re-read a structured question as a whole rather than writing
+`instructions` and `criteria` independently.
+
+## 8. No guaranteed structural invariants
+
+Jev is "extremely consistent [for] semantically similar inputs" but does **not** guarantee the kind
+of structural invariants a person might assume hold:
+
+- The same underlying question asked as a Noul vs. an equivalent yes/no Choice can disagree
+  materially — a documented example gives `noul: 0.22` for one framing and `{choice: "no",
+  probabilities: {yes: 0.01, no: 0.99}, confidence: 0.97}` for the logically equivalent Choice
+  framing of the same input. Neither is simply derivable from the other.
+- A Noul and its logical negation are not guaranteed to sum to 1 — a documented example has
+  `refund: 0.72` and `not_refund: 0.47` for the same message (sum 1.19, not 1.0). There are several reasons `P(noul)` and `1 - P(not noul)` may diverge; don't rely on the
+  identity holding.
+
+**Do instead:** don't rely on structural/arithmetic invariance between separate questions. Don't
+carry a confidence threshold tuned on one question (or question type) over to a different one without
+separately validating it.
+
+## 9. Generation
+
+"[Jev is] not trained to generate text. While you can force it to by chaining choices, this will not
+work well and will be very slow."
+
+**Do instead:** when the answer space is genuinely bounded, use Choice over the explicit option set
+rather than trying to make Jev produce the value as free text. When free text generation is actually
+required, use a generative model for that part of the workflow — Jev is not a substitute for one.
+
+## Closing checklist (from the docs' own "avoid" callout)
+
+Avoid:
+
+- Asking Jev something code can compute exactly (counting, dates, arithmetic).
+- Hiding several separate judgments inside one question — decompose instead.
+- "System Two" tasks — anything needing several layers of indirect reasoning.
+- Giving a question more `state` context than it actually needs — "Jev suffers from context rot,"
+  specifically with respect to unrelated material in the shared state (see item 5 above).
+
+Community feedback channel TypeSafe links from this page: Discord, `discord.com/invite/WUujKYBp8s`.