rote-cli 0.1.0__py3-none-any.whl

rote/skills/rote-graduate/references/llm-judge-extraction.md

@@ -0,0 +1,341 @@
+# LLM Judge Extraction
+
+Phase 4 of the graduator turns every `llm_judge` node's fuzzy prose
+rubric into a typed signature with bounded inputs and outputs. This
+file is the reference for that extraction.
+
+The premise is simple: if the source skill's rubric defines a
+classification (red flags, tiers, decision categories), that
+classification has a natural schema — and *running it as an unbounded
+LLM prompt throws away all the structure*. A typed signature recovers
+the structure, makes the step regression-testable, and lets downstream
+nodes consume a predictable shape instead of parsing free text.
+
+## What a typed signature looks like
+
+A typed signature is a Python class (or BAML function, or DSPy
+Signature) with three parts:
+
+1. **Input model** — a Pydantic BaseModel with every field the LLM
+   needs, sourced from upstream nodes.
+2. **Output model** — a Pydantic BaseModel with enum-bounded fields
+   wherever the rubric implies a discrete choice, plus any structured
+   rationale.
+3. **Forward method** — the call site that dispatches to the LLM. For
+   v0 this is a stub; downstream work fills it in with DSPy or BAML.
+
+The BDR example `signatures/vet_contact.py` is the canonical reference
+implementation.
+
+## The extraction procedure
+
+### Step 1 — Read the source rubric
+
+Find every piece of prose in the source skill that describes how to
+classify the input. For BDR's `vet_contact`, this is
+`references/quality-and-vetting.md`:
+
+- High-signal indicators (boosts)
+- Red flags (discard)
+- The core test ("would this person commission an RWE study?")
+- Tier definitions (ideal / strong / good)
+- Numeric thresholds (accuracy ≥ 85)
+
+### Step 2 — Enumerate the decision space
+
+Ask: what are the possible outputs? For `vet_contact`:
+
+| Dimension | Values |
+|---|---|
+| Decision | `keep`, `discard` |
+| Tier (if keep) | `ideal`, `strong`, `good` |
+| Reason (if discard) | `indication_mismatch`, `msl_role`, `biomarker_discovery`, `translational`, `sales_commercial`, `ops_strategy`, `program_management`, `low_accuracy`, `no_valid_email`, `other` |
+| Evidence | free-form string (1-2 sentences) |
+
+Anything with a small enumerable value space becomes an **enum**. Free
+text is only used for the *evidence* or *explanation* field, never for
+the core decision.
+
+**Rule of thumb:** if you can write the values in a table like the one
+above, it's an enum. If the values would span a paragraph, it's free
+text.
+
+### Step 3 — Design the input model
+
+Walk the IR backward from the node. The input model must contain every
+field the LLM needs to make the decision. For `vet_contact`:
+
+```python
+class VetContactInput(BaseModel):
+    model_config = ConfigDict(extra="forbid")
+
+    contact: EnrichedContact   # from enrich_contact_batch upstream
+    brief: CampaignBrief        # from pipeline input
+    intel: IntelBrief           # from target_research upstream
+```
+
+**`extra="forbid"`** prevents accidental field additions that would
+silently break downstream reads. Strict-by-default.
+
+### Step 4 — Design the output model
+
+```python
+class VetDecision(str, Enum):
+    KEEP = "keep"
+    DISCARD = "discard"
+
+class ContactTier(str, Enum):
+    IDEAL = "ideal"
+    STRONG = "strong"
+    GOOD = "good"
+
+class DiscardReason(str, Enum):
+    INDICATION_MISMATCH = "indication_mismatch"
+    MSL_ROLE = "msl_role"
+    BIOMARKER_DISCOVERY = "biomarker_discovery"
+    # ... one per rubric category
+
+class VetContactOutput(BaseModel):
+    model_config = ConfigDict(extra="forbid")
+
+    decision: VetDecision
+    tier: ContactTier | None = None           # only when decision == keep
+    discard_reason: DiscardReason | None = None   # only when decision == discard
+    relevance_evidence: str
+```
+
+**Optional fields with `None` defaults** are how you express "this
+field only applies in some decision branches" without making the
+schema conditional. Downstream consumers handle the None case.
+
+### Step 5 — Lift hard thresholds into a pre-filter
+
+If the rubric contains any numeric threshold or enum check that can be
+evaluated without the LLM, put it in the `forward()` method as a
+pre-filter that short-circuits before calling the model.
+
+From `signatures/vet_contact.py`:
+
+```python
+MIN_ACCURACY_SCORE: int = 85
+
+class VetContact:
+    async def forward(self, inputs: VetContactInput) -> VetContactOutput:
+        if inputs.contact.accuracy_score < MIN_ACCURACY_SCORE:
+            return VetContactOutput(
+                decision=VetDecision.DISCARD,
+                discard_reason=DiscardReason.LOW_ACCURACY,
+                relevance_evidence=(
+                    f"Accuracy score {inputs.contact.accuracy_score} "
+                    f"below threshold {MIN_ACCURACY_SCORE}."
+                ),
+            )
+        # ... dispatch to LLM for fuzzy cases
+```
+
+**Why this matters:** the pre-filter saves tokens on the obvious cases
+(probably 20-40% of real contacts) and guarantees the hard rule cannot
+drift. The LLM can never "forget" the accuracy threshold because it
+never sees those contacts.
+
+### Step 6 — Scaffold a seed eval set
+
+For every `llm_judge` node, create a seed evals file at the path
+referenced by the node's `eval_set:` field. Each line is one test case.
+
+Harvest examples directly from the source rubric. For every
+discard_reason enum value, construct at least one input that should
+produce that reason.
+
+For BDR `evals/vet_contact.jsonl`:
+
+```jsonl
+{"name": "msl_role_discard", "input": {"contact": {"job_title": "Medical Science Liaison, Respiratory", "employment_history": [{"title": "MSL, Oncology"}], "accuracy_score": 95}, "brief": {"therapeutic_area": "respiratory"}}, "expected": {"decision": "discard", "discard_reason": "msl_role"}}
+{"name": "biomarker_discard", "input": {"contact": {"job_title": "Director, Biomarker Sciences", "accuracy_score": 92}}, "expected": {"decision": "discard", "discard_reason": "biomarker_discovery"}}
+{"name": "low_accuracy_discard", "input": {"contact": {"job_title": "Sr. Director RWE", "accuracy_score": 70}}, "expected": {"decision": "discard", "discard_reason": "low_accuracy"}}
+{"name": "ideal_coe", "input": {"contact": {"job_title": "Sr. Director, Real World Evidence CoE", "employment_history": [...]}}, "expected": {"decision": "keep", "tier": "ideal"}}
+```
+
+These seed examples are not a full eval suite — they're the starting
+point a human can expand as they encounter real cases. But having them
+in the repo means the downstream DSPy/BAML compile step has a regression
+baseline to optimize against.
+
+## Common patterns
+
+### Pattern: "discard categories" rubric
+
+The source skill has an explicit list of categories to reject. Each
+becomes an enum member of a `Reason` enum, and the eval set has one
+example per category.
+
+### Pattern: "tier the keepers" rubric
+
+The source skill defines tiers (ideal / strong / good) with criteria
+for each. Each tier becomes an enum member of a `Tier` field, which is
+optional (only set when the decision is to keep).
+
+### Pattern: "one judge, fan out"
+
+If the source skill applies the same judgment to a list of inputs
+(vet 50 contacts, personalize 10 emails), set `fan_out: true` on the
+node in the IR. The adapter will invoke the signature once per input
+element in parallel. The signature itself handles one input at a time.
+
+### Pattern: "cheap pre-filter + expensive LLM"
+
+Almost every `llm_judge` has at least one numeric or enum constraint
+hiding in the rubric. Always check.
+
+## What does NOT belong in a signature
+
+- **Exploratory work.** If the "classification" requires the agent to
+  decide which external sources to consult, it's an `agent_loop`, not
+  an `llm_judge`.
+- **Unbounded generation.** If the output is a paragraph of free prose
+  with no schema, reconsider whether it's the right kind. Most
+  "generate a paragraph" tasks can be narrowed to a few structured
+  fields (opening line + TA callout + CTA, for example).
+- **Multi-step reasoning across many inputs.** Signatures take one
+  bounded input and return one bounded output. If the step needs to
+  reason across a batch, use `fan_out: true` so each input is its
+  own invocation.
+
+## The output of Phase 4
+
+For each `llm_judge` node:
+
+1. A file at `signatures/<node_name>.py` with:
+   - The input model
+   - The output model
+   - The enum definitions
+   - The signature class with a stub `forward()` method (pre-filter
+     logic included; LLM dispatch raises `NotImplementedError`)
+2. A file at `evals/<node_name>.jsonl` with 3–10 seed examples, one
+   per distinct decision path.
+3. A `signature_spec:` block embedded directly in the
+   `pipeline.yaml` node — see "Cross-runtime signature_spec" below.
+
+Record every signature extracted in the Phase 7 graduation report with
+the source rubric location and a one-line summary of the output schema.
+This is the audit trail for the human reviewer.
+
+## Cross-runtime signature_spec
+
+Python signature files (item 1 above) work for the Temporal adapter,
+which emits Python and can `import` them directly. They do **not**
+work for runtimes that emit a different language — the Cloudflare
+adapter emits TypeScript, can't read Python, and has no way to call
+into a Python BAML/DSPy client (those have native Rust binaries that
+don't run on Cloudflare Workers' V8 isolate).
+
+The IR carries a runtime-agnostic structured form alongside the path:
+`signature_spec`. Every `llm_judge` node should populate it. The
+adapter that consumes the IR converts the schemas to whatever native
+shape its target language expects — Pydantic for Python, Zod for
+TypeScript, etc.
+
+### Field shape
+
+```yaml
+signature_spec:
+  input_schema: { ... }          # JSON Schema for the input model
+  output_schema: { ... }          # JSON Schema for the output model
+  prompt: |                       # Jinja-style {{ var }} interpolation
+    <multi-line prompt template>
+  client: anthropic               # 'anthropic' | 'openai'
+  model: claude-sonnet-4-6        # optional; adapter chooses default
+  temperature: 0.0                # optional
+```
+
+### Deriving the JSON Schemas
+
+The Pydantic models you wrote in step 3 (input model) and step 4
+(output model) already know how to emit JSON Schema:
+
+```python
+VetContactInput.model_json_schema()
+VetContactOutput.model_json_schema()
+```
+
+Embed those dictionaries verbatim under `input_schema` and
+`output_schema`. The `$defs` block stays inline — adapters resolve
+references at emit time.
+
+If you can't run Python, derive the JSON Schema by hand from the
+Pydantic source:
+
+| Pydantic | JSON Schema |
+| --- | --- |
+| `field: str` (required) | `{"type": "string"}` in `properties`, name in `required` |
+| `field: int` | `{"type": "integer"}` |
+| `field: float` | `{"type": "number"}` |
+| `field: bool` | `{"type": "boolean"}` |
+| `field: list[X]` | `{"type": "array", "items": <X>}` |
+| `field: SomeEnum` | `{"enum": [<member values>]}` (or `$ref` to a `$defs` entry) |
+| `field: X \| None = None` | `{"anyOf": [<X>, {"type": "null"}], "default": null}`, optional in `required` |
+| `model_config = ConfigDict(extra="forbid")` | `"additionalProperties": false` on the object |
+
+For nested Pydantic models, hoist the inner model into a `$defs`
+entry and use `{"$ref": "#/$defs/InnerModelName"}` at the use site.
+This matches Pydantic's own emission shape and lets adapters resolve
+references with a single helper.
+
+### Designing the prompt template
+
+The prompt is a Jinja-style template. Variables are addressed by the
+input model's field names (top-level only — adapters use simple
+`{{ contact }}` substitution that JSON-stringifies non-string
+values). Three rules:
+
+1. **Always end with a directive that names the structured-output
+   tool** — the adapter wraps the call in tool-use mode and the LLM
+   needs the cue to invoke it. Example: *"Return your decision via
+   the structured output tool."*
+2. **Reproduce the discard-categories table inline.** The schema
+   already constrains the output enum, but the prompt should still
+   describe each category in prose so the LLM has the rubric.
+3. **Don't paste the source skill's entire reference file** — just
+   the rubric. The skill bundle has plenty of context that's
+   irrelevant at decision time and inflates token cost per call.
+
+### Worked example: vet_contact prompt
+
+```yaml
+prompt: |
+  You are vetting a contact for a BDR outreach campaign.
+
+  Apply this rubric:
+  - Discard if job title indicates MSL, Biomarker/Discovery,
+    Translational Research, Sales/Commercial, Operations/Strategy,
+    or Program Management.
+  - Discard on indication mismatch with the campaign therapeutic
+    area.
+  - Tier surviving contacts: ideal / strong / good based on RWE/HEOR
+    signal density.
+
+  Core test: would this person commission, design, or approve a
+  real-world evidence study?
+
+  Contact: {{ contact }}
+  Campaign brief: {{ brief }}
+  Intel brief: {{ intel }}
+
+  Return your decision via the structured output tool.
+```
+
+### Pre-filter logic and signature_spec
+
+Hard thresholds (the Step 5 pre-filter) live in **Python**, not in the
+prompt. Cross-language emission is the responsibility of the runtime
+adapter. The Temporal adapter calls the Python signature class, which
+runs the pre-filter then dispatches to the LLM. The Cloudflare adapter
+emits a TS function that calls the LLM directly — a future iteration
+will model the pre-filter as a separate `pure_function` node so it
+runs cross-runtime, but for v0.2 the signature_spec is "schema +
+prompt only" and the pre-filter only short-circuits in the Python
+runtime.
+
+If you want a hard rule to apply on every runtime today, model it as
+a separate `pure_function` node *before* the `llm_judge` and route
+short-circuited inputs around the LLM via an explicit edge.

rote/skills/rote-graduate/references/node-kinds.md

@@ -0,0 +1,223 @@
+# Node Kinds — Classification Rubric
+
+Every step in a graduated pipeline is exactly **one** of five kinds. Phase 2
+of the graduator's job is assigning the right kind to every step in the
+source skill. This file is the reference for that classification.
+
+## The five kinds
+
+### `pure_function`
+
+A step whose logic is fully deterministic: same input, same output, no LLM
+reasoning required, no external API calls.
+
+**Signals to recognize one:**
+- The skill's prose describes a fixed transformation (e.g., "format as
+  markdown with these sections", "group contacts by company").
+- The skill includes literal Python, pseudocode, or a concrete formula.
+- The step produces a report, a formatted output, or a structured
+  summary from already-structured inputs.
+- The step is a loop with a clear termination condition over
+  already-gathered data.
+
+**Anti-signals (not a `pure_function`):**
+- The step calls an external service → `external_call`.
+- The step makes a fuzzy judgment → `llm_judge`.
+- The logic varies based on context in a way that can't be enumerated
+  → `agent_loop`.
+
+**BDR example:** `pre_enrollment_report` — takes counts of vetted, passed,
+and excluded contacts and renders a fixed markdown template. No LLM, no
+APIs, just string formatting.
+
+**Common mistake:** don't miss steps where the LLM was being used to
+generate obvious string templates. If the skill's prose *already shows you
+the exact output format*, the LLM was doing pure formatting and the step
+is a `pure_function`.
+
+---
+
+### `external_call`
+
+A step that makes a deterministic call to an external service (HTTP API,
+database, file system) with retry and timeout semantics.
+
+**Signals to recognize one:**
+- The skill uses a tool in a fixed, repeatable way — not exploratory.
+- The call has well-known limits (batch size, rate limits) that appear
+  in the skill's prose.
+- The step is "fetch X from service Y given params Z" with no LLM
+  reasoning about the response shape.
+
+**Anti-signals (not an `external_call`):**
+- The response requires LLM interpretation before being useful →
+  that's a pair of nodes: `external_call` followed by `llm_judge`.
+- The tool is being used exploratorily with variable inputs →
+  `agent_loop`.
+- The step is a pure in-memory computation → `pure_function`.
+
+**BDR example:** `enrich_contact_batch` — calls
+`zoominfo_enrich_contacts` with a fixed output field set and a hard batch
+size of 10. The parameters don't vary run-over-run.
+
+**Common mistake:** treating every tool call as an `external_call`. A tool
+call is only an `external_call` if the *semantics* of calling it are
+deterministic. If the agent decides which tool to call and what to pass
+each iteration, it's an `agent_loop`.
+
+**MCP → deterministic API:** this is the kind that most often originates
+as an MCP tool call in the source skill. The graduation step is to record
+the underlying vendor API the tool wraps (e.g.,
+`POST /crm/v3/objects/contacts/batch/upsert`) in the `impl`'s docstring so
+the trace from skill → MCP tool → REST endpoint is visible in the code.
+
+---
+
+### `llm_judge`
+
+A step that asks an LLM to make a classification or generation against a
+rubric, with typed input and typed output. The fuzzy part is bounded —
+the output space is enumerable or naturally small.
+
+**Signals to recognize one:**
+- The skill describes "apply these rules to decide X" where X is a
+  bounded decision (keep/discard, tier A/B/C, a short piece of
+  generated text).
+- There's an explicit rubric in the skill — red flags list, core test,
+  tier definitions, discard categories.
+- The output has a natural schema: decision + reason + evidence.
+- The input is bounded (a single contact, a single row of data) rather
+  than exploratory.
+
+**Anti-signals (not an `llm_judge`):**
+- The output is free-form prose with no schema → `agent_loop`.
+- The input is exploratory (search queries, research topics) → `agent_loop`.
+- The "judgment" has hard numerical thresholds that could be encoded
+  in Python — lift those into a pre-filter in the signature, but the
+  step itself can still be an `llm_judge`.
+
+**BDR example:** `vet_contact` — takes an enriched contact plus the
+campaign brief, applies the BDR red-flags rubric, returns
+`{decision: keep|discard, tier: ideal|strong|good, discard_reason: ...,
+relevance_evidence: str}`. The decision is fuzzy (reading employment
+history) but the output schema is tight.
+
+**Common mistake:** running an `llm_judge` when the classification is
+actually deterministic. If every dimension of the decision maps to a
+boolean check on the input, it's a `pure_function`.
+
+---
+
+### `agent_loop`
+
+A step that requires genuine LLM orchestration — the agent decides what
+to do next, which tools to call, and when to terminate, based on
+intermediate results. Reserved for genuinely exploratory work.
+
+**Signals to recognize one:**
+- The skill's prose says things like "iterate until...", "try different
+  searches", "backfill when gaps appear".
+- The tool choices vary run-over-run.
+- The termination condition depends on intermediate results, not a
+  fixed iteration count.
+- The step produces a summary or brief from external research where
+  the sources aren't known in advance.
+
+**Anti-signals (not an `agent_loop`):**
+- The step is a fixed sequence of API calls with known inputs → chain
+  of `external_call` nodes.
+- The step is an LLM classification with bounded output → `llm_judge`.
+- Every "iteration" does the same thing on different inputs → a
+  `pure_function` or `external_call` with `fan_out: true`.
+
+**BDR example:** `lead_generation_loop` — starts with three parallel
+ZoomInfo searches, enriches in batches, discards contacts that fail
+vetting, backfills with new targeted searches until the quota is met.
+Both the number of iterations and the specific queries vary per campaign.
+
+**Required fields:** `tools` must be set on every `agent_loop` node (the
+tools the agent may call). `loop_body` is optional and lists the IDs of
+sub-nodes the loop invokes each iteration.
+
+**Common mistake:** leaving things as `agent_loop` when they could be
+crystallized. Most skills over-use this kind on their first pass because
+"the LLM was doing it" is the easiest classification. Fight the urge —
+prefer any other kind when the data supports it. Every step you keep as
+`agent_loop` is a run-time cost and a reliability risk.
+
+---
+
+### `hitl_gate`
+
+A step where the workflow pauses waiting for a human signal before
+continuing. Survives worker restarts; resumes when the signal arrives.
+
+**Signals to recognize one:**
+- The skill explicitly says "present to the user", "wait for approval",
+  "the user may add or remove".
+- The next step is conditional on explicit human decision.
+- The skill says something "must be done manually in the UI" — the
+  gate is the thing that confirms it happened.
+
+**Anti-signals (not a `hitl_gate`):**
+- The step prints output but doesn't wait for a response → that's the
+  end of a pipeline or a terminal `pure_function`.
+- The "human" is conceptual — there's no actual signal → still a
+  `pure_function` that produces a report.
+
+**BDR examples:**
+1. `contact_review_gate` — Phase 3 pause where the BDR reviews the
+   vetted contact table before CRM upload. Signal:
+   `contact_review_approved`.
+2. `manual_enrollment_handoff` — Phase 7 pause where the BDR manually
+   enrolls contacts in HubSpot's UI. Signal: `bdr_enrollment_complete`.
+
+**Required field:** every `hitl_gate` MUST have a `signal`. The adapter
+uses this to generate the corresponding signal handler in the workflow.
+
+**Optional field:** `notify` tells the adapter how to alert the human
+reviewer when the workflow reaches the gate (Slack channel, email, etc.).
+
+---
+
+## Decision rules for ambiguous cases
+
+**When a step could be two kinds, prefer the more deterministic one.** In
+descending order of determinism:
+
+```
+pure_function > external_call > llm_judge > agent_loop
+```
+
+The north star is "keep the LLM at points where the input is unbounded or
+ambiguous, and codify everything else." Every step you move leftward on
+this ladder is tokens saved and reliability gained.
+
+**`agent_loop` vs `external_call`:** could the agent do this with a fixed
+sequence of calls? If yes, it's `external_call`s (possibly chained via
+edges or fan-out). If the sequence itself varies based on intermediate
+results, it's `agent_loop`.
+
+**`llm_judge` vs `pure_function`:** does the decision depend on reading
+prose (job titles, descriptions, employment histories, free text)? If
+yes, it's `llm_judge`. If it depends only on numeric thresholds or enum
+matching, it's `pure_function`.
+
+**`llm_judge` vs `agent_loop`:** does the output have a schema? If yes,
+it's `llm_judge`. Free-form output usually indicates the agent is
+orchestrating something and you haven't found its real boundaries yet.
+
+**`hitl_gate` vs report-producing `pure_function`:** is there an explicit
+signal or approval the workflow waits for? If yes, `hitl_gate`. If the
+step just prints a summary and the pipeline ends or continues
+unconditionally, it's a `pure_function`.
+
+## Cheat sheet
+
+| Question | Kind |
+|---|---|
+| Is there an explicit human signal? | `hitl_gate` |
+| Is it a vendor API call with fixed semantics? | `external_call` |
+| Is it fully deterministic Python? | `pure_function` |
+| Is it a fuzzy classification with typed output? | `llm_judge` |
+| Does the agent genuinely need to decide? | `agent_loop` |