@pikku/inspector 0.12.55 → 0.12.58

package/CHANGELOG.md

@@ -1,3 +1,171 @@
+## 0.12.58
+
+### Patch Changes
+
+- 3ad2131: Name models by what they are for, and switch them all in one place
+
+  A `models` table in pikku.config.json maps an alias to a provider-qualified
+  model, so a declaration can say `model: 'cheap'` and the project repoints every
+  use of that tier at once instead of editing each agent. A model containing `/`
+  is still concrete and used exactly as written, which is how an agent that needs
+  one specific model pins it — aliases are opt-in.
+
+  The table is baked into codegen rather than read at runtime, so it applies to
+  deployed units and not just local runs, and `pikku dev`/`pikku serve` take
+  `--model cheap:openai/gpt-5-nano` to repoint a tier for one run without editing
+  the config.
+
+  Because the inspector already holds every agent's model literal, a bare name
+  with no matching alias now fails the build (PKU146) naming the aliases that do
+  exist, rather than reaching a provider as an unknown model.
+
+  Aliases resolve for every modality, not just agents: image, speech,
+  transcription, embedding and reranking all reach a provider through the same
+  point in the Vercel runner.
+
+- b930dca: Remove the `secretBroker` escape hatch and scope addon secrets and credentials
+
+  `secretBroker` let three named console functions receive the real `SecretService`,
+  against the rule that a function never sees one. It is gone: the inspector allowlist,
+  the `FunctionRuntimeMeta` flag, the runner branches, and the `WiredSecretBrokerServices`
+  type. Console secret administration moved into the console addon, where a
+  `SecretAdminService` holds the `SecretService` and the functions hold none.
+
+  Addons are now scoped rather than trusted. The CLI emits each package's declared secret
+  keys, and the host wraps the `SecretService` in a `ScopedSecretService` and the
+  `CredentialService` in a new `ScopedCredentialService` before the addon's service factory
+  runs — so an addon reads only what it declared, cannot write secrets, and cannot enumerate
+  the app's users. `wireAddon({ globalSecrets, globalCredentials })` waives this, taking the
+  reason as its value; only the consuming app can grant it, and the deploy manifest reports
+  every grant under `unscopedSecretAddons` / `unscopedCredentialAddons`.
+
+- 8978fbd: feat(workflow): let an approval gate declare who may answer it
+
+  `workflow.approval()` gains `approvers` (`'any' | 'owner' | 'not-initiator'`)
+  and `approverScope`, so a gate can require four-eyes sign-off, restrict itself
+  to the run's initiator, or require the decider to hold a named scope.
+
+  Both are enforced when the workflow replays the gate — the same place, and for
+  the same reason, the decision payload is validated: the policy is a value on
+  the workflow, and a decision can be recorded before the run has ever reached
+  the gate. A decision that fails the policy is discarded and the gate stays
+  closed. Where the run has already published its policy, the check also runs at
+  submission time so the caller gets a 403 rather than silence.
+
+  An answer is now recorded where it can be answered for later. The settled
+  decision carries `decidedBy` and `decidedAt` in its `ApprovalOutcome`, so who
+  signed reaches `workflowStep.result` and `workflowStepHistory` rather than
+  living only in mutable run state. Every answer — accepted, refused at the door,
+  or cleared on replay — is also written to the audit sink as
+  `workflow.approval.decided`, which outlives the run: `deleteRun` cascades to
+  steps and history, and a refused attempt never reaches a step at all. Projects
+  with no audit service wired are unaffected.
+
+  **This loosens the default.** `approveStep` previously refused anyone but the
+  run's initiator, unconditionally. A gate that declares no `approvers` now
+  accepts a decision from anyone the approve entrypoint admits — restore the old
+  behaviour per-gate with `approvers: 'owner'`, or gate the approve route with
+  `auth`/`permissions`. Ownership still governs _reads_ of a run unchanged.
+
+- Updated dependencies [063f43a]
+- Updated dependencies [ce66bf8]
+- Updated dependencies [d0307a8]
+- Updated dependencies [ce66bf8]
+- Updated dependencies [3ad2131]
+- Updated dependencies [b930dca]
+- Updated dependencies [b95e77d]
+- Updated dependencies [fd9d834]
+- Updated dependencies [8978fbd]
+  - @pikku/core@0.12.82
+
+## 0.12.57
+
+### Patch Changes
+
+- e110c55: Wire `pikkuAIScorer` / `pikkuAIJudge` through the inspector and codegen, and let an agent name the scorers that grade it.
+
+  The inspector reads a scorer's lane off which constructor was called rather than
+  off a field, so the two lanes cannot disagree with the code that produced them,
+  and refuses a scorer with no name or description — the meta is the only thing
+  that names it at runtime. An agent naming a scorer that was never declared is a
+  build error (`PKU155`) rather than an agent that quietly grades nothing forever.
+
+  Codegen emits a `ScorerName` union, so `scorers` on an agent is checked against
+  the scorers the project actually declares, plus the scorer wirings and meta.
+  `pikku validate` now also flags a scorer declared outside a `*.scorer.ts` file,
+  for the same reason scenarios have to live in files named for them: a rubric
+  buried in an agent definition is one nobody reviews as a rubric.
+
+- e110c55: Add `scenario.expectScore` — grade a finished agent run with a declared scorer and assert on it.
+
+  An agent's answer cannot be matched against a fixed string, so a scenario grades
+  it instead. `expectScore(step, runId, scorer, { atLeast, atMost, reference })`
+  runs one declared scorer against the run the scenario just triggered and fails
+  with the reason the judge gave. The default bound is `atLeast: 0.5`, so an
+  unqualified assertion still fails a run graded zero.
+
+  Grading goes over the new `pikkuScenarioGradeRun` instrumentation RPC, which the
+  dev server registers alongside the coverage and stub RPCs — so it exists only in
+  processes that should have it, and never in a deployed bundle. It grades from
+  the snapshot the runtime already took when the run finished, which is what makes
+  a scenario's grade the same measurement production's sampler makes rather than
+  an approximation of it: a run's prompt, answer and tool calls are spread across
+  a thread's messages, where the boundary of one run is not recoverable.
+
+  Two things differ deliberately from live scoring. The sample rate is ignored — a
+  scorer grading 1% of traffic still grades every scenario run — and the grade is
+  returned rather than recorded, so a test's score never lands among the
+  production figures. `reference` supplies the answer key a `requiresReference`
+  judge grades against, which is the only way such a judge is reachable at all.
+
+- Updated dependencies [e110c55]
+- Updated dependencies [e110c55]
+- Updated dependencies [e110c55]
+- Updated dependencies [acc8077]
+- Updated dependencies [905f737]
+- Updated dependencies [3cc6428]
+- Updated dependencies [c524adf]
+- Updated dependencies [e110c55]
+  - @pikku/core@0.12.81
+
+## 0.12.56
+
+### Patch Changes
+
+- 7e60867: Delete exports nothing references
+
+  A sweep of the `@pikku/core` surface for exports with no consumer anywhere in the repo — no package, template, verifier or e2e project imports them: `ExtractFunctionOutput`, `CLICommandDefinition`, `RequestHeaders`, `HTTPFunctionsMeta`, `HTTPWiringMiddleware`, `JsonRpcError`, `TriggerSourceInfo`, `getMCPResources`, `getMCPPrompts`, `onGraphNodeComplete` and `InputRef`.
+
+  Every one was a compatibility promise with nothing on the other end of it. Removing them narrows what 0.13 has to keep stable.
+
+  `isRef` looked like the twelfth, and isn't. It is the type guard that reads what `createRef` writes — the `__isRef` brand marking a graph node input as "substitute another node's output here". Nothing imported it because neither it nor `RefValue` was reachable from any entry point, so the one consumer that needed it, the inspector's graph serializer, had reimplemented the same four conditions privately as `isRefValue` along with its own structural copy of `RefValue`. Deleting `isRef` would have made that duplicate permanent, with the brand's shape asserted in two places free to drift apart.
+
+  So `isRef` and `RefValue` are exported from `@pikku/core/workflow` instead, and the inspector imports them rather than keeping its own copy.
+
+- a879ab3: **Codegen reports a schema it named but never generated (PKU463).** A named
+  contract type that is declared but not exported is imported by the virtual
+  source file the schema generator compiles, resolves to nothing, and yields no
+  schema — while the function meta still carries the name. `pikku all` exited 0
+  and the first call to that function failed in a deployment with
+  `MissingSchemaError`. The reference is now checked once addon schemas are
+  merged, since an addon supplies its own and checking earlier would report every
+  one of them as unresolved.
+- Updated dependencies [41c1a95]
+- Updated dependencies [ce96383]
+- Updated dependencies [7e60867]
+- Updated dependencies [f8f1244]
+- Updated dependencies [dcf20cb]
+- Updated dependencies [6512384]
+- Updated dependencies [e3b4c14]
+- Updated dependencies [efd0ed1]
+- Updated dependencies [cba98fb]
+- Updated dependencies [ce96383]
+- Updated dependencies [f8f1244]
+- Updated dependencies [f8f1244]
+- Updated dependencies [6e93a35]
+- Updated dependencies [6dada45]
+  - @pikku/core@0.12.80
+
 ## 0.12.55
 
 ### Patch Changes

package/dist/add/add-ai-agent.js

@@ -217,6 +217,29 @@ function resolveIdentifierToAgentName(identifier, checker) {
     }
     return null;
 }
+/**
+ * The scorers an agent asks to be graded by, which are plain names rather than
+ * references: a scorer is resolved out of the registry at grading time, not
+ * imported by the agent, so nothing here has a declaration to chase.
+ */
+function resolveScorerNames(obj, agentName, logger) {
+    const property = obj.properties.find((p) => ts.isPropertyAssignment(p) &&
+        ts.isIdentifier(p.name) &&
+        p.name.text === 'scorers');
+    if (!property || !ts.isPropertyAssignment(property))
+        return null;
+    if (!ts.isArrayLiteralExpression(property.initializer))
+        return null;
+    const names = [];
+    for (const element of property.initializer.elements) {
+        if (!ts.isStringLiteral(element)) {
+            logger.critical(ErrorCode.INVALID_VALUE, `AI agent '${agentName}' lists a scorer that is not an inline string: \`${element.getText()}\`. A scorer is named, not imported, so the name has to be readable from source.`);
+            continue;
+        }
+        names.push(element.text);
+    }
+    return names.length > 0 ? names : null;
+}
 export const addAIAgent = (logger, node, checker, state, options) => {
     if (!ts.isCallExpression(node)) {
         return;
@@ -244,6 +267,7 @@ export const addAIAgent = (logger, node, checker, state, options) => {
         const maxStepsValue = getPropertyValue(obj, 'maxSteps');
         const temperatureValue = getPropertyValue(obj, 'temperature');
         const toolChoiceValue = getPropertyValue(obj, 'toolChoice');
+        const scorersValue = resolveScorerNames(obj, nameValue || '', logger);
         const toolsValue = resolveToolReferences(obj, checker, nameValue || '', logger);
         if (toolsValue) {
             for (const toolName of toolsValue) {
@@ -389,6 +413,7 @@ export const addAIAgent = (logger, node, checker, state, options) => {
             ...(toolsValue !== null && { tools: toolsValue }),
             ...(agentsValue !== null && { agents: agentsValue }),
             ...(workflowsValue !== null && { workflows: workflowsValue }),
+            ...(scorersValue !== null && { scorers: scorersValue }),
             tags,
             inputSchema,
             outputSchema,

package/dist/add/add-ai-scorer.d.ts

@@ -0,0 +1,2 @@
+import type { AddWiring } from '../types.js';
+export declare const addAIScorer: AddWiring;

package/dist/add/add-ai-scorer.js

@@ -0,0 +1,61 @@
+import * as ts from 'typescript';
+import { getPropertyValue } from '../utils/get-property-value.js';
+import { extractFunctionName } from '../utils/extract-function-name.js';
+import { ErrorCode } from '../error-codes.js';
+/**
+ * `pikkuAIScorer` grades in code and `pikkuAIJudge` grades with a model, which
+ * is the whole difference between the two lanes — so the lane is read off which
+ * constructor was called rather than from a field an author could contradict.
+ */
+const LANES = {
+    pikkuAIScorer: 'fast',
+    pikkuAIJudge: 'slow',
+};
+export const addAIScorer = (logger, node, checker, state) => {
+    if (!ts.isCallExpression(node))
+        return;
+    const expression = node.expression;
+    if (!ts.isIdentifier(expression))
+        return;
+    const lane = LANES[expression.text];
+    if (!lane)
+        return;
+    const firstArg = node.arguments[0];
+    if (!firstArg || !ts.isObjectLiteralExpression(firstArg))
+        return;
+    const obj = firstArg;
+    const { exportedName } = extractFunctionName(node, checker, state.rootDir);
+    const nameValue = getPropertyValue(obj, 'name');
+    if (!nameValue) {
+        logger.critical(ErrorCode.MISSING_NAME, "AI scorer is missing the required 'name' property.");
+        return;
+    }
+    const description = getPropertyValue(obj, 'description');
+    if (!description) {
+        logger.critical(ErrorCode.MISSING_DESCRIPTION, `AI scorer '${nameValue}' is missing the required 'description' property.`);
+        return;
+    }
+    const sampleRateValue = getPropertyValue(obj, 'sampleRate');
+    if (sampleRateValue !== null &&
+        (sampleRateValue < 0 || sampleRateValue > 1)) {
+        logger.critical(ErrorCode.INVALID_VALUE, `AI scorer '${nameValue}' has a sampleRate of ${sampleRateValue} — it is the fraction of live runs to grade, so it must be between 0 and 1.`);
+        return;
+    }
+    const requiresReference = getPropertyValue(obj, 'requiresReference') ?? false;
+    const scorerKey = exportedName || nameValue;
+    if (exportedName) {
+        state.scorers.files.set(scorerKey, {
+            path: node.getSourceFile().fileName,
+            exportedName,
+        });
+    }
+    state.scorers.scorersMeta[scorerKey] = {
+        name: nameValue,
+        description,
+        lane,
+        sampleRate: sampleRateValue ?? 1,
+        requiresReference,
+        sourceFile: node.getSourceFile().fileName,
+        exportedName: exportedName || undefined,
+    };
+};

package/dist/add/add-wire-addon.js

@@ -53,6 +53,8 @@ export function addWireAddon(node, state, logger) {
     let secretOverrides;
     let variableOverrides;
     let credentialOverrides;
+    let globalSecrets;
+    let globalCredentials;
     for (const prop of firstArg.properties) {
         if (!ts.isPropertyAssignment(prop) || !ts.isIdentifier(prop.name))
             continue;
@@ -94,6 +96,18 @@ export function addWireAddon(node, state, logger) {
             ts.isObjectLiteralExpression(prop.initializer)) {
             credentialOverrides = parseStringRecord(prop.initializer);
         }
+        else if (key === 'globalSecrets') {
+            // The grant is real at runtime whatever the reason evaluates to, so a
+            // non-literal is recorded as its source text rather than dropped.
+            globalSecrets = ts.isStringLiteral(prop.initializer)
+                ? prop.initializer.text
+                : prop.initializer.getText();
+        }
+        else if (key === 'globalCredentials') {
+            globalCredentials = ts.isStringLiteral(prop.initializer)
+                ? prop.initializer.text
+                : prop.initializer.getText();
+        }
     }
     if (!name || !pkg)
         return;
@@ -108,6 +122,8 @@ export function addWireAddon(node, state, logger) {
         secretOverrides,
         variableOverrides,
         credentialOverrides,
+        globalSecrets,
+        globalCredentials,
     });
     state.rpc.usedAddons.add(name);
     state.rpc.wireAddonFiles.add(node.getSourceFile().fileName);

package/dist/add/scenario-instrumentation.d.ts

@@ -13,8 +13,13 @@
  * `pikku dev` registers the implementations itself. Ignoring by name (rather
  * than deleting the scaffold and requiring every project to regenerate) is what
  * keeps an existing project's checked-in `scenarios.gen.ts` from deploying.
+ *
+ * `pikkuScenarioGradeRun` was never scaffolded — it arrived after the split —
+ * but it belongs here for the same reason: grading a run on demand, ignoring the
+ * sample rate that governs live traffic, is a thing you do to a development
+ * server, and the name is reserved so an application cannot claim it.
  */
-export declare const SCENARIO_INSTRUMENTATION_FUNCTIONS: readonly ["pikkuScenarioTakeLiveCoverage", "pikkuScenarioResetLiveCoverage", "pikkuScenarioResetStubs", "pikkuScenarioGetStubCalls"];
+export declare const SCENARIO_INSTRUMENTATION_FUNCTIONS: readonly ["pikkuScenarioTakeLiveCoverage", "pikkuScenarioResetLiveCoverage", "pikkuScenarioResetStubs", "pikkuScenarioGetStubCalls", "pikkuScenarioGradeRun"];
 export type ScenarioInstrumentationFunction = (typeof SCENARIO_INSTRUMENTATION_FUNCTIONS)[number];
 /**
  * True for a scenario instrumentation function, by name. Versioned ids

package/dist/add/scenario-instrumentation.js

@@ -13,12 +13,18 @@
  * `pikku dev` registers the implementations itself. Ignoring by name (rather
  * than deleting the scaffold and requiring every project to regenerate) is what
  * keeps an existing project's checked-in `scenarios.gen.ts` from deploying.
+ *
+ * `pikkuScenarioGradeRun` was never scaffolded — it arrived after the split —
+ * but it belongs here for the same reason: grading a run on demand, ignoring the
+ * sample rate that governs live traffic, is a thing you do to a development
+ * server, and the name is reserved so an application cannot claim it.
  */
 export const SCENARIO_INSTRUMENTATION_FUNCTIONS = [
     'pikkuScenarioTakeLiveCoverage',
     'pikkuScenarioResetLiveCoverage',
     'pikkuScenarioResetStubs',
     'pikkuScenarioGetStubCalls',
+    'pikkuScenarioGradeRun',
 ];
 const names = new Set(SCENARIO_INSTRUMENTATION_FUNCTIONS);
 /**

package/dist/error-codes.d.ts

@@ -37,6 +37,7 @@ export declare enum ErrorCode {
     DB_FORM_ON_NON_STRING = "PKU484",
     DB_DEFAULT_SCHEMA_NAME_COLLISION = "PKU485",
     SCHEMA_GENERATION_ERROR = "PKU456",
+    SCHEMA_REFERENCE_UNRESOLVED = "PKU463",
     INLINE_SCHEMA = "PKU489",
     FUNCTION_METADATA_NOT_FOUND = "PKU559",
     DUPLICATE_AUTH_DEFINITION = "PKU581",
@@ -61,6 +62,7 @@ export declare enum ErrorCode {
     AGENT_TOOL_UNKNOWN_NAMESPACE = "PKU152",
     AGENT_TOOL_NOT_FOUND = "PKU153",
     AGENT_TOOL_MISSING_DESCRIPTION = "PKU154",
+    AGENT_SCORER_NOT_FOUND = "PKU155",
     SCHEMA_AND_WIRING_COLOCATED = "PKU490",
     SERVICES_NOT_DESTRUCTURED = "PKU410",
     WIRES_NOT_DESTRUCTURED = "PKU411",

package/dist/error-codes.js

@@ -42,6 +42,7 @@ export var ErrorCode;
     ErrorCode["DB_DEFAULT_SCHEMA_NAME_COLLISION"] = "PKU485";
     // Configuration errors
     ErrorCode["SCHEMA_GENERATION_ERROR"] = "PKU456";
+    ErrorCode["SCHEMA_REFERENCE_UNRESOLVED"] = "PKU463";
     ErrorCode["INLINE_SCHEMA"] = "PKU489";
     // Function errors
     ErrorCode["FUNCTION_METADATA_NOT_FOUND"] = "PKU559";
@@ -78,6 +79,8 @@ export var ErrorCode;
     ErrorCode["AGENT_TOOL_UNKNOWN_NAMESPACE"] = "PKU152";
     ErrorCode["AGENT_TOOL_NOT_FOUND"] = "PKU153";
     ErrorCode["AGENT_TOOL_MISSING_DESCRIPTION"] = "PKU154";
+    // Agent scorer reference errors
+    ErrorCode["AGENT_SCORER_NOT_FOUND"] = "PKU155";
     // File structure errors
     ErrorCode["SCHEMA_AND_WIRING_COLOCATED"] = "PKU490";
     // Optimization diagnostics

package/dist/inspector.js

@@ -6,12 +6,12 @@ import { TypesMap } from './types-map.js';
 import { getFilesAndMethods } from './utils/get-files-and-methods.js';
 import { findCommonAncestor } from './utils/find-root-dir.js';
 import { createNestedProjectFilter } from './utils/nested-project-filter.js';
-import { aggregateRequiredServices, stampAuthHandlerServices, validateAgentModels, validateAgentToolReferences, validateSecretOverrides, validateVariableOverrides, validateCredentialOverrides, validateRemoteAddonDependencies, validateRemoteAddonAuth, validateScopeReferences, validateSystemRoleScopes, validatePersonaRoles, computeResolvedIOTypes, computeMiddlewareGroupsMeta, computePermissionsGroupsMeta, computeRequiredSchemas, computeSecretBrokers, validateNoSecretAliasServices, validateSecretUsage, computeDiagnostics, validateSchemaWiringSeparation, validateScenarioServices, validateScenarioSteps, validateWorkflowGraphAddons, } from './utils/post-process.js';
+import { aggregateRequiredServices, stampAuthHandlerServices, validateAgentModels, validateAgentToolReferences, validateAgentScorerReferences, validateSecretOverrides, validateVariableOverrides, validateCredentialOverrides, validateRemoteAddonDependencies, validateRemoteAddonAuth, validateScopeReferences, validateSystemRoleScopes, validatePersonaRoles, computeResolvedIOTypes, computeMiddlewareGroupsMeta, computePermissionsGroupsMeta, computeRequiredSchemas, validateNoSecretAliasServices, validateSecretUsage, computeDiagnostics, validateSchemaReferences, validateSchemaWiringSeparation, validateScenarioServices, validateScenarioSteps, validateWorkflowGraphAddons, } from './utils/post-process.js';
 import { validateExposedFunctionsGated } from './utils/validate-exposed-functions-gated.js';
 import { validateTagsResolveToMiddleware } from './utils/validate-tags-resolve-to-middleware.js';
 import { annotateHttpRouteAuth } from './utils/annotate-http-route-auth.js';
 import { generateOpenAPISpec } from './utils/serialize-openapi-json.js';
-import { pikkuState } from '@pikku/core/internal';
+import { pikkuState } from '@pikku/core/ecosystem';
 import { resolveLatestVersions } from './utils/resolve-versions.js';
 import { finalizeWorkflows } from './utils/workflow/graph/finalize-workflows.js';
 import { finalizeWorkflowHelperTypes, finalizeWorkflowWires, } from './utils/workflow/graph/finalize-workflow-wires.js';
@@ -116,6 +116,10 @@ export function getInitialInspectorState(rootDir) {
             agentsMeta: {},
             files: new Map(),
         },
+        scorers: {
+            scorersMeta: {},
+            files: new Map(),
+        },
         cli: {
             meta: {
                 programs: {},
@@ -336,6 +340,11 @@ export const inspect = async (logger, routeFiles, options = {}) => {
         }
         // Re-load addon schemas (generateAllSchemas replaces state.schemas)
         await loadAddonSchemas(logger, state);
+        // After the addon merge, not before: an addon supplies its own schemas here,
+        // and checking earlier would report every one of them as unresolved.
+        if (options.schemaConfig) {
+            validateSchemaReferences(logger, state);
+        }
         state.manifest.initial = options.manifest ?? null;
         const contracts = extractContractsFromMeta(state.functions.meta);
         const baseManifest = state.manifest.initial ?? createEmptyManifest();
@@ -361,7 +370,6 @@ export const inspect = async (logger, routeFiles, options = {}) => {
         computeResolvedIOTypes(state);
         computeMiddlewareGroupsMeta(state);
         computePermissionsGroupsMeta(state);
-        computeSecretBrokers(state);
         validateNoSecretAliasServices(logger, checker, state);
         validateSecretUsage(logger, state);
         computeDiagnostics(state);
@@ -379,8 +387,9 @@ export const inspect = async (logger, routeFiles, options = {}) => {
         if (options.openAPI) {
             state.openAPISpec = await generateOpenAPISpec(logger, state.functions.meta, state.http.meta, state.schemas, options.openAPI.additionalInfo, pikkuState(null, 'misc', 'errors'));
         }
-        validateAgentModels(logger, state);
+        validateAgentModels(logger, state, options.modelAliases);
         validateAgentToolReferences(logger, state, options);
+        validateAgentScorerReferences(logger, state);
         validateSecretOverrides(logger, state);
         validateVariableOverrides(logger, state);
         validateCredentialOverrides(logger, state);

package/dist/types.d.ts

@@ -9,6 +9,7 @@ import type { QueueWorkersMeta } from '@pikku/core/queue';
 import type { WorkflowsMeta } from '@pikku/core/workflow';
 import type { MCPResourceMeta, MCPToolMeta, MCPPromptMeta } from '@pikku/core/mcp';
 import type { AIAgentMeta } from '@pikku/core/ai-agent';
+import type { ScorerMeta } from '@pikku/core/ai-scorer';
 import type { CLIMeta } from '@pikku/core/cli';
 import type { CLICommandMeta } from '@pikku/core/cli';
 import type { NodesMeta } from '@pikku/core/node';
@@ -229,6 +230,8 @@ export type InspectorOptions = Partial<{
      * tool's own name, so this is a quality gate rather than a correctness one.
      */
     strictMeta: boolean;
+    /** The `models` table from pikku.config.json, alias -> `provider/model`. */
+    modelAliases: Record<string, string>;
     sourceFile: ts.SourceFile;
     /**
      * The project's tsconfig, read for its path mappings only. The inspector's
@@ -518,6 +521,16 @@ export interface InspectorState {
             secretOverrides?: Record<string, string>;
             variableOverrides?: Record<string, string>;
             credentialOverrides?: Record<string, string>;
+            /**
+             * The app's stated reason for handing this instance the whole
+             * `SecretService` instead of one scoped to its declared secrets.
+             */
+            globalSecrets?: string;
+            /**
+             * The app's stated reason for handing this instance the whole
+             * `CredentialService` instead of one scoped to its declared credentials.
+             */
+            globalCredentials?: string;
         }>;
         wireAddonFiles: Set<string>;
     };
@@ -534,6 +547,13 @@ export interface InspectorState {
             exportedName: string;
         }>;
     };
+    scorers: {
+        scorersMeta: ScorerMeta;
+        files: Map<string, {
+            path: string;
+            exportedName: string;
+        }>;
+    };
     cli: {
         meta: CLIMeta;
         files: Set<string>;

package/dist/utils/filter-inspector-state.js

@@ -328,6 +328,14 @@ export function filterInspectorState(state, filters, logger) {
             agentsMeta: JSON.parse(JSON.stringify(state.agents?.agentsMeta ?? {})),
             files: new Map(),
         },
+        // Scorers carry through every filter. They are named by the agents that
+        // survive it, and a deployment that quietly stopped grading would look
+        // exactly like one whose scorers all returned nothing.
+        scorers: {
+            ...state.scorers,
+            scorersMeta: JSON.parse(JSON.stringify(state.scorers?.scorersMeta ?? {})),
+            files: new Map(state.scorers?.files ?? []),
+        },
         rpc: {
             ...state.rpc,
             internalMeta: { ...state.rpc.internalMeta }, // Clone to avoid mutating original

package/dist/utils/post-process.d.ts

@@ -50,7 +50,26 @@ export declare function computeResolvedIOTypes(state: InspectorState): void;
 export declare function computeMiddlewareGroupsMeta(state: InspectorState): void;
 export declare function computePermissionsGroupsMeta(state: InspectorState): void;
 export declare function computeRequiredSchemas(state: InspectorState, options: InspectorOptions): void;
-export declare function validateAgentModels(logger: InspectorLogger, state: InspectorState | Omit<InspectorState, 'typesLookup'>): void;
+/**
+ * Every schema a function's contract names has to exist once codegen is done.
+ *
+ * The way it stops existing is quiet. `generateCustomTypes` hands the schema
+ * generator a virtual source file that *imports* each named contract type, so a
+ * type that is declared but never exported resolves to nothing and yields no
+ * schema — while the function's meta still carries the name. Nothing downstream
+ * reads that name until the first call, which fails with a runtime
+ * `MissingSchemaError` in a deployment rather than in the build that caused it.
+ *
+ * Reported rather than critical: the reference is only *guaranteed* to break
+ * the functions that are actually invoked, so this blocks CI through
+ * `--fail-on-error` without failing a dev server over a function nobody calls.
+ */
+export declare function validateSchemaReferences(logger: InspectorLogger, state: InspectorState): void;
+/**
+ * A model is either concrete (`provider/model`) or an alias, which must appear
+ * in the `models` table from pikku.config.json.
+ */
+export declare function validateAgentModels(logger: InspectorLogger, state: InspectorState | Omit<InspectorState, 'typesLookup'>, modelAliases?: Record<string, string>): void;
 /**
  * Scenarios are pure stories of remote RPCs (same rule as client-side CLI
  * renderers): the func may only destructure logger/config — everything else
@@ -96,8 +115,6 @@ export declare function validateWorkflowGraphAddons(logger: InspectorLogger, sta
  * because the pikku state metadata doesn't exist in the CLI context.
  */
 export declare function validateSchemaWiringSeparation(logger: InspectorLogger, state: InspectorState): void;
-/** Marks the functions that keep the full `SecretService`. */
-export declare function computeSecretBrokers(state: InspectorState): void;
 /**
  * Refuses a singleton service that is a `SecretService` under another name.
  *
@@ -141,6 +158,14 @@ export declare function validateScopeReferences(logger: InspectorLogger, state:
  * a title labels a tool in a UI, it does not tell a model when to reach for it.
  */
 export declare function validateAgentToolReferences(logger: InspectorLogger, state: InspectorState | Omit<InspectorState, 'typesLookup'>, options?: InspectorOptions): void;
+/**
+ * Validates that every scorer an agent asks to be graded by exists.
+ *
+ * The runtime warns and skips an unresolvable name rather than failing a run
+ * that has already answered the user — which means a typo would otherwise cost
+ * nothing visible and simply grade nothing, forever.
+ */
+export declare function validateAgentScorerReferences(logger: InspectorLogger, state: InspectorState | Omit<InspectorState, 'typesLookup'>): void;
 /**
  * Validates that every scope granted by a system role is declared via
  * `defineScope`. Runs after all visitors, so declaration order does not matter.