@pikku/inspector 0.12.56 → 0.12.58

package/CHANGELOG.md

@@ -1,3 +1,133 @@
+## 0.12.58
+
+### Patch Changes
+
+- 3ad2131: Name models by what they are for, and switch them all in one place
+
+  A `models` table in pikku.config.json maps an alias to a provider-qualified
+  model, so a declaration can say `model: 'cheap'` and the project repoints every
+  use of that tier at once instead of editing each agent. A model containing `/`
+  is still concrete and used exactly as written, which is how an agent that needs
+  one specific model pins it — aliases are opt-in.
+
+  The table is baked into codegen rather than read at runtime, so it applies to
+  deployed units and not just local runs, and `pikku dev`/`pikku serve` take
+  `--model cheap:openai/gpt-5-nano` to repoint a tier for one run without editing
+  the config.
+
+  Because the inspector already holds every agent's model literal, a bare name
+  with no matching alias now fails the build (PKU146) naming the aliases that do
+  exist, rather than reaching a provider as an unknown model.
+
+  Aliases resolve for every modality, not just agents: image, speech,
+  transcription, embedding and reranking all reach a provider through the same
+  point in the Vercel runner.
+
+- b930dca: Remove the `secretBroker` escape hatch and scope addon secrets and credentials
+
+  `secretBroker` let three named console functions receive the real `SecretService`,
+  against the rule that a function never sees one. It is gone: the inspector allowlist,
+  the `FunctionRuntimeMeta` flag, the runner branches, and the `WiredSecretBrokerServices`
+  type. Console secret administration moved into the console addon, where a
+  `SecretAdminService` holds the `SecretService` and the functions hold none.
+
+  Addons are now scoped rather than trusted. The CLI emits each package's declared secret
+  keys, and the host wraps the `SecretService` in a `ScopedSecretService` and the
+  `CredentialService` in a new `ScopedCredentialService` before the addon's service factory
+  runs — so an addon reads only what it declared, cannot write secrets, and cannot enumerate
+  the app's users. `wireAddon({ globalSecrets, globalCredentials })` waives this, taking the
+  reason as its value; only the consuming app can grant it, and the deploy manifest reports
+  every grant under `unscopedSecretAddons` / `unscopedCredentialAddons`.
+
+- 8978fbd: feat(workflow): let an approval gate declare who may answer it
+
+  `workflow.approval()` gains `approvers` (`'any' | 'owner' | 'not-initiator'`)
+  and `approverScope`, so a gate can require four-eyes sign-off, restrict itself
+  to the run's initiator, or require the decider to hold a named scope.
+
+  Both are enforced when the workflow replays the gate — the same place, and for
+  the same reason, the decision payload is validated: the policy is a value on
+  the workflow, and a decision can be recorded before the run has ever reached
+  the gate. A decision that fails the policy is discarded and the gate stays
+  closed. Where the run has already published its policy, the check also runs at
+  submission time so the caller gets a 403 rather than silence.
+
+  An answer is now recorded where it can be answered for later. The settled
+  decision carries `decidedBy` and `decidedAt` in its `ApprovalOutcome`, so who
+  signed reaches `workflowStep.result` and `workflowStepHistory` rather than
+  living only in mutable run state. Every answer — accepted, refused at the door,
+  or cleared on replay — is also written to the audit sink as
+  `workflow.approval.decided`, which outlives the run: `deleteRun` cascades to
+  steps and history, and a refused attempt never reaches a step at all. Projects
+  with no audit service wired are unaffected.
+
+  **This loosens the default.** `approveStep` previously refused anyone but the
+  run's initiator, unconditionally. A gate that declares no `approvers` now
+  accepts a decision from anyone the approve entrypoint admits — restore the old
+  behaviour per-gate with `approvers: 'owner'`, or gate the approve route with
+  `auth`/`permissions`. Ownership still governs _reads_ of a run unchanged.
+
+- Updated dependencies [063f43a]
+- Updated dependencies [ce66bf8]
+- Updated dependencies [d0307a8]
+- Updated dependencies [ce66bf8]
+- Updated dependencies [3ad2131]
+- Updated dependencies [b930dca]
+- Updated dependencies [b95e77d]
+- Updated dependencies [fd9d834]
+- Updated dependencies [8978fbd]
+  - @pikku/core@0.12.82
+
+## 0.12.57
+
+### Patch Changes
+
+- e110c55: Wire `pikkuAIScorer` / `pikkuAIJudge` through the inspector and codegen, and let an agent name the scorers that grade it.
+
+  The inspector reads a scorer's lane off which constructor was called rather than
+  off a field, so the two lanes cannot disagree with the code that produced them,
+  and refuses a scorer with no name or description — the meta is the only thing
+  that names it at runtime. An agent naming a scorer that was never declared is a
+  build error (`PKU155`) rather than an agent that quietly grades nothing forever.
+
+  Codegen emits a `ScorerName` union, so `scorers` on an agent is checked against
+  the scorers the project actually declares, plus the scorer wirings and meta.
+  `pikku validate` now also flags a scorer declared outside a `*.scorer.ts` file,
+  for the same reason scenarios have to live in files named for them: a rubric
+  buried in an agent definition is one nobody reviews as a rubric.
+
+- e110c55: Add `scenario.expectScore` — grade a finished agent run with a declared scorer and assert on it.
+
+  An agent's answer cannot be matched against a fixed string, so a scenario grades
+  it instead. `expectScore(step, runId, scorer, { atLeast, atMost, reference })`
+  runs one declared scorer against the run the scenario just triggered and fails
+  with the reason the judge gave. The default bound is `atLeast: 0.5`, so an
+  unqualified assertion still fails a run graded zero.
+
+  Grading goes over the new `pikkuScenarioGradeRun` instrumentation RPC, which the
+  dev server registers alongside the coverage and stub RPCs — so it exists only in
+  processes that should have it, and never in a deployed bundle. It grades from
+  the snapshot the runtime already took when the run finished, which is what makes
+  a scenario's grade the same measurement production's sampler makes rather than
+  an approximation of it: a run's prompt, answer and tool calls are spread across
+  a thread's messages, where the boundary of one run is not recoverable.
+
+  Two things differ deliberately from live scoring. The sample rate is ignored — a
+  scorer grading 1% of traffic still grades every scenario run — and the grade is
+  returned rather than recorded, so a test's score never lands among the
+  production figures. `reference` supplies the answer key a `requiresReference`
+  judge grades against, which is the only way such a judge is reachable at all.
+
+- Updated dependencies [e110c55]
+- Updated dependencies [e110c55]
+- Updated dependencies [e110c55]
+- Updated dependencies [acc8077]
+- Updated dependencies [905f737]
+- Updated dependencies [3cc6428]
+- Updated dependencies [c524adf]
+- Updated dependencies [e110c55]
+  - @pikku/core@0.12.81
+
 ## 0.12.56
 
 ### Patch Changes

package/dist/add/add-ai-agent.js

@@ -217,6 +217,29 @@ function resolveIdentifierToAgentName(identifier, checker) {
     }
     return null;
 }
+/**
+ * The scorers an agent asks to be graded by, which are plain names rather than
+ * references: a scorer is resolved out of the registry at grading time, not
+ * imported by the agent, so nothing here has a declaration to chase.
+ */
+function resolveScorerNames(obj, agentName, logger) {
+    const property = obj.properties.find((p) => ts.isPropertyAssignment(p) &&
+        ts.isIdentifier(p.name) &&
+        p.name.text === 'scorers');
+    if (!property || !ts.isPropertyAssignment(property))
+        return null;
+    if (!ts.isArrayLiteralExpression(property.initializer))
+        return null;
+    const names = [];
+    for (const element of property.initializer.elements) {
+        if (!ts.isStringLiteral(element)) {
+            logger.critical(ErrorCode.INVALID_VALUE, `AI agent '${agentName}' lists a scorer that is not an inline string: \`${element.getText()}\`. A scorer is named, not imported, so the name has to be readable from source.`);
+            continue;
+        }
+        names.push(element.text);
+    }
+    return names.length > 0 ? names : null;
+}
 export const addAIAgent = (logger, node, checker, state, options) => {
     if (!ts.isCallExpression(node)) {
         return;
@@ -244,6 +267,7 @@ export const addAIAgent = (logger, node, checker, state, options) => {
         const maxStepsValue = getPropertyValue(obj, 'maxSteps');
         const temperatureValue = getPropertyValue(obj, 'temperature');
         const toolChoiceValue = getPropertyValue(obj, 'toolChoice');
+        const scorersValue = resolveScorerNames(obj, nameValue || '', logger);
         const toolsValue = resolveToolReferences(obj, checker, nameValue || '', logger);
         if (toolsValue) {
             for (const toolName of toolsValue) {
@@ -389,6 +413,7 @@ export const addAIAgent = (logger, node, checker, state, options) => {
             ...(toolsValue !== null && { tools: toolsValue }),
             ...(agentsValue !== null && { agents: agentsValue }),
             ...(workflowsValue !== null && { workflows: workflowsValue }),
+            ...(scorersValue !== null && { scorers: scorersValue }),
             tags,
             inputSchema,
             outputSchema,

package/dist/add/add-ai-scorer.d.ts

@@ -0,0 +1,2 @@
+import type { AddWiring } from '../types.js';
+export declare const addAIScorer: AddWiring;

package/dist/add/add-ai-scorer.js

@@ -0,0 +1,61 @@
+import * as ts from 'typescript';
+import { getPropertyValue } from '../utils/get-property-value.js';
+import { extractFunctionName } from '../utils/extract-function-name.js';
+import { ErrorCode } from '../error-codes.js';
+/**
+ * `pikkuAIScorer` grades in code and `pikkuAIJudge` grades with a model, which
+ * is the whole difference between the two lanes — so the lane is read off which
+ * constructor was called rather than from a field an author could contradict.
+ */
+const LANES = {
+    pikkuAIScorer: 'fast',
+    pikkuAIJudge: 'slow',
+};
+export const addAIScorer = (logger, node, checker, state) => {
+    if (!ts.isCallExpression(node))
+        return;
+    const expression = node.expression;
+    if (!ts.isIdentifier(expression))
+        return;
+    const lane = LANES[expression.text];
+    if (!lane)
+        return;
+    const firstArg = node.arguments[0];
+    if (!firstArg || !ts.isObjectLiteralExpression(firstArg))
+        return;
+    const obj = firstArg;
+    const { exportedName } = extractFunctionName(node, checker, state.rootDir);
+    const nameValue = getPropertyValue(obj, 'name');
+    if (!nameValue) {
+        logger.critical(ErrorCode.MISSING_NAME, "AI scorer is missing the required 'name' property.");
+        return;
+    }
+    const description = getPropertyValue(obj, 'description');
+    if (!description) {
+        logger.critical(ErrorCode.MISSING_DESCRIPTION, `AI scorer '${nameValue}' is missing the required 'description' property.`);
+        return;
+    }
+    const sampleRateValue = getPropertyValue(obj, 'sampleRate');
+    if (sampleRateValue !== null &&
+        (sampleRateValue < 0 || sampleRateValue > 1)) {
+        logger.critical(ErrorCode.INVALID_VALUE, `AI scorer '${nameValue}' has a sampleRate of ${sampleRateValue} — it is the fraction of live runs to grade, so it must be between 0 and 1.`);
+        return;
+    }
+    const requiresReference = getPropertyValue(obj, 'requiresReference') ?? false;
+    const scorerKey = exportedName || nameValue;
+    if (exportedName) {
+        state.scorers.files.set(scorerKey, {
+            path: node.getSourceFile().fileName,
+            exportedName,
+        });
+    }
+    state.scorers.scorersMeta[scorerKey] = {
+        name: nameValue,
+        description,
+        lane,
+        sampleRate: sampleRateValue ?? 1,
+        requiresReference,
+        sourceFile: node.getSourceFile().fileName,
+        exportedName: exportedName || undefined,
+    };
+};

package/dist/add/add-wire-addon.js

@@ -53,6 +53,8 @@ export function addWireAddon(node, state, logger) {
     let secretOverrides;
     let variableOverrides;
     let credentialOverrides;
+    let globalSecrets;
+    let globalCredentials;
     for (const prop of firstArg.properties) {
         if (!ts.isPropertyAssignment(prop) || !ts.isIdentifier(prop.name))
             continue;
@@ -94,6 +96,18 @@ export function addWireAddon(node, state, logger) {
             ts.isObjectLiteralExpression(prop.initializer)) {
             credentialOverrides = parseStringRecord(prop.initializer);
         }
+        else if (key === 'globalSecrets') {
+            // The grant is real at runtime whatever the reason evaluates to, so a
+            // non-literal is recorded as its source text rather than dropped.
+            globalSecrets = ts.isStringLiteral(prop.initializer)
+                ? prop.initializer.text
+                : prop.initializer.getText();
+        }
+        else if (key === 'globalCredentials') {
+            globalCredentials = ts.isStringLiteral(prop.initializer)
+                ? prop.initializer.text
+                : prop.initializer.getText();
+        }
     }
     if (!name || !pkg)
         return;
@@ -108,6 +122,8 @@ export function addWireAddon(node, state, logger) {
         secretOverrides,
         variableOverrides,
         credentialOverrides,
+        globalSecrets,
+        globalCredentials,
     });
     state.rpc.usedAddons.add(name);
     state.rpc.wireAddonFiles.add(node.getSourceFile().fileName);

package/dist/add/scenario-instrumentation.d.ts

@@ -13,8 +13,13 @@
  * `pikku dev` registers the implementations itself. Ignoring by name (rather
  * than deleting the scaffold and requiring every project to regenerate) is what
  * keeps an existing project's checked-in `scenarios.gen.ts` from deploying.
+ *
+ * `pikkuScenarioGradeRun` was never scaffolded — it arrived after the split —
+ * but it belongs here for the same reason: grading a run on demand, ignoring the
+ * sample rate that governs live traffic, is a thing you do to a development
+ * server, and the name is reserved so an application cannot claim it.
  */
-export declare const SCENARIO_INSTRUMENTATION_FUNCTIONS: readonly ["pikkuScenarioTakeLiveCoverage", "pikkuScenarioResetLiveCoverage", "pikkuScenarioResetStubs", "pikkuScenarioGetStubCalls"];
+export declare const SCENARIO_INSTRUMENTATION_FUNCTIONS: readonly ["pikkuScenarioTakeLiveCoverage", "pikkuScenarioResetLiveCoverage", "pikkuScenarioResetStubs", "pikkuScenarioGetStubCalls", "pikkuScenarioGradeRun"];
 export type ScenarioInstrumentationFunction = (typeof SCENARIO_INSTRUMENTATION_FUNCTIONS)[number];
 /**
  * True for a scenario instrumentation function, by name. Versioned ids

package/dist/add/scenario-instrumentation.js

@@ -13,12 +13,18 @@
  * `pikku dev` registers the implementations itself. Ignoring by name (rather
  * than deleting the scaffold and requiring every project to regenerate) is what
  * keeps an existing project's checked-in `scenarios.gen.ts` from deploying.
+ *
+ * `pikkuScenarioGradeRun` was never scaffolded — it arrived after the split —
+ * but it belongs here for the same reason: grading a run on demand, ignoring the
+ * sample rate that governs live traffic, is a thing you do to a development
+ * server, and the name is reserved so an application cannot claim it.
  */
 export const SCENARIO_INSTRUMENTATION_FUNCTIONS = [
     'pikkuScenarioTakeLiveCoverage',
     'pikkuScenarioResetLiveCoverage',
     'pikkuScenarioResetStubs',
     'pikkuScenarioGetStubCalls',
+    'pikkuScenarioGradeRun',
 ];
 const names = new Set(SCENARIO_INSTRUMENTATION_FUNCTIONS);
 /**

package/dist/error-codes.d.ts

@@ -62,6 +62,7 @@ export declare enum ErrorCode {
     AGENT_TOOL_UNKNOWN_NAMESPACE = "PKU152",
     AGENT_TOOL_NOT_FOUND = "PKU153",
     AGENT_TOOL_MISSING_DESCRIPTION = "PKU154",
+    AGENT_SCORER_NOT_FOUND = "PKU155",
     SCHEMA_AND_WIRING_COLOCATED = "PKU490",
     SERVICES_NOT_DESTRUCTURED = "PKU410",
     WIRES_NOT_DESTRUCTURED = "PKU411",

package/dist/error-codes.js

@@ -79,6 +79,8 @@ export var ErrorCode;
     ErrorCode["AGENT_TOOL_UNKNOWN_NAMESPACE"] = "PKU152";
     ErrorCode["AGENT_TOOL_NOT_FOUND"] = "PKU153";
     ErrorCode["AGENT_TOOL_MISSING_DESCRIPTION"] = "PKU154";
+    // Agent scorer reference errors
+    ErrorCode["AGENT_SCORER_NOT_FOUND"] = "PKU155";
     // File structure errors
     ErrorCode["SCHEMA_AND_WIRING_COLOCATED"] = "PKU490";
     // Optimization diagnostics

package/dist/inspector.js

@@ -6,7 +6,7 @@ import { TypesMap } from './types-map.js';
 import { getFilesAndMethods } from './utils/get-files-and-methods.js';
 import { findCommonAncestor } from './utils/find-root-dir.js';
 import { createNestedProjectFilter } from './utils/nested-project-filter.js';
-import { aggregateRequiredServices, stampAuthHandlerServices, validateAgentModels, validateAgentToolReferences, validateSecretOverrides, validateVariableOverrides, validateCredentialOverrides, validateRemoteAddonDependencies, validateRemoteAddonAuth, validateScopeReferences, validateSystemRoleScopes, validatePersonaRoles, computeResolvedIOTypes, computeMiddlewareGroupsMeta, computePermissionsGroupsMeta, computeRequiredSchemas, computeSecretBrokers, validateNoSecretAliasServices, validateSecretUsage, computeDiagnostics, validateSchemaReferences, validateSchemaWiringSeparation, validateScenarioServices, validateScenarioSteps, validateWorkflowGraphAddons, } from './utils/post-process.js';
+import { aggregateRequiredServices, stampAuthHandlerServices, validateAgentModels, validateAgentToolReferences, validateAgentScorerReferences, validateSecretOverrides, validateVariableOverrides, validateCredentialOverrides, validateRemoteAddonDependencies, validateRemoteAddonAuth, validateScopeReferences, validateSystemRoleScopes, validatePersonaRoles, computeResolvedIOTypes, computeMiddlewareGroupsMeta, computePermissionsGroupsMeta, computeRequiredSchemas, validateNoSecretAliasServices, validateSecretUsage, computeDiagnostics, validateSchemaReferences, validateSchemaWiringSeparation, validateScenarioServices, validateScenarioSteps, validateWorkflowGraphAddons, } from './utils/post-process.js';
 import { validateExposedFunctionsGated } from './utils/validate-exposed-functions-gated.js';
 import { validateTagsResolveToMiddleware } from './utils/validate-tags-resolve-to-middleware.js';
 import { annotateHttpRouteAuth } from './utils/annotate-http-route-auth.js';
@@ -116,6 +116,10 @@ export function getInitialInspectorState(rootDir) {
             agentsMeta: {},
             files: new Map(),
         },
+        scorers: {
+            scorersMeta: {},
+            files: new Map(),
+        },
         cli: {
             meta: {
                 programs: {},
@@ -366,7 +370,6 @@ export const inspect = async (logger, routeFiles, options = {}) => {
         computeResolvedIOTypes(state);
         computeMiddlewareGroupsMeta(state);
         computePermissionsGroupsMeta(state);
-        computeSecretBrokers(state);
         validateNoSecretAliasServices(logger, checker, state);
         validateSecretUsage(logger, state);
         computeDiagnostics(state);
@@ -384,8 +387,9 @@ export const inspect = async (logger, routeFiles, options = {}) => {
         if (options.openAPI) {
             state.openAPISpec = await generateOpenAPISpec(logger, state.functions.meta, state.http.meta, state.schemas, options.openAPI.additionalInfo, pikkuState(null, 'misc', 'errors'));
         }
-        validateAgentModels(logger, state);
+        validateAgentModels(logger, state, options.modelAliases);
         validateAgentToolReferences(logger, state, options);
+        validateAgentScorerReferences(logger, state);
         validateSecretOverrides(logger, state);
         validateVariableOverrides(logger, state);
         validateCredentialOverrides(logger, state);

package/dist/types.d.ts

@@ -9,6 +9,7 @@ import type { QueueWorkersMeta } from '@pikku/core/queue';
 import type { WorkflowsMeta } from '@pikku/core/workflow';
 import type { MCPResourceMeta, MCPToolMeta, MCPPromptMeta } from '@pikku/core/mcp';
 import type { AIAgentMeta } from '@pikku/core/ai-agent';
+import type { ScorerMeta } from '@pikku/core/ai-scorer';
 import type { CLIMeta } from '@pikku/core/cli';
 import type { CLICommandMeta } from '@pikku/core/cli';
 import type { NodesMeta } from '@pikku/core/node';
@@ -229,6 +230,8 @@ export type InspectorOptions = Partial<{
      * tool's own name, so this is a quality gate rather than a correctness one.
      */
     strictMeta: boolean;
+    /** The `models` table from pikku.config.json, alias -> `provider/model`. */
+    modelAliases: Record<string, string>;
     sourceFile: ts.SourceFile;
     /**
      * The project's tsconfig, read for its path mappings only. The inspector's
@@ -518,6 +521,16 @@ export interface InspectorState {
             secretOverrides?: Record<string, string>;
             variableOverrides?: Record<string, string>;
             credentialOverrides?: Record<string, string>;
+            /**
+             * The app's stated reason for handing this instance the whole
+             * `SecretService` instead of one scoped to its declared secrets.
+             */
+            globalSecrets?: string;
+            /**
+             * The app's stated reason for handing this instance the whole
+             * `CredentialService` instead of one scoped to its declared credentials.
+             */
+            globalCredentials?: string;
         }>;
         wireAddonFiles: Set<string>;
     };
@@ -534,6 +547,13 @@ export interface InspectorState {
             exportedName: string;
         }>;
     };
+    scorers: {
+        scorersMeta: ScorerMeta;
+        files: Map<string, {
+            path: string;
+            exportedName: string;
+        }>;
+    };
     cli: {
         meta: CLIMeta;
         files: Set<string>;

package/dist/utils/filter-inspector-state.js

@@ -328,6 +328,14 @@ export function filterInspectorState(state, filters, logger) {
             agentsMeta: JSON.parse(JSON.stringify(state.agents?.agentsMeta ?? {})),
             files: new Map(),
         },
+        // Scorers carry through every filter. They are named by the agents that
+        // survive it, and a deployment that quietly stopped grading would look
+        // exactly like one whose scorers all returned nothing.
+        scorers: {
+            ...state.scorers,
+            scorersMeta: JSON.parse(JSON.stringify(state.scorers?.scorersMeta ?? {})),
+            files: new Map(state.scorers?.files ?? []),
+        },
         rpc: {
             ...state.rpc,
             internalMeta: { ...state.rpc.internalMeta }, // Clone to avoid mutating original

package/dist/utils/post-process.d.ts

@@ -65,7 +65,11 @@ export declare function computeRequiredSchemas(state: InspectorState, options: I
  * `--fail-on-error` without failing a dev server over a function nobody calls.
  */
 export declare function validateSchemaReferences(logger: InspectorLogger, state: InspectorState): void;
-export declare function validateAgentModels(logger: InspectorLogger, state: InspectorState | Omit<InspectorState, 'typesLookup'>): void;
+/**
+ * A model is either concrete (`provider/model`) or an alias, which must appear
+ * in the `models` table from pikku.config.json.
+ */
+export declare function validateAgentModels(logger: InspectorLogger, state: InspectorState | Omit<InspectorState, 'typesLookup'>, modelAliases?: Record<string, string>): void;
 /**
  * Scenarios are pure stories of remote RPCs (same rule as client-side CLI
  * renderers): the func may only destructure logger/config — everything else
@@ -111,8 +115,6 @@ export declare function validateWorkflowGraphAddons(logger: InspectorLogger, sta
  * because the pikku state metadata doesn't exist in the CLI context.
  */
 export declare function validateSchemaWiringSeparation(logger: InspectorLogger, state: InspectorState): void;
-/** Marks the functions that keep the full `SecretService`. */
-export declare function computeSecretBrokers(state: InspectorState): void;
 /**
  * Refuses a singleton service that is a `SecretService` under another name.
  *
@@ -156,6 +158,14 @@ export declare function validateScopeReferences(logger: InspectorLogger, state:
  * a title labels a tool in a UI, it does not tell a model when to reach for it.
  */
 export declare function validateAgentToolReferences(logger: InspectorLogger, state: InspectorState | Omit<InspectorState, 'typesLookup'>, options?: InspectorOptions): void;
+/**
+ * Validates that every scorer an agent asks to be graded by exists.
+ *
+ * The runtime warns and skips an unresolvable name rather than failing a run
+ * that has already answered the user — which means a typo would otherwise cost
+ * nothing visible and simply grade nothing, forever.
+ */
+export declare function validateAgentScorerReferences(logger: InspectorLogger, state: InspectorState | Omit<InspectorState, 'typesLookup'>): void;
 /**
  * Validates that every scope granted by a system role is declared via
  * `defineScope`. Runs after all visitors, so declaration order does not matter.

package/dist/utils/post-process.js

@@ -2,7 +2,6 @@ import { existsSync, readFileSync } from 'node:fs';
 import { join } from 'node:path';
 import { extractTypeKeys } from './type-utils.js';
 import { ErrorCode } from '../error-codes.js';
-import { isSecretBrokerFunction } from './secret-brokers.js';
 import { findSecretAliasServices } from './secret-alias-services.js';
 import { relative } from 'node:path';
 import { AUTH_HANDLER_FUNC_ID } from '../add/add-auth.js';
@@ -606,15 +605,26 @@ export function validateSchemaReferences(logger, state) {
             `is declared but not exported — export it, or inline the type at the call site.`,
     });
 }
-export function validateAgentModels(logger, state) {
+/**
+ * A model is either concrete (`provider/model`) or an alias, which must appear
+ * in the `models` table from pikku.config.json.
+ */
+export function validateAgentModels(logger, state, modelAliases = {}) {
     for (const [, meta] of Object.entries(state.agents.agentsMeta)) {
         const model = meta.model;
         if (!model) {
             logger.critical(ErrorCode.MISSING_MODEL, `AI agent '${meta.name}' is missing the 'model' property.`);
             continue;
         }
-        if (!model.includes('/')) {
-            logger.critical(ErrorCode.INVALID_MODEL, `AI agent '${meta.name}' uses model '${model}', which must be provider-qualified as '<provider>/<model>' (e.g. 'openai/gpt-4').`);
+        // `hasOwn`, not a truthy lookup: every object literal inherits `toString`,
+        // `constructor` and `__proto__`, so a bare model named after one of them
+        // would read as a configured alias and pass a check nothing else repeats.
+        if (!model.includes('/') && !Object.hasOwn(modelAliases, model)) {
+            const known = Object.keys(modelAliases).sort();
+            logger.critical(ErrorCode.INVALID_MODEL, `AI agent '${meta.name}' uses model '${model}', which is neither provider-qualified as '<provider>/<model>' (e.g. 'openai/gpt-4') nor an alias in the "models" table of pikku.config.json. ` +
+                (known.length
+                    ? `Known aliases: ${known.join(', ')}.`
+                    : `No "models" aliases are configured.`));
         }
     }
 }
@@ -793,14 +803,6 @@ export function validateSchemaWiringSeparation(logger, state) {
         }
     }
 }
-/** Marks the functions that keep the full `SecretService`. */
-export function computeSecretBrokers(state) {
-    for (const [id, meta] of Object.entries(state.functions.meta)) {
-        if (isSecretBrokerFunction(meta.pikkuFuncId ?? id)) {
-            meta.secretBroker = true;
-        }
-    }
-}
 /**
  * Refuses a singleton service that is a `SecretService` under another name.
  *
@@ -830,9 +832,6 @@ export function validateNoSecretAliasServices(logger, checker, state) {
             `SecretService is confined to pikkuServices, pikkuWireServices, addon service factories and middleware — give a service the secret value when you construct it and expose only what the function needs.`);
     };
     for (const [id, meta] of Object.entries(state.functions.meta)) {
-        if (meta.secretBroker) {
-            continue;
-        }
         report('Function', id, meta.services?.services ?? []);
     }
     for (const [id, def] of Object.entries(state.permissions.definitions)) {
@@ -1038,6 +1037,25 @@ export function validateAgentToolReferences(logger, state, options = {}) {
         }
     }
 }
+/**
+ * Validates that every scorer an agent asks to be graded by exists.
+ *
+ * The runtime warns and skips an unresolvable name rather than failing a run
+ * that has already answered the user — which means a typo would otherwise cost
+ * nothing visible and simply grade nothing, forever.
+ */
+export function validateAgentScorerReferences(logger, state) {
+    const known = Object.keys(state.scorers.scorersMeta);
+    for (const [agentKey, agent] of Object.entries(state.agents.agentsMeta)) {
+        const where = agent.sourceFile ? ` (${agent.sourceFile})` : '';
+        for (const scorer of agent.scorers ?? []) {
+            if (known.includes(scorer))
+                continue;
+            logger.critical(ErrorCode.AGENT_SCORER_NOT_FOUND, `AI agent '${agentKey}'${where} asks to be graded by scorer '${scorer}', which is not declared in this project. ` +
+                `Declared scorers: ${known.join(', ') || 'none'}.`);
+        }
+    }
+}
 /**
  * Validates that every scope granted by a system role is declared via
  * `defineScope`. Runs after all visitors, so declaration order does not matter.

package/dist/utils/serialize-inspector-state.d.ts

@@ -195,6 +195,8 @@ export interface SerializableInspectorState {
                 secretOverrides?: Record<string, string>;
                 variableOverrides?: Record<string, string>;
                 credentialOverrides?: Record<string, string>;
+                globalSecrets?: string;
+                globalCredentials?: string;
             }
         ]>;
         wireAddonFiles: string[];
@@ -212,6 +214,13 @@ export interface SerializableInspectorState {
             exportedName: string;
         }][];
     };
+    scorers: {
+        scorersMeta: InspectorState['scorers']['scorersMeta'];
+        files: [string, {
+            path: string;
+            exportedName: string;
+        }][];
+    };
     cli: {
         meta: InspectorState['cli']['meta'];
         files: string[];

package/dist/utils/serialize-inspector-state.js

@@ -95,6 +95,10 @@ export function serializeInspectorState(state) {
             agentsMeta: state.agents?.agentsMeta ?? {},
             files: Array.from(state.agents?.files?.entries() ?? []),
         },
+        scorers: {
+            scorersMeta: state.scorers?.scorersMeta ?? {},
+            files: Array.from(state.scorers?.files?.entries() ?? []),
+        },
         cli: {
             meta: state.cli.meta,
             files: Array.from(state.cli.files),
@@ -268,6 +272,10 @@ export function deserializeInspectorState(data) {
             agentsMeta: data.agents?.agentsMeta || {},
             files: new Map(data.agents?.files || []),
         },
+        scorers: {
+            scorersMeta: data.scorers?.scorersMeta || {},
+            files: new Map(data.scorers?.files || []),
+        },
         cli: {
             meta: data.cli.meta,
             files: new Set(data.cli.files),

package/dist/utils/workflow/dsl/extract-dsl-workflow.js

@@ -803,19 +803,28 @@ function extractApprovalStep(call, context, outputVar) {
             step.outputVar = outputVar;
         }
         // The `schema` option is a runtime value validated inside the workflow body,
-        // so it is deliberately not serialized here — only `expiry`, which the graph
-        // and planned-step ladder need in order to describe the gate.
+        // so it is deliberately not serialized here — only the literal options the
+        // graph and planned-step ladder need in order to describe the gate.
         const options = args[1];
         if (options && ts.isObjectLiteralExpression(options)) {
             for (const prop of options.properties) {
-                if (ts.isPropertyAssignment(prop) &&
-                    prop.name.getText() === 'expiry' &&
+                if (!ts.isPropertyAssignment(prop))
+                    continue;
+                const name = prop.name.getText();
+                if (name === 'expiry' &&
                     (ts.isStringLiteral(prop.initializer) ||
                         ts.isNumericLiteral(prop.initializer))) {
                     step.expiry = ts.isNumericLiteral(prop.initializer)
                         ? Number(prop.initializer.text)
                         : prop.initializer.text;
                 }
+                if (name === 'approvers' && ts.isStringLiteral(prop.initializer)) {
+                    step.approvers = prop.initializer
+                        .text;
+                }
+                if (name === 'approverScope' && ts.isStringLiteral(prop.initializer)) {
+                    step.approverScope = prop.initializer.text;
+                }
             }
         }
         return step;

package/dist/utils/workflow/dsl/patterns.d.ts

@@ -35,7 +35,8 @@ export declare function isScenarioStepCall(node: ts.CallExpression): boolean;
 export declare function getScenarioStepPhase(node: ts.CallExpression): ScenarioStepPhaseName | undefined;
 /**
  * Check if a call expression is one of the scenario expectation helpers —
- * `scenario.expectService()`, `.expectError()` or `.expectEventually()`.
+ * `scenario.expectService()`, `.expectError()`, `.expectEventually()` or
+ * `.expectScore()`.
  */
 export declare function isScenarioExpectationCall(node: ts.CallExpression): boolean;
 /**