@spendgraph/prompt 0.2.0

package/README.md

@@ -0,0 +1,240 @@
+# @spendgraph/prompt
+
+Pull a stored prompt or write one in code, render it, and record what it cost.
+Talks to spendgraph through `@spendgraph/sdk` and to nothing else.
+
+```sh
+npm install @spendgraph/prompt
+```
+
+## Two ways in, one prompt
+
+`pullPrompt` fetches the wording from the server. `buildCustomPrompt` declares
+it in code. Both hand back the same `Prompt`, so `render`, `serialize` and
+`call` behave identically and nothing downstream has to know which it was
+given.
+
+```ts
+import { PromptClient, buildCustomPrompt } from "@spendgraph/prompt";
+
+const prompts = new PromptClient({
+  apiKey: process.env.SPENDGRAPH_API_KEY,
+  baseUrl: "https://your-spendgraph.example.com",
+});
+
+const stored = await prompts.pull("billing-explainer");
+const local = buildCustomPrompt({
+  name: "billing-explainer",
+  blocks: [{ title: "Role", body: "You explain invoices to {audience}." }],
+  question: "Explain this line item in {max_words} words.",
+});
+```
+
+`pull` takes an id or a slug — the server resolves either. `pullPrompt(handle,
+opts)` is the same thing without keeping a client around; it builds one per
+call, so a service pulling repeatedly wants the class, which caches.
+
+## Running one
+
+Bind a model and the prompt runs itself:
+
+```ts
+import { Llm } from "@spendgraph/llms";
+
+const llm = new Llm({ client: anthropic, model: "claude-opus-5" });
+const chain = stored.bind(llm);
+
+const answer = await chain.invoke({ audience: "students", max_words: 50 });
+answer.output;
+answer.rolloutId;
+
+await chain.stream(values, { onText: (delta) => res.write(delta) });
+const answers = await chain.batch(cases, { concurrency: 4 });
+```
+
+`bind` asks for a shape, not an import — anything with `call(messages)` returning
+an outcome satisfies `Model`, and `Llm`'s reply already is one, so neither
+package depends on the other.
+
+### Or keep the call yourself
+
+`call` renders, runs the model call you give it, and records what happened
+around it. This package never sees the provider response, which is why the call
+is handed to it rather than made by it.
+
+```ts
+const reply = await stored.call({ audience: "students", max_words: 50 }, async ({ messages }) => {
+  const res = await anthropic.messages.create({ model, max_tokens: 400, messages });
+  return { output: res.content[0].text, model, inputTokens: 400, outputTokens: 12 };
+});
+```
+
+It fills in the version, fields, messages, latency and status; you return the
+text, model and tokens. Return the tokens or every cost on the prompt reads as
+zero — the first call that forgets warns once.
+
+Recording is on for every prompt and is fire-and-forget. There is no
+client-level switch: pass `{ report: false }` on the call that should not be
+recorded, which is where you know whether it should.
+
+What was sent is built once and shared by the success path and the failure path,
+so the two cannot disagree about it. That is the thing hand-written reporting
+gets wrong. A callback that throws is recorded as failed and then rethrown: a
+provider call that throws is otherwise recorded nowhere at all, which leaves the
+run most in need of explaining with no evidence.
+
+### What each kind records
+
+A **server** prompt has a row and a version behind it, so a call writes a
+rollout. A **custom** prompt has neither, and `POST /api/v1/prompts/:id/rollouts`
+needs both — so it records usage instead. The tokens still land on the
+dashboard, attributed to the model and tagged with the prompt's name; only the
+wording goes unversioned. Read `prompt.source` to tell them apart.
+
+A custom prompt is keyed on its name rather than a generated id, because that
+name is what tags the usage event — a random one would file every run of the
+same prompt under a different heading.
+
+Leave `sdk` off `buildCustomPrompt` and the prompt touches the network never.
+`{ report: false }` on any call runs the model and records nothing.
+
+## Conversation history
+
+Pass prior turns and they render between the system turn and the question,
+oldest first.
+
+```ts
+const history = [
+  { role: "user", content: "What is a proration?" },
+  { role: "assistant", content: "A mid-cycle adjustment." },
+];
+
+prompt.format(values, { history });
+await prompt.call(values, ({ messages }) => call(messages), { history });
+```
+
+History is passed through **verbatim** — it is a record of what was already
+said, not a template, so `{placeholders}` inside it are left alone. Only the
+prompt's own blocks and question are substituted. A `system` turn in the history
+is dropped: the prompt owns that turn, and two of them is a different request
+from the one the prompt describes.
+
+Tools are still selected on the question, not on an older turn — the shortlist
+has to serve what is being asked now.
+
+A rollout records at most 50 messages, which is what the route accepts. A longer
+conversation still runs; only its rollout is refused, and that refusal reaches
+`onReportError` rather than your call — so the first time it happens this warns,
+once, while somebody is still watching. `MAX_RECORDED_MESSAGES` is exported if
+you want to trim before it bites.
+
+## Fields
+
+A `{placeholder}` is declared as a `FieldSpec` — a name, a type, whether it is
+required, and how it renders. A missing required field throws before anything is
+sent. Types come from `@spendgraph/sdk`, because a tool's arguments and a graph
+node's inputs are the same problem.
+
+```ts
+prompt.format(values);     // Message[], validated and substituted
+prompt.serialize(values);  // the field map that render used
+```
+
+## Caching, and invalidating it
+
+`pull` serves a stale copy immediately and refreshes behind the caller, so an
+edit takes effect within a TTL without any request paying for the fetch.
+
+`promote` invalidates for you. Doing it by hand needs to drop **both** handles,
+not just the one passed: `pull` keys on whatever string it was given, so the
+same prompt sits under its uuid and its slug at once, and `promote` is only
+reachable with a uuid while the dashboard teaches pulling by slug. Dropping one
+left the other warm for a full TTL, still rendering the wording the promotion
+replaced.
+
+The delete runs even when nothing is stored under the handle: a cold isolate has
+a fetch in flight and no entry yet, and the delete is what stops that in-flight
+result being stored after the promotion.
+
+```ts
+await prompts.promote(promptId, versionId);  // invalidates
+prompts.invalidate(promptId);                // both handles
+prompts.invalidate();                        // everything
+```
+
+`onPullError` is worth setting. Without it a background refresh that has stopped
+resolving is silent, and a rename on the dashboard freezes a running service
+with nothing anywhere saying so.
+
+## Versions and datasets
+
+```ts
+await prompts.versions(promptId, { origin: "user", limit: 20 });
+await prompts.cases(promptId);
+await prompts.setCases(promptId, cases);
+```
+
+`setCases` replaces the whole dataset. A dataset is a set — "which cases am I
+evaluating against" has one answer at a time, and a merge would leave no way to
+remove a case.
+
+## Running server-side
+
+`run`, `sample` and `runAll` execute the prompt on the server, where the spec,
+rendering and pricing already live. This is the batch path. A production request
+should pull, render and report instead: the server-side call adds a hop and makes
+spendgraph a dependency of your uptime, which is the right trade for evaluation
+and the wrong one for a user waiting on a response.
+
+```ts
+await prompts.run(promptId, values);
+await prompts.sample(promptId, values, { k: 5 });
+await prompts.runAll(promptId, values);      // one rollout per saved model
+```
+
+`maxCostMicros` caps what those spend. Omit it and the client warns once that it
+is unbounded; pass `null` to say you meant it.
+
+## Tools
+
+`call` and `invoke` accept anything that hands out a turn — `toolbus()` from
+`@spendgraph/tools`, or your own. The callback gets the turn fully typed; the
+rollout gets the tools offered and the steps taken.
+
+```ts
+await prompt.call(values, ({ messages, turn }) => call(messages, turn), { tools: bus });
+```
+
+Which tools were offered is recorded as well as which were called: without it a
+rollout cannot tell "chose not to" from "was never offered", which are opposite
+bugs.
+
+## What is exported
+
+The main entry is six values, because everything else is already wired into
+`PromptClient`:
+
+```ts
+import {
+  PromptClient,          // pull, run, report, versions, datasets
+  pullPrompt,            // one prompt, without keeping a client
+  buildCustomPrompt,     // a prompt written in code
+  BudgetExceededError,
+  FieldValidationError,
+  MAX_RECORDED_MESSAGES,
+} from "@spendgraph/prompt";
+```
+
+Everything a prompt can do hangs off the object itself — `format`, `serialize`,
+`call`, `bind` — so there is nothing else to import to use one.
+
+Two subpaths hold the rest:
+
+| | |
+| --- | --- |
+| `@spendgraph/prompt/render` | `renderMessages` · `compileSystemPrompt` · `applyVariables` · `findVariables` |
+| `@spendgraph/prompt/internals` | `PullCache` · `Budget` · `makePrompt` · `bind` · `mapLimit` · … |
+
+Reach for `internals` only to build your own client; `PromptClient` already
+wires all of it.
+

package/dist/budget/budget.d.ts

@@ -0,0 +1,31 @@
+/**
+ * What this client has spent, and whether it may spend more.
+ *
+ * A `sample` loop over a dataset is the shape of an expensive accident, and the
+ * guard is a counter and a comparison.
+ *
+ * Governs what this client initiates — `run`, `sample`, `runAll`. Not `report`,
+ * which records money already spent elsewhere.
+ */
+export declare class Budget {
+    private used;
+    private readonly limit;
+    /**
+     * `undefined` means no ceiling and warns once — an unbounded client in a
+     * scheduled script is how a surprise bill happens. `null` means the same
+     * thing deliberately, and says nothing.
+     */
+    constructor(maxCostMicros?: number | null);
+    spent(): number;
+    remaining(): number;
+    add(costMicros: number): void;
+    /**
+     * Throws if the ceiling is already reached.
+     *
+     * Checked before a request rather than after: stopping once the bill has been
+     * incurred is a report, not a limit.
+     */
+    assertAffordable(): void;
+}
+/** Test seam: lets the once-only warning be exercised more than once. */
+export declare function resetBudgetWarning(): void;

package/dist/budget/budget.js

@@ -0,0 +1,53 @@
+import { BudgetExceededError } from "./errors.js";
+/** Warned once per process, not per client — a library that nags gets muted. */
+let warned = false;
+/**
+ * What this client has spent, and whether it may spend more.
+ *
+ * A `sample` loop over a dataset is the shape of an expensive accident, and the
+ * guard is a counter and a comparison.
+ *
+ * Governs what this client initiates — `run`, `sample`, `runAll`. Not `report`,
+ * which records money already spent elsewhere.
+ */
+export class Budget {
+    used = 0;
+    limit;
+    /**
+     * `undefined` means no ceiling and warns once — an unbounded client in a
+     * scheduled script is how a surprise bill happens. `null` means the same
+     * thing deliberately, and says nothing.
+     */
+    constructor(maxCostMicros) {
+        this.limit = maxCostMicros ?? Infinity;
+        if (maxCostMicros === undefined && !warned) {
+            warned = true;
+            console.warn("[prompt] No maxCostMicros set — this client can spend without limit. " +
+                "Pass maxCostMicros, or null to say you meant it.");
+        }
+    }
+    spent() {
+        return this.used;
+    }
+    remaining() {
+        return this.limit === Infinity ? Infinity : Math.max(0, this.limit - this.used);
+    }
+    add(costMicros) {
+        if (Number.isFinite(costMicros) && costMicros > 0)
+            this.used += costMicros;
+    }
+    /**
+     * Throws if the ceiling is already reached.
+     *
+     * Checked before a request rather than after: stopping once the bill has been
+     * incurred is a report, not a limit.
+     */
+    assertAffordable() {
+        if (this.used >= this.limit)
+            throw new BudgetExceededError(this.used, this.limit);
+    }
+}
+/** Test seam: lets the once-only warning be exercised more than once. */
+export function resetBudgetWarning() {
+    warned = false;
+}

package/dist/budget/errors.d.ts

@@ -0,0 +1,12 @@
+/**
+ * Thrown when a client has spent its ceiling.
+ *
+ * Deliberately not a `SpendgraphError`: nothing was refused by the server, so a
+ * caller inspecting `status` or `retryable` would be told a story about HTTP
+ * that never happened.
+ */
+export declare class BudgetExceededError extends Error {
+    readonly spentMicros: number;
+    readonly limitMicros: number;
+    constructor(spentMicros: number, limitMicros: number);
+}

package/dist/budget/errors.js

@@ -0,0 +1,18 @@
+/**
+ * Thrown when a client has spent its ceiling.
+ *
+ * Deliberately not a `SpendgraphError`: nothing was refused by the server, so a
+ * caller inspecting `status` or `retryable` would be told a story about HTTP
+ * that never happened.
+ */
+export class BudgetExceededError extends Error {
+    spentMicros;
+    limitMicros;
+    constructor(spentMicros, limitMicros) {
+        super(`Cost ceiling reached: spent $${(spentMicros / 1e6).toFixed(4)} of ` +
+            `$${(limitMicros / 1e6).toFixed(4)}. Raise maxCostMicros or start a new client.`);
+        this.name = "BudgetExceededError";
+        this.spentMicros = spentMicros;
+        this.limitMicros = limitMicros;
+    }
+}

package/dist/budget/index.d.ts

@@ -0,0 +1,2 @@
+export { Budget, resetBudgetWarning } from "./budget.js";
+export { BudgetExceededError } from "./errors.js";

package/dist/budget/index.js

@@ -0,0 +1,2 @@
+export { Budget, resetBudgetWarning } from "./budget.js";
+export { BudgetExceededError } from "./errors.js";

package/dist/build/build.d.ts

@@ -0,0 +1,19 @@
+import type { Spendgraph } from "@spendgraph/sdk";
+import type { Prompt, PromptSpec, ReportInput } from "../types/index.js";
+export interface BuildOptions {
+    /** Where usage goes. Omit and the prompt is entirely offline. */
+    sdk?: Spendgraph;
+    /** Attached to every usage event this prompt records. */
+    metadata?: Record<string, string | number | boolean>;
+    /** Replaces the once-only warning about a callback that returned no tokens. */
+    onEmptyTokens?: (entry: ReportInput) => void;
+}
+/**
+ * A prompt written in code rather than pulled from the server.
+ *
+ * The same `Prompt` `pullPrompt` returns — same `render`, same `serialize`,
+ * same `trace` — so nothing downstream has to know which one it was handed.
+ * Give it an `sdk` and it records what the call cost; leave it off and it
+ * touches the network never.
+ */
+export declare function buildCustomPrompt(spec: PromptSpec, opts?: BuildOptions): Prompt;

package/dist/build/build.js

@@ -0,0 +1,73 @@
+import { makePrompt, noRecorder } from "../prompt/index.js";
+import { applyVariables } from "../render/index.js";
+function neverBreakTheCallItMeasured() { }
+let warnedAboutTokens = false;
+function warnOnEmptyTokens(entry) {
+    if (warnedAboutTokens)
+        return;
+    if ((entry.inputTokens ?? 0) > 0 || (entry.outputTokens ?? 0) > 0)
+        return;
+    warnedAboutTokens = true;
+    console.warn("[prompt] trace() recorded a call with no tokens. This package never sees " +
+        "the provider response, so return inputTokens and outputTokens from your callback " +
+        "or every cost on this prompt reads as zero.");
+}
+/**
+ * Usage, not a rollout.
+ *
+ * A custom prompt has no row on the server and so no version to reference, and
+ * `POST /api/v1/prompts/:id/rollouts` needs both. The tokens are still real, so
+ * they go to ingest — the spend lands on the dashboard, attributed to the model
+ * and tagged with the prompt's name; only the wording goes unversioned.
+ */
+function usageRecorder(sdk, name, opts) {
+    const onEmpty = opts.onEmptyTokens ?? warnOnEmptyTokens;
+    return {
+        enabled: true,
+        write(_promptId, entry) {
+            if (entry.status === "failed")
+                return;
+            onEmpty(entry);
+            void sdk.ingest
+                .send([
+                {
+                    eventId: entry.rolloutId,
+                    model: entry.model,
+                    inputTokens: entry.inputTokens ?? 0,
+                    outputTokens: entry.outputTokens ?? 0,
+                    cacheReadTokens: entry.cacheReadTokens,
+                    cacheWriteTokens: entry.cacheWriteTokens,
+                    citationTokens: entry.citationTokens,
+                    reasoningTokens: entry.reasoningTokens,
+                    metadata: { ...opts.metadata, prompt: name },
+                },
+            ])
+                .catch(neverBreakTheCallItMeasured);
+        },
+    };
+}
+/**
+ * A prompt written in code rather than pulled from the server.
+ *
+ * The same `Prompt` `pullPrompt` returns — same `render`, same `serialize`,
+ * same `trace` — so nothing downstream has to know which one it was handed.
+ * Give it an `sdk` and it records what the call cost; leave it off and it
+ * touches the network never.
+ */
+export function buildCustomPrompt(spec, opts = {}) {
+    const variables = spec.variables ?? {};
+    const blocks = (spec.blocks ?? []).map((b) => ({
+        title: applyVariables(b.title, variables),
+        body: applyVariables(b.body, variables),
+    }));
+    return makePrompt({
+        id: spec.name,
+        name: spec.name,
+        slug: null,
+        blocks,
+        question: applyVariables(spec.question ?? "", variables),
+        fields: spec.fields ?? [],
+        versionId: null,
+        models: spec.models ?? [],
+    }, "custom", opts.sdk ? usageRecorder(opts.sdk, spec.name, opts) : noRecorder);
+}

package/dist/build/index.d.ts

@@ -0,0 +1,2 @@
+export type { BuildOptions } from "./build.js";
+export { buildCustomPrompt } from "./build.js";

package/dist/build/index.js

@@ -0,0 +1 @@
+export { buildCustomPrompt } from "./build.js";

package/dist/cache/cache.d.ts

@@ -0,0 +1,118 @@
+/**
+ * LRU with a TTL, and staleness reported rather than enforced.
+ *
+ * Returning a stale entry instead of dropping it is what lets `pull` serve
+ * immediately and refresh behind the request. Evicting on expiry puts a round
+ * trip on whichever request arrives a millisecond after the TTL lapses.
+ */
+export interface CacheOptions<T = unknown> {
+    /** Entries older than this are stale, not gone. Default 300 (5 minutes). */
+    ttlSeconds?: number;
+    /** Default 100. */
+    maxSize?: number;
+    /** Injectable clock, so tests do not sleep. */
+    now?: () => number;
+    /**
+     * How long to leave a key alone after its refresh failed. Default 60.
+     *
+     * A handle that has stopped resolving is served from cache and retried on a
+     * schedule, rather than on every call.
+     */
+    retryAfterSeconds?: number;
+    /**
+     * Called when a background refresh throws.
+     *
+     * `report` has `onReportError` because silence there loses spend data; the
+     * same argument applies here, where silence means serving a wording that can
+     * never update. Without it a rename on the dashboard freezes a running
+     * service with nothing anywhere saying so.
+     */
+    onRefreshError?: (err: unknown, key: string) => void;
+    /**
+     * Every name a cached value answers to — for a prompt, its id and its slug.
+     *
+     * The cache is keyed on whatever handle the caller passed, which is why
+     * invalidation kept missing: a promotion knows the uuid, production pulled by
+     * the slug, and a fetch still in flight carries no payload to connect them.
+     * With this, an invalidation is recorded against the prompt rather than
+     * against one string, and a fetch is dropped when the thing it fetched was
+     * invalidated — not when some unrelated key was.
+     */
+    identify?: (value: T) => string[];
+}
+export interface CacheHit<T> {
+    value: T;
+    stale: boolean;
+    ageMs: number;
+}
+export declare class PullCache<T> {
+    private readonly store;
+    private readonly ttlMs;
+    private readonly maxSize;
+    private readonly now;
+    /** Keys with a refresh in flight, so N concurrent readers cause one fetch. */
+    private readonly refreshing;
+    /** When a key's refresh last failed, so a dead handle is not retried per call. */
+    private readonly failures;
+    private readonly retryAfterMs;
+    private readonly onRefreshError?;
+    private readonly identify?;
+    /**
+     * A monotonic counter and the tick at which each name was last invalidated.
+     *
+     * A fetch records the counter when it starts and, when it lands, asks whether
+     * anything it turned out to be — its key, its id, its slug — has been
+     * invalidated since. That is the whole protocol, and it holds for the cold
+     * path, the background refresh, and a handle nobody has seen before, none of
+     * which the previous per-key epoch could cover.
+     */
+    private tick;
+    private readonly invalidatedAt;
+    /**
+     * The tick of the last `clear()`.
+     *
+     * `clear` cannot name a fetch it has never seen — a cold one is in neither
+     * `store` nor `refreshing` — so it raises a floor instead: anything that
+     * started before it is out of date whatever it turns out to be.
+     */
+    private clearedAt;
+    /** Fetches outstanding, so the log above can be dropped when none remain. */
+    private outstanding;
+    constructor(opts?: CacheOptions<T>);
+    get(key: string): CacheHit<T> | undefined;
+    set(key: string, value: T): void;
+    /**
+     * Runs `refresh` unless one is already in flight for this key.
+     *
+     * Without the guard, a burst of requests arriving just after the TTL lapses
+     * each start their own refresh — the stampede the cache exists to prevent.
+     * Failures are swallowed on purpose: a background refresh that throws must not
+     * surface in a request that was already answered from cache.
+     */
+    refreshInBackground(key: string, refresh: () => Promise<T>): void;
+    /** Records a fetch starting, and the tick it must be judged against. */
+    beginFetch(): number;
+    /** Stores a landed fetch unless what it fetched was invalidated meanwhile. */
+    settle(key: string, value: T, startedAt: number): void;
+    /** Ends a fetch that threw, and backs the handle off. */
+    abandon(key: string): void;
+    /** Records that everything under this name is out of date. */
+    invalidate(name: string): void;
+    /** The log only matters while a fetch could still be judged against it. */
+    private forgetLogIfIdle;
+    /** Keeps `failures` inside the same bound as the entries themselves. */
+    private boundFailures;
+    /** Every key whose entry satisfies `match`. */
+    keysWhere(match: (value: T) => boolean): string[];
+    delete(key: string): void;
+    clear(): void;
+    get size(): number;
+    /**
+     * How many keys the cache holds bookkeeping for, entries aside.
+     *
+     * Exposed for the test that this stays bounded: `epochs` and `failures` are
+     * private and grow on paths `size` cannot see, so a test written against
+     * `size` passes whether or not they leak — which is what the first one did.
+     */
+    get trackedKeys(): number;
+}