@prompt-helm/sdk 1.4.0 → 1.6.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -16,6 +16,8 @@ refresh, 60 seconds by default.
16
16
  their values, your provider keys or your LLM traffic.
17
17
  - **Model settings travel with the text.** A version can carry its model, temperature and
18
18
  output limit, so moving to a cheaper model is a publish you can roll back or A/B test.
19
+ - **System and user messages.** A prompt can hold the user message too, with its own variables,
20
+ and ask for an answer in JSON that matches a schema.
19
21
  - **No dependencies.** ESM and CommonJS, Node.js 18 and later.
20
22
 
21
23
  ## Install
@@ -50,6 +52,30 @@ The stored prompt marks each variable in double braces, such as `{{customer_name
50
52
  version uses a variable the call doesn't pass, `get` doesn't send a half-filled prompt: it
51
53
  returns the fallback (or, without one, throws `missing_variables`).
52
54
 
55
+ ## System and user messages
56
+
57
+ A prompt's text is its system message. It can also hold the user message, for a request your app
58
+ sends the same way each time with different values. `resolve` fills both in:
59
+
60
+ ```ts
61
+ const r = await ph.resolve("classify-ticket", {
62
+ vars: { product: "Acme", ticket: ticket.text },
63
+ fallback: { system: CLASSIFY_TICKET, user: "Ticket: {{ticket}}" },
64
+ });
65
+
66
+ const reply = await openai.responses.create({
67
+ model: r.config.model ?? "gpt-5",
68
+ instructions: r.system,
69
+ input: r.user ?? ticket.text,
70
+ });
71
+ ```
72
+
73
+ `user` is `null` when the live version has none: your app sends its own, such as what the person
74
+ typed. `get` returns the system message only and never needs the user message's variables, so an
75
+ app that sends its own user message keeps working when one is added in the dashboard. If the user
76
+ message uses a variable the call doesn't pass, `resolve` returns the fallback, as it does for the
77
+ system message. SDK versions before 1.5 receive the system message only.
78
+
53
79
  ## Options
54
80
 
55
81
  All optional when the environment variables are set.
@@ -63,6 +89,7 @@ All optional when the environment variables are set.
63
89
  | `onError` | `console.warn` | Called with the error and the slug when a fallback or a cached prompt was served instead of failing. |
64
90
  | `reportFallbacks` | `true` | Tells PromptHelm, by error code only, when a fallback or a cached prompt was served, so the prompt's page shows which app fell back and why. |
65
91
  | `reportResults` | `reportFallbacks` | During an [A/B test](#ab-tests), sends PromptHelm totals only: calls served per version, and the outcomes and usage you report. |
92
+ | `reportFailures` | `reportFallbacks` | Sends what [`failed`](#refused-calls) learns: the provider's HTTP status and error codes, by version, never a message. |
66
93
  | `onResolve` | none | Called on every `get` and `resolve` with the version served, for your analytics or traces. |
67
94
  | `fetch` | global `fetch` | Your own fetch, for tests or a proxy. |
68
95
 
@@ -78,6 +105,7 @@ Without a `fallback`, `get` throws them; with one, `onError` receives them.
78
105
  | `missing_variables` | The live version uses a variable the call doesn't pass. |
79
106
  | `timeout` | No response within the timeout. |
80
107
  | `network` | PromptHelm couldn't be reached. |
108
+ | `api_mismatch` | The live version's model can't be called with the call's `api`. |
81
109
  | `server` | Anything else, such as a 5xx response. |
82
110
 
83
111
  After the first successful fetch, a failed background refresh keeps serving the cached prompt
@@ -106,35 +134,96 @@ const system = await ph.get("support-reply", {
106
134
  });
107
135
  ```
108
136
 
137
+ A prompt with a user message is written as `{ system, user }`, the shape `fallback` takes.
138
+
109
139
  Run it in CI before the build, so every deploy carries the text that was live then. It reads
110
140
  `PROMPTHELM_API_KEY` (and `PROMPTHELM_BASE_URL`), and writes JSON for a `.json` path or with
111
141
  `--format json`. A prompt appears in the file once it's published.
112
142
 
113
143
  ## Model settings
114
144
 
115
- A version can carry the model to call it with, its temperature and its output limit, set next to
116
- the text in the dashboard. Moving to a cheaper model is then a publish like any other: one click
117
- rolls it back, and an A/B test can tell whether it does as well. `resolve` hands them over with
118
- the text:
145
+ A version can carry the model to call it with and how: its temperature and output limit, stop
146
+ sequences, an answer in JSON, the provider and how to reach it, a timeout, retries and parameters
147
+ of your own, all set next to the text in the dashboard. Moving to a cheaper model is then a publish
148
+ like any other: one click rolls it back, and an A/B test can tell whether it does as well.
149
+
150
+ Each API wants them in its own place, so the SDK has a request builder for each:
151
+ `toOpenAIResponses`, `toChatCompletions` (OpenAI's and every compatible host's), `toAnthropic`,
152
+ `toGemini` (`@google/genai`) and `toBedrockConverse`. Give it what `resolve` returned and your own
153
+ request; it returns that request with the version's messages and settings in, and the options for
154
+ `create()`:
119
155
 
120
156
  ```ts
157
+ import Anthropic from "@anthropic-ai/sdk";
158
+ import { PromptHelm, toAnthropic } from "@prompt-helm/sdk";
159
+
121
160
  const r = await ph.resolve("support-reply", {
122
161
  vars: { company: "Acme", customer_name: user.name },
123
162
  fallback: SUPPORT_REPLY,
163
+ api: "anthropic-messages",
124
164
  });
125
-
126
- const reply = await openai.responses.create({
127
- model: r.config.model ?? "gpt-5",
128
- temperature: r.config.temperature,
129
- max_output_tokens: r.config.maxTokens,
130
- input: r.text,
165
+ const { params, options } = toAnthropic(r, {
166
+ model: "claude-sonnet-5-5", // when the dashboard sets none, and for the fallback
167
+ max_tokens: 1024,
168
+ messages: history, // your conversation so far; the version's user message follows it
131
169
  });
170
+ try {
171
+ const reply = await anthropic.messages.create(params, options);
172
+ } catch (error) {
173
+ r.failed(error);
174
+ throw error;
175
+ }
176
+ ```
177
+
178
+ What the dashboard leaves unset stays yours; what it sets wins. The builders import no provider
179
+ package and call nothing. When the fallback is served, `config` is empty and your request goes out
180
+ as you wrote it, with the fallback's messages. `r.config` holds the same settings (`model`,
181
+ `temperature`, `maxTokens`, `stop`, `response`, `provider`, `apis`, `connection`, `timeoutMs`,
182
+ `maxRetries`, `extra`) if you'd rather map them yourself. PromptHelm doesn't check a model name
183
+ with your provider, and a name your provider doesn't know fails your calls: try a new model in an
184
+ A/B test first. SDK versions before 1.4 ignore the settings, and before 1.6 receive only the first
185
+ four.
186
+
187
+ `api` names the API your app calls the model with: `openai-responses`, `chat-completions`,
188
+ `anthropic-messages`, `gemini` or `bedrock-converse`. `config.apis` lists the ones the version's
189
+ model takes where it's called (OpenAI's models take Responses and Chat Completions); a version
190
+ whose model your API can't reach isn't served, and the call gets its fallback (`api_mismatch`).
191
+ PromptHelm learns which API each key's app uses, so the prompt's page warns before such a version
192
+ is published.
193
+
194
+ ### Reaching the provider
195
+
196
+ `clientOptions(r, yours)` adds what the dashboard set for reaching the provider (an address, a
197
+ Region or location, a timeout and retries) to your client's options, in the names the official SDK
198
+ of the API you named in `resolve` uses. Keys stay in your code:
199
+
200
+ ```ts
201
+ const bedrock = new BedrockRuntimeClient(clientOptions(r, { credentials }));
132
202
  ```
133
203
 
134
- `config` holds only the settings set in the dashboard (`model`, `temperature`, `maxTokens`), so
135
- keep your defaults in the code; when the fallback is served it is empty. PromptHelm doesn't check
136
- a model name with your provider, and a name your provider doesn't know fails your calls: try a new
137
- model in an A/B test first. Apps on SDK versions before 1.4 ignore the settings.
204
+ Build a client again when `r.config.connection` changes; most apps keep one per connection. A
205
+ Microsoft Foundry resource's address becomes its `/openai/v1/` or `/anthropic/` URL and a
206
+ Databricks workspace its `/serving-endpoints`; for anything else, the dashboard's Base URL is used
207
+ as it is.
208
+
209
+ ### Answers in JSON
210
+
211
+ A version can ask for an answer in JSON with the fields set in the dashboard. `config.response` is
212
+ then `{ type: "json", name, schema }`: a JSON Schema kept inside what OpenAI, Anthropic, Gemini and
213
+ Amazon Bedrock all accept (and, for GPT on Microsoft Foundry, what Azure accepts), so the same
214
+ schema works with each. The request builders put it where each API wants it: `text.format` on
215
+ Responses, `response_format` on Chat Completions, `output_config.format` on Anthropic,
216
+ `responseJsonSchema` on Gemini and `outputConfig.textFormat` on Bedrock, as text. PromptHelm never
217
+ sees the answer, so check it in your code. Without a response format the model answers in text;
218
+ SDK versions before 1.5 ignore it.
219
+
220
+ ### Refused calls
221
+
222
+ `r.failed(error)` tells PromptHelm that your provider refused the call: the HTTP status, the
223
+ error's type and code, and the parameter it names (OpenAI's SDK gives one), never its message.
224
+ The prompt's page shows them per version, so a setting your provider rejects shows up while it's
225
+ still one publish to roll back. Anything without an HTTP status, such as a network error, is
226
+ ignored. `reportFailures: false` turns it off.
138
227
 
139
228
  ## A/B tests
140
229
 
package/dist/cli.js CHANGED
@@ -94,7 +94,11 @@ async function fetchLivePrompts(options) {
94
94
  }
95
95
  function renderFallbacks(prompts, format) {
96
96
  const sorted = [...prompts].sort((a, b) => a.slug < b.slug ? -1 : a.slug > b.slug ? 1 : 0);
97
- const json = JSON.stringify(Object.fromEntries(sorted.map((p) => [p.slug, p.content])), null, 2);
97
+ const entries = sorted.map((p) => [
98
+ p.slug,
99
+ p.user ? { system: p.content, user: p.user.content } : p.content
100
+ ]);
101
+ const json = JSON.stringify(Object.fromEntries(entries), null, 2);
98
102
  return format === "json" ? `${json}
99
103
  ` : `${TS_HEADER}export const fallbacks = ${json} as const;
100
104
  `;